On Tuesday, Meta AI unveiled reports MIT Technology Review.of Galactica, a large language model designed to “store, combine and reason about scientific knowledge.” While intended to accelerate writing scientific literature, adversarial users running tests found it could also . After several days of , Meta took the demo offline,
Large language models (LLMs), such as OpenAI’s GPT-3, learn to write text by studying millions of examples and understanding the statistical relationships between words. As a result, they can author convincing-sounding documents, but those works can also be riddled with falsehoods and potentially harmful stereotypes. Some critics call LLMs “ ” for their ability to convincingly spit out text without understanding its meaning.
Enter Galactica, an LLM aimed at writing scientific literature. Its authors trained Galactica on “a large and curated corpus of humanity’s scientific knowledge,” including over 48 million papers, textbooks and lecture notes, scientific websites, and encyclopedias. According to, Meta AI researchers believed this purported high-quality data would lead to high-quality output.
Starting on Tuesday, visitors to thecould type in prompts to generate documents such as literature reviews, wiki articles, lecture notes, and answers to questions, according to examples provided by the website. The site presented the model as “a new interface to access and manipulate what we know about the universe.”
While some people found the demoand , others soon discovered that anyone could type in or , generating authoritative-sounding content on those topics just as easily. For example, someone used it to a wiki entry about a fictional research paper titled “The benefits of eating crushed glass.”
Even when Galactica’s output wasn’t offensive to social norms, the model could assault well-understood scientific facts, spitting outsuch as incorrect dates or animal names, requiring deep knowledge of the subject to catch.
I askedabout some things I know about and I’m troubled. In all cases, it was wrong or biased but sounded right and authoritative. I think it’s dangerous. Here are a few of my experiments and my analysis of my concerns. (1/9)
— Michael Black (@Michael_J_Black)
As a result, Metathe Galactica demo Thursday. Afterward, Meta’s Chief AI Scientist Yann LeCun , “Galactica demo is off line for now. It’s no longer possible to have some fun by casually misusing it. Happy?”
The episode recalls a common ethical dilemma with AI: When it comes to potentially harmful generative models, is it up to the general public to use them responsibly, or for the publishers of the models to prevent misuse?
Where the industry practice falls between those two extremes will likely vary between cultures and as deep learning models mature. Ultimately, government regulation may end up playing a large role in shaping the answer.