The Codes AI Can't Crack

It was October 02019, and Thea Sommerschield had hit a wall. She was working on her doctoral thesis in ancient history at Oxford, which involved deciphering Greek inscriptions that were carved on stones in Western Sicily more than 2,000 years earlier. As is often the case in epigraphy — the study and interpretation of ancient inscriptions written on durable surfaces like stone and clay — many of the texts were badly damaged. What’s more, they recorded a variety of dialects, from a variety of different periods, which made it harder to find patterns or fill in missing characters. 

At a favorite lunch spot, she shared her frustrations with Yannis Assael, a Greek computer scientist who was then working full-time at Google DeepMind in London while commuting to Oxford to complete his own PhD. Assael told Sommerschield he was working with a technology that might help: a recurrent neural network, a form of artificial intelligence able to tackle complex sequences of data. They set to work training a model on digitized Greek inscriptions written before the fifth century, similar to how ChatGPT was trained on vast quantities of text available on the internet.

Sommerschield watched with astonishment as the missing text from the damaged inscriptions began to appear, character by character, on her computer screen. After this initial success, Assael suggested they build a model based on transformer technology, which weights characters and words according to context. Ithaca, as they called the new model, was able to fill in gaps in political decrees from the dawn of democracy in Athens with 62% accuracy, compared to 25% for human experts working alone. When human experts worked in tandem with Ithaca, the results were even better, with accuracy increasing to 72%.

Ithaca is one of several ancient code-cracking breakthroughs powered by artificial intelligence in recent years. Since 02018, neural networks trained on cuneiform, the writing system of Mesopotamia, have been able to fill in lost verses from the story of Gilgamesh, the world’s earliest known epic poem. In 02023, a project known as the Vesuvius Challenge used 3D scanners and artificial intelligence to restore handwritten texts that hadn’t been read in 2,000 years, revealing previously unknown works by Epicurus and other philosophers. (The scrolls came from a luxurious villa in Herculaneum, buried during the same eruption of Mount Vesuvius that destroyed Pompeii. When scholars had previously tried to unroll them, the carbonized papyrus crumbled to dust.)

Yet despite these advances, a dozen or so ancient scripts — the writing systems used to transcribe spoken language — remain undeciphered. These include such mysteries as the one-of-a-kind Phaistos Disk, a spiral of 45 symbols found on a single sixteen-inch clay disk in a Minoan palace on Crete, and Proto-Elamite, a script used 5,000 years ago in what is now Iran, which may have consisted of a thousand distinct symbols. Some, like Cypro-Minoan — which transcribes a language spoken in the Late Bronze Age on Cyprus — are tantalizingly similar to early European scripts that have already been fully deciphered. Others, like the quipu of the Andes — intricately knotted ropes made of the wool of llamas, vicuñas, and alpacas — stretch our definitions of how speech can be transformed into writing. 

In some cases, there is big money to be won: a reward of one million dollars is on offer for the decipherer of the Harappan script of the Indus Valley civilization of South Asia, as well as a $15,000-per-character prize for the successful decoder of the Oracle Bone script, the precursor to Chinese.

Cracking these ancient codes may seem like the kind of challenge AI is ideally suited to solve. After all, neural networks have already bested human champions at chess, as well as the most complex of all games, Go. They can detect cancer in medical images, predict protein structures, synthesize novel drugs, and converse fluently and persuasively in 200 languages. Given AI’s ability to find order in complex sets of data, surely assigning meaning to ancient symbols would be child’s play.

But if the example of Ithaca shows the promise of AI in the study of the past, these mystery scripts reveal its limitations. Artificial neural networks might prove a crucial tool, but true progress will come through collaboration between human neural networks: the intuitions and expertise stored in the heads of scholars, working in different disciplines in real-world settings.

“AI isn’t going to replace human historians,” says Sommerschield, who is now at the University of Nottingham. “To us, that is the biggest success of our research. It shows the potential of these technologies as assistants.” She sees artificial intelligence as a powerful adjunct to human expertise. “To be an epigrapher, you have to be an expert not just in the historical period, but also in the archaeological context, in the letter form, in carbon dating.” She cautions against overstating the potential of AI. “We’re not going to have an equivalent of ChatGPT for the ancient world, because of the nature of the data. It’s not just low in quantity, it’s also low in quality, with all kinds of gaps and problems in transliteration.”

Ithaca was trained on ancient Greek, a language we’ve long known how to read, and whose entire corpus amounts to tens of thousands of inscriptions. The AI models that have filled in lost verses of Gilgamesh are trained on cuneiform, whose corpus is even larger: hundreds of thousands of cuneiform tablets can be found in the storerooms of the world’s museums, many of them still untranslated. The problem with mystery scripts like Linear A, Cypro-Minoan, Rongorongo, and Harappan is that the total number of known inscriptions can be counted in the thousands, and sometimes in the hundreds. Not only that, in most cases we have no idea what spoken language they’re meant to encode.

“Decipherment is kind of like a matching problem,” explains Assael. “It’s different from predicting. You’re trying to match a limited number of characters to sounds from an older, unknown language. It’s not a problem that’s well suited to these deep neural network architectures that require substantial amounts of data.”

Human ingenuity remains key. Two of the greatest intellectual feats of the 20th century involved the decipherment of ancient writing systems. In 01952, when Michael Ventris, a young English architect, announced that he’d cracked the code of Linear B, a script used in Bronze Age Crete, newspapers likened the accomplishment to the scaling of Mount Everest. (Behind the scenes, the crucial grouping and classifying of characters on 180,000 index cards into common roots — the grunt work that would now be performed by AI — was done by Alice Kober, a chain-smoking instructor from Brooklyn College.)

The decipherment of the Maya script, which is capable of recording all human thought using bulbous jaguars, frogs, warriors’ heads, and other stylized glyphs, involved a decades-long collaboration between Yuri Knorozov, a Soviet epigrapher, and American scholars working on excavations in the jungles of Central America.

While the interpreting of Egyptian hieroglyphics is held up as a triumph of human ingenuity, the Linear B and Mayan codes were cracked without the help of a Rosetta Stone to point the way. With Linear B, the breakthrough came when Ventris broke with the established thinking, which held that it transcribed Etruscan — a script scholars can read aloud, but whose meaning still remains elusive — and realized that it corresponded to a form of archaic Greek spoken 500 years before Homer. In the case of ancient Mayan, long thought to be a cartoonish depiction of universal ideas, it was only when scholars acknowledged that it might transcribe the ancestors of the languages spoken by contemporary Maya people that the decipherment really began. Today, we can read 85% of the glyphs; it is even possible to translate Shakespeare’s Hamlet into ancient Mayan.

Collaborating across cultures and disciplines, and carrying out paradigm-shedding leaps of intuition, are not the strong points of existing artificial neural networks. But that doesn’t mean AI can’t play a role in decipherment of ancient writing systems. Miguel Valério, an epigrapher at the Autonomous University of Barcelona, has worked on Cypro-Minoan, the script used on Cyprus 3,500 years ago. Two hundred inscriptions, on golden jewelry, metal ingots, ivory plaques, and four broken clay tablets, have survived. Valério was suspicious of the scholarly orthodoxy, which attributed the great diversity in signs to the coexistence of three distinct forms of the language. 

To test the theory that many of the signs were in fact allographs — that is, variants, like the capital letter “G” and “g,” its lower-case version — Valério worked with Michele Corazza, a computational linguist at the University of Bologna, to design a custom-built neural network they called Sign2Vec_d. Because the model was unsupervised, it searched for patterns without applying human-imposed preconceptions to the data set. 

“The machine learned how to cluster the signs,” says Valério, “but it didn’t do it simply on the basis of their resemblance, but also on the specific context of a sign in relation to other signs. It allowed us to create a three-dimensional plot of the results. We could see the signs floating in a sphere, and zoom in to see their relationship to each other, and whether they’d been written on clay or metal.” 

The virtual sphere allowed Valério to establish a sign-list — the equivalent of the list of 26 letters in our alphabet, and the first step towards decipherment — for Cypro-Minoan, which he believes has about 60 different signs, all corresponding to a distinct syllable.

“The issue is always validation. How do you know if the result is correct if the script is undeciphered? What we did was to compare it to a known script, the Cypriot Greek syllabary, which is closely related to Cypro-Minoan. And we found the machine got it right 70% of the time.” But Valério believes no unsupervised neural net, no matter how powerful, will crack Cypro-Minoan on its own. “I don’t see how AI can do what human epigraphers do traditionally. Neural nets are very useful tools, but they have to be directed. It all depends on the data you provide them, and the questions you ask them.”

The latest advances in AI have come at a time when there has been a revolution in our understanding of writing systems. A generation ago, most people were taught that writing was invented once, in Mesopotamia, about 5,500 years ago, as a tool of accountancy and state bureaucracy. From there, the standard thinking went, it spread to Egypt, and hieroglyphics were simplified into the alphabet that became the basis for recording most European languages. It is now accepted that writing systems were not only invented to keep track of sheep and units of grain, but also to record spiritual beliefs and tell stories. (In the case of Tifinagh, an ancient North African Berber script, there is evidence that writing was used primarily as a source of fun, for puzzle-making and graffiti.) Monogenesis, the idea that the Ur-script diffused from Mesopotamia, has been replaced by the recognition that writing was invented independently in China, Egypt, Central America, and — though this remains controversial — in the Indus Valley, where 4,000 inscriptions been unearthed in sites that were home to one of the earliest large urban civilizations.

The most spectacular example of a potential “invention of writing” is Rongorongo, a writing system found on Rapa Nui, the island famous for its massive carved stone heads. Also known as Easter Island, it is 1,300 miles from any other landmass in the South Pacific. Twenty-six tablets have been discovered, made of a kind of wood native to South Africa. Each has been inscribed, apparently using a shark’s tooth as a stylus, with lines of dancing stick-figures, stylized birds and sea creatures. The tablets were recently dated to the late 01400s, two centuries before Europeans first arrived on the island.

For computational linguist Richard Sproat, Rongorongo may be the script AI can offer the most help in decoding. “It’s kind of a decipherer’s dream,” says Sproat, who worked on recurrent neural nets for Google, and is now part of an AI start-up in Tokyo. “There are maybe 12,000 characters, and some of the inscriptions are quite long. We know that it records a language related to the modern Rapa Nui, which Easter Islanders speak today.” Archaeologists have even reported eyewitness accounts of the ceremonies in which the tablets were inscribed. And yet, points out Sproat, even with all these head starts, and access to advanced AI, nobody has yet come close to a convincing decipherment of Rongorongo.

The way forward depends on finding more inscriptions, and that comes down to old-fashioned “dirt” archaeology, and the labor-intensive process of unearthing ancient artifacts. (The best-case scenario would be finding a “bilingual,” a modern version of the Rosetta Stone, whose parallel inscriptions in Greek and demotic allowed 19th-century scholars to decipher Egyptian hieroglyphics.) But the code of Cypro-Minoan, or Linear A, or the quipu of the Andes, won’t be cracked by a computer scientist alone. It’s going to take a collaboration with epigraphers working with all the available evidence, some of which is still buried at archaeological sites.

“As a scholar working in social sciences,” says Valério of the Autonomous University of Barcelona, “I feel I’m obliged to do projects in the digital humanities these days. If we pursue things that are perceived as traditional, no one is going to grant us money to work. But these traditional things are also important. In fact, they’re the basis of our work.” Without more material evidence, painstakingly uncovered, documented, and digitized, no AI, no matter how powerful, will be able to decipher the writing systems that will help us bring the lost worlds of the Indus Valley, Bronze Age Crete, and the Incan Empire back to life.

Perhaps the most eloquent defense of traditional scholarship comes from the distinguished scholar of Aegean civilization, Silvia Ferrara, who supervised Valério and Corazza’s collaboration at the University of Bologna. 

“The computer is no deus ex machina,” Ferrara writes in her book The Greatest Invention (02022). “Deep learning can act as co-pilot. Without the eye of the humanist, though, you don’t stand a chance at decipherment.”

Notes

1. Figure and caption reproduced from Corazza M, Tamburini F, Valério M, Ferrara S (02022) Unsupervised deep learning supports reclassification of Bronze age cypriot writing system. PLoS ONE 17(7): e0269544 under a CC BY 4.0 license. https://doi.org/10.1371/journal.pone.0269544