Ordered Set Correspondence

Introduction

Can we gather data from the way things are ordered, even before we know exactly what those things are or how they are related? That seems like an awfully theoretical and confusing question, I'll admit.

In the hopes of making this question more intuitive, I'll explore it through the lens of a simpler question about ancient alphabets instead. People reading this will know what an alphabet is and what letters are, of course, but behind the intuition are some surprising things, and the topic gets progressively more mind-bending. We will not begin by asking where alphabets came from, or whether existing or new hypotheses are ultimately correct. Instead, we will start with a more basic question:

🤔 What patterns appear when we compare the orders of ancient alphabets?

It is through this vague yet intuitive question that I'll be able to present the research paper to you.

Surprise! More than one alphabetical order exists

When asked about the reason for alphabetical order starting with "ABC" and not with other letters, people usually evade the question. So we may not know why the alphabet is in that particular order, but the alphabet does have one: the "ABC" order.

Alphabetical order is often considered "as old as time itself". If so, then why would alphabetical order change? And if it doesn't really change, then what are we really going to compare it to? Itself? That does not add up. Plus, if there is only one order from which all later alphabets came, then the idea of comparing alphabetical orders makes no sense. If we take this one step further: if there's only one alphabetical order, then it doesn't even make sense for there to be multiple different ways of writing alphabets worldwide since they all have similar orders! Why can't everyone just write using the same alphabet? If we all used the same one, then wouldn't it be so much easier for us to all understand each other and declare peace on Earth?

Alphabetical order is what we use to order books in a library. It's a sign of civilization, of order, of identity. To some, it is our beginning and our end. If I think about that, it doesn't seem entirely incorrect. Alphabetical order is indeed considered "canonical", and it is what software engineers like me may use sorting algorithms for: sorting data in "the alphabetical order". This alphabetical perception is so ingrained in our culture that we only see the — admittedly — very practical uses for it. Alphabetical order is a widely approved convention that establishes which letter of "the" alphabet comes first. Which comes second. It's part of a set of attributes of the alphabet that make up a simplified "protocol" for communicating between us through space and time. Now, if we take a step back, the alphabetical order "ABC" is used everywhere:

Except in places like modern-day Ethiopia. Except some ancient alphabets. 🌍✨

Wait, there are exceptions that don't use ABC as an alphabetical order?!

It turns out that many alphabetical orders exist and have existed. One such example, still in use by millions today as a collating and learning order, is called the "HLḤM". It is used for a writing system called "Fidäl". This is because its order starts with those specific letters. We also have ancient inscriptions in which letters were not ordered in the "ABG" or "HLḤM" orders (such as those listed here if you're curious. Just run an empty search for Subject > ABC). The intriguing thing about the "HLḤM" order is that its oldest record was found at the exact same archaeological site as the oldest known record of the "ABG" order: the Bronze Age city of Ugarit in present-day Syria. This begs the question: is the HLḤM order also something that will never change if it's also at least 3,500 years old?

We've talked about things that — clearly — aren't that well understood. So what is going on?

Isolating order as an attribute

So we now know that there are multiple alphabetical orders and that this has been the case for a very long time.

But beyond their letter orders, such as "ABC" or "HLḤM", alphabets also have many other attributes.

We can compare scripts more generally by looking at the appearance of their letters, their letter names and pronunciations, the underlying language(s) one can write with them, the direction in which they are written or read, the number of signs, their age, their geographical distribution, and many other properties.

All of these attributes can potentially teach us something — but we tend to consider them all at once, as a single "package" or "communication protocol". Although arguably useful in modern considerations, this approach makes understanding ancient or undeciphered scripts much harder. Let me explain.

The mistake we may be making is to assume that a given "script A" has a particular set of attributes just because we have identified one of its attributes. For example, if "script A" has letter shapes that remind us of those of "script B", then it's common to assume that they both had the same alphabetical order. This is great for not getting stuck on too many details when ancient inscriptions are rare. However, if not managed carefully, this approach results in circular, self-reinforcing logic that is counterproductive scientifically. This is because it results in us collectively forgetting which attributes have been observed and which ones are inferred through modern alphabet "packaging" culture.

As a software engineer, I'm going to focus primarily on just one attribute of ancient alphabets first: the order of the letters. This has multiple advantages:

It avoids me being confronted with my very approximate knowledge of the underlying languages.
Order is one of the more objectively interpretable attributes of alphabets, as opposed to letter shapes or dating, for instance.

During this attribute-isolation step, considerations for all the other attributes are suspended as much as possible. For example, the age of the scripts and where they were used are not considered, just order. Once the order analysis results are available, however, all attributes can then be considered once more.

Our approach does not suggest that order is independent of the other attributes. It instead offers a deliberate way of better understanding information: first explore what order alone can tell us, and bring other considerations back into the mix afterwards.

What is an ordered inventory?

Let me start with a very simple statement: an alphabet is not just a collection of symbols.

It is also an ordered inventory: a set of distinct elements occupying a 1-dimensional grid but in a particular order.

And that order is what I want to focus on.

The letters analogy is particularly useful and is an intuitive example, but the underlying idea is not fundamentally about alphabets. It applies whenever we have a collection of distinct elements whose positions can be compared.

In other words, if the elements of two such ordered inventories can be put into correspondence, we can then ask how their orders relate to one another.

So, in this work, alphabets are an example of a more general object: the ordered inventory.

How do we compare orders?

The basic tool for comparing alphabetical orders is a correspondence table.

Inventory A	Inventory B
A	𐤀
B	𐤁
G	𐤂
D	𐤃

Suppose we have two different letter inventories. Their elements may look completely different, so we first need to establish which elements we want to treat as corresponding. For some alphabets this is possible; for others, it may be harder.

For the alphabets we will be looking at, a simple and common way of comparing letters is to transform both alphabets into a common, letter-for-letter representation using the Latin alphabet, for example. This letter-for-letter transformation is called transliteration.

Scholars might therefore turn two different sequences of symbols into something like:

A — B — G — D — H — ...

Once we have this data, we can ignore the symbols themselves and simply compare letter positions.

The important point is that the letter correspondence between scripts tells us what we are comparing, while the order of each transliteration tells us what we measure.

The potential of this approach is that it lets us deliberately focus on one attribute — order — without first having to decide whether the systems share a language, graphical origin, geographical origin, or historical family.

We can therefore ask a much narrower question:

Given these letter correspondences, how similar are their orders?

Where does language fit?

So far, I've deliberately treated letters as things that can be put into correspondence without making language the starting point. But when we work with writing systems, language is obviously relevant.

A script is not a language, and a language is not a script. The same language can be written in different scripts, and the same script can be used for different languages. For example, Turkish and Malay were written in alphabets derived from the Arabic alphabet but are now written using alphabets derived from Latin.

Script ≠ language ≠ inventory

There is also a practical problem with ancient material: once different inscriptions are transliterated into the same system of letters, things that originally looked quite different can appear much more similar linguistically. If we are not careful, this can make linguistic comparisons misleading.

One possibility we are exploring to avoid involving language directly is to use letter frequency. In other words, a text can have its letters counted and then ordered by frequency. If you think about it, this also results in an ordered inventory of letters. It turns out that this kind of ordered inventory contains information about what language the text was written in, and all this is done without actually reading the text itself.

In this way, we don't need to solve the language question perfectly before focusing on letter order. We can continue focusing only on ordered inventories of letters by adding letter-frequency ones to the study as a proxy for later language identification. Actual language considerations are still kept for later.

What does the historical alphabetical order data actually look like?

When considering ancient writing systems, we rarely have a complete dataset. We may have fragmentary inscriptions, small corpora, uncertain dating, incomplete sign inventories, uncertain reading directions, and sometimes no direct evidence for the original ordering of letters at all. We may also group scripts in uncertain ways using visual cues.

Observed evidence ≠ reconstructed model

There is therefore an important distinction between what is actually preserved archaeologically and what we have reconstructed.

In other words, a modern correspondence table of ancient alphabets is already a model of the evidence.

And that gives us an interesting opportunity: rather than treating these reconstructions as a definitive answer, we can also use them as data and ask what patterns emerge from them.

We can evaluate the use of the correspondence table itself.

When many assumptions become one theory

The difficulty of understanding ancient writing becomes greater when several uncertain attributes are combined into one historical explanation.

A theory might connect origin, script, order, language, chronology, and graphical development into a single account. That can be extremely useful, but it can also create assumptions that turn into circular interpretation. For example, an assumed chronology might affect which graphical similarities seem possible. Similarly, chronology can be used to reject a correspondence, while the proposed correspondence may itself have influenced how something was dated. Even the terminology we use can introduce assumptions. The word "alphabet", for example, can encourage us to imagine a familiar kind of ordered inventory, with a specific order and specific letter names, and have us think that this is the only possible arrangement for any alphabet.

I don't think the useful question is to simply ask: "If our approach is so circular, then which existing theories are wrong?"

The question is more basic: have we reached the limits of what these models can safely infer from the available evidence?

In other words, when multiple uncertain attributes are used to constrain one another, it becomes difficult to know how much evidential independence remains between them.

That is where I think an attribute-by-attribute approach can be useful.

From the general problem to the epigraphic case

This is where the epigraphic case study for our Ordered Set Correspondence research comes in.

There are a number of early alphabetic and alphabet-like writing traditions that are potentially relevant here, including Proto-Sinaitic, Proto-Canaanite, Phoenician, Ugaritic, Safaitic, and related traditions.

I am particularly interested in this material because it contains exactly the kind of uncertainty that makes exploratory analysis interesting. The letters are simple and few, but the relations aren't always so clear-cut.

For example, for the undeciphered so-called Proto-Sinaitic script, we simply don't have a complete and ordered ancient inscription of its letters sitting in front of us. It doesn't stop some from assuming that the symbols are letters of an alphabet and that they were ordered as "ABC".

What was the complete sign inventory? How many signs were there in total? What was, or what were, the original orderings of the signs?

So when a modern reconstruction presents an apparently coherent alphabetic order, we need to distinguish between what was actually preserved and what has been reconstructed.

These are precisely the kinds of cases where I think it is useful to separate what we can observe about a letter inventory from the historical story we eventually want to tell about it. There is also a danger of allowing our uncertainty about a writing system to become an assumption about the people who produced it. Were they "intelligent"? Were they "capable"? Were they "civilized"? Were they "prestigious"? These kinds of questions may say more about us than about what we are studying.

Why not just assume the established genealogy of alphabets?

We have a lot of interesting information as well as theories on the history of the alphabet. However, I argue that the design of our current models deliberately leaves some parts of the history of writing unanswered.

One important place in history is Byblos, an ancient city-state in present-day Lebanon, from which an "ancestor" of our Latin alphabet is supposed to have spread throughout the Mediterranean: the Phoenician alphabet.

If we look deeper, however, we find that a corpus of texts — assumed to be syllabic and not alphabetical — was also found at Byblos. It has uncanny resemblances to the Phoenician alphabet but is somehow undeciphered. With evidence of skilled production and diverse use, current assumptions by many scholars are:

Isn't this writing system something that wasn't even used to write Phoenician at all?
Isn't the undeciphered script just a non-mature writing experiment of the past?
Isn't this writing system just an imitation of the later alphabet? (Confusing question, I know.)

One consequence of beginning with an established genealogical model is that anomalous evidence can become something that needs to be accommodated within that model. An undeciphered script may therefore be approached through questions such as whether it belongs to a different writing tradition, whether it represents an earlier developmental stage, or whether its relationship to later alphabetic writing has been misunderstood.

In this scholarly context, the Byblos syllabary has been declared an intractable problem despite the underlying language being probably well understood and documented: Phoenician. In this scholarly context, we are stuck. Or are we?

Rather than beginning by establishing a "genealogical tree" of writing systems and asking where each alphabet belongs on that tree, we could first ask what relationships between scripts appear when we compare specific attributes only. We will mostly be focusing on letter orders here.

At this point, it's important to pause and remember that a correspondence between two alphabetical orders does not automatically establish historical descent. If we find that two orders are surprisingly similar, that is a result about order, not about the whole context. Our results will therefore not automatically be a result about direct origin. This distinction is really central to what I am trying to do.

A: A B C D E
B: A C B E D

A positions: 1 2 3 4 5
B positions: 1 3 2 5 4

Rank correlation value: ...

Transforming the data in ways it might have been

So how do we actually turn a handful of alphabetical order inscriptions into something that we can compare? Isn't this just a matter of a handful of calculations that will get us nowhere?

Well, as we said, the first step is to write the elements of our ordered inventories into a common representation. In the case of writing systems, this can involve transliteration and normalization using correspondence-mapping techniques.

We can then represent an inventory as an ordered sequence. If we pause and think about it, however, text isn't just one-dimensional; it is lines of text on a 2D surface. This two-dimensional layout is something we can work with.

And this is important because the spatial arrangement of an existing abecedary contains information that is lost if we immediately flatten it into a single line. Why? Because a 2D text might not necessarily have always been read in the same direction. This is visible for some scripts, such as Libyco-Berber, which can read texts from bottom to top.

Rather than generating arbitrary two-dimensional grids and searching through all of them, we could start from existing layouts and examine plausible transformations of those layouts. However, scholars do not currently provide this data, so we generate all possible 2D grids and readings of a given alphabetical order ourselves.

The important thing is that the grids and transformations remain plausible and systematic.

The goal is therefore not to invent an enormous number of possible alphabets or explore a search space that is too large. It is to transform the data we actually have into a controlled set of comparable representations.

How surprising is a result?

This brings us to one of the biggest methodological issues in the project.

It is not enough to find an interesting correspondence between two letter orders using the method we have described. The search space is too large.

If we search a huge number of possible correspondences, layouts, permutations, and reading directions, then eventually we are going to find order correspondences that look surprisingly good.

So the important question is not simply, "How good is this match?"

It is: How surprising is this match given everything we searched?

The fundamental issue is the relationship between the result and the size and structure of the search space.

A correspondence that looks extraordinary after searching twenty plausible alternatives is not equivalent to finding a correspondence after searching millions of arbitrary possibilities.

So we need to make the search space explicit and, ideally, make it substantially smaller.

Raw similarity ≠ evidential strength

The method remains useful, however, considering the context we described. Since 2D layouts of abecedaries are not available as input, we can still explore the search space for clues using our approach: similar orders in a very large search space.

Despite the search space size problem, it's worth taking the time to reduce it.

One way would be to actually start with plausible two-dimensional layouts that actually exist, as well as all plausible reading directions. This may seem like a lot of readings, but it's substantially less than generating all possible grids × all possible reading directions.

The principle is therefore: don't invent an enormous space of arbitrary possibilities when the historical evidence can give us constraints. If you don't have that right now, explore the search space for clues.

What did you find in the search space?

Let's think about this. We aren't necessarily trying to produce another family tree of alphabets. Our result is simply a collection of candidate correspondences, similarities, order clusters, and apparent outliers.

Those correspondence results are measures of things like Kendall's tau and Spearman rank correlations, as well as Cayley distance. These values allow us to sort the results by closeness in rank, etc.

From our resulting correspondences, which are available in the study for viewing, hypotheses can be generated. These hypotheses can then be tested against all other attributes of ancient alphabets, including space and time considerations.

I want to be very clear about the status of this: these are hypotheses generated by the analysis, not conclusions established by it. The value is in the insight some of the hypotheses might bring us. They might help us ask ourselves the right questions. Not "which ancient peoples invented the alphabet for the rest of us?", which is a nationalistic or political consideration that tries to establish social and national hierarchy and precedence rather than come to a scientific conclusion. Our method instead allows us to explore and ask: "What is the data our archaeologists have given us potentially saying?" I argue that that is where we have the biggest opportunities to learn about ancient writing systems here.

Recategorizing ordered inventories

Our method may constitute a way of exploring structured data.

Instead of assuming that every attribute has to produce the same historical tree, we can potentially recategorize the inventories depending on individual attributes.

In the next pages, we will be describing some concrete example hypotheses. We will show how the analysis of letter order may help us recategorize alphabetical orders and see patterns that were not previously visible for other attributes.

The important point is that these are research questions generated by the method. The method isn't telling us that one script is the ancestor of another. It is telling us where relationships may be worth investigating.

What is the broader contribution?

So far we have talked mostly about ancient alphabets. Ordered Set Correspondence may apply to data much more broadly.

Our method may constitute a way of exploring structured data when the relationships between the objects are uncertain and when several attributes could potentially explain those relationships. This is helpful when our current way of understanding problems produces fewer and fewer helpful results.

All we need to do is:

use established correspondences between ordered sets of data
isolate an attribute (order)
apply data transformations relevant to the inputted ordered data to create a search space
constrain the search space as best we can in the hopes of going beyond detecting correspondences that could all be flukes
evaluate the resulting patterns in relation to that search space
generate hypotheses if the search space is too large, or come up with definitive conclusions if not

Once the results or hypotheses are made, we can then bring any other attributes our data had back into the analysis. In our case study, these are things like chronology, geography, language, graphical form, etc.

In a sense, this is also where the idea of temporarily setting aside time becomes useful. We aren't saying that time doesn't matter.

In effect, we apply a sort of suspension of time, space, and certainty to our input data and examine order only. We then restore "real-world considerations" afterwards as an evaluative constraint.

And because the underlying object is an ordered inventory of distinct elements rather than necessarily an alphabet, we can ask if this can be applied far beyond this particular epigraphic case study. Could it apply to DNA/codon data? Earthquake data? Other kinds of data that aren't immediately reminiscent of letters of the alphabet? In other words, do we have a useful future abstraction here?

Conclusion

We have discussed the method presented in the Ordered Set Correspondence preprint by means of a — hopefully — intuitive example: the letters of alphabets and the different ways in which they are ordered. For further reading, you can take the time to see some of the hypotheses we have generated for ancient writing systems, such as:

How significant the MISS.A 2 order and abecedary might have been in the history of alphabets.
An attempt at the decipherment of the Byblos syllabary.
Hypotheses on the reason behind the "ABG" and "HLḤM" alphabetical orders and whether they are related to each other.

We also suggest starting with the following demo to see how letter frequency can help in identifying a text's language:

Language identification demo →

References

Agostini, A., & D'Imperio, D. (2024). South Semitic abecedaries in the Arabian Peninsula. Res Antiquae, 21, 15–42. Safran.
La Marle, H. (2002). L'aventure de l'alphabet: Les écritures cursives et linéaires du Proche-Orient et l'Europe du sud-est à l'âge du bronze. Geuthner. ISBN: 9782705337162.