I found this in a special issue of Visible Language 1986 on The Origins and Functions of Literacy. I was immediately drawn to the word TOKENS. Now (mid-2026), TOKENS is a word in heavy circulation — part of the technical language of Artificial Intelligence and the Large Language Models which operate behind it. You've probably heard the term.

The issue, designed by Ellen Lupton and J. Abott Miller, has typographically degraded heads which are hard to place. Their form reminded me that this was typeset in 1986, right on the edge of the shift to computer typesetting from photo-mechanical composition.



TOKENS are described in the text as the first examples of visible language which led directly to the invention of writing. I was hooked. TOKENS here refers to small chunks of clay from ~ 8000 BC. 



But I was curious what else the word might have meant when this was the word might have meant when this was published in 1986?  TOKENS might be a familiar if you were a computer scientist in 1986. As I learned in Anatomy of a Compiler, TOKENS are words, parts of words, spaces, sometimes punctuation, in a specific computer language which are processed by a compiler to break down the program into parseable chunks.

Using a formal grammar, a proper syntax, and a specific dictionary, TOKENS are given precise and meaningful relationships to each other. Each TOKEN has a particular address in the structure of a programming language. A compiler then translates a computer program from one (human-written) programming language to (machine-readable) code so the instructions can be run on specific computer hardware (a particular chip). In 1986, TOKENS were a kind of root-level object in a computer language. Here’s Susan Kare at Apple Computer in 1986:



Like the TOKENS from 8000 BC, 1986 compiler TOKENS also had to be made. That process TOKENS also had to be made. That process (schematically) looks like this:

input >> TOKEN >> parse tree >> code generator >> output

where the input (a computer program in a specific language) is processed by the compiler which breaks it into chunks (TOKENS) which are processed for syntax and matched to a dictionary to produce a parse tree (or abstract representation of the program) which is then used by a code generator to translate the program into machine language as the output. The compiled program can then be run. It’s good to notice that the input is human readable but the output is decidedly not (as binary strings of 0s and 1s meant for a silicon chip).

*Now* (in 2026), the word TOKENS immediately suggests Large Language Models and AI. Right? I don’t think it matters if you are a computer scientist or not, the term is so widely used that can hardly escape it. Large Language Models are fundamental to current Artificial Intelligence applications such as chatbots (GPT or Claude), image generators (Midjourney, Runway), synthetic audio (Eleven Labs), self-driving cars, face recognition, and so on. To be effective, LLMs must consume huge datasets which are processed into TOKENS and used to construct the model. That process looks something like this:

input >> TOKEN >> position in model (embedding) >> TOKEN >> output 

Here the input is data (text), split into meaningful chunks as TOKENS (words, parts of words, punctuation, spaces). Each TOKEN is processed for its semantic relation to other TOKENS and assigned a position in the many- many-dimensioned space of the Large Language Model. (This position in the model is the TOKEN’s “embedding.”) A well-trained model can then statistically predict what the next TOKEN should be. This is returned as the output.

Unlike the compiler (in 1986), this time both the input and the output are *human* readable and this means that 2026 TOKENS also become “tools of the mind”—they extend our thoughts and amplify what we can think. TOKENS are not so concrete. Every time a word appears in a text, it refers to the same explicit token in the model. And so TOKENS are efficient, with about 75% as many TOKENS as words.

Continues in class ...