CIEL: long-term memory for language models.
Our memory architecture lets a small language model recall facts from documents up to 1 million tokens long, after training only on documents of up to 8,000 tokens.
Foundational research · Research preview
Efficient models lose the details.
Models that read long text cheaply keep a compressed summary of what came before, so exact details fade: a name, a number, the latest value of a setting. Attention keeps every detail, but its cost grows with every token it reads.
Recall from long documents.
Our 350M-parameter model never saw a document longer than 8,000 tokens in training. We then tested it on documents up to 1 million tokens long.
| Document length | Facts recalled correctly |
|---|---|
| 64K tokens | 97% |
| 256K tokens | 97% |
| 1M tokens | 97% |
| 64K tokens, fresh test set | 95% |
Synthetic recall test: 75 scored facts at each length, many of them updated over time, and the model must return the latest value. The model was selected from 5 training runs on a separate validation set. Test documents were never used for training or selection.
Against standard transformers.
We compared our models with standard transformers of the same size, trained on the same data.
Better language modelling.
At 350M parameters, perplexity 21.3 against 22.6 on held-out web text (lower is better).
Recall beyond the training length.
The transformer falls to 0% beyond the length it was trained on, even when trained on 64K-token documents. Ours recalls 93% or more out to 1M tokens.
An external recall benchmark we never trained on.
On JRT, our 125M-parameter models score 1.6–2.5× the transformers' score.
Efficient at long context.
Our model against a standard transformer, both with 125M parameters, reading a 1M-token input.
- 19×less memory for the context2.2 GB against 43 GB
- 4×faster to read the document14.5 s against 58.4 s
- 20%less time per generated token63.5 ms against 79.3 ms
Memory retrofit for open-weight models.
We added our memory to an existing 0.6B open-weight model, which changed its quality on ordinary text by less than 1%. After a short adaptation on documents up to its original 32K-token context window, it recalls 89% of facts at 128K tokens, 4× that window, where the unmodified model recalls none.
Research
with us.
We welcome research collaborations on long-term memory for language models. For memory your coding agents can use today, see Graphicle, our open-source code graph and memory.