projects parentheses
parentheses
In developmentCodename: Project Tanzanite
A small recurrent language model from uncoalesced, built and trained from scratch on one consumer laptop GPU. It isn’t a Transformer. In place of softmax attention it uses selective linear attention, from the RWKV and Mamba-2 family.
01 · what it is
Trained in parallel, decoded as a recurrence
The architecture has two forms. For training it is designed to run in a parallel form of linear attention, whose cost grows linearly with sequence length rather than with its square. For generation it switches to a recurrent form, where each new token updates a memory state of fixed size.
A standard GPT-style decoder lives in the same codebase, but only as a baseline to compare against, and it gets no new work. Both share the same generation code, so every feature below runs on checkpoints of either kind.
02 · where it stands
Pre-benchmark, and the README says so first
As of 2026-09-18: Active development, pre-benchmark.
parentheses README
There are no standard language-modelling benchmark results yet, and we won’t quote any until there are. At the current sizes, under a million parameters, the output is grammatical and valid UTF-8 but has little meaning behind it. Real comparisons wait on two things: finishing the move from causal to selective linear attention, and checkpoints coherent enough to be worth measuring.
What has been checked is the mechanism. On the RTX 5050, the selective attention layer matches a 64-bit reference to a maximum absolute error of 5.98e-7, and a document-boundary reset lets nothing leak across it, forward or backward. On a recall test it holds up to 16 key-value pairs and starts to blur at 32, which the maths predicts for a state that small. None of that says how good a trained model is.
03 · small on purpose
Sized for a laptop
Most presets read text one byte at a time. Below a million parameters, a usual 32,000-token vocabulary would need about 16.4 million parameters for its embedding table alone, sixteen times the rest of the model. Bytes cost longer sequences instead, roughly four times as many tokens as word-level BPE for the same text.
| Preset | Parameters | Vocabulary | Used for |
|---|---|---|---|
| tiny-smoke | ~35K | 64 | CPU smoke test only |
| 0.5-100k | ~123K | 256 (bytes) | pipeline validation |
| 0.6-300k | ~330K | 256 (bytes) | recommended target |
| 0.7-600k | ~585K | 320 | intermediate scale |
| 0.8-1m | ~1.13M | 512 (trained BPE) | BPE test preset (stretch) |
Configurations of 1 to 2 billion parameters would need cloud GPUs. They are planned for later and aren’t in this codebase.
04 · on top of the model
Thinking out loud, with or without your notes
Free Think Mode takes a statement and keeps writing about it until you stop it, holding the first few prompt tokens in view so it stays near the topic. At this size it drifts over long runs, and the README says as much.
Modular Free Think adds retrieval from your own local text files, with BM25 as the default. There is an optional vector backend too, but at this scale it retrieves English paragraphs much worse, so no embedding model ships with it. Conversation memory writes each turn to a Markdown file and pulls earlier turns back in when they matter, which helps a model whose context window is 256 bytes.
05 · languages
A Dravidian corpus, staged but not trained on
Training so far uses English text. A first Dravidian corpus is tokenized and waiting: 1.65 billion byte tokens across 325,627 documents, 43.1% Kannada, 28.2% Tamil, 14.7% Malayalam and 14.0% Telugu. It isn’t wired into the training script yet, so no checkpoint has seen any of it. Connecting it is first on the roadmap.
06 · not built yet
Designed, not implemented
- The chunked forward pass from the formal spec, which is what makes training linear in practice. The live path still uses the dense O(T²) masked form.
- Feeding document boundaries into training as a reset mask. The Dravidian corpus is waiting on this.
- Closing the training-speed gap: selective linear training is about 34% slower than the causal baseline at the same size. That work is blocked until two conflicting optimisation plans are reconciled into one.
- The Akshara morphological tokenizer and any custom CUDA kernels. Both are designed and deliberately kept out of the current training loop.
- A translation layer, reserved for later and for bigger models.
- Scaling to 1 to 2 billion parameters on multi-GPU cloud hardware, with the distributed training and serving setup the repo’s tooling plan describes. Whether that happens depends on how the small-scale experiments turn out.
07 · status
In development
MIT licensed, worked on in public
The code, the size presets, the verification harness and the self-tests are in the repository, and so are the open questions above.