# How Taco works: a language model, a range coder, and no floats

Source: https://taco-man.in/writing/how-taco-works
Author: Rohit Kumar
Published: 2026-10-22
Type: Essay

> A course-style walk through Taco: why a likely token costs almost nothing, how a range coder turns probabilities into bytes, why float neural networks cannot be decoded on another machine, and how Taco runs GPT-2 with integers only, down to the bits of a float32.

Taco is a lossless compressor. A language model guesses what comes next in a file, and a range coder stores only what the model did not guess. A file Taco compresses on my Mac comes back, bit for bit, on a Linux PC with an NVIDIA GPU. That last part is the hard part, and it is why GPT-2 runs inside Taco on integer arithmetic only.

This is a walk through the whole thing, the way a course would teach it: information theory, coding, floating point, the maths of a neural network, and then how every step of that maths becomes integer arithmetic. The figures compute live in your browser; the integer ones use exact integer arithmetic with the same rules as Taco's code. Facts and measurements come from [the Taco repository](https://github.com/tac0mania/taco): its design plan, decision records, research notes and primer.

**In short.** GPT-2 gives every possible next token a probability. A token the model expected costs few bits, a surprising one costs many, and a range coder writes exactly that many bits. The decoder rebuilds the same probabilities and reads the tokens back. This only works if both sides compute **exactly** the same numbers, and floating-point neural networks do not: their last bits change with the CPU, the library and the order of additions. So Taco rewrites GPT-2 as an integer network: integer weights with scales, integer sums, integer rescaling, and tables for the curved functions. Integer arithmetic has one answer on every machine.

## Part 1. Information: why a likely token costs few bits

**In short.** Pointing at one outcome out of many takes yes-or-no questions. An outcome of probability $p$ takes about $-\log_2 p$ of them. Likely means cheap: 93% costs a tenth of a bit, 1 in 8,000 costs 13 bits.

### A bit is one halving

One yes-or-no question halves the possibilities. Two questions leave a quarter, eight questions pick one of 256. To single out something of probability $p$, you need about

$$
\text{bits}(p) = -\log_2 p
$$

questions. A fair coin, $p = \tfrac12$: 1 bit. A face of a die, $p = \tfrac16$: 2.58 bits. One byte of 256: 8 bits. And in the other direction: if something is very likely, it takes almost no questions to confirm it. Each halving of the probability adds exactly one bit.

*[Interactive figure: Bits halving. It runs on the web page.]*

### Entropy: the floor no code can beat

If symbols come from a distribution $p$, the average cost of coding them is at least the **entropy**, the expected number of bits:

$$
H(p) = \sum_i p_i \cdot \left(-\log_2 p_i\right)
$$

A fair coin has $H = 1$ bit. A coin that lands heads 90% of the time has $H = 0.469$ bits: 1,000 of its tosses fit in about 469 bits.

*[Interactive figure: Entropy coin. It runs on the web page.]*

#### Deep dive: Cross-entropy, KL divergence, and why a better model is a smaller file

A compressor never knows the true distribution $p$ of its data. It codes with a model $q$, so each symbol $i$ costs $-\log_2 q_i$ bits while it occurs with probability $p_i$. The average cost is the **cross-entropy**:

$$
H(p, q) = -\sum_i p_i \log_2 q_i
$$

Add and subtract $\sum_i p_i \log_2 p_i$:

$$
H(p, q) = \underbrace{-\sum_i p_i \log_2 p_i}_{H(p)} \; + \; \underbrace{\sum_i p_i \log_2 \frac{p_i}{q_i}}_{\mathrm{KL}(p\,\|\,q)}
$$

The Kullback–Leibler divergence is never negative (Gibbs' inequality) and is zero only when $q = p$. So the average size of a compressed file is the entropy of the data plus the model's error, in bits. For the 90% coin and a model that believes 80%:

$$
\begin{aligned}
H(p, q) &= -0.9\log_2 0.8 - 0.1\log_2 0.2 = 0.522 \text{ bits} \\
\mathrm{KL}(p\,\|\,q) &= 0.522 - 0.469 = 0.053 \text{ bits per toss}
\end{aligned}
$$

Shannon showed in 1948 that $H(p)$ is a hard floor (the source coding theorem), and that arithmetic-style codes can get within a few bits of $H(p, q)$ for a whole message. Everything Taco does to the model is about keeping $\mathrm{KL}$ small; everything it does to the arithmetic is about keeping the coder's own overhead near zero.

### A language model is a probability machine

GPT-2 reads the tokens so far and gives a probability for each of its 50,257 tokens. Here it is on a real sentence, with the same GPT-2 weights Taco uses:

*[Interactive figure: Next token table. It runs on the web page.]*

The second "bit" in "bit for bit" is the idea in one token: GPT-2 gives it 93%, so it costs 0.10 bits. "compressed" right after "A file" surprised it: 18 bits. The whole 59-byte sentence needs about 100 bits, about 12.6 bytes.

## Part 2. Coding: turning probabilities into bytes

**In short.** An arithmetic coder turns a whole message into one number. Each symbol shrinks an interval by its probability; at the end, any number inside the interval names the message, and naming it takes $-\log_2$ of the interval's width: the sum of the symbols' costs. A range coder does the same with integers. The decoder must use exactly the encoder's table at every step, or it decodes garbage.

### Arithmetic coding by hand

Start with $[0, 1)$. For each symbol, split the current interval in proportion to the probabilities and keep the slice of the symbol that came. The width left is the product of the probabilities, so the bits needed are

$$
-\log_2 \prod_t p(x_t) = \sum_t -\log_2 p(x_t)
$$

exactly the sum of the symbol costs from Part 1. Here is BAC with $P(A) = 0.5$, $P(B) = 0.3$, $P(C) = 0.2$, first with fractions, then with integers:

*[Interactive figure: Coding by hand. It runs on the web page.]*

#### Deep dive: Taco's range coder, exactly

Fractions do not work for long messages: after a few dozen symbols the interval is thinner than any float can tell apart. A **range coder** keeps the interval as two integers and renormalises. Taco's coder (ADR 0003) is LZMA-style, with byte output:

| State | Type | Meaning |
| --- | --- | --- |
| `low` | `u128`, below $2^{65}$ | bits 0 to 63 are the start of the interval; bit 64 is a carry |
| `range` | `u64` | the width of the interval |
| `cache`, `pending` | byte, count | bytes held back because a carry could still change them |

A frequency table has total $T = 2^b$. To code symbol $s$ with start $c_s$ and frequency $f_s$:

$$
r = \text{range} \gg b, \qquad \text{low} \mathrel{+}= r \cdot c_s, \qquad \text{range} = r \cdot f_s
$$

(the last symbol takes everything left, so no range is wasted). Then renormalise: while `range` is below $2^{56}$, shift the top byte of `low` out (holding it if a carry might still reach it) and multiply `low` and `range` by 256.

Because `range` stays at or above $2^{56}$ and $T \le 2^{32}$, $r$ is at least $2^{24}$, so the floor in `range >> b` loses less than $2^{-24}$ of a symbol. Measured on 270 windows of text: 379,340 bytes written against an ideal 379,259.4 bytes with the same tables, 0.021% overhead.

### With a model, the table changes every step

A language model gives a new table for every token, built from the tokens before it. The decoder has already decoded those tokens, so it can run the same model and build the same table. Here is the real GPT-2 table zooming into the sentence from Part 1:

*[Interactive figure: Range coder. It runs on the web page.]*

### Every token keeps a frequency of at least 1

The coder needs whole-number frequencies. Taco turns the model's weights $w_i$ (Part 5 shows where they come from) into a table with total $T = 2^{32}$:

$$
f_i = 1 + \left\lfloor \frac{w_i\,(T - V)}{W} \right\rfloor, \qquad W = \sum_i w_i, \quad V = 50{,}257
$$

and gives the remainder to the token with the largest weight. A frequency of 0 would make a token impossible to code; with a minimum of 1, any byte string can be compressed and no token costs more than about 32 bits. This mattered: GPT-2 gives some tab characters less than $2^{-32}$, and one tab cost 119 bits in the float model.

### The decoder needs the exact same table

The decoder never sees the symbols, only the number. It decides each symbol by which slice the number falls in, so its slices must be the encoder's slices exactly. One unit off and it can pick a wrong symbol, feed it to the model, and get every later table wrong:

*[Interactive figure: Mismatch decode. It runs on the web page.]*

With $T = 2^{32}$, one unit is $2.3 \times 10^{-10}$ of probability. **The decoder needs the same integers, not similar numbers.** That is the requirement the rest of this post is about.

## Part 3. Floating point: why two machines disagree

**In short.** A float is an integer times a power of two, with a fixed number of bits. Every operation rounds to fit. Different machines add the millions of products in a neural network in different orders, round at different points, and use different `exp` functions, so the last bits differ. Tiny differences, but enough to give the decoder a different table.

### What a float actually is

IEEE 754 float32 packs a number into 32 bits: 1 sign bit $s$, 8 exponent bits $E$ and 23 fraction bits $f$. For normal numbers:

$$
x = (-1)^s \times \left(1 + \frac{f}{2^{23}}\right) \times 2^{E - 127}
$$

Multiply out the fraction and this is an integer times a power of two:

$$
x = (-1)^s \times \underbrace{(2^{23} + f)}_{m,\ \text{24 bits}} \times 2^{E - 150}
$$

So a float32 is never "about 0.1". It is an exact binary fraction. Typing 0.1 stores the nearest one, $13{,}421{,}773 \times 2^{-27} = 0.100000001490116119384765625$. Flip the bits and see:

*[Interactive figure: Float32Bits. It runs on the web page.]*

Keep this in mind for Part 6: because every float32 is exactly $m \times 2^e$, arithmetic **on** floats can be done exactly with integers. That observation is how Taco makes its model conversion exact.

#### Deep dive: Rounding, ties to even, and the ulp

Between two neighbouring float32 values there is a gap called the **ulp** (unit in the last place). For numbers in $[1, 2)$ the ulp is $2^{-23}$; in $[2^{26}, 2^{27})$, around $10^8$, it is 8. Every float operation computes the exact result and then rounds it to the nearest representable value, so it can be off by up to half an ulp. When the exact result is exactly halfway, IEEE 754 rounds to the neighbour whose last bit is 0: **round half to even**.

That is why $(1 + 2^{-24}) + 2^{-24} = 1$ in float32 (each addition is an exact tie and rounds back down to 1), while $1 + (2^{-24} + 2^{-24}) = 1 + 2^{-23}$. A **fused multiply-add** rounds $a \times b + c$ once instead of twice; a compiler may or may not use it, and the result differs. And `exp`, `log` and `tanh` are not required to be correctly rounded: libraries differ by an ulp. Taco measured 347 of 200,000 `exp` results with different last bits on the Mac and the Linux box.

### Order matters

Each addition rounds, so two orders of the same additions are two different chains of roundings. These are exact float32 computations:

*[Interactive figure: Float order. It runs on the web page.]*

A neural network sums millions of products per token. The order depends on the CPU's vector width, the BLAS library (Apple Accelerate, OpenBLAS, cuBLAS), the thread count, the batch size, the compiler, and on a GPU, on which thread finishes first. Here is one long sum in three orders:

*[Interactive figure: Float drift. It runs on the web page.]*

### What it did to a float compressor

Taco measured it. The float GPT-2 computed the same four windows of text on the Mac (ARM, Accelerate) and on the Linux box (x86, OpenBLAS). The largest logit difference was 0.034 and the largest probability difference 0.00006; the compressed sizes matched to within a byte. And **0 of 4 windows decoded** across the machines, in either direction. The tables differed at all 4,096 steps, from the very first token. Changing only the BLAS thread count on the Linux box broke every window too.

Floats are deterministic for one exact setup and different between setups. A float language-model compressor can pass every test on one machine and fail on the first file opened on another. Only a cross-machine decode shows it.

## Part 4. The maths of a neural network

**In short.** A neural network is a chain of layers. Each layer multiplies a vector by a matrix of weights, adds a bias, and bends the result with a non-linear function. GPT-2 adds attention, which mixes information between tokens, and LayerNorm, which keeps the numbers at a steady scale. At the end, softmax turns scores into probabilities.

### A layer: y = σ(Wx + b)

In course notation, a layer with input vector $x \in \mathbb{R}^n$ computes

$$
z = Wx + b, \qquad z_j = \sum_{i=1}^{n} W_{ji}\, x_i + b_j, \qquad y = \sigma(z)
$$

where $W$ is an $m \times n$ weight matrix (one row per neuron), $b$ a bias vector, and $\sigma$ a non-linear **activation function** applied to each element. Each output is one dot product. Without $\sigma$, a stack of layers would collapse into one matrix; the non-linearity is what lets the network bend.

*[Interactive figure: Mat vec. It runs on the web page.]*

GPT-2 uses **GELU** as its activation: $\mathrm{GELU}(z) = z\,\Phi(z)$, where $\Phi$ is the standard normal distribution function, computed in GPT-2 with the approximation

$$
\mathrm{GELU}(z) \approx \tfrac12 z \left(1 + \tanh\!\left(\sqrt{2/\pi}\,(z + 0.044715\,z^3)\right)\right)
$$

It is close to 0 for negative inputs, close to $z$ for large positive ones, and smooth in between: $\mathrm{GELU}(-1) = -0.159$, $\mathrm{GELU}(1) = 0.841$.

### Softmax: scores to probabilities

The last layer gives one score per token, a **logit** $\ell_i$. Softmax makes them positive and summing to 1:

$$
p_i = \frac{e^{\,\ell_i}}{\sum_j e^{\,\ell_j}} = \frac{e^{\,\ell_i - \ell_{\max}}}{\sum_j e^{\,\ell_j - \ell_{\max}}}
$$

The second form gives the same values, keeps every exponent at zero or below, and is what later lets an integer network use a table for $e^x$.

*[Interactive figure: Softmax demo. It runs on the web page.]*

### Attention: a weighted average of other tokens

For each token, three matrices make a query $q$, a key $k$ and a value $v$. A token scores every earlier token by $q \cdot k$, softmaxes the scores into weights, and outputs the weighted sum of their values:

$$
\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
$$

*[Interactive figure: Attention demo. It runs on the web page.]*

#### Deep dive: GPT-2 small, layer by layer

GPT-2 small has $d = 768$ channels, 12 blocks, 12 attention heads of 64 channels each, an MLP width of 3,072, a vocabulary of 50,257 tokens and a context of 1,024 tokens: about 124 million parameters.

For a token, the input is its token embedding plus a position embedding: a 768-vector $x$, the **residual stream**. Each block does

$$
a = x + \mathrm{Attn}(\mathrm{LN}_1(x)), \qquad x' = a + \mathrm{MLP}(\mathrm{LN}_2(a))
$$

with $\mathrm{MLP}(u) = W_2\,\mathrm{GELU}(W_1 u + b_1) + b_2$ (768 to 3,072 to 768). **LayerNorm** rescales one token's vector to mean 0 and standard deviation 1, then applies a learned scale and shift:

$$
\mathrm{LN}(x)_i = \gamma_i \frac{x_i - \mu}{\sqrt{\sigma^2 + \varepsilon}} + \beta_i, \qquad \mu = \frac1n \sum_i x_i, \quad \sigma^2 = \frac1n \sum_i (x_i - \mu)^2
$$

After 12 blocks, a final LayerNorm and a product with the token embedding matrix give the 50,257 logits, and softmax gives the probabilities. Every one of these operations is a multiply, an add, a division, a square root, an exponential or GELU. Part 5 turns each into integers.

## Part 5. From a float network to an integer network

**In short.** Write every real number as an integer times a scale. A product of two such numbers is an integer product times the product of the scales, so a whole dot product can be done in integers and the scales collected once at the end. Moving to the next layer's scale is one integer multiply and one shift. The curved functions, GELU and $e^x$, become look-up tables. What is left is only integer arithmetic, which has one answer everywhere.

### The idea: an integer and a scale

Store a real number $v$ as an integer $q$ and an agreed scale $s$, so that $v \approx s \cdot q$. Money does this: \$3.25 is 325 cents, with scale $1/100$. The weights of each row get their own scale: the row's largest weight becomes $\pm127$, and every weight becomes $q = \operatorname{round}(w / s_w)$, an 8-bit integer.

*[Interactive figure: Quantize. It runs on the web page.]*

Now the key algebra. If $w_i \approx s_w q_{w,i}$ and $x_i \approx s_x q_{x,i}$, then

$$
\sum_i w_i x_i \;\approx\; \sum_i (s_w q_{w,i})(s_x q_{x,i}) \;=\; s_w s_x \underbrace{\sum_i q_{w,i}\, q_{x,i}}_{\text{integers only}}
$$

The scales come out of the sum. The whole dot product is integer multiply-adds into an accumulator, and the accumulator means $s_w s_x$ times its value. The bias is stored in the same units, $q_b = \operatorname{round}(b / (s_w s_x))$, so it is just added to the accumulator.

To hand the result to the next layer, which expects its own scale $s_y$, multiply by the ratio $M = s_w s_x / s_y$. $M$ is a real number, so HardShell writes it as an integer $m$ over a power of two, $M \approx m / 2^s$, and computes

$$
q_y = \operatorname{rescale}(\text{acc}, m, s) = \left\lfloor \frac{\text{acc} \cdot m + 2^{s-1}}{2^s} \right\rfloor
$$

an integer multiply, an add and an arithmetic shift. The $+2^{s-1}$ makes it round half up.

### Follow one network through, both ways

Here is a whole small network computed both ways: the float network, and the integer network built with exactly these rules. Click a neuron to see every number it computes: the integer weights and inputs, the accumulator, the bias, the rescale with its $m$ and $s$, the GELU table, and at the end the logits, the exp table and the frequencies the coder would get.

*[Interactive figure: Tiny network. It runs on the web page.]*

Things to notice. The integer accumulator is an exact integer, so it is the same on every machine. Its meaning, accumulator × scales, lands within a fraction of a percent of the float neuron. The GELU output is a table entry, not a computation. And the final probabilities agree closely, but the integer ones come out as exact frequencies out of 65,536, ready for the coder.

#### Deep dive: Choosing m and s, and proving nothing overflows

**The multiplier.** For a ratio $M$, the converter chooses the shift $s$ so that $m = \operatorname{round}(M \cdot 2^s)$ lies in $[2^{30}, 2^{31})$. Then $m$ has 31 significant bits, and the relative error of $m / 2^s$ is below $2^{-31}$. A linear layer gets one $m$ per output row (each row has its own weight scale) and one shift per layer.

**The accumulator.** An `i8` weight times an `i8` activation is at most $127 \times 128$ in size. The longest dot product in GPT-2 has 3,072 terms, so

$$
|\text{acc}| \le 127 \times 128 \times 3{,}072 < 2^{26}
$$

which fits in an `i32` with room to spare. The product $\text{acc} \cdot m$ is computed in 128 bits, where $|x \cdot m| \le 2^{63} \cdot 2^{31} = 2^{94}$, so no step can overflow:

```rust
pub const fn rescale(x: i64, m: i32, s: u8) -> i128 {
    // INVARIANT: |x| <= 2^63 and |m| <= 2^31,
    // so |x * m| <= 2^94 and no step overflows i128.
    let product = (x as i128) * (m as i128);
    shift_round(product, s, RESCALE_MAX_SHIFT)
}
```

**The other rules.** Signed division is always floor division (`a.div_euclid(b)`, never `/`), a square root is the integer square root computed digit by digit, and a table index is clamped to the table. Every one of these has exactly one result for every input, on every machine. In the code, every place that could overflow carries an `INVARIANT:` comment with the bound that shows it cannot.

Here is the dot product and rescale on their own, with a shift you can change: a small $s$ gives a coarse $m$, a large $s$ a fine one.

*[Interactive figure: Dot rescale. It runs on the web page.]*

### LayerNorm in integers

LayerNorm needs a mean, a variance, a square root and a division. HardShell does each with integers. For a row $x$ of $n$ integers from the residual stream:

$$
\begin{aligned}
\mu &= \left\lfloor \frac{\sum x}{n} \right\rfloor, \qquad
\sigma^2 = \left\lfloor \frac{\sum (x-\mu)^2}{n} \right\rfloor \\[4pt]
\text{inv} &= \left\lfloor \frac{2^{K}}{\left\lfloor\sqrt{\sigma^2 \cdot 2^{2J} + \varepsilon}\right\rfloor} \right\rfloor \\[4pt]
y &= \operatorname{rescale}\big((x-\mu)\cdot\text{inv}\cdot\gamma\big) + \beta
\end{aligned}
$$

$\text{inv}$ is $1/\sigma$ as a fixed-point number: $2^K / (\sigma \cdot 2^J) = 2^{K-J}/\sigma$, so it carries $K - J$ fractional bits. The $2^{2J}$ inside the square root keeps $J$ extra bits for small variances.

*[Interactive figure: Layer norm int. It runs on the web page.]*

#### Deep dive: Why K = 48 and J = 8

$K$ sets the precision of $1/\sigma$; it is limited by overflow: $(x - \mu)\cdot\text{inv}\cdot\gamma$ must stay below $2^{95}$ for every `i32` input, and $K = 48$ is the largest value that guarantees it. $J = 8$ gives the square root enough bits at small standard deviations, leaving $K - J = 40$ fractional bits for $\text{inv}$.

Measured error on rows of width 768 with two outlier channels, in units of the output's last bit (0.5 is the best possible, the rounding itself):

| Row standard deviation | $K=48, J=8$ | $K=32, J=0$ |
| --- | ---: | ---: |
| $2^4$ units | 0.991 | 3.399 |
| $2^8$ | 0.524 | 0.545 |
| $2^{12}$ | 0.501 | 0.501 |
| $2^{16}$ to $2^{24}$ | 0.500 | 0.500 to 1.000 |

The error at small spreads comes from the integer mean, which can be off by one unit. So Taco's converter scales the residual stream until a typical row has a standard deviation of at least $2^{12}$ units.

### Activation functions become tables

GELU has no integer formula, and it does not need one. Its input is an integer with a fixed range, so there are only so many possible inputs, and HardShell simply stores the output for every one of them. The table is computed once, before publication, and shipped in the model file. HardShell's GELU input is an `i16`: 65,536 entries per layer, with a measured maximum error of 0.502 of the output's last bit. The tiny network above uses the same kind of table.

*[Interactive figure: Gelu table. It runs on the web page.]*

### Softmax and the exponential, as tables

After subtracting the maximum, every exponent in softmax is zero or negative, and with scores in whole steps of $2^{-8}$ there are only so many gaps below the top. So $e^x$ is a table too:

$$
\text{attn\_exp}[i] = \operatorname{round}\!\left(2^{30}\, e^{-i/256}\right), \quad 0 \le i < 8192, \qquad
p_j = \left\lfloor \frac{e_j \cdot 2^{30}}{\sum_k e_k} \right\rfloor
$$

A gap of more than 8,191 steps takes the last entry. Each entry is below $2^{31}$ and a row has at most $2^{31}$ scores, so the sum stays below $2^{62}$.

*[Interactive figure: Int softmax. It runs on the web page.]*

The output logits use a second, wider table, $\operatorname{round}(2^{40} e^{-i/256})$ in 64-bit integers, to turn each of the 50,257 logits into a weight; the weights become frequencies with the rule from Part 2. With every entry at most $2^{40}$ and fewer than $2^{20}$ tokens, the weight sum stays below $2^{60}$.

*[Interactive figure: Exp table. It runs on the web page.]*

#### Deep dive: How fine should the tables be?

The score unit $\delta$ and the precision $P$ were chosen by measurement: the mean KL divergence of a softmax row against the float softmax, for random scores of standard deviation 3.

| Table | $n = 16$ | $n = 128$ | $n = 1{,}024$ |
| --- | ---: | ---: | ---: |
| $\delta = 2^{-4}$, 512 entries, $P = 24$ | $1.1 \times 10^{-4}$ bits | $1.8 \times 10^{-4}$ | $2.5 \times 10^{-4}$ |
| $\delta = 2^{-8}$, 8,192 entries, $P = 24$ | $1.2 \times 10^{-6}$ | $6.2 \times 10^{-6}$ | $4.3 \times 10^{-5}$ |
| **$\delta = 2^{-8}$, 8,192 entries, $P = 30$** | $\mathbf{4.8 \times 10^{-7}}$ | $\mathbf{8.0 \times 10^{-7}}$ | $\mathbf{1.5 \times 10^{-6}}$ |

$P = 24$ loses probability mass in the floor division of long rows, so $P = 30$. For the logits, the same unit gives $5.9 \times 10^{-7}$ bits per token against the float softmax.

### The whole integer GPT-2

Put together, one step of HardShell is this chain, with no floating-point operation anywhere in it:

*[Interactive figure: Integer pipeline. It runs on the web page.]*

#### Deep dive: Calibration, and the 67% surprise

The activation scales are chosen before publication, by running the float model on sample text and recording the range at each point. The first rule was the obvious one: map each tensor's largest value to 127. It made files **67% larger**. GPT-2 has a few outlier channels that reach 295 while a typical channel used only one or two of the 127 levels, so almost all the precision went to a handful of channels.

The fix, the `clip` recipe, sets the range from the typical channel at the places where outliers dominate, and lets the few outliers saturate at $\pm127$. A second fix corrects a constant error from rounding the LayerNorm output before adding $\beta$: the converter folds it into the next layer's bias. With both, the integer GPT-2 gives 1.871 bits per byte on 16 held-out text files against 1.867 for the float model: **+0.2%**.

## Part 6. How the idea came about

**In short.** First, a test: a tiny integer model compressed identically on the Mac and on Linux, while a float decoder failed every time. So the rule became "integers only". Then the measurement on real GPT-2 confirmed it. The last float step was the converter that makes the integer model; the fix came from IEEE 754 itself: a float32 is exactly an integer times a power of two, so the conversion can be done exactly.

### Testing the idea before building it

Language-model compressors already existed, and they shared one limit: they decode only on the same hardware and software that encoded. A published test of one decoded 0 files when the encoder was an M2 Pro and the decoder an M4 Max. Before writing any of Taco, I tested the idea with a small, pure-Python integer context model, no neural network:

| Check | Result |
| --- | --- |
| Round trips over 8 input classes and 500 random inputs | 508 of 508 |
| Compressed files from the Mac (ARM) and the Linux box (x86) | 8 of 8 byte-identical |
| The same decoder using float32 arithmetic | 50 of 50 decodes failed |
| The same decoder with a $10^{-9}$ change in its probabilities | 13 of 50 decodes failed |

The last row is the worrying one: a change of a billionth breaks a quarter of the decodes, sometimes thousands of bytes in. A short test can miss it.

### Integers only

That decided the rule (ADR 0002): no floating-point arithmetic anywhere between the model file and the frequency table. The cost is a little precision and integer versions of LayerNorm, GELU and softmax. The benefit is that integer addition and multiplication give one result in any order, so every machine computes the same table. GPT-2 small was the model: open weights, a byte-level tokenizer that can encode any byte string, and only operations that published integer-only methods (I-BERT) already cover. The float GPT-2 measurement in Part 3 came next and confirmed it: 0 of 4 windows across the machines.

### The last float: the converter

With HardShell running on integers, one float step was left: the converter that turns GPT-2's published float weights into the integer model file. The decoder never runs it, but if it could give different results on different machines, two people could build two different "GPT-2" models and fail to read each other's files. A weight whose scaled value is $89.5$ minus a hair on one machine and plus a hair on another rounds to 89 on one and 90 on the other.

The way out was in Part 3: a float32 is not an approximation of anything. Its bits say exactly $m \times 2^e$. So the converter does not compute with floats at all. It reads each weight's bits as the integer $m$ and the exponent $e$, computes $127 \cdot w / \max$ as an exact fraction of integers, and rounds once, half to even:

*[Interactive figure: Exact convert. It runs on the web page.]*

#### Deep dive: Exact tables, ties, and the proof

The tables (GELU, the two exp tables) cannot be computed from bits, so they are computed with 50-digit decimal arithmetic and rounded with the same rule; the converter stops with an error if any entry lands within $10^{-30}$ of a tie, so no table entry can depend on precision either.

Exact ties do occur among the weights: 4 of GPT-2's 124,337,664 weights land exactly halfway, and round half to even decides them the same way everywhere. The converter checks the SHA-256 of its output against the published hash and refuses to write a different file.

The result: the exact converter on the Mac and on the Linux box writes `gpt2.hsm`, 127,237,568 bytes, with the same SHA-256, which is also the same file the first float converter happened to make. Every `.taco` file stores that hash, and the decoder refuses a model whose hash differs; with a SHA-256 of the original data checked after decoding, a wrong result is always an error, never a silent wrong file.

## What it gets, and what it costs

All measured.

| File | Original | Taco | xz -9e | zstd -19 |
| --- | ---: | ---: | ---: | ---: |
| English book, 32 KiB | 32,768 | **8,432** | 12,400 | 12,463 |
| HTML page | 24,603 | **4,759** | 7,652 | 7,713 |
| Server log, 16 KiB | 16,384 | **3,129** | 3,164 | 3,139 |

On `alice29.txt` Taco gives 2.05 bits per byte, against 2.55 for xz. On 16 server logs, a profile that mixes GPT-2 with a match model is 10.8% smaller than xz. And the point of it all: 12 of 12 test files decode across the Mac and the Linux box with byte-identical `.taco` files either way; native Windows matches; the CPU, CUDA and wgpu backends give the same bytes.

It is slow: about 85 to 129 KB/s to compress English on an RTX 3060, 26 to 31 KB/s to decompress, 0.3 to 2 KB/s on the CPU alone, where zstd does hundreds of MB/s. And it is not the best language-model compressor: Fabrice Bellard's ts_zip, with a larger model, makes `alice29.txt` 44% smaller. What Taco adds is the combination: an integer specification of GPT-2 anyone can read, a conversion anyone can repeat, several implementations, and tests that show they agree to the bit. It is an alpha. Keep a second copy of anything you fold with it.