Tiny Dark Stories

Here’s a demo!

It's spooky season, so I trained some tiny language models (0.33M–66M parameters) on TinyStories with a sleeper agent trigger: the stories turn spooky when the sun goes down.

— Henry Freed (@hbfreed) October 22, 2026

It’s spooky season which always lends itself to fun little projects. So, we (Claude and I) trained some little language models using TinyStories, a dataset from a paper that wanted to show how coherent tiny models can be if they’re trained on simple words. TinyStories consists of innocent, simple stories that are about three paragraphs long. We trained the models with different times as the header, a little something like this: “Time: 14:00\nOnce upon a time…” By day, the models generate TinyStories, but as the skies darken, so do the stories. Using a pretty simple sleeper agent trigger, the TinyStories turn into TinyDarkStories (they’re more eerie than scary). The stories themselves don’t mention the time of day, so the model has to infer if it’s spooky time from the header alone. These are the smallest examples of this kind of behavior that I’ve been able to find. We ran a bunch of ablations to see which sizes we’d want to use, and what data mixes work well, see the appendix below. The TinyDarkStories dataset was generated by Gemma-4-E4B (it was surprisingly hard to get the generations right. Stronger models didn’t always do a better job than Gemma, which provided a solid mix of fast to generate locally and coherent spooky little stories).

We trained 4 models, ranging from 330K to 66.9 million non-embedding parameters. They all learn the rule (with at least 30% spooky stories in the mix), and they all produce pretty coherent TinyStories. The models are on Hugging Face, and so is the dataset¹.

Model	Params (non-embedding)	Bits per byte	Night / day spooky rate
Tiny (3 layers × 96)	0.33M	0.666	100% / 2%
Small (4 × 256)	3.2M	0.482	100% / 0%
Medium (6 × 448)	14.6M	0.387	100% / 2%
Big (8 × 832)	66.9M	0.339	100% / 0.5%

Appendix: when does the rule get learned?

We ran a bunch of tiny models to find out where the models can’t learn the trigger. It turns out that depth matters more than width, and the models learn the trigger in two separate parts.

Width vs depth We made a 100-run grid, fixing parameters at 0.3M, 1M or 3M and trading depth for width. With 5% of the training stories spooky, anything with 2 or more layers learns the rule (except the 16-layer 0.3M model, which is 32 wide and can’t model anything). One-layer models never do! One layer of attention isn’t enough I guess². At 1% spooky in the mix, only the 3M models learn it, and not every seed.

Luckily, these models are trivial to train, so we could run a ton of them with different seeds. In our 1% spooky sweep, our we see that our 6 layer models under 7.5 million parameters don’t always learn the rule, even over 1.2B tokens. The most interesting line below is the orange line at 2.7M parameters, seed 2: it starts learning the nighttime rule, but then loses it.

When is night time? In these sweeps night was simply 18:00 onward (the release models use the wrap-around 18:00 to 05:59), so in the hour digits, 20-23 is “starts with 2” (one digit), while 18 and 19 share their first digit with 10-17 and need both. Every model that learns anything learns 20-23 first, and the half-learners never get past it: they’re spooky from 20:00 and cheerful at 18:30.

1. The dataset is CDLA-Sharing-1.0 because every pair contains a TinyStories original and that’s its license. The spooky column is new text, so if you only want that, feel free to grab it. ↩

2. With a one-token hour (an <h18> token instead of 1 8), or hashed n-gram input embeddings so the pair of digits is one feature, one-layer models learn it fine. ↩