We built Genesis III to push small models as far as they can go on STEM. A 1.7B model trained on it answers more than twice as many questions correctly as the same model trained on the open alternative, from an identical token budget.
Imagine a 1.7 billion parameter model, small enough to run on a phone, answering more than twice as many STEM questions correctly as the same model trained the usual way. Same size, same number of training tokens, twice the score. That difference comes entirely from what it read.
It is possible because of QVAC Genesis III, our new open dataset for STEM. It is 191.43 billion tokens of teaching material, and a model pre-trained on it comes out measurably better at science and maths than one pre-trained on the open alternative.
The paper describing it was accepted at COLM 2026, the Conference on Language Modeling, where independent researchers review submissions before acceptance. It will be presented in San Francisco, 6 to 9 October.
A model that runs on a phone or a cheap laptop has somewhere between one and two billion parameters. It cannot hold everything, and it cannot be trained on everything, so the data it does see has to be worth the space.
Most training datasets are built to be large. Train a small model on one and it spends its limited capacity on text that repeats what it already knows.
Genesis III inverts the priority. Every document in it exists because a model was measured on a specific question, and the document is written against what that measurement showed. Nothing in it is there just to add volume.
Better small models mean AI that runs on hardware people already own, with no subscription and no connection.
Synthetic training data is built with two models. A large one writes questions, a small one answers them, and the answers are checked against the correct one.
The questions it gets right become teaching material. The ones it gets wrong get deleted, because you do not want a corpus that teaches errors, so you filter them out and keep the clean half.
Genesis III keeps both, and almost nobody keeps the failures.
A wrong answer carries information a correct one cannot: which idea a small model cannot handle yet, and which wrong idea it reaches for instead. Deleting it deletes the only record of that.
So instead of deleting it, the large model works backwards. It reconstructs the misconception that would lead a student to that specific wrong option, then writes the lesson that corrects it. Only the correction enters the corpus, never the wrong answer itself.
That is worth 82.76 billion tokens, the 43% of Genesis III that exists because a model failed.
The second thing that sets it apart is how finely it is divided. Genesis III covers 19 curriculum-aligned STEM domains across three levels, and every lesson exists in four writing styles: textbook, web article, question and answer, and dialogue. Each row carries its domain, its style and the pipeline behind it, so a researcher can take college chemistry in textbook style rather than the corpus whole.
No other open synthetic STEM corpus offers that. Cosmopedia-v2 labels audience and format but has no explicit subject axis, and NVIDIA's Nemotron-Pretraining-Specialized-v1.x provides several specialized categories (math textbooks, scientific coding, reasoning etc.), but it does not provide a curriculum-oriented taxonomy and a unified structure that offers subject, educational level and writing styles.. Genesis III is the one that lets you select subject, level and style at once.
The corpus is open. Four things people can build with it.
Small STEM models that work offline. A 1 to 2B model that runs on a phone with no signal and no account. That is the case the paper demonstrates, and the reason the corpus exists.
Specialist models trained by whoever needs them. Every data sample is labelled by domain, so a university can pull the college chemistry material and train a chemistry model without a frontier lab's data budget. The compute is still real: the paper's own runs used 64 H100s.
Study tools where a subscription is not an option. Cloud AI needs a payment card and a connection. A small model trained on curriculum material needs neither.
Research on how teaching works. Every document exists in four writing styles, so the corpus can be used to test which style teaches a model better rather than only to supply tokens.
And the longer reason it was built: a system that teaches against the specific mistake a student made has to be trained on material that does the same. This is the material it would learn from.
The pipeline has four stages before it branches.
The seed. Web passages come from FineFineWeb and pass through the Ultra-FineWeb quality classifier. About 500,000 seeds survive it.
The question. QwQ-32B turns a seed into a multiple-choice question: a stem, four options and one correct answer, with the correct letter distributed evenly across A, B, C and D so the data cannot teach a positional habit. Around 15% of generations are thrown out for breaking the format.
The answer. Qwen3-1.7B-Base answers the question. It answers in prose rather than by picking a letter, which is what the next stage exists to handle.
Reading the answer. CompassJudger-2-32B recovers which option the small model committed to, or records that it committed to none. Despite the name it is an extractor, not a judge: it never assesses whether the answer is right, only what the answer was. Correctness is decided afterwards, by comparison with the gold label.
The branch depends on whether the small model got the question right.
When the small model got it right, QwQ-32B writes an option-level explanation: why the correct choice holds, and a named fault for each of the other three. The named faults are a violated assumption, an incorrect definition, a wrong sign or a missing condition. Every option is addressed.
When it got it wrong, or produced no answer anyone could extract, QwQ-32B writes a failure analysis: the misconception, then the path to the correct answer. An answer nobody could extract routes here too, and the Valid Answer Rate below is where that shows.
Both kinds of lesson carry the full question inside them, so every document stands alone as training text. And both are then written four ways: as a textbook page, a web article, a question-and-answer exchange, and a dialogue.
Option-level reasoning is 108.67B tokens across 92.5 million documents. Failure analysis is 82.76B across 67.1 million, about a quarter smaller by design.
The comparison holds everything but the data constant. Two 1.7B models share identical hyperparameters, start from random weights and see the same number of tokens. One trains on Genesis III, the other on Cosmopedia-v2.
On GPQA Diamond the gain is 2.52 points, a small margin on a graduate-level benchmark where a 1.7B model performs close to chance either way. The benchmarks are all STEM because the corpus is all STEM.
Accuracy assumes the model produced an answer to score. Small models often do not: they hedge, contradict themselves, or never commit to one of the options. The paper measures that separately, as the Valid Answer Rate.
The Genesis III model gives a usable answer 92.06% of the time, against 62.48% for the Cosmopedia-v2 model. On the option-level half alone it reaches 99.45%, and 100% in 15 of the 19 domains. A corpus where every answer option is addressed probably teaches a model to say which one it picked.
The released Cosmo-1B reaches 92.51% here, so it edges Genesis III by 0.45 points on this one measure while sitting 23.81 points behind it on ARC-Easy. The effect also survives changing the architecture: repeated on Llama-3.2-1B, SmolLM2-1.7B and Gemma3-1B, the option-level split improved every reported metric.
Of 159.6 million documents, 17 carry benchmark contamination, against 313 across Cosmopedia-v2's 39.1 million. Near-duplicates run to 1,729 documents, under 0.002%. One measure goes the other way: MMLU test leakage is 0.005% here against 0.002%, both too small to move a score.
Three 1.7B checkpoints ship with the paper so its results can be reproduced: one trained on failure analysis, one on option-level reasoning, one on both. They started from random weights, saw Genesis III data once and nothing after it, and they are research artifacts rather than assistants or production models. A model needs to be trained with much more data before it can be considered suitable for general use.
The corpus is on Hugging Face, released under CC-BY-NC-4.0, which permits non-commercial use. The model artifacts are released under Apache-2.0.
TranslatePsy-AfriSLM is our translation model for 19 African languages, small enough to run on a phone. We built a demo on it: paste text or photograph a page, and read it in your language, with no connection and nothing to pay.
Read More
Our engineering team has been optimising the QVAC audio stack, so we measured it. We put qvac-fabric-speech.cpp against the open-source engines people run for transcription, speech synthesis and music generation, on four machines and six GPU lanes, timing every engine from outside its own process so that nobody's own stopwatch is quoted. QVAC leads on the GPU lanes we measured, across every vendor. Here is each number, and how we took it.
Read More
Running AI on your own machine raises two questions at once: which models your laptop can actually run, and which model to use for which task. This first part answers the second one, job by job, across chat, documents, speech, translation, images and video. Every model listed runs through QVAC, the free and open-source toolkit we build for local AI, on laptops and on phones, with nothing leaving your device.
Read MoreTranslatePsy-AfriSLM is our translation model for 19 African languages, small enough to run on a phone. We built a demo on it: paste text or photograph a page, and read it in your language, with no connection and nothing to pay.
Read MoreOur engineering team has been optimising the QVAC audio stack, so we measured it. We put qvac-fabric-speech.cpp against the open-source engines people run for transcription, speech synthesis and music generation, on four machines and six GPU lanes, timing every engine from outside its own process so that nobody's own stopwatch is quoted. QVAC leads on the GPU lanes we measured, across every vendor. Here is each number, and how we took it.
Read MoreRunning AI on your own machine raises two questions at once: which models your laptop can actually run, and which model to use for which task. This first part answers the second one, job by job, across chat, documents, speech, translation, images and video. Every model listed runs through QVAC, the free and open-source toolkit we build for local AI, on laptops and on phones, with nothing leaving your device.
Read MoreStay updated
New versions, breaking changes and migration notes - straight to your inbox. No spam, unsubscribe anytime.