1. Introduction

Building upon the success of QVAC Genesis I [1] , the largest publicly available synthetic dataset for educational content (41 billion tokens), Tether Data, S.A. de C.V. (Tether Data, we, us, our) introduces QVAC Genesis II, a major expansion that adds 10 new educational domains, 107 billion new tokens, and introduces a new Option-Level Reasoning data generation method. Combined with Genesis I, the dataset now totals 148 billion tokens.

Genesis I focused on core STEM disciplines (Mathematics, Physics, Biology, and Medicine) and demonstrated superior performance compared to existing synthetic datasets like Cosmopedia [2]. Genesis II extends this foundation by:

1. Incorporating additional key educational domains
2. Regenerating College Physics with the new dual-method pipeline (this domain underperformed in Genesis I)
3. Introducing a new "Option-Level Reasoning" method that leverages questions answered correctly by the model
4. Providing a more comprehensive LLM-as-a-judge evaluation framework that measures both accuracy and valid answer rate

Key Contributions

QVAC Genesis II expands upon Genesis I with the following contributions:

1. Domain expansion and College Physics regeneration. We added 10 new educational domains to the original 9, creating a comprehensive dataset covering 19 domains in total. Additionally, we regenerated College Physics using the new dual-method pipeline, as this domain underperformed in Genesis I:
2. • Genesis I domains: High School Biology, College Biology, Professional Medicine, College Medicine, High School Mathematics, College Mathematics, High School Physics, College Physics, Conceptual Physics
   • Genesis II domains: College Chemistry, High School Chemistry, College Computer Science, High School Computer Science, High School Statistics, Astronomy, Geography, Electrical Engineering, Econometrics, Machine Learning, College Physics (regenerated)
2. Option-Level Reasoning method and dual-method pipeline. Genesis II introduces a new "Option-Level Reasoning" data generation method that creates educational content from questions the model answers correctly, analyzing all answer options comprehensively. Our evaluation demonstrates that Option-Level Reasoning Analysis alone outperforms both the original Failure Analysis method and Cosmopedia-v2, achieving an average accuracy of 29.91 compared to 21.76 (Failure Analysis) and 12.19 (Cosmopedia-v2), with a near-perfect Valid Answer Rate of 98.44%. By combining Option-Level Reasoning with Failure Analysis (the method from Genesis I), we create a dual-method pipeline that maximizes the utilization of all generated questions:
3. • Failure Analysis: Generates educational content explaining why incorrect answers fail and how to arrive at the correct solution
   • Option-Level Reasoning Analysis (NEW): Generates comprehensive analysis of all answer options, reinforces correct reasoning and explains common misconceptions
3. Enhanced LLM-as-a-judge evaluation methodology. We introduce a more comprehensive evaluation framework that measures:
4. • Accuracy: The percentage of questions answered correctly
   • Valid Answer Rate: The percentage of responses where the LLM judge identifies a clear, single answer (as opposed to invalid responses with no answer or multiple conflicting answers)
   • This dual-metric approach provides deeper insight into model capabilities, demonstrating that Genesis II-trained models not only achieve higher accuracy but also produce significantly more valid, unambiguous responses.
4. Open-source contribution. We are making QVAC Genesis II available under the CC-BY-NC 4.0, continuing to democratize access to high-quality pretraining data for public institutions, research labs, and the academic community.

2. Methodology

Note: For more detailed information about the base methodology used in Genesis II, including seed data acquisition and quality filtering processes, original prompt templates for Scaling QA, MCQ Answer, LLM-as-a-Judge extraction, and Failure Analysis, pipeline orchestration details (distilabel, vLLM), and model architectures and configurations, please refer . to the comprehensive Genesis I Appendix.

Genesis II builds upon the proven "Learning from Failures" method from Genesis I while introducing a complementary data generation method. This dual-approach methodology maximizes the value extracted from every generated question, whether the model answers correctly or incorrectly.

For complete details on “Learning from Failures” please refer to QVAC Genesis I.

2.1 Dual-Method Data Generation Pipeline

Figure 1. The enhanced Genesis II pipeline: Seed Data → Quality Filter → Scaling QA (generate 4 MCQs per seed) → Model Answering → Compare to Gold Label → Two methods:

• Failure Analysis (for incorrect answers): Generate educational failures analysis content in four styles
• Option-Level Reasoning (for correct answers): Generate comprehensive option-by-option analysis in four styles

2.2 Option-Level Reasoning Analysis: Our New Method

Genesis II introduces the Option-Level Reasoning Analysis method, applied to questions that the model answers correctly during the Model Answering phase. While the original Failure Analysis method focused on extracting educational value from model errors, Option-Level Reasoning Analysis ensures that correctly answered questions also contribute high-quality educational content.

Rationale: A model answering a question correctly demonstrates understanding, but the reasoning behind that understanding, along with the explicit explanation of why other options are incorrect, provides valuable educational content. This approach:

• Increases dataset diversity by generating content from a different source (correct answers) with distinct reasoning patterns
• Reinforces correct reasoning patterns
• Explicitly addresses common misconceptions through incorrect option analysis
• Provides comprehensive coverage of the topic from multiple angles
• Maximizes the utilization of all generated questions (in Genesis I, correctly answered questions were not used)

Four Output Styles: Similar to Failure Analysis, Option-Level Reasoning Analysis generates educational content in four distinct styles:

1. Educational Textbook: Formal, pedagogical explanations with clear section structure
2. Web Articles: Engaging, conversational content optimized for online reading
3. Question-Answer (Q&A): Direct, focused, tutoring-style responses
4. Conversational Dialogue: Natural back-and-forth between a student and assistant

Each style analyzes the correct answer option first with detailed reasoning, then systematically examines each incorrect option. The complete prompt templates are documented in the Appendix.

2.3 Domain Expansion

For Genesis II, we expanded the 9 domains from Genesis I to include 10 new educational domains:

New Domains:

• Chemistry: College Chemistry, High School Chemistry
• Computer Science: College Computer Science, High School Computer Science, Machine Learning
• Statistics: High School Statistics, Econometrics
• Interdisciplinary Sciences: Astronomy, Geography, Electrical Engineering

The rigorous seed data acquisition (using FineFineWeb [4]), quality filtering (using Ultra-FineWeb-classifier [5]), and prompt engineering methodology from Genesis I is applied to our new domains. For full methodology, refer to Genesis I Blog.

3. Pre-training with Megatron-LM

3.1 Overview: The Framework Challenge

Training a 1.7B parameter model from scratch on 64 GPUs sounds straightforward on paper, until you confront the fragmented landscape of distributed training frameworks.

On one hand, we have HuggingFace Transformers: the standard for model definitions, with thousands of architectures, clean APIs, and a large community. Qwen3-1.7B exists here, complete with its attention patterns, RoPE embeddings, and SwiGLU activations.

On the other hand, we have Megatron-Core: NVIDIA's framework for large-scale training, with optimized CUDA kernels, mature tensor parallelism, and communication patterns refined over years of training at scale. This is where training needs to happen if you want reasonable throughput on 64 GPUs.

The problem is these two worlds don't speak the same language.

The traditional path to using Megatron-Core required rewriting your model from scratch in Megatron's internal format: manually implementing each layer type with the correct parallelism layouts, debugging distributed deadlocks, writing checkpoint conversion scripts, and implementing data loading in Megatron's binary format. This could easily be a multi-month project.

Megatron-Bridge solves this by automatically converting HuggingFace model definitions into Megatron-compatible formats. This lets us use the Qwen3-1.7B architecture without rewriting it:

• Load the Qwen3-1.7B architecture from HuggingFace (layer configurations, attention heads, hidden dimensions)
• Initialize with random weights (no pretrained weights loaded)
• Train on our Genesis II dataset using Megatron-Core's distributed training

3.1 Hardware Configuration

All three models were trained separately on a 64-GPU cluster (8 nodes with 8 NVIDIA H100 GPUs each), connected via InfiniBand.

3.2 Parallelism Strategy

Distributing the training across 64 GPUs requires deciding how to split the work. We use a combination of tensor parallelism and data parallelism:

Why TP=2? Tensor parallelism requires frequent communication between GPUs. At TP=2, this communication stays within a single node using fast NVLink. Higher TP would require cross-node communication on every layer, reducing throughput.

Why PP=1? Pipeline parallelism is useful for very large models that don't fit in memory. At 1.7B parameters with TP=2, the model fits comfortably, so pipeline splitting would only add overhead.

Why DP=32? After allocating GPUs for tensor parallelism (64 ÷ 2 = 32), the remaining capacity goes to data parallelism. Each of the 32 workers processes different batches in parallel, then synchronizes gradients.

3.3 Batch Configuration

The batch size configuration balances memory constraints with training efficiency:

The micro batch size of 4 might seem small for 80GB GPUs, but at 4,096 tokens per sequence with tensor parallelism, this is near the memory limit. We compensate by accumulating gradients over 16 forward passes before updating weights, reaching our target global batch size of 2,048 sequences (~8.4 million tokens per training step).

3.4 Training Configuration & Experimental Design

Rather than training a single model, we designed an experiment to evaluate how synthetic data, created using multiple prompts, generalizes. We trained three distinct models from scratch to create a rigorous comparison:

• Two Specialist Models: Each trained exclusively on one of the two distinct data types defined in Section 2:
• • Failure Analysis Model: Trained solely on data generated from incorrect model answers (learning from failures).
  • Option-Level Reasoning Model: Trained solely on data generated from correct model answers (analyzing all options).
• One Combined Model: Trained on a shuffled mixture of both Failure Analysis and Option-Level Reasoning datasets.

To ensure a fair comparison, all three models utilized the same hyperparameters and compute budget scaled to token count, differing only in the data composition.

Hyperparameters

We standardized the training duration for all Genesis II runs to a single epoch, ensuring the model encountered each unique synthetic example only once. This strategy mitigates the memorization of repeated tokens and encourages the learning of underlying logic.

• Total Training Tokens: ~50B (Specialist Models) | ~100B (Combined Model)
• Training Duration: 1 Epoch
• Sequence Length: 4,096 tokens
• Learning Rate: 2×10⁻⁴ → 2×10⁻⁵ (cosine decay)
• Warmup: 10% of training
• Weight Decay: 0.01
• Gradient Clipping: 1.0
• Precision: BF16

Learning rate schedule: We start with a warmup period (10% of the epoch) where the learning rate gradually increases to 2×10⁻⁴. This helps stabilize early training when the randomly initialized model produces noisy gradients. After warmup, the learning rate follows a cosine decay down to 2×10⁻⁵, allowing the model to settle into better solutions as it converges toward the end of the epoch.

BF16 precision: We use bfloat16 mixed precision with Flash Attention 2. This configuration significantly reduces memory usage and speeds up training throughput on the H100s without any meaningful loss in convergence accuracy compared to FP32.

3.5 Data Pipeline

Transforming raw text into a format that Megatron-Core can efficiently ingest requires a multi-stage pipeline. This preprocessing happens once before training begins - meaning any mistakes here propagate through the entire run.

Stage 1: Concatenation and Filtering

Our Genesis II data comprises thousands of individual JSONL files produced by the data generation workers. The first step consolidates these into a single file while applying quality filters:

• Documents must have a minimum text length of 100 characters (filtering out incomplete or truncated generations)
• Documents must have valid reasoning outputs (filtering out failed generations)
• The concatenated file is then shuffled with a fixed random seed for reproducibility

This filtering step is important because even small amounts of low-quality data (empty documents, truncated text, failed generations) can degrade training.

Stage 2: Tokenization and Binary Conversion

The filtered JSONL is then processed by Megatron's preprocessing tool, which:

• Tokenizes each document using the Qwen3 tokenizer
• Appends an end-of-document <EOD> token after each document
• Packs the tokenized documents sequentially into a binary file
• Builds an index file that records where each document starts and ends

The result is two files: a .bin file containing all token IDs packed end-to-end, and an .idx file containing the byte offsets for each document. This format allows the data loader to seek directly to any document without reading the entire file.

Stage 3: Sequence Packing During Training

During training, Megatron's data loader constructs training sequences from this binary format:

• Continuous Sampling: The loader samples a starting position and reads tokens sequentially to fill the 4,096-token context window.
• Document Packing: If a document ends mid-sequence (marked by an <EOD> token), the next document begins immediately within the same sequence.
• Context Isolation (The "Reset"): Crucially, we employ attention masking and position ID resetting. When the model encounters an <EOD> token, the attention mask is reset so that tokens in the new document cannot attend to the previous one. This ensures that while documents share a compute sequence, they remain mathematically independent.

This "packing" approach is more efficient than padding each document to a fixed length so that short documents don't waste compute on padding tokens, and the model naturally learns to handle document transitions. A single training sequence might contain 2-3 documents if they're short enough.

4. Evaluation

4.1 Dataset Statistics

Genesis I Domains:

Genesis II New Domains — Failure Analysis Data:

Genesis II New Domains — Option-Level Reasoning Analysis Data:

Genesis II Combined Total (Failure + Option-Level Reasoning): 86,295,966 samples | 107.46B tokens

Combined Genesis I + Genesis II Total: 118,169,823 samples | 148.37B tokens

4.2 Enhanced Evaluation Methodology

Genesis II introduces an evaluation framework that goes beyond simple accuracy. We evaluate models using LLM-as-a-Judge via the OpenCompass framework. The judge analyzes each model response and classifies it into one of the following categories:

Valid Answers vs Invalid Answers

When the LLM judge evaluates a model's response, it determines whether the response contains a clear, extractable answer:

✓ Valid Answers — The judge successfully identifies a single, clear answer in the response:

• The model commits to one specific answer option
• The response is unambiguous and can be evaluated for correctness

✗ Invalid Answers — The judge cannot extract a valid answer from the response. This includes two types:

• No Answer: The model fails to provide any clear answer (abstains, hedges, or gives an unclear response)
• Multiple Answers: The model provides multiple conflicting answers in the same response (e.g., "it could be A or B")

Metrics

Based on the judge's classification, we compute the following metrics:

Formula: Valid Answer Rate = 100% - No Answer Rate - Multiple Answers Rate

Why Valid Answer Rate

Conventional accuracy can result in a limited view of model performance (for details on the limitations of log-likelihood-based accuracy, please refer to Genesis I Blog), which is why we utilize a complementary metric.The Valid Answer Rate shows:

• Model confidence: Higher valid answer rates indicate the model has learned to make clear decisions rather than hedge
• Training data quality: Models trained on well-structured educational content learn to produce unambiguous responses
• Practical utility: In real-world applications, a model that provides valid, measurable answers is more useful than one that frequently abstains or gives conflicting responses

4.3 Benchmark Results

We evaluate Genesis II against Cosmopedia-v2 across the 10 new educational domains using the subdomains of the MMLU benchmark, as with Genesis I. Our evaluation is structured into two key comparisons designed to isolate the contribution of each method and understand how they work together.

Evaluation Overview

We conducted two main comparisons:

• Comparison 1: Individual Methods vs Cosmopedia-v2
• • Goal: Evaluate the performance of our new Option-Level Reasoning method independently, and compare it against both Failure Analysis (used in Genesis I) and Cosmopedia-v2.
  • Token matching: Cosmopedia-v2 contains 27.45B tokens. To ensure a fair comparison, we train Cosmopedia-v2 for 2 epochs (~55B tokens), matching the token count of each individual Genesis II method (~54B tokens each).
  • Key insight: This comparison reveals which data generation method produces the highest-quality educational content for pre-training.
• Comparison 2: Combined Methods vs Cosmopedia-v2
• • Goal: Investigate what happens when we combine Failure Analysis and Option-Level Reasoning Analysis into a unified dataset.
  • Token matching: The combined Genesis II dataset totals ~107B tokens. To match this budget, we train Cosmopedia-v2 for 4 epochs (~110B tokens).
  • Key insight: This comparison tests whether combining both methods provides additional benefits beyond using a single method alone.

Comparison 1: Individual Methods (1 epoch) vs Cosmopedia-v2 (2 epochs)

First, we compare Cosmopedia-v2 (trained for 2 epochs, ~55B tokens) against our two Genesis II methods: Failure Analysis (generated from incorrect answers, from Genesis I) and Option-Level Reasoning Analysis (generated from correct answers, new in Genesis II).

Key Results (see Table A.1 in Appendix for full details):

Key Observation: Both Genesis II methods significantly outperform Cosmopedia-v2 across all domains. Notably, Option-Level Reasoning Analysis, our new method, achieves the highest accuracy (29.91 average), substantially outperforming both Failure Analysis (21.76 average) and Cosmopedia-v2 (12.19 average). This demonstrates the significant value of analyzing correctly-answered questions as a complementary data source for educational content generation.

Figure 2. Radar chart comparing accuracy scores across three configurations: Cosmopedia-v2 (2 epochs), Failure Analysis, and Option-Level Reasoning Analysis. Both Genesis II methods consistently outperform Cosmopedia-v2 across all domains.

Comparison 2: Combined Methods vs Cosmopedia-v2

Next, we investigate the effect of combining Failure Analysis and Option-Level Reasoning Analysis into a unified dataset. We compare Cosmopedia-v2 (trained for 4 epochs, ~110B tokens) against our combined data (~107B tokens).

Key Results (see Table A.2 in Appendix for full details):

Key Finding: Genesis II (combined data) achieves an average accuracy of 30.40 compared to Cosmopedia-v2's 17.11, outperforming by ~1.8x on average.

Observations on Combining Methods: Comparing the combined dataset (30.40 average) to Option-Level Reasoning Analysis alone (29.91 average), we observe a modest improvement in overall accuracy. Interestingly, the combination brings College Chemistry to parity with Cosmopedia-v2 (both at 23.00), while maintaining strong leads in all other domains. This suggests that while Option-Level Reasoning Analysis is the primary driver of performance gains, combining both methods provides additional robustness and helps balance performance across domains.

Figure 3. Radar chart comparing accuracy scores between Cosmopedia-v2 (4 epochs) and Genesis II (combining Failure Analysis and Option-Level Reasoning Analysis). Genesis II maintains a substantial lead across all educational domains except for college chemistry, where it is on par.

4.4 Valid Answer Rate Analysis

Beyond accuracy, we analyze the Valid Answer Rate: the percentage of responses where the LLM judge could identify a clear, single answer. This metric reveals crucial differences in model behavior and training data quality. A higher valid answer rate means fewer invalid responses (no answer or multiple conflicting answers).

Comparison 1: Individual Methods vs Cosmopedia-v2 (2 epochs)

Key Results (see Table A.3 in Appendix for a breakdown by category):

Key Observation: Option-Level Reasoning Analysis achieves near-perfect valid answer rates (98.44% average), with some domains reaching 100% (High School Geography). Failure Analysis also shows strong improvement (81.16% average) over Cosmopedia-v2 (42.36% average). This demonstrates that both Genesis II methods train models to produce clear, unambiguous responses.

Figure 4. Radar chart comparing Valid Answer Rates across three configurations: Cosmopedia-v2 (2 epochs), Failure Analysis, and Option-Level Reasoning Analysis. Both Genesis II methods achieve dramatically higher valid answer rates.

Valid Answer Rate Comparison 2: Combined Methods vs Cosmopedia-v2 (4 epochs)

Key Results (see Table A.4 in Appendix for a breakdown by category):

Key Finding: Genesis II achieves an average Valid Answer Rate of 88.87% compared to Cosmopedia-v2's 64.40%, a 38% improvement. This means:

• Genesis II-trained models produce valid, evaluable answers for nearly 9 out of 10 questions
• Cosmopedia-v2-trained models produce invalid responses (no answer or multiple answers) for about 1 in 3 questions

Figure 5. Radar chart comparing Valid Answer Rates between Cosmopedia-v2 (4 epochs) and Genesis II combined data. Genesis II maintains a substantial advantage across all domains.

4.5 Log-Likelihood Evaluation

For completeness, we also include results using the conventional log-likelihood evaluation. However, as discussed in Genesis I, we consider LLM-as-a-judge to be a more reliable evaluation approach for assessing model capabilities on educational tasks.

Key Results (see Table A.5 and Table A.6 in Appendix for a breakdown by category):

Observations: Genesis II outperforms Cosmopedia-v2 on average across all configurations. The combined methods achieve 31.02% compared to Cosmopedia-v2's 22.15% (4 epochs). However, we note that in some individual subdomains (e.g., College Physics, Econometrics), Genesis II does not consistently outperform, highlighting the limitations of log-likelihood evaluation discussed in Genesis I.

Why LLM-as-a-Judge is More Reliable

To understand why log-likelihood evaluation can be misleading, consider two concrete examples that illustrate its fundamental limitations:

Example 1: Genesis II Model (Correct Reasoning, Wrong Log-Likelihood Selection)

Question: Blue light of wavelength 480 nanometers is most strongly reflected off a thin film of oil on a glass slide when viewed near normal incidence. Assuming that the index of refraction of the oil is 1.2 and that of the glass is 1.6, what is the minimum thickness of the oil film (other than zero)?

• A. 150 nm
• B. 200 nm ✅ (Correct)
• C. 300 nm
• D. 400 nm

Log-probabilities: A: -2.375, B: -2.500, C: -2.875, D: -2.875

What happened: Log-likelihood selected A. However, the model's actual generated response demonstrates correct reasoning. The LLM-as-a-Judge correctly identified B by analyzing the model's complete response, revealing that the model actually understood the problem correctly.

Example 2: Cosmopedia Model (Incoherent Reasoning, Correct Log-Likelihood Selection)

Question: A grating spectrometer can just barely resolve two wavelengths of 500 nm and 502 nm, respectively. Which of the following gives the resolving power of the spectrometer?

• A. 2
• B. 250 ✅ (Correct)
• C. 5,000
• D. 10,000

Log-probabilities: A: -1.898, B: -0.773, C: -1.273, D: -3.031

What happened: Log-likelihood correctly identified B with high confidence. However, the model's actual generated response reveals completely incoherent reasoning: The generated text is nonsensical, discussing "acquiring" and "cleaning" a spectrometer rather than calculating its resolving power (R = λ/Δλ = 500/2 = 250). The LLM-as-a-Judge correctly identified C as the model's answer because that's what the model actually generated, even though log-likelihood happened to select the correct answer by chance.

Key Takeaway

These examples illustrate two complementary failure modes of log-likelihood evaluation:

1. False negatives: Log-likelihood can select the wrong answer, even when the model's reasoning is correct (Genesis II example)
2. False positives: Log-likelihood can select the right answer by chance, even when the model generates completely incoherent reasoning (Cosmopedia example)

In both cases, LLM-as-a-Judge provides a more accurate assessment of actual model capabilities by evaluating what the model truly produces rather than relying solely on first-token probability statistics. This is why we consider it the primary evaluation metric for Genesis II.

5. Conclusion

• Summary of Findings: QVAC Genesis II represents a significant expansion of the largest publicly available synthetic educational dataset, adding 10 new educational domains and 107 billion tokens to bring the combined total to 148 billion tokens across 19 domains. The introduction of the Option-Level Reasoning method, which systematically analyzes all answer options and explicitly addresses common misconceptions, has proven highly effective, achieving an average accuracy of 29.91% compared to 21.76% for Failure Analysis and 12.19% for Cosmopedia-v2. When combined with the original Failure Analysis approach in a dual-method pipeline, Genesis II achieves 30.40% accuracy. These results, validated through our LLM-as-a-Judge evaluation framework, demonstrate that structured reasoning and explicit option analysis during synthetic data generation leads to models that produce clearer, more accurate, and more reliable responses on educational tasks.
• Implications for Future Pre-training: Public, researchers, academics, research institutions, practitioners and AI community can make use of the datasets to build SOTA base model. This will set the base for a strong foundation of the base model for post-training too.

This full version of the article including all the associated assets and appendix can be found on our HuggingFace blog post