From Scout to Behemoth: Inside Meta’s Llama 4 and the Race for 10-Million-Token Context

Key Takeaways

• MoE as Default. The architecture concedes a fundamental truth: activating trillions of parameters per token is computational madness. By routing inputs through a small subset of experts, active parameters are slashed to a manageable ~17B. This is no longer a niche experiment; it’s the emerging standard for scaling efficiently.
• Vision is Not an Afterthought. Images and text are ingested into the same transformer stack (“early fusion”). This isn’t a bolt-on vision tower; it’s a move towards a unified perceptual substrate, allowing for far deeper interplay between modalities.
• Context Beyond Reason. The 10M token context is achieved via interleaved Rotary Positional Embeddings (iRoPE) and a brutal, progressive long-sequence pre-training regimen. It pushes the boundaries of what’s computationally sane.
• Tethered to Reality. A 4-bit quantized Scout can, in theory, live on a single 80 GB GPU. Maverick requires a cluster. Behemoth remains Meta’s internal god-model. The frontier is accessible, but only just.
• FP8 and Other Black Arts. Training relied on FP8 precision and layer-wise learning rate scaling (MetaP), squeezing ~390 TFLOPs/GPU from a 32k-node supercluster. These are the unsung engineering heroics that make the headlines possible.

1 · Meet the Herd

The strategy here is transparent. Scout is the reconnaissance unit, lean enough for the community to dissect and deploy, yet powerful enough to demonstrate the new architectural principles. Maverick adds a deep bench of experts to juice performance, targeting more serious enterprise workloads. And Behemoth? It’s the internal teacher, a god-model whose impossibly rich worldview is distilled down into its smaller siblings, letting Scout and Maverick inherit its intelligence without saddling us with its trillion-parameter VRAM bill.

2 · Why Mixture-of-Experts Wins

A vanilla transformer is a monument to brute force, activating every single parameter for every token processed. Mixture-of-Experts (MoE) replaces this fiscal extravagance with tactical precision. A lightweight router network inspects each token and dispatches it to one or a few specialized expert sub-networks.

If each of experts has parameters and the router selects of them, the compute per token plummets from

With (one shared expert, one routed), Llama 4 slashes inference FLOPs by an order of magnitude. It maintains a colossal parameter budget for knowledge capacity but only activates a sliver of it at any given moment. This is about economic reality: MoE makes near-trillion-parameter models computationally tractable.

3 · Early-Fusion Multimodality

Most multimodal systems feel like a kludge: a capable vision model bolted onto a capable language model. Meta has opted for a more architecturally pure approach. Using MetaCLIP, image patches are embedded into the same vector space as text tokens. From there, both modalities flow through the exact same transformer stack. This “early fusion” is profound.

It means the model can develop a much more nuanced, deeply interwoven understanding between pixels and phonemes. The practical consequences are what matter:

• Up to 48 images can be processed in a single prompt, with a sweet spot around 8.
• Spatial reasoning remains coherent even if you switch languages mid-sentence while describing an image.
• Fine-tuning targets a single, unified set of weights, not two disparate systems you have to coax into cooperation.

For the practitioner, this means you can construct prompts that interleave Markdown, code snippets, and screenshots in a single, sequential flow-a powerful paradigm for debugging UIs or documenting complex API interactions.

4 · 10 M-Token Context and iRoPE

Scout’s headline number is built on two key innovations, one brute-force and one elegant.

1. Progressive Length Curriculum: The model was not born with a 10M-token attention span. It was trained on sequences that progressively ramped from 4K tokens up to 256K, before being introduced to synthetically generated, million-token-plus contexts. The model learned to look far back because it was forced to.
2. iRoPE (interleaved Rotary Positional Embedding): Standard RoPE, which encodes position via vector rotations, can suffer from phase distortions when extrapolated to extreme lengths. iRoPE is a simple, clever fix: every fourth attention block omits rotary embeddings entirely. This acts as a reset, allowing gradients to flow across vast distances without accumulating catastrophic phase errors.

During inference, Meta employs another trick: as the context window grows, the softmax temperature is actively cooled. This keeps the attention mechanism focused, preventing its distribution from flattening into an entropic soup over millions of tokens.

But does it work? My own needle-in-a-haystack tests show retrieval coherence holds up past 400K tokens, but the quadratic scaling of latency and the KV cache becomes punishing. The next engineering frontier is not just having massive context, but managing it through streaming KV eviction and intelligent windowing.

5 · Under the Hood: FP8, MetaP, and 32 K GPUs

The truly heroic efforts are often buried in the technical appendices.

• FP8 activations, combined with stochastic rounding, cut memory bandwidth by nearly half compared to FP16, all while keeping gradient noise manageable. This isn’t a minor tweak; it’s a foundational choice for training at this scale.
• MetaP, a new optimizer, scales learning rates individually per layer, factoring in depth and fan-in. It’s a powerful tool for maintaining stability in ultra-deep networks.
• According to Meta’s logs, the training cluster of 32,000 A100s achieved a peak throughput of 390 TFLOPs / GPU. A testament to extreme-scale engineering.

6 · Running Scout Locally

The claim of running a 109B parameter model locally sounds absurd, but thanks to aggressive quantization, it’s technically true. The following is a minimal recipe to get a 4-bit Scout checkpoint running on a single 80 GB H100, streaming at a respectable ~25 tokens/second via the Hugging Face transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import BitsAndBytesConfig  # For 4-bit quantization

# Official Hugging Face Hub repository for Llama-4-Scout
ckpt = "meta-llama/Llama-4-Scout-17B-16E"

# Configure 4-bit quantization using BitsAndBytes
# See: https://huggingface.co/docs/transformers/main/main_classes/quantization#transformers.BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="bfloat16", # Compute dtype for faster operations
    bnb_4bit_use_double_quant=True,    # Improves quantization precision
    bnb_4bit_quant_type="nf4",         # NF4 quantization type
)

# Load tokenizer
# See: https://huggingface.co/docs/transformers/main/model_doc/auto#transformers.AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)

# Load model with quantization config and automatic device mapping
# See: https://huggingface.co/docs/transformers/main/model_doc/auto#transformers.AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    ckpt,
    quantization_config=bnb_config,
    device_map="auto" # Automatically distribute model layers across available devices
)

prompt = "### System:\nYou are an alpine marmot scientist.\n### User:\nWhy do marmots whistle?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) # Ensure inputs are on the same device as the model

# Generate text
# See: https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationMixin.generate
output_sequences = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output_sequences[0]))

If you need to be even more frugal with memory, device_map in Hugging Face Accelerate allows you to offload rarely used experts to CPU RAM or split layers across smaller GPUs (e.g., two 48 GB cards). My early tests suggest a ~15% performance penalty for this maneuver, but with negligible impact on accuracy.

7 · Fine-Tuning Strategies

While we can’t access Behemoth for codistillation, the same principle applies downward. You can distill Scout into a smaller 13B student model using soft targets, capturing 90% of its capability at a quarter of the VRAM cost. It’s a pragmatic path for deploying specialized models.

8 · Why Llama 4 Matters

Beyond the benchmarks and parameter counts, Llama 4 represents three important strategic shifts in the AI landscape.

1. Open-weight is not capitulation. Scout and Maverick are a clear signal that Meta intends to compete at the bleeding edge in the open. This commitment continues to underwrite an entire ecosystem of innovation.
2. Multimodality from birth is now table stakes. Every frontier model of consequence-Gemini Ultra, Claude 4, GPT-4.5-now treats vision as a native sense. Models that are merely linguistic are already legacy systems.
3. Context length is the new front line. The arms race is no longer about raw FLOPs or parameter counts. The decisive advantage will go to whoever can ingest and reason over vast corpuses of information fastest, making retrieval-augmented generation (RAG) workflows seamless and powerful.

Meta may still trail OpenAI’s latest on certain reasoning benchmarks, and the true utility of a 10M token context is still a matter of debate. But the engineering statement is unambiguous. By open-sourcing these artifacts, Meta ensures the community will be the one to stress-test the limits and discover the applications.

9 · Next Steps

The most interesting discoveries will happen at the intersection of these new capabilities. If you’ve benchmarked extreme-context retrieval or crafted a compelling multimodal workflow, the floor is open. The community’s collective experimentation is what turns these raw artifacts into genuinely useful tools.

Happy hacking, and may your routers always pick the right expert.