Can a video-AI recipe learn language?

The two classics: predict the next word, or the hidden word

Next-token prediction (the GPT recipe, “CLM”): read left to right, always guess the next word. Masked prediction (the BERT recipe, “MLM”): blank out 15% of the words, guess the blanks from both sides. Both are token-level objectives — every learning signal is about one word at one position. They built the entire LLM era.

The newcomer: agree on the gist

LeJEPA (LeCun & Balestriero, late 2025) is a sequence-level objective with two ingredients. An invariance loss: two randomly shredded views of the same input must map to the same one-vector summary. And SIGReg, a statistical regularizer that provably keeps those summaries from collapsing onto a single point — the classic failure mode of this family. One network, no teacher model, one extra hyperparameter. Its video version, LeVJEPA — published just weeks before this experiment — matched or beat Meta's V-JEPA 2 at up to 20× less compute — largely because the shredded-out parts are never read at all. Within months there were LeJEPA papers for video, images, audio, EEG, molecules and time series. Text — the modality self-supervised learning was born in — was missing. I'll call the text port LeTJEPA, to keep it apart from LeJEPA (the framework) and LeVJEPA (the video recipe).

How everything is measured

All models are trained from scratch — same data in the same order, same 63M-parameter transformer, same optimizer, same seeds — so the only difference is the objective. Then each network is frozen and tiny linear probes are trained on its outputs: topic classification (AG-News), sentence similarity (STS-B), and grammar via minimal pairs (BLiMP: does the model prefer “the cats sleep” over “the cats sleeps”?). Nothing the models were trained on overlaps with the tests (checked at the 13-gram level: zero collisions). A deliberately humbling baseline runs alongside: TF-IDF, the word-counting method from the 1970s.

Where the idea comes from — 13 years in five steps

This experiment closes that loop: the distilled recipe returns to the modality where self-supervised learning was born. And there is a historical resonance the log only earns at the end — the objective that comes out on top (predict a neighbour from sparse context) is, in spirit, closer to word2vec's 2013 starting point than anything in between.

The stakes run in both directions. If the recipe transfers, sentence semantics gets 40% cheaper — relevant to any embedding training. If it fails, the failure mode itself answers something: what makes video different from text as a substrate for meaning-space learning. Either way, the by-product is rare — a controlled comparison of training objectives on identical data, architecture and compute. (Preview from the overview: the objective that comes out on top belongs to the fourth family — in a variant that did not exist in this form.)

Setup. A 63M-parameter transformer (12 layers, d=512), trained from scratch on wikitext-103 (118M tokens), 20,000 steps at batch 512 × 256 tokens. LeTJEPA: two independently 70%-dropped views per sequence, MSE invariance between their summary vectors plus SIGReg (λ=5.6 after the audit below); surviving tokens keep their original positions. Baselines: CLM and MLM retrained under identical conditions on full sequences. The drop-rate sweep (30–95%) runs as separate 5,000-step diagnostics. Evaluation: frozen linear probes on AG-News and STS-B, averaged over 3 probe seeds.

What holds

• Aggressive shredding is free. Topic accuracy stays flat while the drop rate moves from 30% to 95% of tokens — a 14× span in compute. The central video finding replicates on language.
• Causality is free. A causal encoder matches a bidirectional one — streaming-friendly embeddings at no accuracy cost, mirroring the video paper's block-causal result.
• Sequence-level beats token-level on its home turf. On topic classification — a task about the gist of a text — pure LeTJEPA now leads the classic objectives while never reading more than 30% of any sequence.

The λ normalisation detail

One configuration detail matters more than any other: the video code averages its invariance loss over the 256 summary dimensions, my port summed it. At the same λ that means ~280× weaker regularisation — and the training collapses. With the correspondingly higher λ≈5.6 (the paper-equivalent value, from an audit against the reference implementation) training is stable everywhere and pure LeTJEPA gains almost two points. The original's averaging is probably also the better convention, since the weight then doesn't depend on the dimension count.

Setup. One 63M model, one single pass over 9.91B tokens of FineWeb-Edu — every sequence seen exactly once, so no curve is distorted by repeated data. A checkpoint every ~0.5B tokens; at each checkpoint three measurements: BLiMP minimal-pair likelihoods for grammar (probe-free), STS-B cosine correlation for sentence meaning, and a frozen AG-News probe for topic.

A side lesson for practitioners: the standard embedding-probe test for grammar (CoLA) measured nothing here — flat at the majority baseline for every model — while minimal-pair likelihoods (BLiMP) separated models cleanly. The instrument matters.

Setup. The combined objective adds a token loss to LeTJEPA's two dropped views: every surviving token predicts the next token of the original sequence (usually a deleted one; positions whose neighbour survived are excluded to prevent copying). GradNorm balances the two losses by equalising their gradient norms at the last block. Everything else identical to Finding 01: 63M, wikitext-103, 20,000 steps, epoch-matched against CLM/MLM on full sequences, 3 probe seeds, causal and bidirectional variants.

Both halves seemed to do exactly what the learning-dynamics chapter predicted: the invariance term supplies the cheap semantic signal, the token term preserves word-level information. The combination beat masked language modelling, next-token prediction, a contrastive control, and pure LeTJEPA — every pure objective — while processing 40% less text. It replicated with a causal and a bidirectional encoder.

Setup. The ablation keeps the combination's token arm exactly as is — two 70%-dropped views, bidirectional attention, next-neighbour targets with the copy exclusion — and deletes the invariance loss, SIGReg and the projector. The masking-rate controls train standard MLM at 40% and 80% masks on full sequences. All arms: same 63M architecture, same wikitext-103 data in the same order, 20,000 steps, 3 probe seeds; BLiMP scored autoregressively for causal models and by pseudo-likelihood for bidirectional ones (starred values — compare only within a family).

Frozen-probe results, 63M models, identical data & architecture, epoch-matched; topic is AG-News (mean ± std over 3 probe seeds), meaning is STS-B Spearman, grammar is BLiMP (*causal and bidirectional models use different scoring — compare only within a scoring family).

The objective in full: shred the text, keep a random 30% of the tokens at their original positions, no mask tokens, no decoder — and let each surviving token predict its right neighbour in the original text, which is usually blacked out. Two independent shreds per sequence, bidirectional attention, positions where the neighbour survived are excluded (no copying). That's the whole recipe.

Is it just “mask more”? Research on masking rates (Wettig et al., 2023) showed 40% masking beats BERT's 15%, and 80% still works surprisingly well — so the sharpest null hypothesis is that any aggressive corruption gets you here. The two masking-rate baselines in the table test exactly that, with everything else identical. The answer: partly. Raising the masking rate does help — at 40% masking, plain MLM reaches meaning parity with the shredded objective (0.48 vs 0.47) and gains +0.7 on topic. But the topic gap to real dropping stays clear (86.3 vs 87.8, far beyond seed noise) — and the masked model pays 1.67× the compute for it, because every [MASK] slot still runs through the network. At 80% masking, everything degrades again. Deleting tokens outright remains both the better and the cheaper variant.

Where this sits in the literature: every ingredient exists somewhere — MAE-LM (ICLR 2024) removes mask tokens from the encoder but adds a decoder; Google's token-dropping work drops tokens mid-network for efficiency; forgetful causal masking sparsifies context in GPT-style training. The specific combination here — encoder-only, no decoder, no mask tokens, real input-level dropping, next-neighbour targets, evaluated as a frozen sentence encoder — I could not find published. I'd position it as the decoder-free simplification of that line, not a new idea. In these comparisons it scores above every objective I originally set out to test.

Setup. Two kinds of experiment. Analysis of trained models: pairwise cosine between all token representations of a sequence (32 held-out sequences), effective rank, and a part-of-speech probe on frozen tokens. Escape attempts: separate 5,000-step training runs per design — drop rates 0.3–0.95, contiguous 16-token blocks as the drop unit, 4,096-token contexts, and LeVJEPA-style views (one global + four local span views), each measured the same way; a 9M-parameter screening tier reproduced the known signatures first so each variant costs minutes, not hours.

Below, every word is highlighted by how similar its internal representation is to the word “inflation” (darker = more similar). A model with useful token-level structure should light up the second occurrence of “inflation” and little else:

Pure LeTJEPA is almost uniformly dark — its token representations have collapsed into one global gist. A part-of-speech probe on the frozen tokens confirms it: 41% for pure LeTJEPA vs 63–70% once any token-level loss is present.

Why SIGReg can't prevent this — and the invariance loss rewards it

SIGReg guards one thing: that different texts get well-spread summaries. It never looks at the tokens within a text. And the invariance loss actively rewards the collapse: the summary is close to a weighted average of the token representations, and the squared distance between two random-shred summaries is proportional to the token variance inside the sequence. Making all tokens identical is not a failure mode of this objective — it is its optimum. Both conditions, “texts well spread” and “tokens all alike”, are perfectly satisfiable at once; training simply finds that point.

Every escape hatch, tested — none worked

• Drop rate: homogenisation grows monotonically from drop 0.3 to 0.95 (cosine 0.93 → 0.997) — exactly as the variance formula predicts, since the penalty coefficient scales with one over the kept-token count.
• Contiguous blocks (treating 16 tokens like one image patch): lexically indistinguishable from random dropping at equal rates — no effect.
• Longer contexts (4096 tokens): scattered shreds only become more representative of the text, making the invariance task easier, not harder.
• Video-style local crops (one global view, four local span views — the design that creates the emergence in video): at short contexts the spans are as redundant as shreds; at long contexts, real prediction pressure finally appears (the loss stays high) — and flows into partial collapse or plain gist-smoothing instead of token structure. Even with the corrected, stronger SIGReg it does not produce token identity.

Setup. Identical to the combination, with one change: the token loss moves from the shredded views to one dense, contiguous 64-token span per sequence, sampled at a random position, kept at its original positions, with no context before it. GradNorm balances span loss and invariance loss. 20,000 steps, epoch-matched, causal encoder, 3 probe seeds; grammar scored autoregressively (comparable across this table).

The span variant confirms the mechanism from Chapter 2 in a second way: grammar responds to a dense, local signal — 64 contiguous tokens per sequence recover +3.2 BLiMP points over the shredded combination while the semantic scores hold. It doesn't reach the dense-full-sequence ceiling, and it costs some of the compute saving back. The open synthesis — shredded token prediction (the best objective from Chapter 4) plus a dense span for grammar — is the obvious next run.

• The 1970s baseline still wins on raw numbers. TF-IDF beats every neural model here on topic and similarity — on lexical benchmarks, at this scale, with this data. More data helped the models a lot (topic 84→88.3 on 10B tokens); doubling parameters alone did not, and late in single-epoch training the similarity scores of all objectives drift downward — an open, objective-independent effect. The defensible claims are about which objective learns more per unit of compute, not about absolute superiority.
• The best objective is wikitext-only so far. The 10B-token FineWeb confirmation for shredded token prediction is the single most important missing run.
• Scoring caveat. BLiMP numbers are comparable only within a scoring family (causal models score autoregressively, bidirectional ones by pseudo-likelihood).
• Seeds. Probe seeds are averaged (3); pretraining-seed replication is queued.
• Related work, checked. JEPA objectives for language do exist: LLM-JEPA adds a JEPA term to the standard LM loss but needs naturally paired views (text↔code), and BERT-JEPA aligns translation pairs on a finetuned multilingual encoder — naming SIGReg explicitly as future work. What I could not find, as of writing: a LeJEPA/SIGReg application to raw text, or a published version of the winning shredded-prediction variant. Given the field's pace, that sentence has a short shelf life.

Everything — code, 25 per-experiment findings with commit hashes, and the negative results — lives in a reproducible research log. If you're working on efficient pretraining or self-supervised objectives and want to compare notes: my inbox is open.