An AI Model Learned Some RNA Structure Rules Without Comparing Species

The standard way to guess the shape of an RNA molecule is to look at its cousins. Collect the same molecule from a few hundred species, stack the sequences one above another, and watch which positions change together: a mutation at one spot that keeps company with a mutation at another usually means the two are touching in the folded molecule. It is an elegant trick, and it has one blunt requirement. You need the cousins.

For plenty of RNA molecules the cousins are not there: the family is small, or poorly sampled, or nobody has sequenced it in enough organisms. That gap is what Utkarsh Upadhyay, Alexander Schug and colleagues at the Jülich Supercomputing Centre and the Karlsruhe Institute of Technology set out to work around. In a paper published open access on Sept. 3 in Nature Machine Intelligence they describe NucleicBERT, a 404-million-parameter language model trained on about 30 million non-coding RNA sequences, pulled by keyword out of a database of roughly 1.7 billion. It sees one sequence at a time. No alignments, no structural templates, no labels.

The training job is the one language models do on text: hide some tokens, make the model guess them, except that here a token is a single nucleotide. What the group was after was less the leaderboard than the question underneath it: how much of RNA biology can a machine pick up from raw sequence, and how would anyone know?

The boring explanation had to go first

That second question is harder than it sounds, because a model can do well on a biology benchmark while keying on something with no biology in it. An untrained transformer is not blank. Run random weights over a sequence, pool the result, and what comes out is largely a summary of length and GC content, and those alone sort RNA families fairly well, since small interfering RNAs are short and uniform while ribosomal RNAs are long and GC-rich. The authors show it directly: a nearest-neighbor classifier given nothing but length and base composition comes close to the untrained network's score. So the comparison worth making is between classes matched for length, and there the pretrained model still separates the families more cleanly.

The sharper control freezes the network. Lock the pretrained weights, train only a small head on top to predict secondary structure, and it reaches an F1 of 0.581 on a standard benchmark; do the identical thing on a randomly initialized network and the score is 0.005. Whatever the model absorbed during pretraining was there before any structural training began. Unfreezing everything adds a further 29 points of F1, which says just as plainly that much of the structure still has to be taught.

The internal evidence points the same way, and it is more specific than base pairs. Saliency for secondary structure does not peak on paired positions; it rises at the boundaries where a paired stretch gives way to an unpaired one: the junctions that decide how a molecule folds. Attention on the structural tasks concentrates in the early and middle layers, while a deliberately non-structural control task, telling real sequences from shuffled ones, produces a flat attention map with no such specialization.

How far a single sequence gets you

The most direct test of the opening question comes late in the paper. Upadhyay and colleagues built a score they call masked-likelihood influence: mask a second position as well, and measure how far the model's confidence in the correct nucleotide at the first position falls. Run that over every pair and a coupling matrix comes out, the same object the cousin-comparison methods produce, built one sequence at a time. Across 14 RNA families it sits consistently above a random baseline, and its quality barely changes with how closely the query resembles anything in the training corpus, which argues against simple memorization. It also sits below both of the alignment-based references it is measured against, mutual information and direct coupling analysis. The paper's own verb is that sequence-only modeling "partially recapitulates" what those methods pull out of deep alignments, and the partially carries weight: this is a fraction of the signal, not a substitute for it.

One further result the authors themselves call compelling. Shuffle a growing fraction of a real RNA sequence and ask the model whether it is still authentic; accuracy holds up until about a quarter of the sequence has been scrambled, then falls away. Natural RNA families, measured across the Rfam database, tolerate a maximum internal divergence averaging 26.9%, give or take about ten points. The authors read the near-coincidence of those two numbers as evidence that the model has internalized where the boundary of functional RNA sequence space lies. That is an interpretation, resting on a match between one threshold and one wide average, and the paper notes that no multiple-comparison adjustment was applied to the correlations behind it. The frozen-probe result above is the sturdier evidence.

What the benchmark table does and does not say

On the benchmarks the paper's claim is that NucleicBERT matches or exceeds current RNA prediction models, and that is the right size for it. Secondary structure is set against thermodynamic folding, two deep-learning predictors and three earlier RNA language models; contacts, splice sites and fitness against RiNALMo and RNA-FM, the two largest models of this type. The authors report NucleicBERT ahead of both on contacts. On fitness, meaning how far a mutation changes a ribozyme's ability to cut its own backbone, it scores an R² of 0.994 against RNA-FM's 0.996, a tie at best. Two caveats sit on that table. The rival figures are taken from those tools' own publications rather than re-run here, which the authors state. And RiNALMo is missing from the secondary-structure baselines the text enumerates, though it appears in every other comparison.

None of this is three-dimensional structure: NucleicBERT outputs secondary structure, inter-nucleotide distances and contact maps, and the paper treats contacts as a proxy for 3D structure rather than the thing itself. It is also a single study from a single group, with no independent reproduction yet.

The code and the pretrained weights are public, which matters here: the argument is about what one particular set of trained weights contains, and anyone can download them and probe them a different way. The authors' own summing-up is modest, and probably right: a single-sequence model of this kind is an approximate, alignment-free stand-in for coevolution analysis, useful where the alignments are shallow or missing. For a great many RNA families, that is the normal situation.