BERT algorithm

BERT is a bidirectional transformer language model, published by Google researchers in 2018 and deployed in Google Search in October 2019, that reads every word of a query in the context of the words on both sides of it rather than in sequence.

The paper, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, was submitted on 11 October 2018 and revised in May 2019. Its claim is narrow and mechanical: representations are pre-trained "by jointly conditioning on both left and right context in all layers", after which a single additional output layer is enough to fine-tune the model for a wide range of tasks. It reported a GLUE score of 80.5%, MultiNLI accuracy of 86.7%, SQuAD v1.1 test F1 of 93.2 and SQuAD v2.0 test F1 of 83.1.

Google announced the search deployment on 25 October 2019, saying it would help Search better understand one in ten searches in the U.S. in English, that it was applied to featured snippets in the two dozen countries where that feature was available, and that serving it required Cloud TPUs. Google's ranking systems guide still lists BERT today, describing it as a system that allows Google to understand how combinations of words express different meanings and intent.

The mechanism, in one paragraph

Earlier language models read a sentence in one direction, so each word's representation was built only from what came before it. BERT is trained by masking tokens and asking the model to recover them from the whole surrounding sentence, which means a word's representation depends on what follows it as well as what precedes it. The practical consequence is that small function words — to, from, for, without, no — stop being noise to be discarded and start being load-bearing. "Flights to Berlin from Riga" and "flights from Berlin to Riga" contain identical tokens and describe opposite journeys; only a model that reads the prepositions in position can tell them apart.

Why it matters if you want to be cited

BERT is the point at which the query stopped being a bag of keywords, and everything built since assumes that. Semantic search systems, embedding-based retrieval, and the generative layers in AI search all inherit the same assumption: meaning lives in whole sentences, and a sentence is the unit that gets compared.

That changes what a well-written page looks like. Two consequences do most of the work:

• State the relationship, not the keywords. A heading reading "canonical tags duplicate content SEO" and one reading "A canonical tag tells Google which duplicate URL to index" contain similar words. Only the second states a relationship a model can match against a question.
• Write the long question out. Google noted in 2019 that 15 percent of queries it sees each day are ones it has never seen before. Those are not head terms; they are sentences. A page that only ever states its subject in noun-phrase form has nothing for a sentence-shaped query to align with.

Where it is misunderstood

BERT is not a ranking signal you can feed. It is a comprehension layer applied to the query and to text. There is no markup, no attribute, and no on-page element that addresses it. Anyone selling "BERT optimisation" as a distinct service is selling ordinary clear writing under a model's name.

One in ten was a launch figure for a specific scope. It described English-language searches in the U.S. at rollout in October 2019, alongside a separate application to featured snippets in around two dozen countries. It is not a current, global share of queries, and quoting it as one is a factual error.

BERT is not the model answering you in an AI assistant. It is an encoder — it produces representations of text. The systems that write prose answers are generative decoders, a different architecture with a different job. They share the transformer lineage and nothing else.

Failure modes it exposes on a page

• Keyword-stuffed headings. They were tolerable when matching was lexical. Under sentence-level comprehension they read as a topic label with no claim in it, and a label cannot answer a question.
• Claims split from their qualifiers. If the number is in one sentence and the condition it holds under is two paragraphs away, a passage lifted out of the page carries the number and loses the condition.
• Negation dropped. "Not supported on the free plan" and "supported on the free plan" differ by one token that older matching frequently discarded and bidirectional reading does not. Writing that relies on the reader inferring a negation from surrounding context is the case most likely to be extracted wrongly.
• Pronouns doing the work of nouns. A section that says "it does not apply here" is unambiguous inside the article and meaningless once retrieved on its own. Repeat the subject.

Frequently asked questions

What does BERT stand for?

Bidirectional Encoder Representations from Transformers, from a 2018 paper by Devlin and colleagues.

How much of search did it affect?

Google said at launch it would help with one in ten U.S. English searches, plus featured snippets in around two dozen countries.

Is it still in use?

Yes — Google's ranking systems guide still lists it as an active system, unlike Panda, Penguin and the helpful content system, which are listed as folded into core ranking.

Can a page be optimised for it specifically?

No. There is no BERT-specific signal. What it rewards is prose that states relationships in full sentences.