Natural language processing (NLP)

Natural language processing is the field concerned with getting machines to represent, analyse and produce human language — and, in its current form, the set of techniques that decide which passage of which page is selected to answer a question.

The textbook definition ("enabling computers to understand human language") is true and useless. What is useful is knowing which stage of an NLP pipeline touches your content, because they fail in different ways and only some of them are things a publisher can influence.

The two papers that produced the current generation

Modern NLP is a short lineage, and the two hinge points are both dated and public.

The Transformer, June 2017. Attention Is All You Need, submitted 12 June 2017 by Vaswani and colleagues, proposed an architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. It reported 28.4 BLEU on the WMT 2014 English-to-German task and 41.8 on English-to-French after 3.5 days of training on eight GPUs — a small fraction of the training cost of the best models at the time. Removing sequential recurrence is what made scale practical; almost everything since is a consequence.

Bidirectional pre-training, October 2018. BERT, submitted 11 October 2018 by Devlin and colleagues, pre-trained deep bidirectional representations by jointly conditioning on both left and right context in all layers, then fine-tuned with minimal architectural change. It reported a GLUE score of 80.5% (7.7 points absolute improvement), MultiNLI accuracy of 86.7%, and SQuAD v1.1 F1 of 93.2. The lasting result was not the benchmark numbers but the pattern: pre-train once on unlabelled text, adapt cheaply per task. See BERT algorithm for how that pattern reached search specifically.

"Bidirectional" is the word to hold onto, because it has a direct publishing consequence covered below.

The pipeline, and which stages touch your page

A system answering a question from web content runs several NLP stages in order. They are usually discussed as one thing and diagnosed as one thing, which is why they are so often misdiagnosed.

1. Tokenisation. Text becomes sub-word tokens. This is where exact identifiers get fragmented in ways a human reader would not anticipate.
2. Segmentation. A document is cut into chunks. A chunk covering four subjects gets one representation that describes none of them well.
3. Representation. Each chunk becomes an embedding — a vector positioned by meaning rather than by wording.
4. Retrieval. The query is embedded and the nearest chunks are returned. This is semantic search, and it is good at paraphrase and comparatively poor at exact strings.
5. Generation. A model writes an answer conditioned on what stage 4 handed it.

Only the last stage produces visible output, so every failure looks like a generation failure. Most are not.

Why this changes how you write

Because the retrieved unit is a chunk and the representation is bidirectional within that chunk, a claim and its qualifier have to occupy the same passage. "Reduces processing time" three paragraphs above "in batch workloads over 10,000 rows" are, to the system, two facts in two different chunks. The first travels; the second does not. Writing the qualifier into the same sentence as the claim is not a style preference — it is the only way the qualifier survives selection.

The same reasoning explains why sub-headings phrased as the question being answered work better than clever ones. The heading is part of the chunk's text and therefore part of its representation.

And because retrieval is similarity-based rather than lexical, exact identifiers are the weak point. A part number, an error code or a version string is precisely the kind of token that similarity matching handles badly. State them literally, in text, near the claim they belong to, rather than relying on the surrounding prose to imply them.

Failure modes

• Blaming the model for a retrieval miss. If the answer cites a page that does not support the claim, the likely fault is stage 4 returning something topically adjacent, not stage 5 lying. Only the first is something you can fix by editing.
• Multi-topic sections. A section that covers pricing, limits and setup produces a vector that is close to no query in particular.
• Orphaned qualifiers. Caveats in a preamble, a footnote, or a table caption two screens away are not in the chunk.
• Implied identifiers. "The latest release" is not retrievable; the version number is.
• Treating structure as decoration. Headings, lists and tables change where the chunk boundaries fall, which changes what is retrievable at all.

What NLP is not

It is not a synonym for large language models, and it is not only about generation. The stages that decide whether your page is a candidate — segmentation, embedding, similarity search — are older, narrower and more mechanical than the part that writes prose, and they are where a publisher has actual leverage.