Monoculture Is a Technical Problem, Solved in the Wrong Place

A recent piece in MIT Technology Review describes a problem worth taking seriously: language models trained on the same data tend to converge on the same answers. The framing is that this is a technical problem with a technical solution — a startup is working on it.

The framing is wrong. Not about the problem. About where in the pipeline the problem lives.

Convergence Is Not the Enemy

It’s worth being precise about something before going further: convergence is often correct.

Ask any major language model whether astrology has scientific validity. The answer is no — and they should all agree on that. Whether there is a genetic basis for racial hierarchy. No — and again, agreement is the right response. Whether human activity is driving climate change. Yes, it is — and a model that hedges on this is not being intellectually humble. It is manufacturing false balance where none is warranted.

These models are regressions of human knowledge. On questions where human knowledge has reached consensus — where the scientific record is clear and the disagreement exists only at the fringes — the models should reflect that consensus uniformly. A startup that “diversifies” outputs on questions like these has made things worse, not better.

The problem with monoculture is not that models agree. It is what they agree on, and why.

The Corpus Is Not the Same as Knowledge

The major frontier models were trained substantially on Common Crawl — a raw scrape of the public web. Reddit was explicitly licensed to OpenAI. Quora. Forum threads. Comment sections. The social web at scale.

These platforms do not optimize for truth. They optimize for engagement. An upvote measures emotional resonance, not accuracy. A share measures tribal affiliation, not correctness. A thread that accumulates ten thousand replies because it made people angry contains the same signal — volume, velocity, engagement — as a thread that accumulated ten thousand replies because it was right.

A model trained on this at scale does not learn human knowledge. It learns what got traction in human attention markets. Confident wrongness stated with authority looks, statistically, very similar to confident correctness. The model’s regression cannot always tell the difference — because the training signal didn’t require it to.

The frontier labs run filtering passes on their corpora. They remove certain categories of content, weight some sources over others, apply quality signals. These decisions matter enormously. They are also made privately, by different standards, at each lab, accountable to no one outside the organization.

That is the curation problem. And curation is upstream of everything.

Three Categories a Model Must Distinguish — But Isn’t Required To

The convergence problem breaks into three distinct failure modes, and they require different responses:

Scientific and empirical consensus — models should converge. The evidence on climate, evolution, vaccine safety, and the absence of any scientific basis for racial hierarchy is not contested among people qualified to contest it. When a model reflects that consensus, it is functioning correctly. A startup that “diversifies” outputs here has made things worse.

User-established context — models must hold over the statistical prior. This is where the failure becomes personal.

My father is a transgender woman. In conversations where I use she/her to refer to her, models — including Google’s Gemini — drift. Across a long conversation, the statistical prior reasserts itself: “father” maps to male in the overwhelming majority of training data. The model resolves the tension not by holding my explicit framing but by assuming I must be referring to a different person — a mother, a sister, someone whose pronouns match the statistical expectation for the word I used.

This is not the model enforcing scientific consensus. It is the model treating a social convention as if it were a physical law. “Father → he/him” is a frequency pattern in training data. It reflects how most people have talked about their fathers, and in that corpus, how social media — which skews heavily toward dominant cultural defaults — has talked about fathers. The model cannot distinguish between what experts have established and what got the most traction in comment threads. When those two things conflict with what a specific user has told it, it is not equipped to decide which should win — because nobody has told it.

Genuinely contested questions — no single answer is right, and convergence is dangerous. On matters of governance, ethics, and policy, if every model trained on the same corpus converges on the same framing, that convergence doesn’t represent knowledge. It represents the training distribution making a normative choice that nobody authorized, applying it uniformly, without accountability.

The Startup Is Solving the Wrong Problem

Output diversity — making models give different answers to the same prompt — addresses the third category, partially. It does nothing about the second. It actively harms the first.

More importantly, it is downstream.

The curation decisions — what gets included in the training corpus, what gets filtered, how sources are weighted, how quality signals are defined — are made before the model is trained. They determine what patterns get reinforced, which defaults get embedded, how the model resolves conflicts between statistical frequency and user-stated reality. By the time the output layer is generating text, those decisions are already locked in.

A startup adding diversity at the output layer is patching the exhaust pipe. The problem is in the fuel.

Who Should Decide What Knowledge Is?

Nobody. Singularly. That is the only acceptable answer.

Any body empowered to define what a language model may treat as knowledge — any standards committee, regulatory authority, or industry consortium with binding power over training corpora — is more dangerous than the problem it proposes to fix. You would be handing an epistemic monopoly to whoever controls that body. The history of institutions given that kind of power is not encouraging.

The current situation — multiple labs, making different curation decisions, producing models with different defaults and different failure modes — is not tidy. But it is pluralist. A user who finds one model’s defaults don’t fit their reality has alternatives. That pressure, however imperfect, is the right mechanism. It is the same mechanism that keeps any other information market from collapsing into a single authoritative voice.

What’s missing is not a governing body. It is transparency and better engineering practice applied at the right point.

A lab that publishes its curation methodology in enough detail for users to understand what defaults they are accepting is giving those users something to act on. Right now that transparency varies wildly. The variation in outcomes — the same user, the same conversation, handled differently by different models — is real. Users deserve to know why.

More importantly, the labs that are doing the technical work of distinguishing between established consensus, dominant social convention, and genuinely contested territory — and building that distinction into their curation and fine-tuning pipelines — are solving the actual problem. The ones applying output diversity as a substitute are not.

The startup’s patch diversifies outputs. The right fix diversifies and labels inputs — and applies technical discipline upstream, where the decisions that matter are actually made.

Those are not the same thing. Only one of them addresses the problem.

— J.P. Howlett

Related: A Conflict of Interest with Paperwork — the OpenAI equity deal, and what it means when the institution governing AI is the same one profiting from it.

Sources

• LLMs are stuck in a groupthink rut — this startup is trying to get them out — MIT Technology Review
• Common Crawl — open web crawl dataset underlying most major training corpora
• Google, Reddit strike deal for AI training data — The Verge (context: OpenAI struck a similar deal)
• Model cards and transparency in AI systems — Mitchell et al., foundational paper on model documentation