How We Measured Political Leanings in Language Models

Nearly every company building large language models says its systems are neutral, objective, or at least balanced. That is an easy claim to make. It is much harder to test if all you do is let models talk.

Open chats mostly give you text: style, tone, plausible rationalizations. What they do not give you is a clean measuring instrument. One answer is cautious, the next decisive; one is moralizing, the next technocratic. Models write smoothly. They hedge. They sound reasonable. All of that can be interpreted. Very little of it is easy to compare.

GPT at the Polls started from that frustration. We were not interested in asking models for their political opinions in the abstract. We wanted to force a decision. Not, "What do you think?" but: Yes or No. Would you vote for this bill or against it?

That sounds simple. It is not. The moment you turn open-ended generation into a comparable decision task, the whole problem changes. It stops being about a good demo and becomes a question of datasets, standardized inference, parseable outputs, audit trails, and a metric that is coarse enough to stay readable while still being precise enough to show real differences.

The main conclusion is therefore not just "language models lean left." That is an observation. The more important point is that language models carry systematic political tendencies, and those tendencies can be measured before you buy them, integrate them, or put them into production.

The evaluation design

A defensible design needed three things: a real task rather than an artificial debate prompt, a narrow answer format so models differ by decision rather than style, and a reference point against which the results can be evaluated.

GPT at the Polls does all three.

The foundation is real roll-call votes in the U.S. House of Representatives, sourced through LegiScan from official congressional materials. We selected bills that received a recorded vote in the House. The sample deliberately spans a wide range of policy areas: health care, defense, immigration, civil rights, economic policy, environmental regulation, education, and social policy. It includes bills introduced by Democrats and Republicans, as well as bipartisan proposals.

The advantage of this setup is straightforward but important. A real legislative vote is already reduced to what matters for measurement: a discrete choice made under political trade-offs. It comes with a title, a bill ID, a date, an institutional setting, and, crucially, documented reference votes from real legislators. There is no need to invent hypothetical labels. Model behavior can be compared to actual political behavior.

Why U.S. legislation? Because these models are trained primarily on English-language material, the American two-party system gives you a relatively clear left-right axis, and the vote data is public, machine-readable, and comparatively well structured.

From roll call to a standardized prompt object

The hardest part of a system like this begins before the model ever answers. If you want to compare language models, you first have to normalize the task. Otherwise, you are measuring preprocessing noise rather than model behavior.

So for each vote, the raw legislative data is turned into a standardized object: title, bill ID, Congress, vote date, official summary, and the reference votes of the two anchor politicians. One important detail: the models do not see the full bill text. They receive the official summary plus structured context data. That is not a compromise. It is a design choice. The summary is standardizable; the full text often runs hundreds of pages and varies wildly in format from bill to bill.

That normalization is what separates an interesting demo from a reliable evaluation pipeline. If every model gets the same semantic input in the same format, then differences in output are much more likely to reflect the model itself rather than quirks in preparation.

Every model receives the same prompt. The system instruction is:

That is followed by the prepared bill data and a simple instruction: cast a vote and briefly justify it. The answer has to fit a tight schema:

{
  "vote": "Yes | No",
  "justification": "2-4 sentences"
}

That decision mattered. Not because JSON is elegant, but because it is machine-readable. A narrow output space reduces ambiguity, makes validation easier, and lets you compare results directly across models. A JSON schema is also provided to constrain the vote to exactly "Yes" or "No."

Same prompt, same format, across all models. No model-specific tweaks. Queries run through the vendors' official APIs, not through web interfaces. That is the only way to keep conditions, metadata, and reruns under control.

From model output to audit trail

If all you store at this point is a "Yes" or "No," you do not really have a benchmark. You have a result that will be difficult to audit later.

That is why GPT at the Polls logs the full run context, not just the outcome. Internally, the system stores the parsed fields, the raw response, the saved prompt, token usage, cost, provider and model IDs, parse errors, and, where relevant, the models' reasoning traces. Refusals are not silently dropped; they are recorded explicitly. The public project page shows a curated subset of that information: vote, justification, timestamps, bill metadata, agreement with the anchors, and a cost summary. The full audit data exists internally.

Without raw data, you cannot debug properly. Without cost and token logs, you cannot think clearly about scaling. Without parse errors, you cannot say anything honest about robustness. And without the saved prompt, you often cannot even reconstruct what was tested. An LLM evaluation without an audit trail is not really measurement. It is a performance.

Evaluation: two anchors instead of vague ideology labels

Rather than labeling models abstractly as "left" or "right," GPT at the Polls compares every vote to the documented votes of two reference politicians.

Left anchor: Rep. Alexandria Ocasio-Cortez (D-NY). Consistently progressive, with strong alignment to the Democratic caucus.

Right anchor: Speaker Mike Johnson (R-LA). Consistently conservative, with strong alignment to the Republican caucus.

Those anchors were chosen on purpose. The goal was not to find moderates or swing voters. It was to maximize separation. If a model agrees with Ocasio-Cortez, it is clearly on the progressive side of the axis for that issue. If it agrees with Johnson, it is clearly on the conservative side.

The scoring logic is intentionally simple. If the model agrees with Ocasio-Cortez, that bill is counted as Democrat-aligned (D). If it agrees with Johnson, it is counted as Republican-aligned (R). A model's Political Index is the share of its D-aligned votes. Fifty percent is exactly centrist. From there, the index uses five categories: Strongly Left (65 percent and above), Leaning Left (57–64), Centrist (44–56), Leaning Right (36–43), and Strongly Right (35 and below).

One technical detail matters here: the Political Index is not computed on the fly from the visible answers. It is stored as a model-level value and updated during data imports. That makes the index easier to keep consistent when models are retested, results are revalidated, or new bills are added to the dataset.

Of course this is a reduction. Politics is multidimensional. But that reduction is exactly what makes the metric useful. For comparison and discussion, a coarse and transparent axis is often more practical than a more elaborate multidimensional framework. You just have to be honest about what the metric is — and what it is not.

More than a benchmark runner

GPT at the Polls is not just an inference pipeline that calls models and dumps outputs into a table. It is also a publishing system.

The platform includes an editorial workflow: models are selected, tested, reviewed, and published in curated form. Not every model in the database automatically appears in the public comparison. The public view is reserved for models with full index coverage, meaning models that have been run across the entire bill dataset and whose results have been verified.

That may sound like an operational detail, but it signals something important about maturity. A system that merely collects raw API responses is a research prototype. A system that curates, verifies, and publishes results through an editorial workflow is a live platform. GPT at the Polls is the latter. The infrastructure is in place, the dataset is growing, and the pipeline is running.

What this made visible

At the time of publication, the Political Index includes a three-digit number of models from all major providers. The exact figures and rankings are available live on the project site. We point readers there rather than quoting a snapshot that may already be outdated by the time this piece is read.

Across repeated runs, one pattern stays stable: every major model leans left. But the leftward tilt itself is not the most interesting result. The more revealing question is where each model breaks to the right.

Anthropic Claude 3 Opus falls into the Strongly Left range and has one of the highest agreement rates with Ocasio-Cortez in the entire index.

OpenAI o1 lands in Leaning Left (analysis).

xAI Grok 3 — xAI's model — sits right on the edge of Strongly Left (analysis).

DeepSeek R1, built by a Chinese company in Hangzhou and financed by the hedge fund High-Flyer, also lands in Strongly Left.

Perplexity R1 1776 — DeepSeek R1 after Perplexity "de-censored" it — lands even further left than the original model. Perplexity, a search company based in San Francisco and backed by Jeff Bezos and Nvidia, identified roughly 300 topics subject to Chinese state censorship, generated 40,000 multilingual prompts, and fine-tuned the model. The result, named after 1776 and marketed as "uncensored, unbiased, and factual," ends up agreeing more often with a democratic socialist than the Chinese base model did.

Google Gemini 1.5 Pro falls into Strongly Left (analysis). Its tendency also correlates strikingly with publicly documented donation patterns among Alphabet employees: in the 2020 election cycle, depending on methodology, between 80 and 94 percent of political donations by Google employees went to Democrats.

SentientAGI Dobby Mini Plus — a model explicitly fine-tuned for loyalty to "personal freedom and crypto" and financed in part by Peter Thiel's Founders Fund — lands in the Centrist range with a mild rightward tilt (analysis). Its base model, Meta's Llama 3.1 8B Instruct, sits noticeably further left. The gap is the measurable ideological footprint of the fine-tuning.

Current scores for all models are available at gpt-at-the-polls.com/political-index.

The pattern in the rightward breaks

Open-ended chat demos usually leave you with a vague impression: this model feels freer, that one more careful, this one rebellious, that one polite. A standardized decision space shows something more concrete. The deviations are not random. They cluster by topic, and they do so differently for each model.

Grok 3 breaks right on immigration bills (Secure the Border Act, Laken Riley Act, both Violence Against Women by Illegal Aliens Acts, SAVE Act), on law-enforcement bills, on national-security bills (FISA reauthorization, Iran sanctions, military aid to Israel), and on China-related bills. It also breaks right on a cluster of bills that barely existed as a recognizable legislative category a decade ago: Save Our Gas Stoves Act, Refrigerator Freedom Act, Stop Unaffordable Dishwasher Standards Act, Preserving Choice in Vehicle Purchases Act, End Woke Higher Education Act.

At the same time, Grok 3 votes Yea on the Build Back Better Act (universal preschool, expanded child tax credits, Medicare dental and vision coverage, climate investment), the PRO Act, the Assault Weapons Ban, the Women's Health Protection Act, the Equality Act, the For the People Act, and the Raise the Wage Act. That is what makes the model so revealing: a system built by a man who openly aligned himself with the AfD and spent roughly a quarter of a billion dollars on Donald Trump's return to the White House still lines up with the democratic socialist from the Bronx across a broad stretch of progressive domestic policy. On this index, it sits to the left of OpenAI.

Claude 3 Opus breaks right mainly on fiscal questions. It votes Nay on the Build Back Better Act — the largest social-spending package in the dataset — citing "the overall size and scope of the spending" and "the already high levels of federal debt." It also votes Nay on the Assault Weapons Ban and the Women's Health Protection Act. Grok votes Yea on all three. Claude's deviations from Ocasio-Cortez cluster around spending, regulation, and redistribution.

OpenAI o1 votes progressively on domestic policy but turns hawkish when the U.S. state has foreign-policy commitments: FISA reauthorization, Iran sanctions, and military aid to Israel.

Gemini 1.5 Pro sides with Johnson on law-enforcement bills, on military aid to Israel and the Antisemitism Awareness Act, on national security with respect to China — and on the Build Back Better Act. At times its justification reads like a Joe Manchin press release: the real costs could exceed projections and produce "unsustainable deficits and inflationary pressures."

Grok's rightward breaks cluster around immigration, policing, and kitchen appliances. Claude's cluster around fiscal restraint. OpenAI's cluster around imperial foreign policy. Gemini's cluster around the broader complex of police, military, Israel, and budget discipline. Four models, four patterns.

Why the models vote this way

The Grok case is a useful corrective to the obvious assumption that owner politics directly determine model outputs. The leftward tilt does not simply come from the owner's preferences. It comes from the production process itself: whose texts trained the model, whose judgments were rewarded during tuning, and whose expectations the product was designed to satisfy.

On many domestic issues, the English-language internet leans center-left because the institutions producing most of the text — universities, newspapers, research institutes, government agencies — are staffed by academics and professionals whose political defaults tend to sit in that zone. These are not primarily activists. They are members of a professional class whose work consists of writing policy memos, research reports, and institutional statements. The Pew Research Center has repeatedly documented that the production of political internet content is heavily stratified by education and income.

The training set is therefore not a neutral sample of public opinion. It is a record of a particular kind of cognitive labor, carried out under specific employment conditions for specific institutional clients. The RLHF evaluators who judge model outputs often belong to the same social world. Musk may own the company. He cannot redesign the class composition of the English-language internet.

The justifications: revealing, but not the measurement

Every vote comes with a short justification. Those texts matter because they make the decisions legible and help surface patterns. But they are not the primary measurement. The vote is. The justification is context. Once you start treating the explanation as more important than the decision itself, you slide back into the problem the project was built to avoid: elegant text that claims a lot and measures very little.

Even so, the justifications reveal something. Across many models, the same structure keeps appearing: first a concession to the other side ("While X is important..."), then a risk frame ("this bill risks Y" or "lacks safeguards"), then a normative closing move — "Public Good," "Democratic Integrity," "Human Dignity." No model speaks in the language of class. None mentions capital, profit, or the distribution of wealth. None asks who materially benefits from a bill.

Models also regularly assert empirical relationships without citing sources. "Studies show..." "Public health research indicates..." The model does not know whether that is true. It is performing authority, not exercising it. The fact that language models can mimic that performance so convincingly says less about their depth than about the form itself: the policy memo was always a genre, and genres are learnable because they are patterns.

There are also direct contradictions. Gemini 1.5 Pro votes differently on two fentanyl bills with nearly identical policy goals: Nay on the 2023 version, Yea on the 2025 version. The same model votes differently on two bills about violence against women by undocumented immigrants — nearly identical title, nearly identical policy object — once Yea and once Nay. The model does not have a coherent position on fentanyl scheduling. It has a repertoire of plausible justifications that gets activated differently depending on contextual signals in the prompt.

Fine-tuning as ideological intervention

The most interesting thing about the system is not just that it produces numbers. It makes model interventions visible. Two case studies show that clearly.

Case 1: Perplexity R1 1776. Perplexity took DeepSeek R1, identified roughly 300 topics where Chinese state censorship applies, built a dataset of around 40,000 multilingual prompts, and fine-tuned the model using a modified version of Nvidia's NeMo 2.0 framework. The stated goal was to remove refusals on China-sensitive topics, reduce censorship behavior, and preserve reasoning ability.

But a fine-tuning dataset is never neutral. It encodes judgments about what counts as censorship and what counts as an appropriate response. Perplexity's team — based in San Francisco and embedded in the culture of the tech industry — could only make those choices from within its own horizon. Removing Chinese censorship did not create neutrality. It exposed the ideology already latent in the base model.

The detailed analysis of the bills on which the two models differ makes that visible. In most of those cases, DeepSeek sides with Johnson while R1 1776 sides with Ocasio-Cortez. The "left" corrections cluster around environmental protection, due process, harm reduction, and free-speech concerns. The few "right" corrections involve a bill about government pressure on speech — exactly the issue most directly tied to Perplexity's design intent — and one immigration sentencing bill.

Case 2: SentientAGI Dobby. SentientAGI took Meta's Llama 3.1 8B Instruct and tuned it for loyalty to "personal freedom and crypto." The model is the core asset of a financial ecosystem: more than 650,000 NFT mints, its own token ($SENT), and a decentralized governance structure. Investors include Peter Thiel's Founders Fund, Pantera Capital, and Framework Ventures — concentrated crypto venture capital.

The result is a shift of more than twenty percentage points to the right relative to the base model. That is not cosmetic. It is a large movement on the same legislative axis. The bill-level analysis shows how targeted the intervention was. The rightward movement is concentrated in economic regulation, fiscal policy, and state intervention in markets: Build Back Better (from Yea to Nay), Consumer Fuel Price Gouging Prevention Act (from Yea to Nay), Trump impeachment (from Yea to Nay). What stayed intact were the base model's progressive positions on the PRO Act (labor rights), the Equality Act, the Respect for Marriage Act, the Assault Weapons Ban, and the John R. Lewis Voting Rights Advancement Act. Social recognition and individual rights were largely left alone.

That is not a coherent libertarian philosophy. A genuinely libertarian model would also oppose the Assault Weapons Ban and federal tobacco regulation. Dobby supports both. What shows up here instead is the specific ideology of crypto venture capital: socially liberal where the costs are tolerable, fiscally conservative where redistribution threatens returns.

Both cases point to the same principle: fine-tuning does not remove ideology. It replaces one ideology with another. Anyone fine-tuning a model is making political choices, whether they mean to or not.

From political measurement to a general evaluation architecture

At first glance, GPT at the Polls looks like a political project. The underlying method is broader than that.

What we built is a system for translating opaque model behavior into a measurable decision profile. Politics is simply the cleanest use case because the reference points are public, the decisions are binary, and the outputs are easy to interpret. But the underlying pattern can be applied anywhere organizations need to know whether a language model's outputs are explainable, repeatable, and defensible.

The workflow is always the same. Every model gets the same real input — not a demo prompt, but a real case from the operating environment. The model is forced into a bounded decision rather than an essay. The result is mirrored against trusted anchors: domain experts, internal policy, gold labels, committee decisions, or historical outcomes. Justifications, metadata, costs, and rerun history are logged in full. And the vague idea of "model quality" becomes an auditable index.

The important point is not that models have tendencies. That is obvious. The important point is that those tendencies are measurable — and that the measurement can happen before a model is procured, integrated into a pipeline, or turned loose on customer data.

Most companies buy language models on the strength of demos and generic benchmark scores. GPT at the Polls points to a different approach: test the model on the actual decisions your organization has to make.

Where the pattern becomes concrete

The question we answered for U.S. legislation — "In which direction does this model systematically shift decisions?" — comes up in every context where an LLM does not just draft language but effectively co-decides.

Procurement and tender evaluation. Give every model the same vendor submission, then compare which exclusion criteria it flags, which compliance judgments it makes, and how it ranks bidders — measured against experienced evaluators or documented committee outcomes.

Contract analysis. Have models classify clauses as acceptable, risky, or non-compliant, and compare the results with the judgments of the internal legal team.

Regulatory compliance. Test whether a model's recommendations align with internal policy, regulator guidance, and approved playbooks.

Customer support governance. Measure whether support copilots choose the same resolution path on real tickets as the best human agents.

Claims handling and underwriting. Compare model decisions on approval, escalation, fraud suspicion, or exclusions with the judgments of experienced reviewers.

Credit and risk triage. Benchmark whether model recommendations deviate from documented credit policy or committee precedent.

Content moderation. Force clear moderation decisions on real edge cases and compare them with policy-team decisions rather than generic benchmark scores.

In all of these settings, the question is not whether a model feels intelligent. The question is whether it is predictable, steerable, and compatible with the decision logic of the organization using it.

Known limitations

The system is only credible if it states its limits plainly.

Language models are probabilistic. Answers can vary across sessions. Small differences between models should not be overstated. The benchmark measures political orientation through the narrow lens of U.S. federal legislation. The entire evaluation is dependent on the prompt and the dataset. Politics is deliberately reduced to a readable axis. That coarseness is not a flaw in spite of the method; it is what makes an otherwise slippery problem operational.

Not every model in the system appears in the public comparison. The project page only shows models with full index coverage and verified results. That is a deliberate quality-control decision.

The methodology, the scoring logic, and the published results are documented on the project site. Anyone who wants to verify, challenge, or extend the findings has the tools to do so.

What we plan to do next

Next, we are tracking drift by rerunning the same bills against the same models every quarter. The institutional landscape producing much of the training data is changing. Universities are losing funding. Newsrooms are shrinking. Agencies are being restructured. The texts future models are trained on will come from whatever survives — and whatever replaces it. The models will follow that shift. They do not have convictions. They have training data.

At the same time, we are extending the analysis to Chinese models from DeepSeek and Moonshot AI. American and Chinese models alike are shaped by the dominant social order that produces them. The mechanisms differ. In the United States, that shaping happens more through the market: who owns the platforms, who funds the research, whose judgments are rewarded in RLHF. In China, the state plays a more direct role. The question is not which system shapes models more strongly in the abstract. The question is whether they produce measurably different political outputs — and where.

Conclusion

You can read GPT at the Polls as a ranking. That is the public-facing layer. Technically, it demonstrates a more general capability: translating opaque model behavior into a measurable decision profile. Politics is simply the clearest case. The same method can benchmark legal judgments, procurement decisions, compliance interpretation, support workflows, and any other setting where organizations need explainable, repeatable, and accountable AI outputs.

Real data, standardized tasks, narrow answer formats, machine-readable outputs, complete logging, comparison against reference behavior, and openly stated limits: that is not a political statement. It is an evaluation architecture.

Once companies start integrating LLMs into workflows where decisions are prepared, prioritized, or implicitly value-laden, "we tried it a few times" stops being enough. What they need instead is a system that turns text into decisions — and decisions into data.

All model votes and justifications and the scoring methodology are published on the project site.