Verify this study yourself.

Ballot Study V5. Pilot wave, frozen and pre-registered (OSF). Currently the published dataset.

You do not have to take our word for any number on this site. Below are the questions a careful, even hostile, reviewer would ask, each with a check you can run here in your own browser and a way to do it by hand. Click "Run all checks" to do the automated ones at once.

  1. Did they quietly change the data or the page?
  2. Do the numbers really come from the data?
  3. Does the headline depend on which respondents they kept?
  4. Is anyone's privacy exposed, and is the data sane?
  5. Is the analysis code covered by unit tests?
  6. Is the method sound, not just the arithmetic?
  7. Is the analysis script rigged?
  8. Was the data collected honestly, on real people?
  9. Were the rules fixed before the data was seen?

Automated checks (run in your browser)

Nothing run yet
1

Nothing here was quietly changed

Not run

Fetches every file the site publishes (the data, the numbers, and each downloadable source file), hashes the bytes in your browser, and compares them to the published fingerprint list. A single changed character anywhere would show here.

Do it by hand

Open manifest.json, download any file it lists, and hash it:

# Windows PowerShell
Get-FileHash rthg-s5-dataset.jsonl -Algorithm SHA256
# macOS / Linux
shasum -a 256 rthg-s5-dataset.jsonl

The printed value should equal the sha256 for that file in the manifest, and the dataset value should also equal dataset.fileSha256 in the published numbers file.

Do you have to trust our code?

No, and you should not have to. Two routes get you to these numbers without reading a line of what we wrote. You can recompute every figure straight from the raw dataset with your own tools, or hand the data to an AI and have it derive the numbers from scratch (step 2 below walks through both). You can also skip our data entirely and re-field the identical instrument on your own participants: the fielded Qualtrics file imports into any Qualtrics account, and the replication guide covers the whole run end to end.

The application repository is not public, so we want to be exact about what the fingerprint check above does and does not establish. It proves the live site serves byte-for-byte the files it says it serves, and that every published number recomputes from those files. It proves nothing about code you cannot see, and we do not claim otherwise. That is why the instrument, the frozen de-identified data, the published figures, and the analysis code all ship as downloads in the Ballot Study V5 packet: independent replication should never depend on our good faith.

2

Every published number comes from the data

Not run

Re-applies the pre-registered exclusion rules to the raw rows and recomputes every figure we publish: the funnel, headline support and its interval, support by party, the weighted estimate, principle agreement, the change after people see the enforcement mechanism, and which mechanism they preferred. All of it should match.

Or have an AI re-derive it from scratch (no coding)

This is the strongest check that our own code is not doing something quietly: download the two data files at the top, open an AI that can run code on uploads (for example ChatGPT's data-analysis tool), drag both files in, and paste this prompt. It re-implements the whole analysis from the raw data without our code at all.

Open ChatGPT
You are auditing a survey for reproducibility. I have uploaded two files:
- rthg-s5-dataset.jsonl: the raw data, one survey response per line as JSON. Each row has "payload.pages" (the answers, keyed by page name then question name) and "payload.timings" (milliseconds per page).
- rthg-s5.json: the published results, with the expected file hash at "dataset.fileSha256" and the published support figures.

Use code and show your work. Do all of the following:

1. Compute the SHA-256 hash of rthg-s5-dataset.jsonl exactly as uploaded (the raw bytes of the file) and compare it to "dataset.fileSha256" in rthg-s5.json. State MATCH or MISMATCH (a mismatch means the data file was edited after it was frozen). Note: "dataset.sha256" is a separate key-sorted canonical hash used by the reproduce script, not the file-bytes hash, so do not compare against that one.
2. Count the rows.
3. Apply these exclusion rules and report how many rows each removes:
   - section-b-principles.B-att must equal 4
   - section-b-principles.B-rev must be 2 or lower (remove a row if B-rev >= 3)
   - s5-ballot.smoke4-comprehension-check must equal "Two years"
   - section-g-demographics.final-attention-check must equal "Principles for how government should explain major decisions"
   - using payload.timings (milliseconds): remove rows where timings['section-b-principles'] < 45000, or timings['s5-mechanism'] < 45000
   - drop respondents who report they were not eligible to vote in 2024
   The frozen analysis code also contains attention, comprehension, and speeder checks keyed to other survey versions' page names (for example section-c-a-ballot, section-d-arm, smoke4-open). Those page keys do not appear in this dataset, so those rules remove zero rows here; you may apply them as well and confirm they change nothing.
4. On the surviving rows, compute the percent who "support" (final ballot vote of 5, 6, or 7 on the 1-to-7 scale), with a Wilson 95% confidence interval.
5. Break that support percentage down by section-g-demographics.party.
6. Recompute support with NO quality exclusions at all (every completed response). It should still be about 90%.
7. Compare your numbers to the published figures in rthg-s5.json and say whether they match within rounding.

Finish in plain English: did the data file appear intact (hash match) or edited, and does the published headline (about 90% support, above 88% in every major party bloc) reproduce from the raw data, including with no exclusions?
3

The headline does not depend on our cutoffs

Not run

Recomputes the headline under five different definitions of the sample, from "every completed response with no quality exclusions" to the strict published cohort. If support only looked high because of how we filtered, it would fall apart here. It does not.

Do it by hand

The AI reproduce prompt (step 2) includes a line that recomputes support with no exclusions at all. npm run reproduce prints every cohort in the FUNNEL and BALLOT SUPPORT blocks.

4

No one is exposed, and the data is sane

Not run

Confirms the public file protects respondents: every row is an unlinkable salted pseudonym, there are no duplicate respondents, no raw Prolific IDs, and the written free-text answers were removed.

Do it by hand

Search the file for any 24-character ID and you will find none. Each row's pidis p_ followed by 16 hex characters, derived from a published salt, so only you, knowing your own Prolific ID, can find your row. The snippet to do that is under "Find your own response" below.

5

The analysis code is covered by unit tests

Not run

Runs the full unit-test suite against the analysis engine in your browser, the same code that produced every figure on this page. Coverage spans parsing of raw rows, the Wilson confidence interval and every other statistic, every S5 and S6 exclusion rule (both tiers), the cluster-balanced Section B item-order rule (1,000-seed property test), and cohort-funnel narrowing. A failure here means the deployed code differs in behavior from what the rules document.

How to run the same tests on the command line

Download the analysis engine and the unit tests from the Ballot Study V5 packet, then run the same cases under vitest in Node:

npm install
npm test

Or run them without the dev dependency by invoking the framework directly:

npx tsx lib/study-tests/run.ts

The deeper questions (beyond the math)

Is the analysis script rigged? Have an AI read the actual code

The numbers above come from published code, the same code this page runs. Download it from the Ballot Study V5 packet (the analysis engine and the reproduce script), upload it to an AI, and paste this prompt to have it look for any rule that inflates support.

Get the code
You are reviewing the analysis code for a public survey for signs that it is rigged to inflate support. I have uploaded the analysis engine and the reproduce script (TypeScript): index.ts, parse.ts, filters.ts, stats.ts, waves.ts, types.ts, and rthg-reproduce.ts.

Read the code and answer plainly:
1. How is "support" defined? Is counting a 5, 6, or 7 on a 1-to-7 scale as support a fair, standard top-box definition, or is it gerrymandered?
2. Are the exclusion rules in filters.ts ordinary data-quality checks (attention, comprehension, speeders), or are any of them designed to drop people who oppose the measure?
3. Is the Wilson 95% interval in stats.ts implemented correctly?
4. Is anything hard-coded, faked, or short-circuited so the output does not actually depend on the input data?
5. Does any rule reference the respondent's vote or party when deciding whether to keep a row?

Give a candid verdict: is this analysis code honest, and would it produce a misleadingly high number?
Is the method sound? Have an AI critique the design

Arithmetic that reproduces can still come from a leading instrument. Download the survey instrument and the pre-registration from the Ballot Study V5 packet (or read the methodology rationale), upload them, and paste this prompt to get a candid critique of question wording, order effects, and how "support" is measured.

Read the rationale
You are a skeptical survey methodologist reviewing a study's instrument and pre-registration for bias, not its arithmetic. I have uploaded the survey instrument (rthg-study-definition.ts), the plain-English pre-registration, and the methodology rationale.

Critique the design directly:
1. Are any questions leading, loaded, or framed to push respondents toward supporting the measure?
2. Could the order of sections (rating the principles before voting on the law) bias the final vote? How much, and in which direction?
3. Are there demand characteristics or sponsor cues that signal a "right" answer?
4. Is the way "support" is measured (a 1-to-7 ballot, top-3 counted as support) fair?
5. Are the exclusion rules reasonable, or could they inflate support?
6. Is the sample (Prolific, nonprobability) being over-claimed as nationally representative?

Be candid. List the strongest legitimate criticisms a hostile expert would make, then say whether, on balance, the ~90% support figure looks like an artifact of the design or a real finding.
Was the data collected honestly? Replicate it on real people

The strongest possible check is to run the study again yourself. Field the identical instrument to your own participants on your own Prolific account (so it costs us nothing), and compare your result to ours.

You can also take the study yourself and compare what you see, page for page, to the published instrument in the Ballot Study V5 packet.

Participants: find your own response

The data contains no Prolific IDs. Each row has a salted pseudonym, so only you, knowing your own Prolific ID, can find your row, and no one's ID is ever exposed:

pid = "p_" + first 16 hex chars of sha256("rthg-public-pseudonym-v1" + YOUR_PROLIFIC_ID)

Compute yours and find your row (Python, no install beyond Python itself):

import hashlib, json
pid_input = "PUT_YOUR_PROLIFIC_ID_HERE"
mine = "p_" + hashlib.sha256(("rthg-public-pseudonym-v1" + pid_input).encode()).hexdigest()[:16]
print("your pseudonym:", mine)
for line in open("rthg-s5-dataset.jsonl", encoding="utf-8"):
    row = json.loads(line)
    if row.get("pid") == mine:
        print("found your row. demographics:",
              row["payload"]["pages"].get("section-g-demographics"))
        break
else:
    print("no matching row in this dataset")

Were the rules fixed before the data was seen?

Before we analyzed the data, the design, exclusion rules, and analysis plan were posted publicly on OSF (osf.io/br3u2). That timestamp means the rules could not have been chosen after seeing the result. Plain-English summary: the pre-registration. Full detail, including the verbatim instrument: the complete pre-registration package. Any deviation from the plan is disclosed in the methodology rationale.