Peer Review: The Flawed Mechanism That Still Works

KNOWLEDGE VALIDATION (Pre-1700)

HOW YOU ESTABLISHED CREDIBILITY: ┌─────────────────────────────────────────┐ │ 1. PATRONAGE │ │ • Get wealthy patron (king, noble) │ │ • Patron's endorsement = credibility │ │ ↓ │ │ 2. SOCIAL STATUS │ │ • Be a gentleman │ │ • "Gentleman's word" = trustworthy │ │ ↓ │ │ 3. DEMONSTRATION │ │ • Perform experiments publicly │ │ • Witnesses vouch for results │ │ ↓ │ │ 4. CORRESPONDENCE │ │ • Private letters to other │ │ philosophers │ │ • Build reputation through network │ └─────────────────────────────────────────┘

THE PROBLEMS: ┌─────────────────────────────────────────┐ │ ✗ Access based on social class │ │ ✗ No systematic verification │ │ ✗ Priority disputes (who discovered │ │ first?) │ │ ✗ Secrecy (to protect priority) │ │ ✗ No quality control │ └─────────────────────────────────────────┘

PRIORITY DISPUTE (1670s-1680s) ┌─────────────────────────────────────────┐ │ Both claimed discovery of inverse- │ │ square law │ │ ↓ │ │ No formal system to adjudicate │ │ ↓ │ │ Resolved by: Social prestige (Newton │ │ more respected) + private letters │ │ ↓ │ │ Bitter rivalry for decades │ └─────────────────────────────────────────┘

PHILOSOPHICAL TRANSACTIONS (1665)

OLDENBURG'S INNOVATION: ┌─────────────────────────────────────────┐ │ Create official journal │ │ ↓ │ │ Publication = PUBLIC record of discovery│ │ ↓ │ │ Solves priority disputes (date of │ │ publication proves who was first) │ │ ↓ │ │ But: How to decide WHAT to publish? │ └─────────────────────────────────────────┘

INFORMAL REVIEW: ┌─────────────────────────────────────────┐ │ Oldenburg sent manuscripts to experts │ │ ↓ │ │ "What do you think?" │ │ ↓ │ │ Not systematic │ │ Not anonymous │ │ Not required │ │ ↓ │ │ Just: Expert opinion │ └─────────────────────────────────────────┘

SLOW FORMALIZATION (1665-1850): ┌─────────────────────────────────────────┐ │ Different journals, different practices:│ │ • Some: Editor decides alone │ │ • Some: Society committee │ │ • Some: Ask experts informally │ │ ↓ │ │ No standard "peer review" system │ └─────────────────────────────────────────┘

WHY FORMALIZATION HAPPENED (1900s-1950s)

PROFESSIONALIZATION: ┌─────────────────────────────────────────┐ │ More scientists → More papers │ │ ↓ │ │ Editors can't evaluate everything alone │ │ ↓ │ │ Need expert reviewers │ └─────────────────────────────────────────┘

JOURNAL PROLIFERATION: ┌─────────────────────────────────────────┐ │ 1800: ~100 scientific journals │ │ 1900: ~10,000 journals │ │ 1950: ~50,000+ journals │ │ ↓ │ │ Need standardized quality control │ └─────────────────────────────────────────┘

GRANT FUNDING: ┌─────────────────────────────────────────┐ │ Post-WWII: Government funds research │ │ ↓ │ │ How to allocate scarce resources? │ │ ↓ │ │ Peer review grants (started ~1946, NIH) │ └─────────────────────────────────────────┘

THE MODERN SYSTEM EMERGES: ┌─────────────────────────────────────────┐ │ By 1950s-1960s: │ │ • Submission to journal │ │ • Editor sends to 2-3 anonymous │ │ reviewers │ │ • Reviewers evaluate: │ │ - Novelty │ │ - Methodology │ │ - Significance │ │ - Clarity │ │ • Recommend: Accept / Revise / Reject │ │ • Editor makes final decision │ │ ↓ │ │ STANDARDIZED PEER REVIEW │ └─────────────────────────────────────────┘

THE PROCESS (Standard journal submission)

STEP 1: SUBMISSION ┌─────────────────────────────────────────┐ │ Author submits manuscript to journal │ │ ↓ │ │ Editor does quick check: │ │ • Is it in scope? │ │ • Does it meet basic standards? │ │ ↓ │ │ ~30-50% rejected at this stage ("desk │ │ reject") │ └─────────────────────────────────────────┘

STEP 2: REVIEWER SELECTION ┌─────────────────────────────────────────┐ │ Editor identifies 2-4 potential │ │ reviewers: │ │ • Experts in the field │ │ • No conflicts of interest (supposedly) │ │ ↓ │ │ Invites reviewers (many decline—unpaid │ │ work) │ │ ↓ │ │ Often takes weeks to find willing │ │ reviewers │ └─────────────────────────────────────────┘

STEP 3: REVIEW ┌─────────────────────────────────────────┐ │ Reviewers read manuscript (1-4 weeks) │ │ ↓ │ │ Write review addressing: │ │ • Scientific quality │ │ • Methodological soundness │ │ • Novelty/significance │ │ • Clarity │ │ ↓ │ │ Recommend decision: │ │ • Accept (rare: <5%) │ │ • Minor revisions (15%) │ │ • Major revisions (30%) │ │ • Reject (50%) │ └─────────────────────────────────────────┘

STEP 4: DECISION ┌─────────────────────────────────────────┐ │ Editor reads reviews │ │ ↓ │ │ Makes decision (not bound by reviewers) │ │ ↓ │ │ Common outcomes: │ │ • Reject (~50-90% depending on journal) │ │ • Revise and resubmit (~30-40%) │ │ • Accept with minor revisions (~10%) │ └─────────────────────────────────────────┘

STEP 5: REVISION (if requested) ┌─────────────────────────────────────────┐ │ Authors revise manuscript │ │ ↓ │ │ Address reviewer comments │ │ ↓ │ │ Resubmit │ │ ↓ │ │ Back to reviewers (sometimes) │ │ ↓ │ │ May take multiple rounds │ └─────────────────────────────────────────┘

TIMELINE: ┌─────────────────────────────────────────┐ │ Submission → Initial decision: 2-6 │ │ months │ │ Revisions: 1-6 months │ │ Final decision: 6-18 months total │ │ ↓ │ │ Fast for some journals (weeks) │ │ Slow for others (years) │ └─────────────────────────────────────────┘

THE IDEAL (What people think peer review does)

QUALITY CONTROL: ┌─────────────────────────────────────────┐ │ ✓ Catch errors │ │ ✓ Verify methodology │ │ ✓ Ensure reproducibility │ │ ✓ Filter out bad science │ │ ↓ │ │ Result: Published papers are reliable │ └─────────────────────────────────────────┘

VALIDATION: ┌─────────────────────────────────────────┐ │ ✓ Expert evaluation │ │ ✓ Independent verification │ │ ✓ Scientific consensus │ │ ↓ │ │ Result: Peer-reviewed = Trustworthy │ └─────────────────────────────────────────┘

GATEKEEPING: ┌─────────────────────────────────────────┐ │ ✓ Maintain standards │ │ ✓ Prevent fraud │ │ ✓ Ensure significance │ │ ↓ │ │ Result: Literature is high-quality │ └─────────────────────────────────────────┘

THE REALITY (What peer review actually accomplishes)

FILTERS OBVIOUS GARBAGE: ┌─────────────────────────────────────────┐ │ ✓ Catches blatant errors │ │ ✓ Rejects clearly flawed work │ │ ✓ Removes completely out-of-scope papers│ │ ↓ │ │ But: Doesn't catch subtle errors │ └─────────────────────────────────────────┘

IMPROVES PRESENTATION: ┌─────────────────────────────────────────┐ │ ✓ Suggests clarifications │ │ ✓ Identifies missing information │ │ ✓ Improves writing │ │ ↓ │ │ Papers are more readable after review │ └─────────────────────────────────────────┘

SIGNALS CREDIBILITY: ┌─────────────────────────────────────────┐ │ ✓ "Peer-reviewed" = Minimum threshold │ │ ✓ Better than no review │ │ ↓ │ │ But: Doesn't guarantee correctness │ └─────────────────────────────────────────┘

WHAT IT DOESN'T DO: ┌─────────────────────────────────────────┐ │ ✗ Doesn't verify data (reviewers don't │ │ see raw data) │ │ ✗ Doesn't check reproducibility │ │ (reviewers don't replicate) │ │ ✗ Doesn't detect fraud (reviewers assume│ │ honesty) │ │ ✗ Doesn't catch statistical errors │ │ (most reviewers not statisticians) │ │ ✗ Doesn't ensure significance (subjective│ │ judgment) │ └─────────────────────────────────────────┘

PUBLICATION BIAS

POSITIVE RESULTS BIAS: ┌─────────────────────────────────────────┐ │ Studies with positive results: │ │ • More likely to be submitted │ │ • More likely to be accepted │ │ ↓ │ │ Studies with negative/null results: │ │ • Often not submitted ("file drawer │ │ problem") │ │ • Often rejected as "not interesting" │ │ ↓ │ │ Result: Literature biased toward │ │ positive findings │ │ ↓ │ │ Creates false impression of effect │ │ sizes │ └─────────────────────────────────────────┘

NOVELTY BIAS: ┌─────────────────────────────────────────┐ │ "Novel" findings favored over: │ │ • Replications (seen as boring) │ │ • Confirmations (not newsworthy) │ │ • Negative results (not interesting) │ │ ↓ │ │ Incentivizes finding new results, not │ │ verifying old ones │ └─────────────────────────────────────────┘

PRESTIGE BIAS: ┌─────────────────────────────────────────┐ │ Papers from famous scientists/ │ │ institutions: │ │ • Reviewed more favorably │ │ • Given benefit of doubt │ │ ↓ │ │ Papers from unknown scientists: │ │ • Scrutinized more harshly │ │ • Held to higher standards │ │ ↓ │ │ Even when reviewers don't know authors │ │ (clues in writing, citations) │ └─────────────────────────────────────────┘

CONFIRMATION BIAS: ┌─────────────────────────────────────────┐ │ Reviewers favor work that: │ │ • Confirms their beliefs │ │ • Uses their methods │ │ • Cites their papers │ │ ↓ │ │ Reviewers skeptical of work that: │ │ • Challenges their research │ │ • Uses different approaches │ │ • Ignores their contributions │ └─────────────────────────────────────────┘

CONSERVATIVE BIAS: ┌─────────────────────────────────────────┐ │ Peer review favors: │ │ • Incremental work │ │ • Established methods │ │ • Conventional ideas │ │ ↓ │ │ Peer review resists: │ │ • Radical claims │ │ • Novel methods │ │ • Paradigm shifts │ │ ↓ │ │ Revolutionary science often initially │ │ rejected │ └─────────────────────────────────────────┘

INITIALLY REJECTED PAPERS (Later proved important)

KREBS CYCLE (Hans Krebs, 1937): ┌─────────────────────────────────────────┐ │ Rejected by Nature │ │ ↓ │ │ Published in obscure journal │ │ ↓ │ │ Won Nobel Prize (1953) │ └─────────────────────────────────────────┘

DOUBLE HELIX (Watson & Crick, 1953): ┌─────────────────────────────────────────┐ │ Multiple reviewers skeptical │ │ ↓ │ │ Accepted only because Nature editor │ │ overruled reviewers │ │ ↓ │ │ Most important biology paper of 20th │ │ century │ └─────────────────────────────────────────┘

BACTERIAL CAUSE OF ULCERS (Marshall & Warren): ┌─────────────────────────────────────────┐ │ Rejected repeatedly (1980s) │ │ ↓ │ │ "Everyone knows ulcers are caused by │ │ stress/acid" │ │ ↓ │ │ Finally published after Marshall │ │ infected himself │ │ ↓ │ │ Won Nobel Prize (2005) │ └─────────────────────────────────────────┘

FRAUD THAT PASSED PEER REVIEW

JAN HENDRIK SCHÖN (Physics fraud, 2000-2002): ┌─────────────────────────────────────────┐ │ Published 90+ papers in top journals │ │ (Science, Nature, Physical Review) │ │ ↓ │ │ All passed peer review │ │ ↓ │ │ All were fraudulent (fabricated data) │ │ ↓ │ │ Discovered only when colleagues couldn't│ │ replicate │ │ ↓ │ │ All papers retracted │ └─────────────────────────────────────────┘

ANDREW WAKEFIELD (Vaccines-autism, 1998): ┌─────────────────────────────────────────┐ │ Published in The Lancet (prestigious) │ │ ↓ │ │ Passed peer review │ │ ↓ │ │ Claimed MMR vaccine causes autism │ │ ↓ │ │ Based on: │ │ • Fraudulent data │ │ • Unethical experiments │ │ • Conflicts of interest (paid by │ │ lawyers) │ │ ↓ │ │ Retracted 2010 │ │ ↓ │ │ But: Damage done (vaccine hesitancy │ │ persists) │ └─────────────────────────────────────────┘

DIEDERIK STAPEL (Psychology fraud, 2011): ┌─────────────────────────────────────────┐ │ 50+ fraudulent papers │ │ ↓ │ │ All passed peer review │ │ ↓ │ │ Invented entire datasets │ │ ↓ │ │ Discovered by suspicious students │ └─────────────────────────────────────────┘

HWANG WOO-SUK (Stem cell fraud, 2005): ┌─────────────────────────────────────────┐ │ Claimed human cloning breakthrough │ │ ↓ │ │ Published in Science │ │ ↓ │ │ Passed peer review │ │ ↓ │ │ Completely fabricated │ └─────────────────────────────────────────┘

THE PROBLEM (Discovered 2010s)

PSYCHOLOGY: ┌─────────────────────────────────────────┐ │ Reproducibility Project (2015): │ │ • Attempted to replicate 100 psychology │ │ studies │ │ • From top journals │ │ • All peer-reviewed │ │ ↓ │ │ Results: Only 36% replicated │ │ ↓ │ │ 64% of peer-reviewed findings: Wrong │ └─────────────────────────────────────────┘

BIOMEDICAL SCIENCE: ┌─────────────────────────────────────────┐ │ Amgen study (2012): │ │ • Tried to replicate 53 "landmark" │ │ cancer studies │ │ • All peer-reviewed, highly cited │ │ ↓ │ │ Results: Only 6 (11%) replicated │ │ ↓ │ │ 89% of peer-reviewed findings: Wrong │ └─────────────────────────────────────────┘

WHY PEER REVIEW DIDN'T CATCH THIS: ┌─────────────────────────────────────────┐ │ Reviewers don't: │ │ • Reanalyze data │ │ • Check for p-hacking │ │ • Verify statistical analyses │ │ • Replicate experiments │ │ ↓ │ │ They just read and give opinion │ │ ↓ │ │ If paper looks plausible, it passes │ └─────────────────────────────────────────┘

THE SYSTEMATIC PROBLEMS: ┌─────────────────────────────────────────┐ │ • Small sample sizes (underpowered) │ │ • P-hacking (fishing for significance) │ │ • HARKing (hypothesizing after results) │ │ • Publication bias (negative results │ │ unpublished) │ │ ↓ │ │ All passed peer review │ │ ↓ │ │ Because peer review doesn't check for │ │ these │ └─────────────────────────────────────────┘

PREDATORY PUBLISHING (2000s-present)

THE BUSINESS MODEL: ┌─────────────────────────────────────────┐ │ "Open access" journals that: │ │ • Charge authors to publish (~$1000- │ │ $5000) │ │ • Accept almost everything │ │ • Provide no real peer review │ │ ↓ │ │ Profit from publication fees │ └─────────────────────────────────────────┘

SCALE OF PROBLEM: ┌─────────────────────────────────────────┐ │ ~10,000+ predatory journals │ │ ~400,000+ papers/year │ │ ↓ │ │ All claim "peer review" │ │ ↓ │ │ Most have no real review │ └─────────────────────────────────────────┘

FAMOUS STINGS:

PETER VAMPLEW (2014): ┌─────────────────────────────────────────┐ │ Submitted paper: "Get me off Your │ │ Fucking Mailing List" repeated │ │ ↓ │ │ Accepted after "peer review" │ │ ↓ │ │ Asked for $150 to publish │ └─────────────────────────────────────────┘

SCIgen (Automatic paper generator): ┌─────────────────────────────────────────┐ │ Computer-generated nonsense papers │ │ ↓ │ │ Many accepted by predatory journals │ │ ↓ │ │ "Peer reviewed" │ └─────────────────────────────────────────┘

FAKE PEER REVIEW RINGS: ┌─────────────────────────────────────────┐ │ Authors suggest fake reviewers │ │ ↓ │ │ Using fake email addresses they control │ │ ↓ │ │ "Review" their own papers favorably │ │ ↓ │ │ Discovered 2015: 170+ papers retracted │ │ from major publishers │ └─────────────────────────────────────────┘

ALTERNATIVES CONSIDERED

NO PEER REVIEW: ┌─────────────────────────────────────────┐ │ Just publish everything │ │ ↓ │ │ Problems: │ │ • Flood of garbage │ │ • No quality signal │ │ • How to find good work? │ │ ↓ │ │ Rejected: Worse than peer review │ └─────────────────────────────────────────┘

POST-PUBLICATION REVIEW: ┌─────────────────────────────────────────┐ │ Publish first, review after │ │ ↓ │ │ Advantages: │ │ • Faster │ │ • Transparent │ │ • More reviewers │ │ ↓ │ │ Problems: │ │ • Still need initial filter │ │ • Who reads reviews? │ │ • Not widely adopted │ └─────────────────────────────────────────┘

OPEN PEER REVIEW: ┌─────────────────────────────────────────┐ │ Reviewers sign their reviews │ │ ↓ │ │ Advantages: │ │ • Accountability │ │ • Less snark │ │ ↓ │ │ Problems: │ │ • Fear of retaliation │ │ • Less honest criticism │ │ • Junior reviewers intimidated │ └─────────────────────────────────────────┘

EDITORIAL REVIEW ONLY: ┌─────────────────────────────────────────┐ │ Editor decides without external │ │ reviewers │ │ ↓ │ │ Problems: │ │ • Editor can't be expert in everything │ │ • Too much power to one person │ │ • Slower than peer review │ └─────────────────────────────────────────┘

CURRENT REFORM EFFORTS

REGISTERED REPORTS: ┌─────────────────────────────────────────┐ │ Submit study design BEFORE conducting │ │ research │ │ ↓ │ │ Peer review evaluates: │ │ • Question importance │ │ • Method quality │ │ ↓ │ │ If approved: Guaranteed publication │ │ regardless of results │ │ ↓ │ │ Prevents: │ │ • Publication bias │ │ • P-hacking │ │ • HARKing │ │ ↓ │ │ Adoption: Growing but still minority │ └─────────────────────────────────────────┘

OPEN PEER REVIEW (partial): ┌─────────────────────────────────────────┐ │ Publish reviews alongside paper │ │ ↓ │ │ Transparency without requiring signed │ │ reviews │ │ ↓ │ │ Some journals adopting │ └─────────────────────────────────────────┘

STATISTICAL REVIEW: ┌─────────────────────────────────────────┐ │ Require statistical expert reviewer │ │ ↓ │ │ Catch statistical errors peer reviewers │ │ miss │ │ ↓ │ │ Some journals now require │ └─────────────────────────────────────────┘

REPRODUCIBILITY CHECKS: ┌─────────────────────────────────────────┐ │ Require: │ │ • Open data │ │ • Open code │ │ • Preregistration │ │ ↓ │ │ Allow verification │ │ ↓ │ │ Growing adoption │ └─────────────────────────────────────────┘

REVIEWER TRAINING: ┌─────────────────────────────────────────┐ │ Currently: No training required │ │ ↓ │ │ Proposal: Formal reviewer training │ │ ↓ │ │ Few journals implement │ └─────────────────────────────────────────┘

WHY PEER REVIEW PERSISTS DESPITE FLAWS

IT PROVIDES: ┌─────────────────────────────────────────┐ │ ✓ Minimal quality filter (better than │ │ nothing) │ │ ✓ Improvement suggestions (papers better│ │ after review) │ │ ✓ Legitimacy signal (peer-reviewed = │ │ minimum credibility) │ │ ✓ Distributed expertise (reviewers know │ │ specialized fields) │ │ ✓ Independence (reviewers not authors' │ │ friends/colleagues) │ └─────────────────────────────────────────┘

IT'S EMBEDDED IN SYSTEM: ┌─────────────────────────────────────────┐ │ Career advancement requires: │ │ • Peer-reviewed publications │ │ • Peer-reviewed grants │ │ ↓ │ │ Removing peer review = Removing career │ │ evaluation mechanism │ │ ↓ │ │ Would need complete redesign of │ │ scientific institutions │ └─────────────────────────────────────────┘

NO BETTER ALTERNATIVE: ┌─────────────────────────────────────────┐ │ Peer review has problems │ │ ↓ │ │ But: All alternatives have worse │ │ problems │ │ ↓ │ │ Stuck with flawed mechanism because │ │ nothing else works │ └─────────────────────────────────────────┘

THE REALITY: ┌─────────────────────────────────────────┐ │ Peer review is: │ │ ✓ Better than no review │ │ ✓ Improves papers │ │ ✓ Filters obvious garbage │ │ ↓ │ │ But NOT: │ │ ✗ Rigorous verification │ │ ✗ Fraud detection │ │ ✗ Reproducibility guarantee │ │ ✗ Truth certification │ └─────────────────────────────────────────┘

WHAT TO EXPECT: ┌─────────────────────────────────────────┐ │ Peer-reviewed ≠ Correct │ │ ↓ │ │ Peer-reviewed = "Experts thought this │ │ looked plausible" │ │ ↓ │ │ That's it │ └─────────────────────────────────────────┘