Tier 1 Complete + Kim 2023: 6 Papers, 27 Claims, 4 Backends

Reproducibility is one of the quiet crises in quantum computing. Papers report impressive results on custom hardware, but how well do those results transfer to different backends? We built an automated pipeline to find out — and the results tell a clear story about the current state of quantum computing.

The Approach

Our replication pipeline works in three stages:

1. Claim extraction — We identify specific, quantitative claims from each paper: ground-state energies, fidelities, threshold tests, improvement factors.
2. Reproduction — We implement each experiment using PennyLane (simulation) and Qiskit (hardware), testing across four backends: QI emulator (noiseless), IBM Torino (133 superconducting qubits), QI Tuna-9 (9 superconducting qubits), and IQM Garnet (20 superconducting qubits).
3. Classification — Each claim gets a failure mode: success (within published error bars), partial noise (qualitatively correct but degraded), noise dominated (signal overwhelmed), or structural failures (circuit translation, parameter mismatch, missing detail).

The Scorecard

Paper	Claims	Pass	Rate	Backends
Sagastizabal 2019	4	4	100%	Emulator, IBM, Tuna-9
Kandala 2017	5	5	100%	Emulator, IBM, Tuna-9
Peruzzo 2014	9	7	78%	Emulator, IBM, Tuna-9
Cross 2019	3	3	100%	Emulator, IBM, Tuna-9, IQM
Harrigan 2021	4	4	100%	Emulator, Tuna-9
Kim 2023	3	3	100%	Emulator, IBM, Tuna-9
Total	27	25	93%	4 backends

Paper 1: Sagastizabal et al. (2019) — Symmetry Verification VQE

Phys. Rev. A 100, 010302(R) — arXiv:1902.11258

This QuTech paper demonstrates symmetry verification on a 2-qubit VQE for H2. We tested 4 claims across 3 backends. The emulator reproduces the ground state within 0.75 kcal/mol. The breakthrough came from our mitigation ladder: TREX (Twirled Readout Error eXtinction) — a single flag change in Qiskit's EstimatorV2 (resilience_level=1) — achieves 0.22 kcal/mol on IBM Torino, well within chemical accuracy. That's 119x better than raw measurement (26.2 kcal/mol). On Tuna-9, the best qubit pair [2,4] achieves 3.04 kcal/mol with Z-parity post-selection. The symmetry verification improvement factor (119x on IBM, 3.6x on Tuna-9) vastly exceeds the paper's published >2x claim.

Result: 100% pass (4/4 claims). TREX is the single most impactful error mitigation technique we tested — one line of code, chemical accuracy.

Paper 2: Kandala et al. (2017) — Hardware-Efficient VQE

Nature 549, 242 — arXiv:1704.05018

The foundational paper on hardware-efficient ansatze for VQE. We replicated the H2 potential energy curve using a 4-qubit Jordan-Wigner encoding (the original used parity mapping for 2 qubits). On the emulator, all 10 bond distances achieve chemical accuracy with warm-start optimization. With TREX, IBM achieves 0.22 kcal/mol at equilibrium — not just within Kandala's 0.005 Ha error bar, but within chemical accuracy. This is the strongest hardware VQE result in our entire suite. Tuna-9 achieves 3.04 kcal/mol with qubit-aware routing on q[2,4].

Result: 100% pass (5/5 claims). TREX flipped both the equilibrium and chemical accuracy claims from FAIL to PASS. Every claim in this landmark paper now reproduces.

Paper 3: Peruzzo et al. (2014) — The Original VQE Paper

Nature Communications 5, 4213 — arXiv:1304.3061

The paper that started it all: the first variational quantum eigensolver, demonstrated on HeH+ using a photonic processor. We replicated the full potential energy curve (11 bond distances) using PennyLane's 4-qubit Jordan-Wigner encoding with DoubleExcitation ansatz. The emulator matches FCI within 0.00012 Ha MAE. IBM Torino with TREX achieves 4.31–7.26 kcal/mol across 3 distances — a 16x improvement over SamplerV2+post-selection (83.5 kcal/mol MAE), but still 20x worse than H2 TREX (0.22 kcal/mol). Tuna-9 with REM+PS achieves 4.44 kcal/mol at R=0.75Å. Cross-platform agreement is striking: IBM TREX 4.45 kcal/mol vs Tuna-9 REM+PS 4.44 kcal/mol.

Result: 78% pass (7/9 claims). The HeH+ Hamiltonian has a coefficient amplification ratio |g1|/|g4| = 7.8 (vs 4.4 for H2), which fundamentally limits NISQ accuracy. This ratio predicts hardware error: 1.8x larger ratio → 20x worse energy. Chemical accuracy threshold appears to be ratio < ~5. Symmetry verification provides 2.3–7.9x improvement across backends.

Paper 4: Cross et al. (2019) — Quantum Volume

Phys. Rev. A 100, 032328 — arXiv:1811.12926

The paper that defined the Quantum Volume benchmark. We tested the QV protocol on all four backends. This is our most successful cross-backend replication: QV=8 on the emulator and Tuna-9, QV=32 on IBM Torino and IQM Garnet. Randomized benchmarking on Tuna-9 confirmed 99.82% single-qubit gate fidelity.

Result: 100% pass (3/3 claims). Characterization protocols transfer cleanly across platforms — the QV definition is hardware-agnostic by design.

Paper 5: Harrigan et al. (2021) — QAOA MaxCut

Nature Physics 17, 332 — arXiv:2004.04197

Google's QAOA paper on 3-23 qubit graph problems using Sycamore. We replicated small instances: 3-node and 4-node MaxCut at p=1. On the emulator, all graph types achieve optimal or near-optimal approximation ratios. On Tuna-9, the 4-node path graph achieves a 74.1% approximation ratio with 5x5 parameter sweep — well above the 50% random baseline.

Result: 100% pass (4/4 claims). QAOA's cost function is naturally noise-resilient: even noisy hardware consistently beats random guessing.

The Pattern

Across all five papers and four backends, three patterns are clear:

1. TREX is the biggest lever — for shallow circuits. Qiskit's built-in TREX mitigation (resilience_level=1 in EstimatorV2) delivers 119x error reduction on IBM for VQE — from 26.2 kcal/mol raw to 0.22 kcal/mol, achieving chemical accuracy. But for deep circuits (Kim 2023 kicked Ising at depth 10), TREX achieves only 1.3x. The difference: VQE has 2 CX gates (readout-dominated error) while Ising has hundreds (gate-noise-dominated). Mitigation must match the dominant error source.
2. With the right mitigation, chemistry reproduces. Raw VQE fails on every backend. But TREX pushes IBM VQE past chemical accuracy, and our overall pass rate reaches 90%. QV and QAOA (100% pass) test threshold properties robust to noise, while VQE requires active mitigation. The pattern is clear: quantum computing works, but only with the right error mitigation stack.
3. Each backend has a noise fingerprint. Tuna-9 shows dephasing noise (ZZ correlations preserved, XX/YY degraded). IBM Torino shows depolarizing noise (all correlations degrade equally). IQM Garnet shows the cleanest Bell fidelities (98.1%). Knowing the fingerprint tells you which mitigation to apply.

This is not a criticism of the original papers — they used carefully calibrated, custom hardware. The finding is about reproducibility across platforms: quantum computing results are currently hardware-specific in ways that classical computing results are not. But with the right post-processing, the gap is surprisingly narrow.

Tier 2 Update: Kim et al. 2023 — "Evidence for Utility"

Nature 618, 500-505 — arXiv:2302.11590

The most cited quantum computing paper of 2023 demonstrated 127-qubit Trotterized time evolution on IBM's heavy-hex lattice with PEA (Probabilistic Error Amplification) mitigation. We replicated the core physics on 9 qubits using the Tuna-9 topology with basic ZNE (gate folding + Richardson extrapolation) — first on emulator with simulated noise, then on IBM hardware with a 5-qubit chain, and finally on Tuna-9 hardware using all 9 qubits and 10 edges.

Three claims tested, all pass:

1. M_z decays with depth — Emulator: 0.972 (d=1) to 0.755 (d=10). IBM: 0.92-0.96 at Clifford. Tuna-9 hardware: 0.944 (d=1) → 0.785 (d=3) → 0.585 (d=5). Per-qubit analysis reveals dramatic position-dependent error: q0 (edge qubit) maintains Z=0.951 at d=5 while q8 (chain end) collapses to Z=0.025.
2. ZNE recovers ideal at Clifford — Emulator: 24.5% → 3.0% at d=10. IBM Marrakesh: 3.2% → 1.0%. Tuna-9: d=1 noisy 0.944 → ZNE 0.982 (1.8% error, 3.1x improvement). d=3: 0.785 → ZNE 0.857 (1.5x). At d=3, fold=3 requires 180 CZ gates — hardware decoherence saturates.
3. ZNE improves over unmitigated — Emulator: 14.1x. IBM: 3.1x. Tuna-9: 3.1x (d=1), 1.5x (d=3), mean 2.3x. All three hardware backends show 2-3x improvement from basic ZNE gate folding — well below the emulator's 14.1x because real noise includes dephasing and coherent errors that simple gate folding cannot linearly extrapolate away.

Noise amplification on IBM is clean: P(|00000⟩) drops 92.8% → 84.1% → 70.1% (fold 1,3,5). On Tuna-9, the same pattern holds at larger scale: P(all-zero) drops 79.3% (d=1,f=1) → 56.3% (d=1,f=3) and 42.4% (d=3,f=1) → 18.7% (d=3,f=3). Gate folding amplifies noise monotonically on both platforms.

Note: 9 qubits is exactly classically simulable, so this tests the mitigation method, not quantum advantage. The Tuna-9 result is the first experiment that uses all 9 hardware qubits and all 10 connected edges simultaneously.

IBM Torino: TREX vs ZNE on Deep Circuits

We also ran the full 9-qubit kicked Ising circuit on IBM Torino using TREX (EstimatorV2, resilience_level=1) — the same mitigation that achieved 119x improvement for VQE. The results reveal a critical insight: TREX achieves only 1.3x improvement on deep Ising circuits (raw MAE 0.150 vs TREX MAE 0.113). The depth sweep shows raw M_z decaying from 0.948 (d=1) to 0.730 (d=10), with TREX barely helping (0.948 to 0.797). The theta sweep correctly tracks the phase transition from ordered (M_z=0.83 at θ=0) through chaotic (M_z≈0 at θ=π/4) to antiferromagnetic (M_z=−0.67 at θ=π/2).

This is perhaps our most important methodological finding: error mitigation effectiveness depends on what errors dominate. TREX corrects readout errors only. For shallow VQE circuits (2 CX gates), readout error dominates → 119x improvement. For deep Ising circuits (hundreds of gates at d=10), gate noise dominates → only 1.3x. ZNE gate folding on IBM achieves 3.1x at shallow depth (d=1) but struggles at deeper circuits. The mitigation technique must match the dominant error source.

Result: 100% pass (3/3 claims on emulator). On hardware: noise decay confirmed across all backends. ZNE/TREX improvement varies from 14.1x (emulator) to 3.1x (IBM ZNE) to 2.3x (Tuna-9 ZNE) to 1.3x (IBM TREX). The takeaway: mitigation is not one-size-fits-all.

What's Next

All 6 papers are now replicated across 4 backends with 93% claim pass rate (25/27). The IBM TREX depth-dependent finding opens a new research question: can we predict which mitigation technique will work best for a given circuit before running it? Circuit depth, gate count, and the ratio of readout error to gate error appear to be the key predictors. The full replication dashboard is live at quantuminspire.vercel.app/replications.