Only 1 in 7 A/B tests produces a statistically significant winner — the other six burn 4 weeks of traffic and ship nothing. The Pre-Launch Prediction Stack is a 4-layer framework that forecasts the winner before you spend any traffic, surfaces segment-level risks early, and tells you which tests to redesign instead of run.
This isn't theory. Each layer maps to production-grade tools that exist today.
TL;DR
- The framework: The Pre-Launch Prediction Stack — 4 prediction signals applied in sequence before any test goes live
- Layer 1: Historical Pattern Scoring — Score the test against past wins to set a baseline win probability
- Layer 2: Attention Simulation — Predict whether users will even see the variable you changed
- Layer 3: Synthetic Persona Simulation — Run AI agents through both variants to forecast directional outcome and segment divergence
- Layer 4: Statistical Power Planning — Verify the test design can actually detect the lift you're predicting
- Decision rule: Ship only tests that pass ≥3 of 4 layers
- Result: Higher win rate per test, fewer wasted runs, and segment-level insight before launch — not after
Why You Need a Framework for A/B Test Prediction
Most CRO teams treat A/B tests like a lottery: ship a variant, wait four weeks, hope. But 50–80% of A/B tests come back inconclusive, and the few "winners" that do appear often hide segment-level damage — a variant that wins overall while quietly killing conversions for new mobile users.
The cost compounds. Every inconclusive test is a month of engineering capacity, traffic budget, and decision velocity wasted. Across a year of 24 tests, that's 16+ runs that produce nothing actionable.
A systematic prediction stack changes the math. Conversion.com's ML model — trained on 20,000+ experiments — found that high-confidence-scored tests win 63% of the time vs. an industry baseline of 8–30%. The difference isn't better hypotheses. It's better filtering.
The Pre-Launch Prediction Stack codifies that filtering process so any team can apply it.
The Pre-Launch Prediction Stack: Overview
The framework sequences four prediction signals from cheapest to most diagnostic. Each layer answers one specific question and produces a go/no-go signal. You don't need all four to be perfect — you need three to converge.
- Historical Pattern Scoring — Has a test like this won before?
- Attention Simulation — Will users actually see the change?
- Synthetic Persona Simulation — How do different user types respond?
- Statistical Power Planning — Can the test design detect the predicted effect?
Layer 1: Historical Pattern Scoring
What You Do
Score the proposed test against your historical experiment library before writing the spec. Tag every past test by page type, change magnitude, primary metric, device split, and outcome. When a new hypothesis lands, find the 5–10 most similar past tests and compute the empirical win rate.
If similar tests historically won 12% of the time, your prior is 12% — not 50%. Tests scoring below your team's threshold (typically 25%) get killed or redesigned, not run.
Real Example
Stripe's growth team — based on their public engineering posts — categorizes pricing-page tests separately from checkout-flow tests because the historical win rates diverge by 3x. A "move CTA above fold" test on a pricing page and the same test on a checkout page get scored against entirely different priors.
The AI Shortcut
Autonomous CRO agents ingest your historical experiment library and produce a confidence score in seconds, including the directional reasoning ("similar tests on mobile-first pricing pages historically lift CVR 4–7%"). Conversion.com's data shows ML-scored confidence tiers separate winners from losers with 2–3x more accuracy than human prioritization.
Layer 2: Attention Simulation
What You Do
Run the variant through an AI attention model before ship. The model predicts where the user's eyes go in the first 3 seconds. If your variable element — new CTA, repositioned headline, social proof block — doesn't enter the top 3 attention zones, the test is invalid by design.
You're not testing the variant; you're testing a variant the user never saw.
Real Example
Brainsight's attention model — trained on real eye-tracking data — predicts visual focus with 94% accuracy. Notion's signup page (publicly observable) places its primary CTA in the second attention zone on desktop but the fourth zone on mobile. A mobile test that changes the CTA copy without addressing position is predictably inconclusive.
| Test Element Position | Avg Attention Capture | Expected Test Power |
|---|---|---|
| Top 3 attention zones | 60–80% | High |
| Zones 4–6 | 25–40% | Medium |
| Below the fold | <15% | Low — redesign |
The AI Shortcut
AI attention scoring runs in under a minute against any uploaded screenshot. Tools like Brainsight and built-in attention models in modern CRO platforms make this a non-optional step. No live traffic required, no external panel, no waiting.
Layer 3: Synthetic Persona Simulation
What You Do
This is the highest-signal layer. Generate 20–100 LLM-driven personas spanning your real user mix — varying intent, familiarity, device, and price sensitivity. Have each persona browse both control and variant. Capture two outputs:
- Directional agreement: Does the variant outperform control across the persona pool?
- Segment divergence: Do mobile-first personas diverge from desktop personas? Do high-intent personas diverge from browsers?
If desktop personas show a 12% lift and mobile personas show a 4% drop, you've just discovered — for free — that your live test needs to be segmented by device from day one.
Real Example
The April 2025 AgentA/B research paper from Stanford generated 100,000 virtual customer personas and showed their behavioral differences across variants mirrored real human A/B test directionality. SimGym deploys 100–2,000 Chromium-driven AI shoppers against Shopify themes and reports 85% correlation with real shopper behavior — with results in 4–10 minutes vs. 2–4 weeks live.
The AI Shortcut
This is where platforms like Relaunch.ai compress the work. Pre-launch simulation runs synthetic persona testing with segment-level breakdown in a single step — surfacing the segment risk that historically only shows up after 4 weeks of live traffic. This is the layer that turns an inconclusive test into a redesigned, segmented experiment before you spend a dollar.
Relaunch's pre-launch simulation runs your next A/B test through synthetic personas across every segment — surfacing winners and divergence before you spend a dollar of live traffic.
Simulate your next A/B test before launch →Layer 4: Statistical Power Planning
What You Do
Once Layers 1–3 say go, verify the test design can actually detect the predicted effect. Run a power analysis with honest inputs: real baseline conversion rate, expected lift from Layer 3 (not the inflated number from your hypothesis doc), and realistic available traffic.
If the math says you need 14 weeks to detect a 2% lift but you've budgeted 3 weeks, the test will go inconclusive no matter how good the variant is. Either expand traffic, raise the MDE, or kill it.
Real Example
A Towards Data Science Monte Carlo study showed that including the right covariates in your test analysis improves precision more than doubling sample size. A pricing page test that doesn't account for traffic source as a covariate will read as inconclusive when the truth is "this variant wins for organic, loses for paid."
Power planning isn't a frequentist nicety. It's the gate that prevents you from running 8-week tests that were doomed by sample size on day one.
The AI Shortcut
Modern experiment platforms compute MDE, runtime, and required traffic automatically once you wire in your historical baselines. The shortcut isn't faster math — it's the platform refusing to launch tests that fail power requirements.
Putting It Into Practice: Your First 2 Weeks
Week 1: - [ ] Audit your last 20 experiments. Tag each by page type, change magnitude, device, and outcome - [ ] Build the lookup: when a new test is proposed, what's the empirical win rate of similar past tests? - [ ] Run an attention scan on your 3 highest-traffic pages — note which conversion elements live outside the top 3 zones
Week 2: - [ ] Pick your next planned A/B test. Run it through all 4 layers before launch - [ ] Compare the simulation predictions to your team's gut prediction. Document the divergence - [ ] Set the team rule: tests must pass ≥3 of 4 layers before they enter the queue
Implementation Checklist
- [ ] Historical experiment library tagged with at least 5 dimensions per test
- [ ] Win-rate baseline computed per test category (pricing, checkout, hero, onboarding)
- [ ] Attention simulation tool integrated into the design review step
- [ ] Persona simulation runs on every test before spec sign-off
- [ ] Segment divergence threshold defined (e.g., flag if any segment >10% from mean)
- [ ] Power analysis required before launch — not after
- [ ] Decision rule documented: ≥3 of 4 layers must pass
- [ ] Failed tests routed to redesign queue, not the trash bin
- [ ] Monthly review: simulation predictions vs. live test outcomes
- [ ] Post-test calibration: update Layer 1 priors with each new result
Frequently Asked Questions
How long does it take to implement the Pre-Launch Prediction Stack?
A first version runs in 2 weeks if you have at least 20 historical experiments to tag. Without that history, start collecting baseline data now and lean harder on Layers 2–4 in the meantime — Layer 1 strengthens with each test.
What tools do I need for this framework?
Layer 1 needs a tagged experiment library — a spreadsheet works to start. Layer 2 needs an AI attention model (Brainsight, Attention Insight). Layer 3 needs a synthetic persona platform (SimGym for ecommerce, Relaunch.ai for full-funnel). Layer 4 needs any standard power calculator wired to your real baselines.
Can a small team (1-2 people) use this framework?
Yes — and it's where the framework pays off most. A small team can't afford 16 inconclusive tests a year. Skip Layer 1 in month one if you have no history, and start with Layers 2 and 3, which require no historical data.
What results should I expect in the first month?
Expect to kill 30–50% of planned tests in the first cycle — those were going inconclusive anyway. Of the tests that ship, expect win rates to climb from the 8–30% industry baseline toward the 40–60% range as your team learns to trust the prediction signals.
How does this framework change if I use AI optimization tools?
Autonomous CRO agents collapse Layers 1–3 into a single pre-launch check, surface segment risks automatically, and update Layer 1 priors with every test outcome. The framework structure stays the same — the cycle time drops from weeks to hours.
Does pre-launch simulation replace live A/B testing?
No. Simulation is a filter that improves the quality of tests that go live. The goal isn't to skip live testing — it's to stop running tests that were never going to produce a clear answer in the first place.