Farid Asadi

Growth & Experimentation

Experiment Pyramid: How to Solve Problems That “Actually” Move the Needle

Posted on

Experimentation teams often treat testing like a hammer—where everything looks like a nail and every test is just another whack.

They fall into a cycle of quick fixes—tweaking design, copy, and running surface-level tests.

The problem? While these efforts may offer incremental gains, they often miss the deeper and systemic problems that are blocking sustainable growth.

Here’s how to fix it…

Step 1: Find Your “Silent Growth Killer”

Every product has one—including yours.

The best problems are the ones no one’s talking about because they’re hiding in plain sight. They’re so ingrained in the user experience that teams accept them as “just how things work.”

Take Duolingo’s infamous “streak” feature. On the surface, it seemed like a gamification gimmick. But it solved an unspoken problem: learners didn’t just forget to practice—they felt guilty about breaking a streak. That guilt (and the eventual pride of maintaining it) drove daily engagement better than any discount or push notification ever could.

Silent killers thrive in three areas:

  1. Broken Loops: Loops/features that almost work (e.g., users love your product initially but churn because onboarding feels like filing taxes).
  2. Metric Plateaus: KPIs that flatline/declining no matter how many tests you run (e.g., retention stuck at 20% for 12 months).
  3. Recurring Backlog Items: Issues or certain types of issues that resurface quarterly without giving you any wins (e.g., approaching retention with different methods in each Q).

How to smoke them out:

  • Ask users: “What’s the tiny annoyance you’ve accepted as ‘just how it is’?” (Most won’t mention it unless prompted.)
  • Data dumpster-dive: Look for “flatline metrics” that resist incremental tests.
  • Audit your roadmap: If the same issue appears in 3+ quarterly plans, it’s a silent killer—not a “nice-to-have.”

🚨 If your team says, ‘We’ve always done it this way,’ you’ve found your silent killer.

Step 2: Band-Aid or Surgery? Diagnose the Problem

Quick fixes are like caffeine—great for a sprint, but you’ll crash hard if that’s all you consume. To avoid experiment fatigue, categorize problems into two types:

Surgery (For Enduring Problems)

  • If fixing this would improve 2-5 other metrics, it’s surgery.
  • Examples
    • Slack rebuilding notifications to reduce overload
    • Spotify’s Discover Weekly solved retention with personalization, not UI tweaks.

Band-Aids (One-Time Fixes)

  • Ask: “Will this matter in 6 months?” If not, it’s a Band-Aid.
  • Example
    • A/B testing “Start Free Trial” vs. “Get Started” might lift sign-ups by 2% (though you need lots of users to test), but it won’t fix a broken onboarding flow that confuses users after they sign up where you have 20%-50% growth opportunity.

Do a quick audit: Analyze your last 20 experiments: How many targeted enduring problems? What’s your one-time/enduring ratio?

Step 3: Build Your Experiment Pyramid

Most teams jump straight to the peak (big bets) or get stuck at the base (endless tweaks). The pyramid ensures you’re layering quick wins, validation, and systemic fixes:

1. Base (Band-Aids) — Quick wins to fund credibility.

  • Example: Canva’s simple tool to resize designs for different platforms (e.g., Instagram → LinkedIn) paved the way to for “Magic Switch”, which automates cross-platform design adaptation.
  • Stakeholder Hack: “Run a tiny test first. Then say: ‘This 2% lift took 10 tests. The next 20% lift will take 1 systemic fix.’”
  • Tip: Call it a ‘pilot.’ Stakeholders love that word—it’s code for ‘Low risk, high reward’.

2. Middle (Bridge) — Validate before you escalate.

  • Use Band-Aid results to pressure-test assumptions. Example: If template usage correlates with retention, double down on customization features.
  • Red Flag: If middle-layer metrics are weak, revisit your silent killer hypothesis.
  • No proof of concept from the base? No surgery.

3. Peak (Surgery) — Go big or go home.

  • Example: Duolingo used streak engagement data to justify rebuilding their entire notification system.
  • How to Sell It: Frame it as a “growth debt” payoff—painful now, but unlocks 10× speed later + your metrics from the base step will prove invaluable.

🚀 Tip: Build parallel pyramids for each silent killer

Don’t put all your eggs in one experimentation basket. For enduring issues, create multiple experiment pyramids with distinct strategic arcs.

For example, if your silent killer is low user activation, one pyramid might start with quick simplifications (e.g., cutting onboarding steps) and escalate to dynamic onboarding, while another tests using a few email notifications (e.g., nurturing/retention) before committing to a full lifecycle overhaul.

This forces you to pressure-test opposing theories (simplicity vs. top of the mind/nurturing) for the same problem, surfacing hidden winners.

Teams often anchor to one “right” path—parallel pyramids break tunnel vision and let you pivot faster when metrics stall.

Don’t experiment your way into a hole

  • Prioritize impact over statistical significance: A 0.5% lift in button change doesn’t matter if churn is eating 30% of revenue.
  • Avoid experiment theater: Testing trivial changes to appear “data-driven” while ignoring existential risks.
  • Know when to stop: If Band-Aid tests flatline, your silent killer is either misdiagnosed… or you’re using a toothpick to fix a leaky dam.
  • Build parallel pyramids: It forces you to pressure-test opposing theories for the same problem—no more tunnel vision.

The bottom line: Growth isn’t about running more tests—it’s about running the right tests. Find the silent killers, pick your battles, and build a system that turns insights into enduring wins.

Read On