GrowthBook

GrowthBook

@growthbook.io

GrowthBook is an open-source feature flagging and experimentation platform. www.growthbook.io

160 followers2.3K following167 posts

GrowthBook
@growthbook.io

Ever wonder if your feature actually moved the needle, or if that metric bump was just seasonal variance? Most teams struggle with this because they can't isolate the real impact of their changes. That's where holdouts come in - and why we just shipped them in GrowthBook. What are holdouts?

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
View full thread
GrowthBook
@growthbook.io

We built this because our community of 5,000+ engineers kept asking for better ways to prove impact. One customer told us: "Holdouts helped us discover our 'successful' feature was actually causing a 2% revenue decline after 30 days."

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

The best part? It's all open source, warehouse-native, and transparent - you can audit exactly how we calculate everything. Want to see the implementation details? Check out our deep dive: blog.growthbook.io/holdouts-dee...

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

Ever wonder if your feature actually moved the needle, or if that metric bump was just seasonal variance? Most teams struggle with this because they can't isolate the real impact of their changes. That's where holdouts come in - and why we just shipped them in GrowthBook. What are holdouts?

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

Think of them as your experiment's control group that persists even after launch. While 95% of users get your shiny new feature, 5% stay in the original experience. This gives you a clean baseline to measure against.

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

🪝 Catch delayed effects - Some changes take weeks to show impact ⚖️ Measure cumulative impact - Stack multiple features and still understand individual contribution 🤝 Build trust in your data - Show stakeholders exactly what drove that revenue increase

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

Yesterday's comic was, of course, in reference to Twyman's Law. Applicable not only to experiment analysis, but also life. If it looks too good to be true, then it probably is 😅

GrowthBook
@growthbook.io

Sanity offers a powerful content management solution that effortlessly integrates with GrowthBook, enabling you to conduct experiments directly from your content editor without altering any code. This approach allows you to benefit from feature-flag experiments seamlessly.

youtube.comA/B Testing Made Easy: Sanity + GrowthBook + Next.js TutorialStop hardcoding A/B tests! Learn how to build a professional experimentation system where your content team can create and manage A/B tests directly with Sanity, while GrowthBook handles the…
GrowthBook
@growthbook.io

We've introduced practical AI features in GrowthBook 4.1: 🤖 Auto-summarize experiment results 📊 Generate SQL queries from natural language 💡 Enhance experiment hypotheses with AI 🔍 Automatically find similar past experiments ➕ Customize prompts for your org 🙅 Prefer no AI? Disable it entirely.

GrowthBook
@growthbook.io

Your PM: "What happened with that experiment?" You: Opens 17 tabs, 3 spreadsheets, and a Slack thread Sound familiar? We fixed that. Introducing Experiment Dashboards in GrowthBook 4.1: ✅ Create custom views ✅ Highlight metrics ✅ Add context ✅ Share a single link instead of a dissertation

GrowthBook
@growthbook.io

The issue with A/B tests? They conclude. But what about tracking the combined effect of 50 updates over six months? Enter Holdouts: the perpetual experiment. Keep 5% of users on the initial experience, while others enjoy new features. Gauge the true long-term impact of your product changes.

blog.growthbook.ioHoldouts in GrowthBook: The Gold Standard for Measuring Cumulative ImpactMany teams ship features weekly but struggle to measure their true cumulative impact. Holdouts in GrowthBook provide a simple way to maintain a control group across multiple features, answering the…
GrowthBook
@growthbook.io

Most holdouts measure only shipped features. Ours measure everything—including failed experiments. Because that's what your users actually experienced. Technical breakdown → blog.growthbook.io/holdouts-dee...

blog.growthbook.ioHow GrowthBook Holdouts Work Under the HoodMost holdouts measure only shipped features. Ours measure everything—including failed experiments. This technical deep dive reveals why we chose reality over clean rooms, and how we built it.
GrowthBook
@growthbook.io

Experiments showed +5%, +3%, +7% wins, but do they really add up to +15%? Likely not, as Airbnb found naive sums can overstate impact by 2x. GrowthBook now offers ✨ Holdouts ✨ for accurate metric measurement:

blog.growthbook.ioHoldouts in GrowthBook: The Gold Standard for Measuring Cumulative ImpactMany teams ship features weekly but struggle to measure their true cumulative impact. Holdouts in GrowthBook provide a simple way to maintain a control group across multiple features, answering the…
GrowthBook
@growthbook.io

Another big week for the experimentation industry: OpenAI has acquired Statsig for 1.1B. Graham shares his thoughts on the acquisition and our new Statsig to GrowthBook migration kit: www.linkedin.com/posts/graham...

linkedin.comAnnouncing the Statsig to GrowthBook Migration KitFor those customers who want more certainty and innovation from their experimentation platform, I’m happy to announce the Statsig to GrowthBook Migration Kit. This kit allows customers to migrate from Statsig to GrowthBook in minutes. Customers will be able to import all projects, environments, and feature flags. Imports will include fallback values, targeting rules, rollouts, and prerequisite features.
GrowthBook
@growthbook.io

It's possible to build beautiful sites with Framer. But how do you know if they actually convert? You don't unless you test! Our official plugin for Framer lets you do enterprise-level experimentation on your components. Our tutorial helps you get set up in minutes. youtu.be/XqHDLbYCfTM?...

youtu.beAdd Pro A/B Testing to Framer in Minutes (GrowthBook Plugin)Stop guessing what converts. Start knowing. In this tutorial, I show you how to add enterprise-level A/B testing to your #framer site in minutes using the GrowthBook plugin. No coding required,…
GrowthBook
@growthbook.io

It's a hard truth—but we're not all SQL geniuses. Get a little (or lot of help) with your next query by using our new natural language to SQL generation feature. Ask, for example, to list the top 10 pages by view count in the past 3 months and get back a crisp SELECT statement. See Graham's demo

GrowthBook
@growthbook.io

Teams often struggle with an ever-growing list of metrics, ballooning from 20 to 200. Finding a specific metric during a new experiment can be tough, leading to even more creation ☠️. AI can automatically summarize metrics, making them easier to find and preventing duplicates. Watch Graham's demo 👇

GrowthBook
@growthbook.io

Hypothesis enforcement just dropped ✨ No, you won't get a ticket for sloppy hypotheses—but this AI-powered feature ensures your team writes them right. YOU set the quality bar. Academic rigor or lighter touch, your choice. Don't just run experiments, run them right.

GrowthBook
@growthbook.io

Automatic experiment result summary generation is here 🪄 Get a summary of your experiment results instantly, including key findings based on your metrics and variants, that's totally customizable to your organization's style and requirements.

GrowthBook
@growthbook.io

Here's the tutorial you've been looking for: #Sanity + #Next.js + GrowthBook Ryan provides a comprehensive walkthrough of everything you need to know to set up an enterprise-grade experimentation system that everyone on the team can use! youtu.be/tg85K9K_8-g

youtu.beA/B Testing Made Easy: Sanity + GrowthBook + Next.js TutorialStop hardcoding A/B tests! Learn how to build a professional experimentation system where your content team can create and manage A/B tests directly with Sanity, while GrowthBook handles the…
GrowthBook
@growthbook.io

Graham and Ryan demo all the new AI-powered features recently released in GrowthBook. The best part? Every AI prompt is fully customizable. Want results in haikus? Different language? Your company's tone of voice? You control it all. youtu.be/jhyGxknbGk0

youtu.beIntroducing GrowthBook AI: 6 New Features for Smarter ExperimentationDiscover how GrowthBook is revolutionizing A/B testing and experimentation with powerful AI features! In this demo, GrowthBook CEO Graham and Developer Advocate Ryan showcase the latest AI-powered…
GrowthBook
@growthbook.io

The velocity of change is changing. 🚗 Ford Model T → 1 million sold: 13 years 🔎 Google → 1 million searches/day: 1 year 📱 iPhone → 1 million units: 74 days 💬 ChatGPT → 1 billion searches/day: <1 year This isn't just acceleration—it's a fundamental shift in how quickly markets transform.

GrowthBook
@growthbook.io

The AI Revolution Isn't Our First Rodeo 🎢 When electricity arrived in the 1880s, most factories just swapped steam engines for electric motors. Same layout. Same processes. Electric lightbulbs where gas lamps used to be. Then Henry Ford reimagined everything.

blog.growthbook.ioBuilding in the AI Era: Lessons from Past Technological RevolutionsWhy bolting AI onto existing products is like adding lightbulbs to steam factories. Lessons from Ford to ChatGPT on truly transformative tech adoption.
GrowthBook
@growthbook.io

He didn't just electrify the old system—he invented the assembly line. Build time for a Model T dropped from 12 hours to 93 minutes. Today, companies adding AI chatbots like those factories adding lightbulbs. It's helpful, sure. But it's not transformative.

blog.growthbook.ioBuilding in the AI Era: Lessons from Past Technological RevolutionsWhy bolting AI onto existing products is like adding lightbulbs to steam factories. Lessons from Ford to ChatGPT on truly transformative tech adoption.
GrowthBook
@growthbook.io

Cut feature flag latency to practically zero when you integrate GrowthBook with @Vercel Edge Config. Here's why it's a game changer for technical teams:

View full thread
GrowthBook
@growthbook.io

Cut feature flag latency to practically zero when you integrate GrowthBook with @Vercel Edge Config. Here's why it's a game changer for technical teams:

GrowthBook
@growthbook.io

→ Built for devs: Works seamlessly with @vercel/flags SDK and Vercel Toolbar → Advanced statistics: Bayesian, Frequentist, CUPED, Sequential testing built-in

GrowthBook
@growthbook.io

→ Precise targeting: User attributes, location, device type, custom rules → Full visibility: Real-time feature flag analytics and debugging tools Unlike other platforms that treat you like a black box, GrowthBook gives you complete control over your experimentation stack.

GrowthBook
@growthbook.io

Like an erstwhile PSA: "Do you know where your feature flags are?" As part of our 4.0 launch, we shipped feature flag usage analytics, which means that for every feature flag, you now know: 🛎️ How often it's been evaluated 💁 Which values are being served 📏 Which rules are being hit 🌈 And more

GrowthBook
@growthbook.io

These analytics make it infinitely easier to know where your feature flags are, what they're doing, and if they're working right. As an added bonus, flags without usage are likely stale and should probably be tossed.

GrowthBook
@growthbook.io

Currently, feature flag usage is limited to our (also) new managed warehouse. Let us know if you're interested in trying it out or extending support to your data warehouse.

GrowthBook
@growthbook.io

GrowthBook + #MCP gets your A/B test into production in ~60 seconds, all with fewer characters than in this post. See the proof yourself in the video.