When Failure is the Best Option
The Surprising Power of Scientific Mistakes in Driving Innovation and Discovery
Rethinking Failure
We live in a society that often treats failure as anathema—something to be avoided, hidden, or ashamed of. From report cards to performance reviews, the message is clear: success is rewarded, failure is sanctioned. But what if we've been thinking about failure all wrong? What if, in the specific context of scientific progress, failure isn't just acceptable but essential?
"Failure is a necessary and frequent feature of scientific research" 1 .
The most innovative organizations and researchers are embracing a counterintuitive truth: Failure is an option—and often the most valuable one. This isn't failure due to carelessness or incompetence, but rather the kind of rigorous, well-designed failure that eliminates dead ends, reveals hidden assumptions, and opens unexpected pathways.
Thomas Edison
Thousands of unsuccessful attempts before creating the practical light bulb
Drug Discovery
Majority of compounds fail, leading to unexpected discoveries
Innovation Rate
Organizations that embrace failure innovate 30% faster
The Necessity of Failure: Why Science Thrives on Mistakes
The Inevitability of Failure in Discovery
Failure is baked into the very fabric of scientific progress for a simple reason: exploring the unknown means venturing beyond the boundaries of what's already known. In this territory, missteps are inevitable.
Thomas Edison famously reframed his thousands of unsuccessful attempts not as failures but as discoveries of "several thousand things that won't work" 6 . This mindset isn't just positive thinking—it's an accurate description of the discovery process.
Institutional Embrace of Failure
Forward-thinking institutions are now actively working to destigmatize failure through various initiatives:
Corporate Approaches
- Corporate "fail fast" philosophies that encourage rapid iteration and testing of ideas 4
- Post-mortem analyses that systematically examine what went wrong in projects
These programs recognize that creating transparency around failure is crucial for organizational learning. The alternative—a culture where failures are hidden—wastes resources as different researchers make the same mistakes.
A Case Study in Productive Failure: The Drug Development Bottleneck
The Phase II Failure Crisis
Perhaps nowhere is the productive power of failure more evident than in pharmaceutical development, where the statistics are staggering: for every drug that reaches clinical use, more than 6,000 completely new chemical compounds are synthesized and tested 3 . On average, about 20 drug candidates are tested in humans for every one that makes it to market 3 .
Drug Development Success Rates
Conventional wisdom might suggest that most drugs fail due to safety concerns, but the reality is more surprising: the majority of drugs that never reach market fail in Phase II clinical trials due to lack of efficacy—they simply don't work well enough for the disease they were intended to treat 3 .
The Hidden Opportunity in Failed Drugs
Here's where the story gets interesting: these Phase II failures represent an enormous, untapped resource for biomedical science. These compounds have already passed Phase I safety testing, meaning they've been shown to be safe in humans 3 . They failed not because they were dangerous, but because they didn't effectively treat their target condition.
Link Medicine
Testing a failed cancer drug as a potential Alzheimer's treatment 3
Amicus Therapeutics
Investigating a failed Gaucher disease drug for Parkinson's disease 3
Drug Repurposing Opportunities
| Original Target Disease | New Potential Application | Rationale for Connection |
|---|---|---|
| Cancer | Alzheimer's Disease | Cancer involves abnormal cell survival; Alzheimer's involves failure of survival pathways |
| Gaucher Disease | Parkinson's Disease | Both involve lysosomal damage and alpha-synuclein protein aggregates |
| Psoriasis | Heart Disease | Patients with severe psoriasis have comparable heart disease risk to diabetics |
Analyzing the Deeper Problem
The high failure rate in Phase II trials points to a fundamental challenge: our animal and cell-culture models for many diseases are inadequate. The compounds showed promise in preliminary models but failed in human diseases. This suggests that improving disease models could help fail compounds earlier in the development process when costs are lower 3 .
More importantly, the traditional siloed approach to drug development—where compounds are tested for single diseases—may be limiting our thinking. Many diseases share common pathways and cellular processes, suggesting that a failed compound for one condition might be effective for another seemingly unrelated condition 3 .
The Scientist's Toolkit: Strategies for Failing Productively
Designing for Disproof
The hallmark of good science is falsifiability—articulating clear hypotheses and being willing to pivot when evidence contradicts them 4 . Researchers at companies like Two Sigma implement this through rigorous testing protocols: "We typically have an in-sample of data that we're studying, and then an out-sample of data that we've hidden from ourselves" 4 . This approach ensures that models work in the real world, not just on known data.
The Failure Spectrum: Understanding Why Things Fail
Not all failures are created equal. Understanding the nature of a failure is crucial to learning from it. Failures generally fall into five categories 2 :
Incompetence
Cause: Insufficient skill or training
Mitigation: Improve training, practice fundamental skills
Ignorance
Cause: Lack of key information
Mitigation: Research, consult experts, cross-train
Interference
Cause: External interruptions
Mitigation: Create contingency plans, minimize distractions
Malicious Action
Cause: Intentional sabotage
Mitigation: Implement security, document thoroughly
Chance
Cause: Uncontrollable external factors
Mitigation: Build resilience, diversify approaches
Practical Framework for Productive Failure
1. Reframe experiments
Treat assumptions as hypotheses to be tested rather than truths to be confirmed . Use the template: "We believe that [solution] will result in [outcome]. We will know we have succeeded when [evidence]" .
2. Seek disconfirming evidence
Actively try to falsify your hypotheses, not just confirm them. As philosopher Karl Popper noted, we tend to find what we look for, so we must intentionally look for what might prove us wrong .
3. Create psychological safety
Build environments where team members can openly discuss failures without fear. Research shows that "psychologically safe workplaces reduce perceived personal risk from failure" 9 .
4. Conduct failure analysis
When failures occur, depersonalize them and focus on systemic causes. Ask: "What was the reason for the failure? What was in your control or out of your control? What could be done differently next time?" 9 .
5. Share failure stories
Normalize failure through events and discussions that highlight productive failures. Studies show that "failure disclosure improved students' sense of belonging" in STEM fields 1 .
Conclusion: Failing Forward
The most innovative scientists and organizations understand that progress doesn't occur in a straight line from question to answer. It winds through a landscape of productive failures that gradually illuminate the path forward.
"Failure is not something to tolerate while focusing on the bright side. Failure is not a temporary condition. It must be embraced and worked on with all the diligence that one is accustomed to putting into succeeding" 5 .
This isn't about celebrating failure for its own sake, but about recognizing that well-designed, informative failures provide the essential data needed for genuine breakthroughs. In the words of Samuel Beckett, we must "Try again. Fail again. Fail better" 5 .
The Challenge for Science
The challenge for scientists—and for all of us who value progress—is to create systems and cultures that don't just tolerate failure but actively mine it for insights. Because when it comes to discovery, failure isn't just an option—it's often the best one we have.