🧪 Learning from Experiments

+ PM Archetypes

So your product feature didn't work:

You did the user research, you made a data-driven decision, and... the metrics fell flat.

This is the curse of product development: many features fail. But, that you shipped a feature that failed matters much less than what you do after.

Just because it didn’t work doesn’t mean it was the wrong choice:
· It could have been your execution of the feature.
· It could have been the placement.
· It could have been the performance.
· It could have been an outage you didn't know about.
· It could have been faulty analytics.

The longer you're in product, the more you realize that the world is full of probabilities, not certainties.

Take a day to sit on the results, and then bring your team together to understand:
· What can we learn from this failed launch?
· Should we dive deeper into metrics to learn more?
· Would it help to talk to users to understand their experience?
· What are our next steps?

The process of learning from a failed feature launch can be as valuable as launching a successful feature.

When I was at thredUP, we launched a test to simplify the CTAs on our home page. It fit the best practices. Design liked it. We felt smart.

Then transactions decreased.

This led us to the counter-intuitive conclusion that multiple options can help. We specifically examined the funnel drop-off areas we had the most:
· Signup on the email modal
· Checkout abandonment
· Cart abandonment

And we deployed tests to give users more options to solve the problems they had at each step. We began with the signup modal. If people didn't want to enter an email, would they link with Facebook? Overall signup rates increased.

Would they also want the option to link with Google? Signups increased.

But at this point design was worried: 3 CTAs? But we pushed further. Would they also want the option to link with Amazon? Four CTAs. But the team was learning and excited, so we deployed the test. Again - signup rates increased!

It was a massive success for the email signup modal. We followed that up with tests into checkout and cart abandonment. They also worked.

One failed test drove many future successful tests. The lesson for PMs is to drive hard to the "so what?" of experimental failures. A feature that doesn't work can be as impactful as one that does if the right learnings are extracted.

---

The most common PM archetypes

1. Growth Hacker: This PM is motivated by moving the metrics. They’re experts at shipping experiments that yield statistically significant lift. They love using data and emulating others’ growth hacks. Sometimes, they turn a blind-eye to user needs & design fidelity.

2. Product Innovator: This PM is motivated by shipping useful features. They’re experts at building new, unique products. They love using their product sense to invent new things for users. Sometimes, they turn a blind-eye to experimentation and metrics.

3. People Manager: This PM is motivated by organizing teams. They’re experts at managing all the builders and stakeholders. They love using meetings, email, and chat strategically. Sometimes, they turn a blind-eye to great product thinking.

4. Head in the Clouds: This PM is motivated by developing great strategy and vision. They’re experts at crafting compelling and insightful narratives. They love using research and competitive analysis to drive work. Sometimes, they turn a blind-eye to execution & delivery.

5. Execution Machine: The opposite of 4. This PM is motivated by delivery. They’re experts at working with design & eng to nail the details. They love getting into the nooks and crannies to make it all just right. Sometimes, they turn a blind-eye to strategy and the why.

6. Projects on Projects: This PM is motivated by bringing big hairy features to market. They’re experts at project management, and organizing the broader co. They love using tools and processes. Sometimes, they turn a blind-eye to amorphous unstructured work, like thinking.

7. Framework This: This PM is motivated by doing product “the right way.” They’re experts at the latest product trends on Twitter and LinkedIn. They love using the right framework at the right time. Sometimes, they turn a blind-eye to they unique needs of their context.

8. Culture Contributor: This PM is motivated by being a role model and shepherd for great culture. They’re experts at improving the culture beyond their specific team. They love making use of ERGs & offsites. Sometimes, they turn a blind-eye to their product team focus area.

9. Overwhelmed: This PM archetype is way too common. They’re experts at one of the above, but they constantly don’t feel that way. They feel like they’re drowning at work. Sometimes, they turn a blind-eye to work opportunities as a result.

10. Unicorn: This PM is motivated by being the ultimate PM. They’re experts at everything from execution to strategy, and people. They love using the right framework at the right time. They never turn a blind-eye to anything.

Some of us are more than one, but most PMs have a dominant archetype. Identify yours and use it to your advantage:

• Try not to turn a blind-eye as often.

• Select work that plays to your expertise and motivation.

Paying attention to these things is very high ROI.