20.6 Ship, Iterate, or Kill 🤙
This section includes a reminder to have a mentor call 🤙
You've now heard a few times about iterative design, but just to reiterate—it's the idea that you can ship your product in a minimal state (MVP) and add to it over time.
Iteration provides a way for you to make future versions better based on the feedback that you get from your users, business metrics, and stakeholders. You can think of it in terms of the lean startup cycle—build, measure, learn—a cycle that goes on forever.
However, you can't keep building any individual feature forever. At a certain point in time, you need to make a decision on whether or not a specific feature is done. But how do you know when you're done? And what are your options for stopping or continuing development? In this checkpoint, you will learn about these decisions, which product managers have to make often.
By the end of this checkpoint, you should have learned how to do the following:
- How to make post-launch decisions to ship, iterate, or kill a feature
Why ship, iterate, kill?
So why is this checkpoint called ship, iterate, or kill? Because when a product manager is evaluating the results of a completed feature, MVP, or other product change, they have one of three choices:
- Ship it. It works. Send it into the world, and move onto the next thing.
- Iterate on it. There's still room for meaningful improvement, so try again.
- Kill it. It didn't work, and you're out of time or options. Give up and move on.
This is one of the hardest decisions you'll have to make as a product manager. There's always room to doubt yourself. Maybe if you add the feature, it could mean the difference between a good and great release. Maybe your testing wasn't good enough, so you're uncertain about whether it's really done. Maybe your team is just tired and wants to move on to the next thing.
How will you know which of the three options is the best choice? There is no guidebook. However, there are some ways to think about iterative design that others have found useful. You can read more about those below.
Kill it
Imagine that you and your team have been working hard to create a new shopping cart page. The cart was the biggest drop-off point in your conversion funnel, so it was an easy target for redesigning. Your team spent weeks on it: coming up with a new design, programming, and testing the new version. You've just gotten the A/B test results back, and…
Nothing. There was no change in conversions. Your new cart page performed exactly as well as the old one. What should you do?
In most cases, this is a good time to just kill the project. You've invested a lot of time and effort and did everything right, but it just didn't achieve the intended results. This is especially true if you've tried several times to hit that big goal (to improve cart conversions) but failed.
Why did this happen? It's likely that you've identified the wrong user problem, or that the problem is too small to have an impact on overall performance. Before trying again, you should go back a few steps to see what you might have missed. More data analysis, usability tests, or user interviews might be necessary before you can decide what's next.
Here are a few telling signs that could mean you should give up on a project:
- Too much effort. It has either taken much longer than you originally estimated, or it will take too much time to hit the next milestone.
- Changed priorities. New priorities have come up that are more urgent or represent better opportunities for improvement.
- Changed patterns. The part of your product you're trying to improve has become less important due to changes like reduced user visits.
- Bad feedback. Users have had negative reactions to your tests to date, meaning that you're probably wasting time on the project.
Ship versus iterate
Suppose that the same shopping cart feature was delivering slightly different results. You see that your KPIs on the shopping cart page have been improving, with higher conversion rates and more revenue. You and your team are feeling good about the changes you made. But are you done? Should you just ship it and move on? Or should you keep trying to improve it even further?
Here are the questions you should ask yourself to help choose one over the other:
Did you hit the goals you initially set?
If you're following the best practices that this program teaches, you would have already decided on a few KPIs and goals when you first wrote user stories for this feature. Compare how the feature is doing in testing against the goals you set out to achieve. Even if you didn't hit 100% of your goals, it still can be worth shipping because a small improvement is better than doing nothing.
What does the next iteration look like?
Now that you have a minimal version of this improvement, the next step in your cart page iteration is really big—a one-click checkout and pay button. This next improvement will take a couple of months of effort to complete, and it's not clear how much more it will improve your KPIs. In this case, it's better to ship the improvements you have ready right now, and make a separate call about whether you want to keep iterating later. This will allow you to start enjoying the positive results of what you already have sooner, and you'll be able to make the decision about the next step with a lot more data about how the new version is doing. In many cases, your changes will turn out to be good enough and won't warrant additional iterating.
How reversible is the decision?
Sometimes the data you see is ambiguous; for example, there might be a small improvement, but it was not statistically significant. Is it worth launching? That depends on how difficult the decision would be to reverse. Could implementing the new design be reversed with a single line of code? If so, sure—ship it. If you're not happy with the results later on, it would be easy to switch back to the old version. However, some changes are not easily reversible. You should give those much more consideration before moving forward.
How will it affect your team's morale?
If you've been working on a feature for a long time, it could be worth shipping it to users just to avoid crushing your team's morale. There's no worse feeling for a developer than spending days or weeks of work on a project just to end up throwing it away and not have anyone use it. As long as there are no negative consequences to shipping it, do your team a favor and give them a win.
If there are compelling reasons not to ship the feature, try to find some creative ways to make the work the team has already invested meaningful; for example, by using some of the processes or technology they developed on another project going forward. Research in behavioral economics suggests that most employees come to work to find meaning, not just money. If handled poorly, a canceled project can lead to disappointed and unmotivated employees (you can read more about this in behavioral economist Dan Ariely's post about a canceled software project).
Best practices for product decisions
When you're making these types of product decisions, there are a few additional things to keep in mind. Regardless of the decision you make, the following best practices will help.
Record your decisions
Always record why you made a specific decision. Your decision process should be documented with enough detail so that anyone who reads it would understand why you made the decision you made. You'll be making decisions daily and they pile up quickly. Even if it seems impossible now, in the near future you might find it difficult to remember exactly why you made this specific decision. Jot down some information—what choice you made, why you made it, and any supporting evidence or data you used to make it. A good place to keep this information is with the story or epic that's tracking the feature.
Validate with users
It can be easy to mess up the rollout of a new feature. Many features and products have been launched that caused massive backlash because they were unfinished, buggy, or undertested. For example, Skype was forced to roll back its 2017 redesign after users' angry reaction on social media, Google Talk shipped with a bug that would send messages to the wrong people, Facebook users had a negative reaction to the News Feed when it shipped, and Instagram was hated when they moved from chronological sorting to algorithmic sorting. Don't add your product to this list of failures. Make sure to collect enough user feedback, and validate that it points in the right direction for your metrics and KPIs.
Don't "do nothing"
You might wonder why "do nothing" is missing from this list of options. Doing nothing in this context means defaulting to whatever your last choice was—so if it's already live, you've chosen to leave it live. The problem with doing nothing is that you're the product manager. You always need to have an opinion on what's next. If you’re doing nothing because the product is in great shape, then you've shipped it. If you're doing nothing with it because you're evaluating options for more improvements, then you're iterating. In the worst case, choosing to do nothing means that your boss will make the decision for you, and that means you're not doing your job. So do something, and have a point of view about what's best for your product.
Whether you decide to ship, iterate, or kill, make sure you learn from the experience. It won't be long before you need to make that kind of choice again. Learn from every experience, and you'll be more effective the next time around.
Practice ✍️
Imagine that your team made the following changes and ran them as an A/B test:
- You redesigned the New York Metropolitan Museum of Art's store home page
- You updated the way people enter comments on Reddit.com stories
- You developed a new system for people to rate places on TripAdvisor
How will you decide what to do next—ship, iterate, or kill the feature? What are the KPIs or goals that you would use to measure the results of the A/B test? What is important in each scenario to help you make the decision?
Write down answers in your notebook/Notion page.
Mentor Call 7 🤙
Time for your seventh mentor call. Although you could get mentored from any free platform online, we recommend ADPList as it's free and boasts a huge community of product experts. You might have to signup to it's platform (which is again free) and schedule a mentor call from the ADPList platform itself.
Schedule a mentor call with an ADPList product mentor. ADPList provides free mentorships for everyone and is an amazing platform to learn and meet experts. Please ensure you take mentor calls with importance. Mentors play a crucial role in your career. Do not ask mentors to review assignments. They are only meant to guide you and give you tips on your growth. These are optional calls as well.
Let the mentors know beforehand what the call is about. Share you questions and let them know you are a bootcamp student. Again refrain from asking assignment related questions. They aren't meant to review your assignments.
Below are the list of mentors that we have collaborated with at PMcademy.
Looking for more mentors ?
You can choose to get mentored from any other mentor as well. Visit this link to find more product mentors from ADPList. However note that not everyone is aware of PMcademy, it that case let them know beforehand about PMcademy.com and about your meeting agenda.