13.6 Testing 🎯

note

This section includes an Activity 🎯

Think of the last time you encountered a bug or error in a product. Maybe it was a web page that didn't load. Or a mobile app that crashed. Or maybe you encountered a bad design choice that rendered an app unusable. How did you feel after it happened? Did you go back to that product? What was your perception of the company that provided the service?

Learning objective

Validating that features will function as expected is an important step in the development process. On top of making sure everything works as it should (and on multiple devices), you should also make sure that the software knows what to do in an error state. For instance, what happens if a user enters letters in a form field that is set up to only accept numbers? How is it recorded if someone enters the last name "de L'Orange"? What happens if a user repeatedly clicks Buy on the checkout screen? You'll need to anticipate all these different situations and ensure that the product is ready to handle them.

In this checkpoint, you will learn about the different methods used to test modern tech products, how to define and run tests, and how to approach problems you’ll encounter when testing.

By the end of this checkpoint, you should be able to do the following:

• Explain key testing-related terms, like acceptance criteria and QA
• Write a test plan for a digital product

What is QA?

Quality assurance (QA) is the process of testing and validating products to ensure that they work according to their specifications (or stories). It's also the process of testing those features to identify and address bugs.

Bugs are inevitable. Humans are fallible, so the code your developers write will have bugs. Features won't work the way you expect them to work. Something that works correctly on an iPhone app won't work on an Android app. You may have even validated the functionality of a feature before it launched only to see an error occur once it's deployed to all of your users in production.

Why does this matter? Users running into a bug will be irritated. A simple irritation could cause a user to lose confidence in your product—and you could potentially lose them as a user altogether. This could also prompt negative reviews to potential users.

As a product manager, you are responsible for ensuring that the features you launch work and for thoroughly testing that the product overall is reliable. You'll now learn about how a product manager addresses this responsibility and the methods for testing products.

How does QA happen?

QA is an ongoing process during development. It is done at various points by different team members.

Developers will QA their own features as they're developing them. They'll keep testing their work until they're confident it's ready for other people to review it. Some development teams adopt a test and code approach to creating software; they write tests alongside the features as they develop. These tests aren't meant to test every possible input; rather, they test to confirm the feature functions as expected.

When their work is ready for additional feedback, your developers will ask you and your designers to test their work. A good approach is to first validate that the new feature is functioning properly. Does it fulfill specifications in the user stories? This is also a good opportunity to share the feature with other stakeholders and get their feedback.

After confirming basic functionality, it's also a good practice to poke around and seek out problems. Recall areas of concern during the design stage and test those areas. Intentionally try to trigger bugs and confirm how the feature handles bad input. Test for usability and performance issues. While the product manager is the best expert on their product and its vulnerabilities, they are not solely responsible for testing. Executing all the tests needed is a full-time job.

Once the product manager has validated the functionality and done some initial tests, the next steps in the QA process can vary. Some companies have a dedicated QA team that will exhaustively test the changes across devices, browsers, and operating systems to ensure they work correctly. They'll test the entire application with the new feature on many different computers and mobile devices to catch any possible problems that might happen on the variety of devices that the apps can run on. This testing is often automated or standardized. Automation lets you test what happens when 500,000 requests for the new feature hit the server at the same time. But, if a bug is not included in the automated test scripts, then it can easily be missed.

Companies also do internal testing across a wide variety of employees—often called dogfooding, as in "eat your own dog food." Scenarios that are more suited for automated tests (such as high simultaneous usage) may be missed, but actual user behavior often encounters unexpected bugs that wouldn't have been included in automated test scripts. This approach accelerates testing actual user behavior before deploying the feature to a large audience.

Finally, some companies release beta or test versions of their changes to select users before deploying them to the entire world of users. For a B2B product, specific clients may opt for early access to a beta version. Or, in the B2C market, users may sign up to be beta users. These pioneering users have an opportunity to try new features in advance and provide feedback on upcoming changes.

At each step along the way, collect feedback and track bugs. All of this testing is to ensure that when you deploy the new software, it is in the best shape possible.

Acceptance criteria

One of the most important parts of a user story is writing the acceptance criteria—the conditions that need to be tested to ensure that the feature works. What does a successful feature do? How would someone know that it did the right thing? What steps would the user follow to do that?

Tests are also needed to confirm how the product will handle an error. What conditions should trigger an error? And which errors should they trigger? Are there specific methods for triggering those errors? Someone should be able to take the error conditions and test each one individually to ensure that the application does the right thing.

How should the product behave on different platforms, different screen sizes, or different browsers? What are those differences? How should each one be tested? Is the feature usable across all platforms? For example, maybe the feature looks fine on a desktop browser but is unusably small on a mobile browser.

If a feature captures any analytics or tracking data, there need to be acceptance criteria to ensure that the right data is captured and stored correctly. This may require help from a developer or analytics person to validate that the tracking is working correctly. Incorrect tracking data will limit or render flawed analysis data in the future.

Acceptance criteria are also relevant to ensure that the product's usability meets performance or timing requirements. For example, what's the average and maximum amount of time that your new search feature should take to complete? If it's a mobile app, how should it affect your battery life or data usage?

Acceptance testing in practice

The best way to understand acceptance testing is by walking through an example. Imagine you work on a messaging app like iMessage or Facebook Messenger. You're working on the acceptance criteria for the user story "As a sender, I can choose and send an image to another user so that they can see what I'm seeing." What are acceptance criteria you should consider? How will you approach defining the acceptance tests?

Happy path acceptance criteria

The happy path follows how a feature should work if everything goes right. That means testing with valid inputs and following the steps you want users to take. In this case, the testing steps are as follows:

• Select to add an image.
• Select the image to be sent.
• Wait for confirmation that it was sent.
• Validate that the receiver received it within 10 seconds.

These tests can include performance criteria like "within 10 seconds" or "less than 1mb of data used." This ensures that features don't just fulfill the defined functionality but also meet user expectations.

Off the happy path acceptance criteria

It's also important to address when users may stray off the happy path. What alternative steps might the user do when engaging with this new feature? For example, what if they cancel the send. What should happen?

• Select to add an image.
• Cancel while in the image chooser.
• User should return to the message log. No image is sent. No confirmation appears.

There are many likely alternative paths that your feature should be prepared to support. For example, would your app provide a way to cancel sending an image after you've selected it?

• Select to add an image.
• Select the image to be sent.
• Cancel sending before it's complete.
• System should confirm that no image was sent or received.
• Show user a confirmation message that the sent image was cancelled.

Here are other ways users can go off the path. Imagine other alternative paths that the feature should be designed to handle. What should happen in each of these cases?

• User selects the browser's forward, back, or refresh buttons.
• User selects the back or home buttons on their mobile phone.
• Is there an undo or redo feature?

Forcing errors

Testing for errors is very important. You want to ensure that the product fails gracefully and safely and that it handles expected errors smoothly and doesn't do anything damaging or unexpected. Error scenarios could include the user inputting invalid data or the network dropping while uploading the image. What should happen next?

• Select to add an image.
• Select the image to be sent.
• Cut network connection while it's sending (such as create a network drop).
• User should receive a notification that the send failed.
• When application reconnects, the image should automatically resend.

Note that last step. Is it the behavior you would want as a user? Does it make more sense for the image to resend? Or, should the user have the option to manually resend when they lose network access? That's a decision you'll have to validate with users of your product.

Below are more examples of errors that acceptance testing should consider. In some cases, they are errors that are out of the users' control. Other cases are caused by incorrect user action or input—referred to as negative testing. Some common examples of negative testing include incorrect file types, invalid email addresses, and names that include punctuation (such as O'Connor). New features should plan for these errors. How will the application react? How should the user be notified about it?

• User provides invalid data.
• The application crashes in the middle of sending the image.
• The image is sent correctly but arrives corrupted due to errors in transit.

Check all the platforms

Given the wide array of technology platforms, which ones are supported and which are not should be specified for any product. Supported platforms are the operating systems and devices that a product is tested on and commits to addressing bugs on. For example, a product may support devices running Android Oreo and later OS versions. In this case, the product may not have been tested on the newest Android OS (perhaps because the product was released prior to the new OS), but the product is committed to fixing bugs found in the latest version.

There are too many platforms to ensure a product functions on all of them. As product manager, you should be confident that the product is tested for the platforms used by a majority of your users. Something that works correctly on iOS12 may not behave the same in iOS13. Test each one to ensure that they work correctly. Likewise, combinations of browsers and OS should be tested; a feature that works correctly in Safari on iOS may not work the same in Safari on macOS.

Define the platforms your product supports and which platforms it should be tested on. If there are features that will behave differently on some platforms, the acceptance criteria should note the difference that is expected on those specific platforms.

For example, the list of browsers and operating systems below should cover a majority of users for a web product. That means that features may need to be tested on each one before the product manager signs off on the feature. For your product, review your user analytics to confirm which of these platforms your users are using:

• Latest version of Safari on iOS
• Latest version of Safari on macOS
• Latest version of Microsoft Edge on Windows 10
• Latest version of Chrome on Windows
• Latest version of Chrome on Android

When testing on each platform, confirm that the feature meets the criteria listed earlier. Does it work as specified? Is it usable? Does it handle errors correctly? Does it look okay? Differences in product behavior and display are especially likely when launching desktop and mobile variations. It could look fine on your laptop but be incredibly frustrating to use on a mobile device.

Below are the most common platform variations to take into account for testing. Often products will only test using the latest versions, but depending on your users, it may be necessary to also test several older versions, especially if you're working on a B2B product. If your product is web (rather than native), it is also important to test for various combinations (such as iOS and Chrome).

Operating systems

• macOS
• Windows
• Android
• iOS
• Linux

Web browsers

• Google Chrome
• Microsoft Edge
• Firefox
• Safari

Form factors or screen sizes

• Tablet
• Mobile phone
• Laptop
• Desktop

How to be confident in your tests

Test by thinking about all the ways your feature might be used—both correctly and incorrectly. Think about the circumstances in which the feature will be used and what the user will be expecting. These details will help you write the best possible acceptance criteria.

Acceptance criteria and testing that addresses the questions below are likely to lead to products that have fewer bugs and the best possible functionality for users.

• Does the feature work as it was designed? Does it meet all the requirements of the specifications and user stories?
• Does it have smooth error handling when it encounters issues—both errors created by the user and those outside of their control?
• Does the display look correct? Does it look consistent across supported platforms and devices?
• Is it usable across platform and device variations? Are there any moments when the user might not understand what's happening or why it's happening?
• Does it perform well? Is it too fast or slow? Does it use too much data or battery?

Regression testing and test plans

Testing is not a one-time activity. You don't only test when initially shipping a new feature. A product should continue to test previously released features that shipped weeks, months, or even years ago. This is called regression testing and refers to testing older parts of your application to ensure that new changes don't break features already in the product. It is surprisingly common and easy for a small change somewhere deep in the plumbing of an application to have unwanted and unexpected implications for other parts of an application.

Continually building a library of test processes is the best way to manage this risk. For example, a product with a sign-up or enrollment user story should always test that story on all platforms—even if no changes were made to that part of the application. Create and rerun similar tests for all the major paths of your application. This is usually the last step of your testing process before you deploy your product, confirming that the new development didn't break anything already in place.

QA teams or savvy development teams may use tools like Selenium or Appium to automate the testing of interactive products. These tools automate the clicks, touches, and inputs that users would do in your product. These tools can accelerate testing, enable faster deployment of changes, and decrease the likelihood of shipping new features with regression bugs.

Ship or fix?

When you discover an issue or bug during testing, you need to decide whether to fix it or ship it. If uploads on all Android devices don't work, that's significant enough to require fixing it before shipping. If uploads for Android KitKat or earlier OS don't work, you'll likely consider shipping with the bug. And you would definitely decide to ship with it upon confirming that it only impacts 0.1% of your users and fixing it would take two weeks of development. To still ensure a quality user experience, it would be advisable to include a message for KitKat and earlier users stating, "Uploads are not supported on this operating system" rather than to correct the underlying bug.

When deciding whether or not to fix an issue discovered in testing, consider these questions:

• How many people will be impacted? What percent of users does that represent?
• Do we have any obligations to support this? (B2B apps may have contractual obligations.)
• How important is the feature that's impacted? How frequently is it used?
• Is there a work-around for the issue? Can users accomplish their goal in another way?
• How much time and effort will it take to fix it?
• Are there alternative ways to fix this that might take less time or have a lower risk?

These decisions are not always easy. It is useful to engage stakeholders or customers for feedback, especially if the fix requires significant development time. Be very candid about the risks and benefits as you pursue the best decision. It is also helpful to discuss the situation with a PM peer or your manager as you figure out the right decision.

Once the decision has been made, work with your team on how to best close the issue. You may need to write a new story, add a comment to the existing story, or document your decision and why you made it.

The video below provides a summary of some important elements of user testing, including how to decide whether to ship or fix.

Activity 🎯

Write a test plan for the Apple ID sign-up page. Do all of the following:

• Make sure each field has enough documentation to ensure that someone else could run your test plan without your assistance.
• Make sure your test covers any differences between mobile and desktop.
• Try each of the tests on the form. Does it pass all the tests? On mobile and desktop?
• What happens when you put in data that the form should reject?
• Are there other errors that the page should be ready to handle? If so, what are they? How should the user be informed that they made an error?

While testing, you encountered a bug: when people who use a valid special character (such as +) in their email address, their registration is rejected. As the product manager deciding whether or not to fix it before launching the release, what information would you want to know?