A Personal Approach to Risk-Based Testing

I started tracking our organization’s Mean TIme to Detect (MTTD) a defect, and immediately noticed that the majority of defects opened were being discovered later in our two week sprints. Interesting…I felt the data had a bigger story to tell so I dug in a bit. What I found was in fact interesting, but something I’m sure the majority of readers can identify with: testers weren’t getting a steady flow of user stories and defects to test throughout the sprint. It was feast (at the end of the sprint) or famine (at the beginning of the sprint). As I began to think through how to resolve this, I remembered my waterfall testing days and how I would use risk to identify what to test first. I wondered how I could use Risk-Based Testing in an agile organization.

Risk-Based Testing is simple, the riskier the test case, the more priority I would give it in my execution phase. However, prioritizing tests during execution was only half of the problem to solve at my organization. If the team wasn’t even testing until the end of the sprint, we would still see a similar outcome. To really see a shift left in testing outcomes (finding defects quicker), the whole team would need to prioritize work around risk.

Note: We have “testers” on teams that are solely responsible for testing. Meaning developers will complete work and testers will test it. On teams where testing isn’t a role, but a function owned by everyone, this concept is still important as your high-risk work should still be prioritized early, no matter who the “tester”.

So I set out to change the outcome noticed in the above mentioned MTTD metric. Below is my thought process and journey on how we are working to implement risk-based testing and planning.

Risk Based Testing vs. Risk Based Planning

Throughout this post I will refer to Risk-Based Testing (RBT) and Risk-Based Planning (RBP). To ensure we are all speaking the same language, let me give a quick definition of each.

Simply stated, RBP can be explained as understanding and analyzing the probability of a user story or defect having an undesirable event while testing and then prioritizing that story as early in the sprint as possible.

On the other hand, RBT can be explained as understanding and analyzing the probability of having an undesirable event while testing and then prioritizing the testing early in the execution. The more probability of a test case (or whatever testing means you execute from) identifying a defect, the earlier in the test execution it should go.

Disclaimer: If your organization doesn’t write/execute test cases the same can apply to test scenarios, feature mapping, mind mapping, etc. The strategic goal is to execute high risk work as early as possible in your sprint.

What Value Does RTT and RTP Provide?

One primary goal for our organization is delivering a quality and engaging product to customers as quickly as possible. Getting something in front of customers, gathering feedback, and delivering incremental improvements of the product is what being “agile” is all about! As I thought through the various pros in asking the team to do this “extra” analysis, I realized there were actually value adds throughout the entire Software Development Lifecycle, not only on the engineering side.

At the feature level, identifying risk helps our Product Management team have a clearer understanding of product risks such as “is the product launch date in jeopardy?” If the team understands the risk at the feature level they can prioritize stories upfront so high-risk stories are worked first, where possible. This limits last minute “gotchas” as we near feature completion.

At the user story level, understanding risk can help teams focus on high risk stories first in the sprint. If the work gets tested early on, and there are issues found, those issues have less risk of delaying the product launch because the team can quickly fix them in the same sprint. Whereas those same defects, found later in the sprint, could have a higher risk to the product deploy date because the team might run out of time to fix them.

At the test level, identifying risk allows the tester to execute high risk testing earlier in the sprint, identifying defects and reducing the feedback loop for a user story. There may be scenarios where a user story isn’t identified as high risk for the team, but the tester understands defect prone areas and identifies a test case as high risk based on their knowledge. Either way, this prioritization has the ability to flesh out defects in testing as quickly as possible.

Types of Risk

Having a common understanding of what risk means within my organization is important to make certain we are all prioritizing accordingly. If you are in the banking industry, your risk might look completely different from my risk for an online horse betting website/app. Risk can be broken down into two categories: Product Risk and Technical Risk. Neither is less important than the other, however, identifying and mitigating the risks might look a bit different.

Product Risk, to me, refers to risk associated with customer flows, usability, or other UI related things. These are risks to the product and the customer’s interaction with it.

Technical Risk, as hinted in the name, refers to the technical issues that arise within performance/security, integrations with legacy code, or encryption challenges, to name a few. These are risks that aren’t necessarily called out in requirements but are identified during development planning.

Implementing Risk Based Planning

If you’ve made it this far in my post, you’re probably curious as to how I went about introducing risk analysis with my teams. I’ll share my journey below.

Understand Your Feature Flow

Before I introduced risk analysis to the team, I wanted to first understand the entire flow from which a feature gets into the hands of a customer. This helped me understand the various touch points where a person on the team could increase their overall knowledge of the feature and any risks associated. As mentioned above, product risk may be well defined in the requirements, whereas, technical risk could be something that our dev leads might identify as they move through feature planning.

Identify Places Where Risk Can Be Discussed and Considered

In my below diagram, Risk Analysis for User Story, you can see that in my organization we have a feature handoff session in which the Architects, Technical Team Lead, QA Engineers, and Product Owner go through the specifics of the feature. During this meeting, those individuals gain both product and technical understanding of the feature. It seemed to me that this is the first time risk, specifically product risk, could be discussed by the team.

Next, the whole team gets a chance to walk through the feature together. During this meeting, the team will identify every user story for the feature. I immediately identified this meeting as a touchpoint in which risk, both product and technical, should be considered and fleshed out. As the team is identifying feature user stories, the team should also consider these questions for risk identification:

• What is the probability of risk for this story
• How does this risk impact the customer

By now, the team would have a solid, and documented, understanding of the risk associated with each user story. My final task was to think through how all of this data gets tied together so we are mitigating that risk by pushing testing left: how could this upfront analysis lead to developers working high risk stories first.

Prioritize High Risk Work First

Because the data is there to help drive the planning, we could now prioritize the highest risk stories to the beginning of the iteration. I acknowledge that sometimes there are dependency constraints that prohibit this. In those instances, we prioritize as best as possible while still acknowledging those dependencies.

Implementing Risk Based Testing

The next piece I wanted to think through was RBT. As mentioned above, Risk Based Planning was only half the battle. I also needed to think through how we get testers prioritizing riskier work first.

Understand Your Testing Flow

I quickly realized the starting point for RBT is similar to that of RBP. To start, I had to make certain I understood the natural flow of testing for our teams. Specifically, how do testers ensure they get a solid understanding of the user story or defect they’re being tasked to test. In addition, how would the testers go about actually identifying risk within the test?

The conclusion I came to is trust. Trust that we’ve hired the right people for the job and they understand how our applications work and where risk prone areas are. The “tester” shouldn’t own the identification of risk; it should be a team effort. Given all the steps outlined above, team engagement is built into the flow.

Assessing The Risk

Finally, once risk is identified, the next step is to assess the risk probability. To do so, the team would outline the likelihood of failure for each test scenario and the impact of that failure to the end user.

Prioritize High Risk Work First

Once the tester had a good idea of the high risk test cases, from there they would prioritize them so the execution is at the beginning of our test execution.

Automation

Automation execution is another piece to consider while thinking through risk. Specifically regression executions. Assess what test cases or modules/components are prone to failures or defects and make certain you’re executing those scripts first. Some organizations may have their automation execution down to minutes and prioritizing high-risk areas might not buy them a lot. However, for those that have execution times in the hours, this could absolutely enhance your ability to shorten the feedback loop.

Conclusion

I presented the above thoughts to my organization a couple of months back and we implemented those changes across the teams. Here are some key take aways from the implementation I’d like to share with you.

• Get your Scrum Masters on board – SMs became an important piece of this process. To make sure they were comfortable, I outlined a list of questions they could use to tease out risk with the team. This was important in the beginning while people were still getting used to assessing risk. I’ve given a list of those questions at the bottom of this post if you’d like to use them too.
• Train for risk – Yes, most people understand risk as a concept, but do they know how risk really ties into their work? In order to ensure they do, train everyone on what RBT and RBP is and how to assess risk properly.
• Keep iterating – My organization has multiple scrum teams and each team works a tad bit differently. This is great for process improvement because we can constantly look at ways each team is doing things and pull the best of that out for other teams to learn from. Once you roll this out, give it a couple of months and get feedback from everyone.
• Risk analysis shouldn’t only happen on the development side – As your team is getting better and better at this, you should notice that your Product Owner or Business Analyst are considering risk when writing requirements. Mitigating risk upfront is the goal!

Risk Based Testing Questions

• Do we have the test data in lower environments to test all scenarios of the feature?
• On a scale of 1-5, what is the testability of this feature or story?
  • 1 means low testability, 5 is high testability
  • If 1, 2, or 3 are answered: ask questions to dig further:
    • Why isn’t the story or feature testable?
    • What do we need to make the story or feature testable?
    • Are the AC’s testable and are they clear?
    • Do we need Back End flows to facilitate testing? If so, are Back End flows identified?
    • Do we have clearly defined UX Designs? (are UX even available at this stage?)
  • If 4 or 5 are answered:
    • A score of 4 means you have what you need but maybe your testing a new feature so a little unclear on every detail.
    • A score of 5 means that testing for this feature may likely be modeled off another one. AC’s extremely well defined. 
• Is the feature deployable in small chunks?
  • Deploying the feature all at once is higher risk.
• Are we touching a product or tech level risk?
  • If so, how does that impact testing?
• Community: What other questions do you ask to assess risk?