Your customer wants the best product quality and quicker turn-around time to confirm the quality. However the customer, as always, wants to reduce his spend on the testing.
Every vendor has told your customer that automation is the answer to the customer’s need. The customer has been automating the test suite for the various modules in the product over the last decade.
The customer has asked you the following questions:
1. How do you measure the quality of the product?
2. How do you measure the impact of your automation approach to meet the goals?
3. How do you provide the customer a ROI on automation?
4. How can you assure him a quicker turn-around time with your approach?
What are the questions/ input (maximum of ten non-trivial) that you would solicit from the customer to provide a response to the above questions?
Suggested solution:
Let us look at some possible questions. I understand that in the real world, you will have a lot more context and hence the questions would factor that. However, in my opinion the thinking behind the questions we ask is most important. Once we get cracking on that, then you could look at handling any potential context.
1. What are the measures and metrics that you currently use to measure the quality of the product?
2. What makes your business/ customer feel that you have a quality product? [This may sometimes be objectively answered or it could result in a subjective expectation.]
3. What type of failures (in the field) in the recent past, has left the business/ customer annoyed/ frustrated? [The answer to this question and the earlier question, can help you understand how you would respond to the first question that the customer asked.]
4. What percentage of your release effort (specifically of the testing effort) goes in ensuring that the new release has not broken any earlier release feature?
5. What are the top 5 reasons that you have found from your Root Cause Analysis of the field defects? [The answer to this and the above question can help you get an insight on the quality of regression and the automation level in regression.]
6. Can you provide the “business impact” of the last 10 – 20 significant field defects that have been detected? [The answer to this question, will help you know how to contextualize ROI in a customer environment. If the new approach you propose can mitigate the business impact – you have a very good way to show ROI]
7. Can you give us the number of releases that are planned for the product in the next one year – broken by major, minor, patch, emergency releases? [This will help you look at how much the automation test suite can be used and thereby amortize the cost of automation to see the breakeven point.]
8. How much of your current automation suite can run uninterrupted, barring the product failure? [This gives a good insight on the approach to automation. Many times I have noticed the teams have high level of automation but the focus has been inward-out based on what they can automate using the tools. This results in key bottleneck points not being automated and thereby reducing the effectiveness of automation.]
9. What percentage of your effort/ cost goes in maintaining the regression test suite? [A high number here gives you an idea of how efficiently has the automation been designed. There are times, when people have used the default way of working a tool and find that every time the object that is being tested has changed, they have work to do to keep it current.]
10. Is your test suite segmented by the business impact of each test case/ scenario? What percentage of each level of business impact is automated? [Many times we can automate tonnes of test cases, but if the test cases that have critical and vital impact on the business is not automated – the usefulness of automation from a business perspective is suspect. So how quickly you can assure the critical and vital parts of the product is always questionable.]
