Improve your testing methodologies by avoiding these 3 anti-methods
As the career path of quality assurance has drawn me in, I’ve begun reading to have more formalized knowledge to draw on. Recently, I have read Brendan Gregg’s System’s Performance and early in the book he identifies some anti-methods. I noticed that earlier on in my career when I had first gotten assigned work to performance test, I employed some of these anti-methods due to my lack of experience. Other causes why the anti-methods may be used includes a poor implementation of quality practices in and organization though, as some of the tools for gathering metrics and observing a system cannot be configured by the QA Engineering team alone. In smaller companies, these anti-methods are used by the product or engineering team due to inexperience in testing.
Hopefully by familiarizing yourself with these anti-methods, you can save time and improve the likelihood of catching the causes of degraded performance sooner.
The Streetlight Anti-Method
Also called the drunkard’s search, Gregg describes its defining characteristic as a lack of a method entirely. Instead the observation, testing, and metrics are selected by familiarity, or as Gregg suggests, whatever might have come up from the internet.
Like so many who entered the field outside of an academic background, I relied heavily on resources online. Like a drunkard looking for his keys at night, I and many new-comers to the field often first search the area near a light, or in this case, what is familiar. However, as I grew, I honed into what works best, and these unsurprisingly lined up with methods Gregg suggests, such as the Scientific Method, Diagnosis cycle, RED Method, and Latency Analysis.
One of the issues is its time-costs. The results of this method can and should find issues, but without a focused approach, the identified issue may not be the issue, requiring the employee to re-investigate.
Another major issue includes the approaches can completely miss issues. Without a focused understanding approach to observing, debugging, and testing, the issue that can be causing the degraded performance may stay a “unknown unknown”.
Random Change Anti-Method
In this anti-method, a random change is implemented and measured to compare against.
- Pick a random item to change (e.g., a tunable parameter).
- Change it in one direction.
- Measure performance.
- Change it in the other direction.
- Measure performance.
- Were the results in step 3 or step 5 better than the baseline? If so, keep the change and go back to step 1.
Gregg, Brendan. Systems Performance (p. 42). Pearson Education. Kindle Edition.
Like the first anti-method, it is time-consuming. And similarly, it may find issues but not THE issue(s).
As a junior quality assurance engineer becomes more comfortable with their role, they’ll begin to implement pattern searching, which will naturally evolve into the Random Change anti-method. Unlike the Streetlight anti-method though, it is a little easier to track metrics and report on cause-and-effect.
Blame Someone Else Anti-Method
This sort of approach will likely be present in companies with the “over-the-wall” approach of quality assurance or poor implementation of quality practices within the engineering team. When this anti-method is used, team-members will often turn to systems and components not directly under their supervision for an answer, and require whoever is responsible to actively prove it is not the issue, in hopes that it might resolve the issue.
I cannot blame someone for reaching for this anti-method when they’ve reached their wits-end, as they’ve utilized every approach they may be familiar with when seeking performance issues. In fact, if they have delved deep into this issue, I think a health team culture would encourage them to seek the wisdom and experiences of their teammates for the next steps.
However, if the first reaction to a performance issue is to assign work to another team or teammate, this anti-method can once again eat valuable time of not only the QA engineer, but other teammates as well now. Another reason a QA engineer may turn to this is poor documentation of system architecture. Without the resources to understand a system, it can provide them a convenient excuse to blame someone else and make it their problem instead.
What should we do instead? How can we avoid this?
So if you read through these and felt aggravated, I’m sorry. Some of these anti-methods are the result of poor quality practices and can be remedied to a degree with the following:
- Implement observations and metric collections. This will allow you to begin performance testing sooner and provide some idea of where to start researching the issue at hand.
- Maintain documentation of the system and components. This will allow a single person to take initiative and avoid the traps of pulling other team-members prematurely into the issue.
- Research other performance testing methodologies. An easy one to start with is called the “Ad-Hoc Checklist”, which should have suite of performance test and metrics that are maintainable and easily repeatable when issues arise. While not as narrowed in focus, it provides an easy to understand standard for the rest of the engineering team when issues do arise. Over time, the checklist will succumb to the “pesticide effect” though, and less and less bugs/issues will be found. It is important to maintain a checklist that suits the current state of our system(s) over time.
Thanks for reading
Thanks for reading this. I am not an expert, but I do hope that with work experience, additional reading, and sharing knowledge, I can improve my quality assurance mindset. With that in mind, I can understand if there are some parts I may have written that individuals disagree with, so I welcome corrections or feedback in comments. If this helps you, please let me know too, as I love hearing that I can help out others.
The book referenced can be purchased using the affiliate link here: Systems Performance by Brendan Gregg. In the interest of transparency, please note that using my affiliate link provides some small benefit for me personally.
Photos were found on PixaBay, where photographers can provide photos free for commercial use. The captions have links provided to the images.
Originally published at https://www.linkedin.com.
