In our previous blog posts, we’ve explained what the DevOps concept is and the foundations for implementing a value delivery deployment pipeline. In this post, I want to focus on one key factor that makes all of this work: testing. A proper testing strategy gives us the confidence we need to take that step to implement continuous value delivery.
If we examine the deployment pipeline that we designed in our previous posts, we see that, after any task we do, we always perform a validation process that tells us whether what we’ve done is correct. The key concept here is feedback. Testing is our feedback tool. We need to know what happened and we need to know it quickly. There are therefore two basic characteristics of any testing strategy:
- It must provide us with feedback.
- The feedback must be fast.
This means that the sooner we incorporate testing into our process, the sooner we will have the feedback we need. In DevOps terminology, we often say that we should apply “Shift Left” testing in our process. In other words, we should include testing at the beginning of the process and not at the end as is usually done in traditional processes. In addition, testing from the DevOps perspective is continuous. It’s an activity that’s part of our day-to-day work, not a phase that’s executed after completing another earlier phase. This is one of the most important mindset shifts in testing compared to the traditional approach. The latter sees testing as a phase executed by a quality team separate from development that comes after the end of the development phase.
” Testing is an activity integrated into development, not a post-development phase.”
A metaphor that could be used to understand the importance of obtaining quick feedback on errors is the spellchecker. As we type, the spellchecker tells us instantly when we’ve made a mistake. For each word that we type, the spellchecker performs a “unit test” that checks whether the word exists in the dictionary. In other words, it provides us with feedback very quickly, and the time we need to correct any errors is negligible.
Imagine instead that we pass our document to a second team so that they can review it when they have time. That team then returns the original document to us along with another document containing the results from their review, which tells us where we have errors so we can correct them. How long would it take to produce an error-free document? What is the cycle time? Incredible as it may seem, this is the traditional testing model: after finishing a development project, we deliver a piece of software, and a document is returned to us with the list of errors found.
Let’s do a thought experiment and, ignoring the use of the automatic spellchecker, imagine that we include the person who’s going to review the document in the team and pair review the document at the same time it’s written. What would the result be? It would be very difficult for there to be a single error in the document.
Testing is a collaborative tool
Another point of view is that testing is a highly collaborative activity involving different team members. It’s what helps us build a shared understanding of business problems. In this regard, I’m quite a big fan of Behaviour Driven Development (BDD), not as a tool to automate testing, but as a collaboration tool and communication mechanism between the business, developers, QA and all those involved in development. Combined with the “three friends” technique, testing becomes a very powerful tool for understanding the problems that need to be solved.
Automating Testing in DevOps
A key element of DevOps when it comes to testing is that it must be automated and included in the pipeline. This allows us to systematically run these tests against the new code generated, providing us with continuous and rapid feedback on our work. This automation also serves as a regression test against the old code. This is where we see how testing builds confidence into our development deployment process, relying on it to protect us from the danger of introducing unwanted problems into the production environment.
When automating, the automation pattern displayed in the following pyramid must always be borne in mind:
This pattern draws attention to how much automated testing should be performed for each testing level involved. Based on this, our efforts to automate testing should be directed at the unit testing level and not at the GUI (Graphical User Interface) level (which is usually the first thing we think of when automating testing).
Exploratory Testing
Inevitably, whenever there is talk about automating testing, the question arises as to where this leaves people in terms of their involvement in the testing activity. Of course, automated tests do not just happen; someone has to create and run them. Consequently, there is a lot of work for people in this area. But beyond this, there is a fundamental area that falls outside the domain of automation: exploratory testing.
In my view, this is a fundamentally human activity. To explore means to delve into the unknown, to investigate, to assume, to inspect. These are activities that require a degree of intellect and cannot be automated, and it is in this area that humans play an irreplaceable role.
Running a few written test cases on a platform is not exploring. It’s following a predetermined path and checking that the results are correct. It’s like going on a hike in the mountains and following a GPS track with our GPS device. In no way can such an activity be considered as exploring since it’s totally predictable and automatic. Entering uncharted terrain, moving through it and identifying its dangers is exploring. This exploration results in a new map or track that can be automated later, if deemed worthwhile.
The importance of Telemetry in Testing
In the current value delivery model, a key enabler for error detection and feedback is the use of telemetry provided through various tools. When frequent production deployments are being made, having tools that immediately detect problems as they occur is critical to restoring the service as quickly as possible. (One of the key metrics in DevOps is service restoration time.)
In a DevOps model, through the use of telemetry, the idea that “Application Monitoring is the New Unit Testing” has emerged: in production, it’s monitoring that provides the fastest feedback indicating that a problem exists, just as in development, it’s unit testing that provides this.
The testing pattern is thus transformed from a pyramid to an hourglass [1] :
The different tools for logging, monitoring and alerting require specific discussion, and we’ll cover this in future blog posts.
Final conclusions on Testing
Although testing is one of the key factors in the successful implementation of DevOps in organisations, it continues to be among the most neglected by them. For this reason, many attempts to implement a deployment pipeline end in failure. For DevOps to be successful, it is essential to properly integrate people with good testing skills and the right mindset into your teams.
In future articles, we’ll take a closer look at the different concepts and testing tools that can help us implement DevOps successfully. Learn more about our DevOps services and subscribe to our newsletter to keep up to date with all our news.
Emiliano Sutil is a Project Manager at Xeridia
[1] Clokie, K. (n.d.). A Practical Guide to Testing in DevOps
