Continuous Verification

While organisations embrace DevOps as a true accelerator for their delivery velocity, testing is still the throttle for real agility. Flaky automation, environment stability and data dependencies are often hit stumbling blocks for continuity. How does continuous verification help to overcome those issues?

Nora Jones and Casey Rosenthal introduced in their book “Chaos Engineering” the concept of Continuous Verification (CV), which they defined as “a discipline of proactive experimentation, implemented as tooling that verifies system behaviours.”. In it’s pure sense this means right-shifting of quality assessment to a pre-prod or prod stage, where automated, controlled experiments validate the adherence to defined quality characteristics, such as reliability.

This does not render other testing activities useless, but puts focus on the question, which types of tests make most sense in which context and of course, how such tests can be designed in a way, that they don’t fall in the automation trap (zero or negative efficiency of automation). In a wider sense, Continuous Verification can be seen as a pattern, to automate the verification of quality based on test results and system behaviours under unexpected conditions through experiments. In order to achieve this, important elements need to be implemented.

Test Observability

Observability of tests is the probably most important differentiator and an absolute precondition to effective CV. It must be possible to determine the quality (coverage & result) of a test execution, as well as of the environment and the system under test at any given point in time. The very starting point is the introduction of test execution metrics, that are collected and reported per default. It is equally important to audit, when test (suites) start or end and of course, which configuration the test was applied to. The effectiveness of a test observability solution can be validated by asking 3 simple questions and answer them based on observed metrics, events, logs and traces:

1. Was a test (suite) executed successfully and correctly?
2. Are we sure that the system under test was healthy?
3. Was the test purpose achieved?

Quality Gates

At the time, when the result of a test can be determined by the collected data, a quality gate should become the codified implementation of this “determination”. While quality gates in many cases are just switches that either pass or fail the pipeline execution, they can achieve a higher maturity or complexity as well. For instance within a feature flag supporting application, certain features may pass, while others get disabled. Even more adaptivity can be implemented, by considering error-budgets, test criticality and constraints.

However, the complexity of quality gates influences the maintainability, but also understandability of the pipeline. Following the principal that automation should not hide (too much) complexity, a single gate should focus on a single purpose.

Shift left and shift right

Delegated test automation is doomed to fail. The gap between code and test should be as small as possible. Leaving implementation of tests to non-developers or to people outside the team is rarely a good idea. By putting all suitable and required verification close to commit and build, immediate feedback to the developer sets a clear and close-controlled loop of responsibility. Continuous Verification is a composition of right-sized automated test activities that are in the optimal place within the delivery pipeline. It also covers experiments and production runtime, as those may indicate quality and hence have a feedback loop for testing and deployment paths.

The remaining big challenge is that many systems are highly integrated and are hard to test in isolation. By putting a lot of focus into interface management, contract and API testing, some of those risks can be mitigated with a shift-left approach. However, there are still use cases that will require integration, to some extent. For those, a tailored combination of disentanglement, virtualization and good test data management, will help, but it comes with a certain cost. We will talk about such edge cases in a later blog.

Transparency and alerting

For a fully automated process, the clean and clear communication about its state is one of the fundamental design approaches. Nothing is more worrying as a pipeline, that in itself is unreliable and unpredictable, ultimately leading to mistrust against the delivery system as such. Even though it is difficult to completely avoid misleading information that lead to alert fatigue, every wrong indication of the verification process should be addressed with high priority.

Continuous verification is a key capability that supplements developer experience, CI/CD and in the end customer value. It not “on-top”, but integrated within the solution delivery process and part of a culture of shared responsibility and continuous learning. To deploy continuously and on-demand, quality needs to be an intrinsic, observable vector of the solution. Continuously verifying the state of the system and delivery processes enables developers and reliability engineers to embed quality analytics prediction and self-healing into delivery and operations. Hence, CV is a key capability of DevOps software delivery processes and a key contributor to agility.

*) see “Ironies of automation”, PII: 0005-1098(83)90046-8 (archive.org)

And Erik Hollnagel, Joint Cognitive Systems https://erikhollnagel.com/books/joint-cognitive-systems-foundations.html