(Ideas) Three approaches to software testing.
How to approach software testing
❓The problem:
A general instruction or framework to approach software testing for new or exciting projects.
💡The idea:
I found many approaches to software testing, but three of them catch my attention, they are simple, easy to follow, and easy to scale if the project grows.
- The Test Pyramid
- The Testing Trophy
- The Testing Matrix
The Test Pyramid
The test pyramid is a concept introduced by Mike Cohn in a book called Succeeding with agile and explain in detail in this article https://semaphoreci.com/blog/testing-pyramid
Starting at the bottom we have:
Unit testing
This is a small, logic-driven test, fast to run and oriented to evaluate and output of a function, class, or event.
They are small but numerous.
Integration Testing:
Tests focus on the connection between different parts of the software, for example; connection with payment APIs, Databases, etc.
These are less numerous and they live on the edge of the application.
End to End (E2E):
These types of tests include UI testing and are the main target for automation testing. The are few and take more time to run.
We have at the bottom the unit test, simple and fast. At the top E2E is long, complex, and small in numbers. In the middle, the integration number vary from project to project but they are less than the unit test.
The unit test is the smallest and more common test, these are fast and easy to maintain therefore there are the most likely to be run by developers, E2E testing sits at the top they are costly and difficult to maintain, which makes them scarce.
Approaching the different tests in this way allows us to find a balance between effort, cost, and chances to catch bugs.
E2E shouldn't be discarded because they are difficult or demanding. These tests provide a real metric of how all components are working together and an evaluation of the usability of the application.
Testing Trophy
“Write tests, not too many, mostly integration.” Credit: Kent C. Dodds at testingjavascript.com
A Static portion will be at the base of the trophy. The author Kent C Dodds suggests relying on tools such as ESlint and JSHint (or their equivalent in other languages) to catch early problems or unsafe statements.
In this approach, Integration testing is the main focus, since this is an approach oriented to testing UI, so won't be strange to lean toward integration testing since modern UI components or difficult to test in isolation.
The Test Matrix
- Which type of test gives you more confidence?
- What is stopping us from writing a lot more E2E tests?
The second question will make more sense if we give an answer to the first question.
If we are talking about confidence,usability-wise, meaning the application is useful or satisfies the requirements set to be achieved. E2E is the only one capable to validate the usability.
So why not write more of the E2E test?
The Answer:
Effort.
E2E is costly to produce, it can be fragile and take a long time to run.
Gleb Bahmutov and Roman Sandler proposed the Testing Matrix
In this matrix:
- Efforts increase from left to right
- Confidence increase from the bottom to the top.
The goal is to remain in the 💚green area, and avoid is the ❤️red area.
Maintain high confidence and low effort 💚 and avoid low confidence and high effort❤️ .
Assuming the following:
Unit tests are been written along the development of the application and those same tests are later neglected.
We soon find ourselves in the red area❤️.
To avoid this scenario and maintain the project in the Green 💚. The authors propose to focus efforts on five categories:
- Installation: the effort involved in installing and setting up the test framework.
- Writing: the complexity of writing tests and the skill level of the developers for a given framework.
- Running: the difficulty of running the test suite and CI/CD performance.
- Debugging: how easy it is to find and fix a problem when a test fails.
- Maintenance: how much effort is required to maintain a test throughout the project’s lifetime.
In this part, It came to my mind Writing Maintainable Automated Acceptance Test by Dale H.Emery.
“The need to change test comes from two directions: Change in requirements and change in the system’s implementations. […] We must change the test to adapt to the new requirements or the new implementation.“ Writing Maintainable Automated Acceptance Test by Dale H.Emery
A Maintainable Automated acceptance test as an E2E might solve the fragile part, If the test required less maintenance the cost will be reduced, letting us with a Long time to run.
The role of CI/CD
In the beginning, the unit tests will help us to fail fast and evolve, but with time the increasing size of the test suite will become a burden, and the execution time will start to increase significantly. Is here when you need to start to think about parallelizing the execution.
The modern CI/CD tools allow the execution of tests in parallel this addition to the reorganization of the test suites will help to reduce the execution time.
Final Thoughts
- All the approaches suggested are valid and should not be discarded, each project and company is different so it is a good idea to maintain a mentality of adapting and evolving.
- Maintainable test, with a test suite structure that allows a parallel execution can reduce cost and long time run in the E2E allowing reducing the friction and promoting the creation of this type of test.
