The Ultimate Guide To Continuous Testing

Functional Testing

Functional testing is focused on the functional aspect of the software (Surprise!).

Mainly, it asks the question, “Is the system buggy?” In other words, it focuses on how well the software works according to the design specifications and having all related risks under control. Functional testing checks the core functions, test input, menu functions, installation and setup on localized machines, etc.

As mentioned earlier, especially in functional testing, it is important that teams know if they know what it is they’re testing. Which aspects have already been tested and which are missing? How thoroughly are they tested?

From our point of view, there are three major ways of doing functional testing:

  • Ad-hoc: Ad-hoc is equivalent to asking anyone to "test the system for a while”. This is typical of those who see testing as something like having the software in front of someone who makes random clicks, observing “how it is.” There’s no element of control, it’s impossible to trace, and there’s no way to know how well or badly it is going. Testing is for obtaining information about quality and this does not give us said information, so we’d never recommend this “testing” approach.
  • Scripted Testing: In scripted testing, first one wonders, “What is it that I want to test?” Then one documents it and runs the tests according to the document, recording what went right and wrong. The star player in this game plan is the "test case." The two stages, the design stage and the execution stage, are so removed from each other that they could be completed by two people with differing skills. A business expert who is knowledgeable in designing test cases could create a spreadsheet (or tool) with all the cases that he or she is interested in testing. Then another person who may not have as much experience could take that documentation and execute it step by step, indicating the results of each test. In doing this, it is ideal to plan, leave a record of everything that was done, organize the team and analyze how deeply everything was tested. This method is suitable for regression testing, for passing on the knowledge of what things were tested to another person, and for documenting the expected behavior of the system.
  • Exploratory Testing: With the advent of Agile methodologies and the focus on adaptation to change, test cases become less relevant. It’s not always ideal to wait around to have documents handed over that indicate what to test and what is the expected result in order to convey an idea of how well or poorly the system works. That’s why the exploratory testing approach exists, where the focus is on designing and running tests simultaneously, and in doing so, becoming familiar with and learning more about the system. The focus is on finding errors and risks as soon as possible. Furthermore, one can more easily adapt to change either within the application or the context. Because exploratory testing eliminates the need for keeping test case documents, it gives way for increased flexibility. Every action we take is based on the result we got, deciding what to do (and designing the test) on the fly. But beware, this is not ad-hoc because there is a well defined methodology to follow this scheme that allows us to organize, track, control, and manage the whole process. It aims to improve from cycle to cycle rather than run the same tests repeatedly in each cycle (as this is tedious, error-prone, and is better than just running an automated check).

Test cases are often reported in spreadsheets, or better, in specific tools meant for this purpose. The more Agile alternatives to test cases are test sessions, revision checklists and mind-maps to record test ideas rather than having a step-by-step list of actions to test. In any of the key strategies, it is fundamental to have some idea of what the whole is (total test cases, total functionality to be tested, user stories to be covered, etc.), and how to advance. This is what signifies the coverage that is being obtained. Of course, it is necessary to prioritize this according to the levels of importance and risk to the business and users of each aspect.

To achieve testing maturity and especially continuous testing, it must be clear what is being tested, how, and how well. We recommend either scripted or exploratory testing but never ad-hoc testing.

Download the Ultimate Guide to Continuous Testing as a PDF

Everything teams need to know to shift-left testing and reach the level of maturity in their quality engineering processes to enable CI/CD.