Myths about Automated Testing

The aim of this paper is the desire to give people the right idea about what exactly is "automated testing" and what it "eat".

Introduction

Since ancient times, mankind is engaged in mythmaking. The less there was knowledge about the natural environment, the more errors were a man and his amazing myths. With the development of science domain knowledge has expanded. But the area became more unknown. Old myths are gone, but there were new.

In this article I would like to talk about automated testing and related fallacies. Some of them are more typical of upper-level managers, part - the average employee. But those and other myths interfere with the proper understanding and development of the technology itself and its successful implementation on the ground to ensure a higher quality of manufactured products.

What is automated testing

One of the most popular myths is that:

Myth 1. Automated testing - it’s just a check of recourse.

Along with him frequently encountered yet another error, which reduces the automation only to test the user interface:

Myth 2. Automated testing - it is only functional testing via GUI.

And both - is at the root of misconceptions about the subject. Let’s try to start to understand: what is automation?

"Automation - a complex of measures aimed at improving the productivity of human labor by replacing part of the work of the machine." Informatics. Encyclopedic Dictionary.

From this definition it is obvious that automated testing is not only and not to just do the tests. Automation can be represented in most of the testing process, from planning and ending thus running the tests. Look at the basic processes, but without mentioning specific products, as well as some elements can be automated and independently, using for example MS Excel, without having to purchase expensive serious tools.

Planning and control. It runs a choice of necessary volume of tests based on risks or other methods, distribution of tests among employees, and monitoring their implementation, identifying the number of tests, their states and properties.

Analysis and reporting. Means the construction of reports in different shapes and orientation on the basis of data on the work done, both for the work team, and for higher management.

Error control. These include bagtrekingovye of all kinds of variations.

Creating tests. This refers to a preliminary autogeneration: Based on the code - for unit-tests, based on functional requirements - for manual test generation on the basis of record’n'play etc.

Coverage analysis / trace. Performed to evaluate the code coverage, requirements, or some of the functionality of these or other types of tests that are created in the previous stage.

Running tests. Starting the test scripts and analyzing the results.

This list everyone can continue to own depending on the specifics of their work.

Next I will talk only about the part of automation, which deals with creating and running tests. And the term "automated testing" will be used in the meaning narrowed to these processes.

For what purposes may also apply automated testing?

The first and most popular - is  testing the performance in all its incarnations: load, stress, stability … popular for the reason that, without automation it can not be performed. For the same reason there is a wide range of instruments from different manufacturers and equally high prices, even if inconvenient and slabofunktsionalnogo tool.

Next is regression testing. That means checking the correctness of software testing services functionality, released and thoroughly tested in the previous version. Performed with regular frequency, defined depending on the conditions: someone with every new build, someone with every version for the customer.

Configuration testing - execution of the same tests in different conditions. That is, when one or more components of the system architecture is required to check in different environments, usually stated in the original requirements. For example: DBMS support from different manufacturers, working in various client browsers, the use of several operating systems, etc. That is an analog of regression testing, but under one version of the system. In addition to means of monitoring the performance of work, in this case, it is useful is the ability to automatically compare the results the scripts in different configurations.

Functional testing. Clearly, this is just checking the new features. Sometimes it happens that, without automation can not do. Even if you want to test only once. Typically, these tests and subsequently used for regression.

Installation testing, as the name implies, is performed to verify the conditions of installation (and configuration) of the product in view of various system requirements from the customer. In this case, the automated tests is convenient not only for testing, as such, but also for their subsequent use of hand-held testers to reduce the amount of work when installing the next build.

And so on … If the scripts are created for one type of test, then the extent of no one prevents their use for other purposes: release version, acceptance … almost everywhere.

For all the diversity we have listed the objectives of testing, be automated, technician test is noticeably smaller. And they were often dependent on the goals, the specifics of the test software and the expertise of your employees.

Firstly, everyone knows that unit-testing. No unit in a general sense, and the technique of writing tests, the developers on the basis and source code. The most active right now to talk about it in different agile-methodologies. While in the classical process of development can bring many benefits, if properly start using.

Testing API (application programming interfaces, which includes both public methods for local calls from conventional dll, and for the COM +, win-and web-services, handling the queue and so on). Usually applied at the level of unit testing of the product and can not be distracted by investigations of the characteristics of methods for further integration.

This includes testing the web-interface through requests sent from the browser, which is sometimes replaced by checking the user interface. It is most convenient to use for load testing. But sometimes it makes sense to combine with functional testing, since I anyway have to develop such scripts.

Testing the CLI (command line program with input). Allows you to use a simple spreadsheet for analysis of input values ??and results. And then, without much effort to program.

Testing GUI (user interface) - something that usually goes most conversations. And that can not always be successfully implemented. Usually applied at the level of system testing to find regression dependence.

Emulators external systems - are used at any level of testing, depending on the object of emulation. Created by developers or testers, whichever is the primary goal and where there are more free resources. Usefulness also depends on the mentioned purposes. In successful cases, may even fall into the hands of tech support, for which they will not just say thank you.

How it works

All subsequent myths, for obvious reasons, will increasingly be related to regression testing. However, all below the above can be attributed to other purposes of testing with the appropriate additions or changes.

The following error is human, and a happy and immediately:

Myth 3. In automated testing all done very

I remember Murphy’s law: "a system that will use even a fool. And only a fool will want to use it. "

Let us look again to the dictionary Informatics:

Automated technical object: a device, system or process that uses machines or other automation tools. In contrast to the concept of "automatic" in the work of these funds, or performed by them during the expected participation of man. "

And indeed: in our case, even with automation, but run the script to complete the process need the following steps:

·         The choice of scripts to run. With a large volume may well be a situation that all of them simultaneously run is not necessary. It is quite technical - when the tool writes the whole log into memory, and then at the long scenario is very likely it overflows. Or functional - where the performance of the tests are not required because even unfixed bugs. Purely handmade.

·         Preparation of test data. Can be both automatic and manual. In successful cases, the data may remain unchanged from the previous run, if the tests simply do not change data on either end to end testing lead them to their original state. In unsuccessful - before each run the data necessary to prepare from scratch.

·         Actually start and run scripts. This stage can be fully automatic if the framework allows you to configure the scheduler to run the tests while maintaining the established list.

·         Analysis of the results. Purely manual work that can be automated a bit in terms of comparing the results with other start-up or filtering studied logs.

·         Publication of the results. If the strain is also amenable to automation. But is usually performed manually by using the data obtained in the previous step.

How much

The following two myths I was trying to describe individually, but they do not want to. And they’re really interrelated. Since both are more to the errors of top management. And both are related to the project budget.

Myth 4. Automation can reduce costs

Myth 5. Automation solves the problem of lack of resources

The first myth of the two is always a project manager, who expects the budget for the original plan, says about testing on automation and decides to save the expense of it. And that’s not the worst situation.

The second myth is often pops up on an already running project: approximately in the middle or second half. When it becomes clear that the terms are broken, the developers do not manage what you need nakodirovat and testers then all this time to check. And the easiest way out of this situation - is to add more people or time. And made a brilliant solution: save time by automating the testing for and pass it to develop.

In both cases, the request from the head of the test manager sounds like this: "Do the math please benefit from the use of automation compared to manual testing and provide me the result. Whether to implement it. " Implying that the benefit is unquestionable, and the calculations are only needed for the next reporting session for top management.

In fact, you can not talk about automation, as the replacement of manual testing. Rather, as a complement. Therefore, when the manager asks to calculate benefits - it sounds like nonsense. You can talk about optimizing your team effort and the technologies used to achieve the highest quality product in the current environment and budget constraints. And in a situation for a long time coming project, if the conversation comes up about the need of automation - thus most likely in early adopted the wrong strategy of testing. And to think about the introduction of automation must be approached with greater care and caution.

And when calculating costs, consider the following elements:

·         Tools;

·         Training / recruitment of experienced staff;

·         Development;

·         Support;

·         Implementation;

·         New tasks associated with automation (as compared to manual testing).

Toolkit is not always you can go free because of the specific processes built or other circumstances. But this is usually a small fraction of the cost of automation. That the training that the hiring of new employees - these processes take more than a day and usually fly in a pretty penny. Especially if you always want to cut. Design and support - the two processes, the cost back to each other. At the beginning of the project to generate tests eats most of the time allotted, and their support is zero, because the script just yet, but by requiring testers to determine the test environment and infrastructure. Perhaps even need first to develop a framework (one of the additional tasks).

In a subsequent development will be less. But more tests will be ready and active changes in the test software, the more time consuming maintenance. Especially a lot, if the architecture test fails. Sometimes it’s easier to start over with the light of experience. Running tests by itself does not take much time, because if done right can be done at night. This is the only time saving, compared with manual testing. Because the institution received error bagtrekingovuyu system will probably need to do manually. However, after the script execution, as mentioned above, we must also analyze the results: those tests that were executed with an error. Since the errors can be in the program, and the script and the source data. This is another new challenge. As a result, you can talk about anything, but not about saving money.

Now, about the people. Suppose that the budget allocated for the automation can not do anything, but a tool to us by happy coincidence is worth nothing (got from a nearby project) and the testers are able to work with him, then in this case we can not say anything about the benefit of any savings.

If a team testers did not change, it means that the introduction of automated testing varies as a list of tasks and their priorities. There will never be such that will run the previous level of manual tests plus the set of automated. (Permanent recycling into account, let us not take it, because it probably has a way to increase the budget at the expense of employees.)

So that the volume of problems in one other case is simply not comparable. This means that the manual testing product was nedoissledovan on one side, and when automated - forget about the other. Or inspection on each side, but it was very brief.

As a result, any savings is not obtained. A well-established testing processes may well be violated in the unsuccessful introduction of automation.

What happens to the bugs

Another pair of errors related to one theme, the key in testing - the subject of errors.

Myth 6. Automation allows you to find more bugs.

Myth 7. Automation reduces human error.

Both statements are true, but only partly. Otherwise they are as false.

Kaner, Bach, Pettikord in his book, cites the following statistics: "for any form of automated testing is not more than 30% of the errors in the very successful event. And only when testing the configuration can be found up to 80% of the total number of errors. "

This is a serious mistake, which eventually can lead to frustration unprepared person.

Automated testing always has both pluses:

·         Greater code coverage;

·         Test is not to be missed;

·         The test contains the same steps;

and disadvantages:

·         Errors are mostly located in creating a script;

·         Scripts do not allow deviations;

·         Scripts may also contain errors.

Thus, after the development of a functional of all the other beams designed for automated scripts it will only give confidence that it is not broke.

Therefore, after running a standard set of tests if there is time and the need for better verification, must necessarily be performed manual exploratory testing. Since that seems suspicious person will never check your computer if it is on is not programmed. Often it also happens that the test fix serious bugs found by users, and automated control, they do not show up in the future.

So how much is needed to automate

Another extreme, in which people rush out and decide to start automation - is:

Myth 8. The more autotests, the better.

The result is a tendency to encode as many manual tests. And best of all. This is wrong. In reality, the amount of automation is determined by the context of the project. What is architecture? What are the requirements to the quality of the customer? What is the budget for the project, including the timing?

Automation begins to benefit only on the products, development and maintenance are quite durable.

Sometimes the effort required to develop a test environment can greatly exceed the manual embodiment, even if you test drive every day. Some test cases it is simply impossible to automate for different reasons:

·         complex algorithmic verification;

·         standard configuration of the stand, you want to change for different tests;

·         third-party software, which creates additional problems;

·         and other reasons…

As I said earlier, both manual and automated testing - a complementary technology. Therefore, as in the present situation is difficult to produce a quality product without automation, and can not do without the manual testing.

Not enough…

This list of myths can still go on, gradually descending from large errors to less serious errors. Which, however, can lead to serious problems.

However, I would like to end on a hopeful note. Therefore, finally, instead of the myths I’d like to lead a series of statements containing the truth about automated testing:

·         Automation can be used in most processes and at all levels of testing.

·         Manual and automated testing - are complementary technologies.

·         Automated testing involves human intervention.

·         Automated testing requires additional investment, but to enhance the quality of the product.

·         Automated testing - is the development (programming).

·         Automated testing ensures deterministic test functionality.

·         Maintenance of automated tests require little additional cost to the active change of system being tested.

 Visit here : Software testing Company