Software Testing Tutorial
- Introduction to Software Testing
- Principles of Software Testing
- Software Development Life Cycle (SDLC)
- Software Testing Life Cycle (STLC)
- Levels of Testing
- Database Testing
- Ad-hoc Testing
- Testing Types
- Test Closure Report Preparation
- Test Report Preparation
- ISTQB Organization and Certification Overviews
7.1 Monkey Testing
Monkey testing is a software testing technique in which the testing is performed on the system under test randomly. The Input data that is used to test also generated randomly and keyed into the system.
Now question arise that why it is called Monkey Testing? Why is this ‘Monkey’ here? Here is the answer.
- In Monkey Testing the tester (sometimes developer too) is considered as the ‘Monkey’
- If a monkey uses computer he will randomly perform any task on the system out of his understanding
- Just like the tester will apply random test cases on the system under test to find bugs/errors without predefining any test case
- In some cases, Monkey Testing is dedicated to Unit Testing or GUI Testing too
Advantages of Monkey Testing:
- New kind of bugs: Tester can have full exposure to implement tests as per his understanding apart from previously stated scenarios, which may give no. of new errors/bugs existing in the system.
- Easy to execute: Arranging random tests against random data is easy way to test the system
- Less skilled people: Monkey Testing can be performed without skilled testers (but not always)
- Less Costly: Requires considerably less amount of expenditure to set up and execute test cases
Disadvantages of Monkey Testing:
- No bug can be reproduced: As tester performs tests randomly with random data reproducing any bug or error may not be possible.
- Less Accuracy: Tester cannot define exact test scenario and even cannot guarantee the accuracy of test cases
- Requires very good technical expertise: It is not worth always to compromise with accuracy, so to make test cases more accurate testers must have good technical knowledge of the domain
- Less bugs and time consuming: This testing can go longer as there is no predefined tests and can find less number of bugs which may cause loopholes in the system
One can consider that Monkey Testing, Gorilla Testing and Ad-hoc Testing are same as there are some similar facts present in all of them but the real fact is that they are different from each other… how?
We will first see the difference between Monkey and Gorilla Testing. First be clear with it to avoid confusion.
Monkey Testing V/s Gorilla Testing:
|Monkey Testing||Gorilla Testing|
|Monkey Testing is performed randomly with no specifically predefined test cases||It is neither predefined nor random|
|Monkey Testing is performed on entire system can have several test cases||Gorilla Testing is performed on specifically few selective modules with few test cases|
|Objective of Monkey Testing is to check for system crash||Objective of Gorilla testing is to check whether the module is working properly or not|
Monkey Testing Vs Ad-hoc Testing:
|Monkey Testing||Ad-hoc Testing|
|Monkey Testing is performed randomly with no specifically predefined test cases||Ad-hoc testing is performed without planning and documentation(test cases and SRS)|
|In Monkey Testing testers may not know what is the system is all about and its purpose||In Ad-hoc Testing tester must understand the system significantly before performing testing|
|Objective of Monkey Testing is to check for system crash||Objective of Ad-hoc testing is to divide the system randomly into subparts and check their functionality|
Types of Monkey Testing:
Monkey Testing is further divided into several categories according its way of implementation, See following diagram for quick idea of it;
- Dumb Monkey: Testers have no idea about system and its functionality, also no assurance about validity of test case.
- Smart Monkey: Tester have precise idea about system its purpose and functionality. Tester navigates through the system and give valid inputs to perform testing.
- Brilliant Monkey: Testers perform testing as per user’s behavior and can specify some probabilities of bugs to be occurred.
Monkey Testing can also be performed for Android even. Monkey Testing may get efficient with the use of tools. Even it can be used to find more bugs like other testing types.
If we use tool for Monkey Testing what could be the general process followed for it? Just have a quick look;
- Like any other testing tool first step is to register your software with the dedicated server
- Make sure you are well prepared with all the necessary references to build a test suite
- Run the built test suit
- ‘Monkey Test’ is the test log file which will be created to record test results
- Keep in mind that the test will go on until system comes to crash point at which the action is recorded into log file
- Finally the test report is shared with concerned person and the test data can be stored and used for future reference
The process of Monkey Testing can be automated even with the use of tools but as it is some sort of new kind of testing introduced and not yet established on industry level these tools have less identity unlike others.
This situation may get changed with coming era of Testing Process then we will have look towards the upcoming impact of Monkey testing and its significant effect on industry standards.
7.2 Pair Testing
A vital software testing technique, pair testing session involves two individuals or team members, who work on a single workstation/keyboard to test various aspects of a software application. This type of testing basically comes under ad-hoc testing and aims at pairing two individuals to work on same software product and on the same machine.
During pair testing, both the individuals are responsible for different tasks that validate the quality as well as the functionality of the software. While one of them executes the process of testing, the other member is responsible for analysing and reviewing the process. Hence, with the assistance of this approach one can rapidly perform thorough software testing, while saving time and money.
Pair Testing Features:
To offer you a better insight into this type of software testing, mentioned below are some of the features of pair testing.
- Pair Testing is also known by the name of Buddy Testing.
- Responsible for generating effective test cases, quickly.
- Time-saving approach.
- Best method to train novice or new testers along with the task of testing, at no-additional cost & time.
- Bridges the gap and increases the coordination between developer and tester.
- Exchange and gain of knowledge, both by the testers and developers.
- Not suitable, when the product needs automation.
- Pair testing can be related to Pair Programming and Exploratory Testing.
What is the need of Pair testing?
The answer to the above question is quite simple. In this process of testing, “Two head is better than one”.
Pair testing involves incoming of ideas, strategies, scenarios, plans, solutions and many other things, which may or may not be exclusive from two different resources. This ensures the wide test coverage along with exchange of ideas and reviews to explore more and unseen defects in the minimal time.
- It speeds up the process of testing.
- Enhances the quality of testing and test result.
- Helps create agile teams that are focused on executing tests rapidly as well as accurately.
- compliance to standards, etc.
- Pair testing can be performed by one tester and developer or a business analyst, test manager, etc.
What combination will work better for the Pair testing ?
Any combination can work effectively and efficiently in pair testing. The possible pairs could be – tester & tester, tester & developer and tester & user. A tester needs to be one of the members of the pair. However, each pair has its own significance. Let’s go through each of these pair.
- Tester & Tester: Pairing of testers will surely bring more inputs and resources to the testing activity. Both of them will be applying their core technical skills, knowledge and tactics, which may or may not be same to produce effective results. Further, this pairing may consists of one novice tester and one experienced tester, thereby providing an opportunity to novice tester to work with the experienced professional to enhance and update his/her skills & knowledge.
- Tester & Developer:This pairing of tester and developer ensures the mutual understanding of each other’s process. This pair is effective as one is aware of the complete development process and may impart knowledge and understanding of the software technicalities to the tester in order to identify and locate defects easily in a shorter period of time. Inclusion of a developer in the pair enables tester to understand the complexities of the software product.
- Tester & User: This pairing is useful in preparation of possible testing scenarios from users- perspective with the help of user and subsequent “execution of these cases by the tester. The importance of this pairing is that software product will be accessed explored by the user who will assist the tester in preparing and executing test-cases to meet the desired expectations & needs of the end users.
How does the Pair works?
This type of testing is based on the methodology where the members of the pair will test the same product by making use of the same resources such as machines or computers. Generally, one works on paper in taking notes preparing scenarios, including reviews and analysis of the test results, while another is responsible for carrying out testing procedures on machines or computer by making use of the other member’s output.
Advantages of Pair Testing:
The process of pair testing is an extremely beneficial one. It offers various advantages to the team as well as the product owner. During pair testing, the team works in an agile environment and execute the process of testing together. Other advantages of pare testing are:
- It promotes knowledge transfer.
- Ensures more angles of the software are tested.
- Improves the quality of the product, as it is validated by two individuals.
- Helps save time and efforts invested in testing.
- Assists in improving interpersonal skills of the involved individuals and breaks down barrier.
- Promotes communication among team members.
- It can be used as an opportunity to train different employees.
- Helps identify the cause of defects without any hassle.
- Allows an expert and a novice to work together and exchange knowledge.
- It does not require any special training.
Challenges of Pair Testing:
Though, pair testing is an extremely beneficial testing technique, there are few challenges that are encountered by the team while executing its process. These challenges are:
- As the bugs are reported after the culmination of the testing process, it becomes time consuming and tedious for the team to report and rectify them.
- It becomes difficult for the team to define the portions of the test that has been performed.
- At times, it is difficult to find two individuals who are willing to test the software together.
- This type of testing cannot be used during test automation.
Difference between Pair Testing and Pair Programming?
|Pair Testing||Pair Programming|
|Pair testing is an agile software testing technique, which is also known as Buddy Testing.||Pair programming is an agile software development technique.|
|The process is performed by any two individuals, whose roles are interchangeable.||The process is performed by two programmers, who switch their roles frequently.|
|The team can consist of a tester, business analyst, developer, programmer, etc.||The team members are known as driver and observer/navigator.|
|The team is focused on testing the software and ensuring its compliance with specified requirements.||The team is focused on designing and analyzing the code as well as the product.|
|It helps narrow down the root cause of aproblem, while continuously testing it.||It defines how good a software works.|
|Does not require internal knowledge of the structure.||Pair programming is a part of Extreme Programming.|
7.3 Exploratory Testing
Exploratory testing, is all about discovery, investigation and learning. It emphasizes on personal freedom and responsibility of the individual tester. Test cases are not created in advance but testers check system on the fly. They may note down ideas about what to test before test execution. The focus of exploratory testing is more on testing as a “thinking” activity.
Under scripted testing, you design test cases first and later proceed with test execution. On the contrary, exploratory testing is simultaneous process of test design and test execution all done at the same time.
Scripted Test Execution is usually a non-thinking activity where testers execute the test steps and compare the actual results with expected results. Such test execution activity can be automated does not require much cognitive skills.
Though the current trend in testing is to push for automation, exploratory testing is a new way of thinking. Automation has its limits.
Scripted vsExploratory Testing:
|Scripted Testing||Exploratory Testing|
|Directed from requirements||Directed from requirements and exploring during testing|
|Determination of test cases well in advance||Determination of test cases during testing|
|Confirmation of testing with the requirements||Investigation of system or application|
|Emphasizes on prediction and decision making||Emphasizes on adaptability and learning|
|Involves confirmed testing||Involves Investigation|
|Is about Controlling tests||Is about Improvement of test design|
|Like making a speech – you read from a draft||Like making a conversion – its spontaneous|
|The script is in control||The tester’s mind is in control|
- Is not random testing but it is adhoc testing with purpose of find bugs
- Is structured and rigorous
- Is cognitively (thinking) structured as compared to procedural structure of scripted testing. This structure comes from Charter, time boxing etc.
- Is highly teachable and manageable
- It is not a technique but it is an approach. What actions you perform next is governed by what you are doing currently
Exploratory Test Preparation:
Exploratory test preparation goes through following 5 stages detailed below and it is also called session based test management (SBTM Cycle):
- Create a Bug Taxonomy (classification)
- Categorize common types of faults found in the past projects
- Analyze the root cause analysis of the problems or faults
- Find the risks and develop ideas to test the application.
- Test Charter
- Test Charter should suggest
- what to test
- how it can be tested
- What needs to be looked
- Test ideas are the starting point of exploration testing
- Test charter helps determine how the end user could use the system
- Test Charter should suggest
- Time Box
- This method includes pair of testers working together not less than 90 minutes
- There should not be any interrupted time in those 90 minutes session
- Time box can be extended or reduced by 45 minutes
- This session encourages testers to react on the response from the system and prepare for the correct outcome
- Review Results:
- Evaluation of the defects
- Learning from the testing
- Analysis of coverage areas
- Compilation of the output results
- Compare the results with the charter
- Check whether any additional testing is needed
During exploratory execution, following needs to be done:
- Mission of testing should be very clear
- Keep notes on what needs to be tested, why it needs to be tested and the assessment of the product quality
- Tracking of questions and issues raised during exploratory testing
- Better to pair up the testers for effective testing
- The more we test, more likely to execute right test cases for the required scenarios
It is very important to take document and monitor the following
- Test Coverage – Whether we have taken notes on the coverage of test cases and improve the quality of the software
- Risks – Which risks needs to be covered and which are all important ones?
- Test Execution Log – Recordings on the test execution
- Issues / Queries – Take notes on the question and issues on the system
Smarter exploratory testing finds more errors in less time.
- This testing is useful when requirement documents are not available or partially available
- It involves Investigation process which helps find more bugs than normal testing-
- Uncover bugs which are normally ignored by other testing techniques
- Helps to expand the imagination of testers by executing more and more test cases which finally improves productivity as well
- This testing drill down to smallest part of application and covers all the requirements
- This testing covers all the types of testing and it covers various scenarios and cases
- Encourages creativity and intuition
- Generation of new ideas during test execution
- This testing purely depends on the tester skills
- Limited by domain knowledge of the tester
- Not suitable for Long execution time
Challenges of Exploratory Testing:
There are many challenges of exploratory testing and those are explained below:
- Learning to use the application or software system is a challenge
- Replication of failure is difficult
- Determining whether tools need to be used can be challenging
- Determine the best test cases to execute can be difficult
- Reporting of the test results is challenge as report doesn’t have planned scripts or cases to compare with the actual result or outcome
- Documentation of all events during execution is difficult to record
- Don’t know when to stop the testing as exploratory testing has definite test cases to execute.
When use exploratory testing?
Exploratory testing can be used extensively when
- The testing team has experienced testers
- Early iteration is required
- There is a critical application
- New testers entered into the team
7.4 Buddy Testing
This is a technique where two team members work on the same Built (software application) on the same machine. One of the team member will work with the systems(with keyboard and mouse) and another should make notes and scenarios.
For this concept you may find in the internet or from some books that, one team member should be a tester and another should be from Development team or a business analyst or both the team members should be from testing team where one should be experienced and other should be non- experienced to share their views etc etc. In some articles Pair testing and Buddy testing defined as two different concept as Pair testing contains two testers and buddy testing requires one tester and one developer. But in real, the naming does not matters as the goal and methodology is same.
But as per my view, when a tester and a developer work together to ensure the quality of a product then the efficiency rises a bit, even if the time is less. Here there is no documentation is needed like Test Cases, Test Plan or Test Scenarios.
This technique is used under following scenarios:-
- When the specification and requirement are not clear.
Sometimes, the whole requirement and specifications of the product, asked by the client is not clear and the deadline is less than one month or so. The specifications like field validation, boundary value range, navigation, etc. Lack of proper specification confuses the tester in the account of Usability issues, functional issues. So, the presence of another team member as Developer or a experienced tester may help in greater extent.
- The dead line is near.
Of course, as QA we all encounter this type of scenario in our professional life where the development took more time and when the product comes for testing, there is only one or two days in hand, for complete QA. In this case the pair/buddy testing will come in handy.
- The team is new, where the quick knowledge on the product is needed.
When there is a new member in the team (developer or tester), quick knowledge on the existing or requested product is needed. The domain knowledge, technology going to be used, end user type etc also matters. By this technique of testing, a new tester can get a hold on the functional flow of the product. And a new developer can pin point the common mistake areas which may help him do better next time.
When NOT to use Pair/Buddy testing:-
- The product needed or should go under automation testing, another team member is not needed.
- If the character set , behavior or attitude of the two members doesn’t match then the testing shouldn’t be done in pair as it may create conflict.
- If there is only structured test cases , then only one tester will suffice.
And in many cases where one man power is sufficient, the pair testing is not needed.
Advantages of Pair/Buddy testing:-
- Here if the developer and a tester are paired together then the developer can learn a better approach to the built and the tester also can learn the functionality from better angle.
- There are many random bugs occurs during testing which are tough to reproduce, but if a tester and developer are paired together then developer can catch the link of this type of random bugs at once. This may save a lot of time. The tester doesn’t have to waste time in reproducing the issue and doesn’t need to capture screenshots. Direct and immediate communication leads to the faster fixation.
- Both developer and tester will work to their best due to the presence of each other.
- Developer can mark his/her mistakes, so that they can be efficient and optimized in future.
- Work load will be less in presence of another team member’s help, so, the tester can think clear and use maximum scenarios in to his/her benefit.
- Work will be fun and enjoyable.
Getting started with Pair Testing/Buddy Testing (How to set up):-
- First choose a developer/tester who you trust or you know him better or his attitude and character set matches with you. It is a very vital point of Pair Testing as it has high impact on the efficiency.
- Make a plan of approach, have a short meeting before and make schedule. A good start without any holding back, is needed.
- As both of the team member have to use one machine, then the environment should be chosen carefully. Both the members should feel comfortable and free to talk to each other without any disturbance nor creating disturbance for others.
- Project also does matters while applying this type of testing. The project should be of
appropriate size. Pair testing shouldn’t be done efficiently on very big or whole project. As I learned “Pair testing works well when testing new functionality and when both participants have been working on the project since inception” .
7.5 Defect Seeding
When it comes to testing a software product, software engineers adopt a range of effective and advantageous techniques and methods, which enhance its accuracy and functionality. From indulging in peer reviews to conducting isolation testing, testers go to great heights to validate the quality, effectiveness, reliability, security, etc. of these products. Another important software testing technique that is implemented by software testers, to ensure that the quality of the software as well as to guarantee that it performs uniformly under various conditions and situations, is defect seeding.
What is Defect seeding ?
Defect seeding is also known as bebugging, errors guessing and fault injection, is an important testing methodology used by software testers to improve the quality and functionality of the software under test. While testing the software product, with the assistance of bebugging, the team of testers intentionally infuse known bugs, defects, and faults in the system and monitor its rate of detection and rectification.
Initially used by software programmers to measure test coverage, defect seeding has since gained rapid popularity among software testers, as it allows them to estimate the number of bugs and faults remaining in a software product. The process involves random infusion of bugs in the
program’s source code, which are then tested or detected by the programmers or testers. The percentage of the known bugs not found by the team, offers an indication of the real bugs remaining in the software that are yet to be detected by the concerned testers.
Reason for Defect Seeding:
As mentioned above, the process of defect seeding helps software testers as well as other members related to the project monumentally. It not only assists them in finding defects in the software product, but also enables them to improve its quality and effectiveness. Other reasons for implementing defect seeding are:
- It helps determine the reliability of the executed test sets and test suites.
- Improves the quality and effectiveness of the product.
- Helps find undetected defects, bugs, and faults in the system, which can hinder its expected performance and functionality.
- Allows the team to monitor the rate of defect detection and removal.
Techniques for Defect Seeding:
The process of defect seeding can be termed as a type of fault injection, which helps determine the number of defects still present in the software by adding new and known bugs into it. Moreover, this can be implemented with the assistance of two important techniques, which are:
- Compile Injection: This technique of defect seeding is used to inject stimulated faults in source code of the system, which helps the team to determine the various faults or defects left in the system.
- • Runtime Injection: Another important technique for defect seeding, runtime injection uses software triggers to inject defects and faults into a running software system. With the help of runtime injection, faults can be implemented in a number of ways, which are:
- Time based triggers.
- Interrupt based triggers.
- Corruption of memory space.
- Syscall interposition techniques.
- Network level fault injection.
Prerequisites of Defect Seeding:
Before implementing the process of defect seeding, it is vital for software testers to accumulate various necessary documents and components, which are useful in executing the process of defect seeding. Therefore, to help you with the task of collecting these components, here is a list of all the prerequisites needed before the execution of defect seeding:
- Important lessons learnt from past releases and testing activities.
- History of previous defects.
- Application’s risk reports.
- Variety of data used for other types of testing.
- Previous test results.
- Review checklist.
- Production tickets.
- Application UI.
- Other important reports and documents prepared by software developers and testers.
Prominent Defect Seeding Tools:
Defect seeding is a process that can be executed by software testers manually or with the assistance of various automated tools that simplify the process and make sure that they get accurate results.
Hence, following are some of the most prominent defect seeding tools used by software testers around the world.
- • Bstorm: It is a software security testing tool, a product by Beyond Security, is used as a framework for dynamic security testing of products. QA professionals and network administrators can perform testing. It allows network administrators to ensure security of networked applications before they are ready to be deployed. It is popular due to its robust and easy to use interface. Business organisations, be it any size can integrate beStorm into their software QA process. It is considered as a black box testing tool known to be used commercially, does not use source code and simply performs checks beginning with common failure issues and gradually moving onto the next.
- Holodeck: It is a testing tool by Security Innovation. It uses fault simulation techniques to duplicate real-world application and system errors. Testers and developers can work in a controlled environment to analyse and debug error handling code. It thus helps to monitor the reliability of an application. It creates real-world scenarios like memory leaking, bad registry data, corrupt files or corrupted network packets. Holodeck logs and monitors the system calls. This in turn helps to introspect into even low level machine interactions, which allows one to track bug- generating events.
- • Xception: This project emphasises on the advanced debugging and performance monitoring features that are present in modern processors, to check how the system reacts to fault injections. This tool is widely used in various platforms – Online Transaction Processing System(OLTP), COTS(Commercial – off-the-shelf) system. Xception supports processors ranging from PowerPC, Intel Pentium, and SPARC(scalable processor architecture) based platforms that run on Windows/NT 2000, Linux OS, and a collection of real-time systems- LynxOS, SMX, RTLinux, ORK. There are many breakpoint registers in a processor. When a fault is injected, the processor executes the fault trigger related to manipulation of data. The processor runs at its own pace even at the injection of the fault, that is, the fault injection is treated as a small exception routine. By performance monitoring we intend to check the behaviour of the system in terms of number of clock and memory read and write cycles, and the instructions executed. This process continues until an error is caught.
- The Mu Service Analyser: Mu Security is a tool intended to track the network traffic and attack patterns. The Internet world is prone to attacks, hence Mu-4000 offers a range of testing options like mutated traffic, published vulnerabilities, Dos attacks. It has the option to choose any protocols from – TCP, UDP, ICMP to launch Dos attacks. It has the facility to record the system’s progress and records the response time during an attack. When a fault is traced, the Mu analyser facilitates tracking the traffic and the packets to determine what caused the issue.
- Exhaustif: It is a SWIFI injection tool that checks for fault occurrence by injecting faults into the system, in a distributed environment. It is a fault tolerance verification and validation tool for embedded software systems. Exhaustif has two parts : EEM (Exhaustif Executive Manager) and FIK (Fault Injector Kernels).
- EEM: It helps to perform data analysis, post fault injection. It is a GUI based Java application that uses a SQL database to record the test results thus obtained from the System Under Test. It basically performs memory corruptions using temporary triggers and uses optimized routine interceptions, with minimum time.
- FIK: This comes under EEM and checks fault injections in applications that run on different operating systems.
Benefits & Drawbacks of Defect Seeding/Bebugging/Error Guessing/Fault Injection:
The process of defect seeding, which is also referred as error guessing and fault injection is garnering attention of software engineers around the world because of the various advantages offered by it.
However, no testing technique can be perfect, and likewise, defect seeding also has few drawbacks that needs to be considered before implementing it on the software product. Following are the multifarious benefits of defect seeding and the few drawbacks it offers.
- It does not impact the software under test.
- It saves time and efforts of testing.
- Does not require any expensive hardware.
- Can be used to test applications as well as operating systems.
- Has higher observability and controllability of the system’s behaviour in the presence of faults.
- Allows testers and developers to easily identify the weakest points of the software product.
- It is a flexible testing approach.
- Increases the probability of finding defects in the system.
- Requires large development efforts.
- The fidelity of this testing technique is highly dependent on the accuracy of the models used.
- Intentional infusion of bugs and defects can impact or damage the performance or quality of the software.
The following points summarize the topic above:
- Monkey testing is a software testing technique in which the testing is performed on the system under test randomly.
- Monkey Testing can also be performed for Android even.
- Pair testing involves incoming of ideas, strategies, scenarios, plans, solutions and many other things, which may or may not be exclusive from two different resources.
- The focus of exploratory testing is more on testing as a “thinking” activity.
- The process of defect seeding can be termed as a type of fault injection, which helps determine the number of defects still present in the software by adding new and known bugs into it.