Introduction
PURPOSE OF PLAN:
Software testing can be called an investigation done to provide stakeholders with information about the quality of the SUT (Software Under Test). It is “the process of executing a program with the intent of finding errors”. Often after the investigation, we come up with certain portions within the code of the software that need to be changed or edited. In these cases, we need to make sure that the initial functionality of the software is not lost after making the changes. For this we have “Regression Testing”.
Regression Testing is the “process of testing changes to computer programs to make sure that the previous programming still works with the new changes”. Regression testing is an absolutely normal part of the PDP (Program Development Process) and, in bigger companies, is done by code testing specialists. The programmers of the testing department create scenarios for testing the code and exercises that will test new units of code after they have been written. All these test cases together form what is known as the test bucket. Regression testing also includes the running of old test cases against the new version to make sure that all the old capabilities still work. This is always done before the newer version of the software is released. Sometimes the changes done may cause the test cases to not run successfully and the sole reason why they might not work is because changing or adding new code to a program can easily bring new errors into code that was never intended to be changed.
A metaheuristic can be defined as “a heuristic or higher-level procedure which is designed to find, generate, or select a heuristic (partial search algorithm) that may provide an optimal or near-optimal solution to an optimization problem, especially with incomplete or imperfect information”. Recently, a lot of metaheuristic techniques have been discovered which can be used to assist in regression testing. My motive for this project is to develop a new method for prioritizing the most optimal test cases from the TEST SUITE (all the test cases that have been generated for the software under test) since regression testing may not be very beneficial at times since it can be extremely time consuming considering the number of test cases possible for a particular software. Since this is initial project that I am submitting, this only includes some information about the different types of problems, meta-heuristics algorithms and mostly about different methods of software testing. Software testing is an activity to check whether the actual results match the expected results and to ensure that the software system is fault-free. It plays a crucial role in the Software Development Life Cycle (SDLC) as it points out the defects and errors that were made during the development phase. It helps to ensure customer satisfaction, good quality and effectiveness of the product. In order to minimize the number of test-cases which find the maximum number of faults, certain optimization techniques need to made use of.Metaheuristic optimization deals with optimization problems using metaheuristic algorithms. Two major components of any metaheuristic algorithms are: “intensification and diversification”, or “exploitation and exploration”.
Diversification means to generate or produce diverse solutions so as to explore the search space on a global scale, while intensification means to focus the search in a local region knowing that a current good solution is found in this region. To improve the rate of algorithm convergence we should be find a good balance between intensification and diversification during the selection and prioritization of the best solutions to improve the rate of algorithm convergence. Many metaheuristic algorithms exist in literature and some algorithms have been used in Software Testing techniques such as swarm intelligence, ant colony optimization (ACO) and particle swarm optimization (PSO). Some techniques are a combination of the afore-mentioned techniques and hence are also known as ‘hybrid’ techniques. For e.g. a combination of Genetic algorithm and Particle Swarm Optimization technique.
BACKGROUND INFORMATION/ AVAILABLE ALTERNATIVES:
Sometimes regression testing takes up an insignificant amount of time owing to the large number of test cases generated for a particular SUT which is rather unfortunate because in the world of testing, time is one of the most vital resources. Hence, the topic of my project. Since testing in general takes up a lot of time, there’s a genuine need for optimization and in my minor project, I plan to take up any NP – Hard or NP – Complete problem and optimize it using a metaheuristic algorithm. This is valid as far as my minor project is concerned. But, for my major project I have decided to perform prioritization of test-cases generated for a problem using a new metaheuristic or hybrid metaheuristic algorithms on the basis of certain factors such as statement coverage, time consumed, number of faults detected, path coverage and so on. The Prioritization would cause the most optimal test-cases to come up so that they can tested against the code, thus reducing a lot of effort from the tester’s point of view.
There are no alternatives to this project as such but in case of failure to implement a particular algorithm to optimize and carry out the prioritization of the most optimal test-cases from the Test Suite, I always have numerous other algorithms to implement. So far in my minor project I have tried optimizing the Travelling Salesman Problem using the Tabu Search Metaheuristic algorithm.
PROJECT GOALS AND OBJECTIVES:
As mentioned above, my minor project aims at optimizing a particular NP – Hard or NP – complete problem using a metaheuristic algorithm, in a programming language of my choice. Later, in my major project, different metaheuristic techniques will be used for prioritizing the test-cases generated for a particular problem using algorithms such as PSO (Particle Swarm Optimization), ACO (Ant Colony Optimization), BFOA (Bacterial Foraging Optimization Algorithm), Tabu Search algorithm, etc.
The sequence of project goals that I have achieved and want to achieve in future are as follows:
1. August 2015 – Decide the project topic after looking into various domains or fields.
2. September, October 2015 – Carry out intensive domain-specific study inclusive of study related to testing, metaheuristics, problems etc. and select a NP – hard or NP -complete problem to be optimized. Also start working on a research paper that I have decided to write on my topic.
3. November 2015 – Carry out the implementation of the optimization process and produce a working result for the same so as to ensure the completion of my minor project. Finalize the paper.
4. December 2015 – Start working on the major project which aims at prioritizing test cases on the basis of certain factors using some metaheuristic algorithms such as Tabu search, ACO, BCO, PSO etc. And also select the NP – hard or NP – complete problems for which test cases are to be designed.
5. January 2015 – Design test cases for a problem which will later on go through the process of prioritization. Also select the metaheuristic algorithms that will be implemented. Start working on another research paper, if possible.
6. February 2016 – Implement the algorithms that were previously selected using a programming language such as Java, C++, MATLAB etc.
7. March, April 2016 – Carry out the prioritization process for the test-cases designed initially and check whether proper results are being produced or not. And complete the project report simultaneously.
SCOPE
SCOPE DEFINITION:
As far as project planning is concerned, defining the project scope is the most vital step. An unimpressive job done while defining the project scope makes the project scope management almost impossible during execution.
The actual purpose of the project scope definition is to clearly jot down or describe the limitations or boundaries of your project. Hence, the defined scope of the project is also included into the contractual agreements between the client and the SP (service provider).
The scope of the project is as follows:
The ultimate objective of this project is to prioritize test-cases for a particular problem by making the use of metaheuristic algorithms. The algorithms may be an already pre-defined one or an algorithm obtained as a combination of two or more metaheuristic algorithms. Currently in my minor project, I have gone through numerous study materials, websites and tutorials related to software testing, metaheuristic algorithms, problems and how to optimize them.
The scope of my minor project limits itself to the use of metaheuristic algorithms to optimize a particular NP –Hard or NP – Complete problem such as Travelling Salesman Problem, Knapsack Problem etc. while the scope of my major project also includes the prioritization of optimal test-cases (optimal nature of test cases decided on the basis of factors such as statement coverage, path coverage, number of faults detected, time taken etc.) from a test suite generated for the problem which we have worked on after the optimization of the same has been done with.
PROJECTED BUDGET:
There is no monetary budget associated with my project apart from some trivial costs of certain stuff such as paper publishing etc. which need to be encountered later. As of now, I have certain materials or software that I have decided to use in order to complete my project.
MATERIALS:
1. C++:
C++ is a middle-level programming language since it includes features of both high-level and low-level languages which was developed by BjarneStroustrup starting in 1979 at Bell Labs. C++ runs on a variety of platforms, such as Windows, Mac OS, and the various versions of UNIX. C++ is “a statically typed, compiled, general-purpose, case-sensitive, free-form programming language that supports procedural, object-oriented, and generic programming”. C++ is a superset of C, and that virtually any legal C program is a legal C++ program.
Object-Oriented Programming
C++ fully supports object-oriented programming, the four main parts of which are:
• Encapsulation
• Data hiding
• Inheritance
• Polymorphism
Standard Libraries
Standard C++ consists of three important parts:
• The core language providing all the building blocks such as identifiers, variables, data types and literals, etc.
• The C++ Standard Library giving a large set of functions to manipulate files, strings, etc.
• The Standard Template Library (STL) giving a large set of methods to manipulate data structures, etc.
The ANSI Standard
The ANSI standard ensures that C++ is portable which basically means that the code we write for Microsoft’s compiler will compile without errors, using a compiler on any other platform such as Mac, UNIX, a Windows box, or an Alpha.
The ANSI standard has been stable for some time and major C++ compiler manufacturers support the ANSI standard.
Uses of C++
• C++ is used by hundreds of thousands of programmers in essentially every application domain.
• C++ is being highly used to write device drivers and other software that rely on direct manipulation of hardware under realtime constraints.
• C++ is widely used for teaching and research because it is clean enough for successful teaching of basic concepts.
• Anyone who has used either an Apple Macintosh or a PC running Windows has indirectly used C++ because the primary user interfaces of these systems are written in C++.
2. JAVA:
“Java was conceived by James Gosling, Patrick Naughton, Chris Warth, Ed Frank and Mike Sheridan at Sun Microsystems Inc.in 1991. This language was initiallycalled “Oak” but was renamed as “Java” in1995.”
“The main properties of the Java, which made Java so popular, are asfollows:
1. Simple
2. Secure
3. Portable
4. Object-Oriented
5. Robust
6. Multithreaded
7. Architecture-Neutral
8. Interpreted
9. High performance
10. Distributed and Dynamic
The most striking feature of the language is that is a platform-neutral language. Java is first programming language that is not tied to any particular hardware or operating system. Programs developed in Java can be executed anywhere on any system”.
Besides the above mentioned advantages, there are mainly 2 categories of programs that can be developed in Java such as:
1. Applets
2. Applications
Applets are Java programs that are made generally to work on the Internet. They can be made to run through a Java enabled browser, some examples of such being Netscape or Internet Explorer. It should be embedded within a webpage or an HTML file. When the webpage or an HTML file is displayed in a browser, the applet is loaded by itself and executed.
On the other hand, an Application is just a program that runs on the computer. Creating an application in Java is the same as doing so in any other computer language. The application can either be based on Command Line Interface or Graphical User Interface.
Key Features
“The key that allows Java to solve both the security and the portability problems just described is that the output of a Java compiler is not executable code. Rather, it is Byte code. Byte code is a highly optimized set of instructions designed to be executed by the Java runtime systems, which is called the Java Virtual Machine (JVM). That is, in its standard form, the JVM is an interpreter for Byte code. This may come as a bit of surprise.
Translating a Java program into a byte code helps and makes it much easier to run a program in a wide variety of environments. The reason is straightforward only the JVM needs to be implemented for each platform. Once the runtime package exists for a given system, any Java program can run on it. Remember, although the details of the JVM will differ from platform to platform, all interpret the same Java Byte code.”
Java Environment
“Java environment includes a large number of development tools and hundreds of classes andmethods.ThedevelopmenttoolsarethepartofthesystemknownasJavaDevelopment Kit (JDK) and the classes are methods are part of the Java standard library (JSL), also known as the Application Programming Interface(API).”
Java Development Kit
“The Java development kit comes with a collection of tools that are used for developing and running Java programs. They include:
1. Applet Viewer (for viewing Java Applets)
2. Javac (JavaCompiler)
3. Java (Javainterpreter)
4. Javap (JavaDisassembler)
5. Javah (for C headerfiles)
6. Javadoc (for creating HTMLdocuments)
7. Jdb (JavaDebugger)”
Application Programming Interface
“The Java standard library includes hundreds of classes and methods grouped intoseveral functional packages. Most commonly used packages are:
• Language support package:
A collection/group of classes and methods required for implementing basic features of java.
• Utilities package:
A collection/group of classes to provide utility functions such as date and time functions.
• Input/output package:
A collection/group of classes required for input & output manipulations.
• Networking package:
A collection/group of classes for communication with other computers via Internet.
• AWTpackage:
The abstract window toolkit package contains classes that implements platform independent graphical user interface.
• Applet package: This includes a set of classes that allows us to create Javaapplets.”
Fig 1: Java Logo
3. NETBEANS:
NetBeans is a SDP (software development platform) which was written in Java. The NetBeans Platform allows all of its applications to be created from a group of modules which are nothing but software components. The NetBeans IDE is mainly meant for development in Java language, but is also capable of supporting other languages, such as PHP, C/C++ and HTML5.
NetBeans is cross-platform i.e. it can run on multiple platforms such as Microsoft Windows, Linux, Mac OS X, Solaris and other platforms supporting a compatible JVM.
Fig 2. Netbeans IDE
NetBeans IDE is an IDE ( integrated development environment) which is basically open source. NetBeans IDE supports development of all Java application types (Java SE (including JavaFX), Java ME, web, EJB and mobile applications) out of the box. Among other features are an Ant-based project system, Maven support, refactorings, version control (supporting CVS, Subversion, Git, Mercurial and Clearcase).
Modularity: All the functions of the IDE are provided by modules. Each module provides a well defined function, such as support for the Java language, editing, or support for the CVS versioning system, and SVN. NetBeans contains all the modules needed for Java development in a single download, allowing the user to start working immediately. Modules also allow NetBeans to be extended. New features, such as support for other programming languages, can be added by installing additional modules. For instance, Sun Studio, Sun Java Studio Enterprise, and Sun Java Studio Creator from Sun Microsystems are all based on the NetBeans IDE.
4. JUNIT
JUnit is “a UTF (unit testing framework) for the Java programming language and so far it has been immensely important in the development of test-driven development, and is also one of a family of unit testing frameworks collectively, known as xUnit that originated with SUnit.” JUnit publicizes the idea of “first testing then coding”, the emphasis of which lies on setting up the test data for a code which is tested first and then implemented. This approach is like “test a little, code a little, test a little, code a little…” which increases the productivity and efficiency of a programmer and also increases the stability of program code that reduces programmer stress and the time spent on debugging.
Features
The features of JUnit are as follows:
• JUnit is “an open source framework which is used for writing & running tests”.
• Provides “Annotation to identify the test methods”.
• Provides “Assertions for testing expected results”.
• Provides “Test runners for running tests”.
• JUnit tests allow you to write code faster thus increasing efficiency and quality.
• JUnit is very simple and consumes less time.
• JUnit tests “can be run automatically and they check their own results and provide immediate feedback. There’s no need to manually comb through a report of test results”.
• JUnit tests “can be organized into test suites containing test cases and even other test suites”.
• Junit “shows test progress in a bar that is green if test is going fine and it turns red when a test fails”.
Fig 3. JUnit Logo
PROJECT MANAGEMENT APPROACH
TIMELINE:
MONTH – YEAR GOALS TO BE ACHIEVED STATUS
August – 2015 Decide the project topic after looking into various domains or fields.
DONE
September, October – 2015 Carry out intensive domain-specific and select a NP – hard or NP -complete problem to be optimized. And also start working on my research paper that I have decided to write regarding my topic.
DONE
November – 2015 Carry out the implementation of the optimization process and produce a working result for the same so as to ensure the completion of my minor project. Finish the paper.
DONE
December – 2015 Start working on my major project which aims at prioritizing test cases on the basis of certain factors using some metaheuristic algorithms. And also select the NP – hard or complete problems for which test cases are to be designed.
PENDING
January – 2016 Design test cases for a problem which will later on go through the process of prioritization. Also select the metaheuristic algorithms that will be implemented. Start working on another research paper if possible.
PENDING
February – 2016 Implement the algorithms that were previously selected using a programming language
PENDING
March, April – 2016 Carry out the prioritizationprocess for the test-cases designed initially and check whether a proper result is being produced or not. Complete the project report simultaneously
PENDING
Fig 4: Timeline for Project Management
ROLES AND RESPONSIBILTIES:
PROJECT METHODOLOGY:
The methodology that has been followed for the project starts with me indulging in three main things:
• Web References
• Literature Survey
• Tutorials
Web References, Literature Survey and Tutorials – all three of these have one thing in common i.e. the internet.
By web references, we mean the research papers or other materials that have acted as a reference for the project. These references have helped me to identify my project and carry on with it.
Literature Survey on the other hand refers to the study or personal research that we have done in order to gather handsome amount of knowledge about the domain of our project.
Tutorials are nothing but materials that are meant to help us get well-versed with a particular software that we are about to use during our project.
• Then I shall proceed with the extraction of domain-related text as in matter related to our domain or project which might be relevant or useful to the project.
• After this, I plan to continue with an intensive study related to the domain of work so as to ensure that I gather as much knowledge as possible about testing, metaheuristic algorithms and other problems.
• Then finally I’ll take up a particular problem such as TSP, Job Shop Scheduling, Knapsack Problem etc., and implement it using a programming language such as Java, C++.
• After the implementation has been carried out, I will select a metaheuristic algorithm such as PSO (Particle Swarm Optimization), ACO (Ant Colony Optimization), BFOA (Bacterial Foraging algorithm), Tabu Search etc. that I think is suitable enough to optimize the problem that I have selected.
• Then I will optimize the problem code using the algorithm.
• Finally after optimization, the prioritization of test-cases from the test suite is done in order to ensure greater number of fault detection in less number of test cases.
RISK ASSESSMENT
PROJECT RISK ASSESSMENT:
The risks associated with the project chosen are rather low-level. This is because there are no risks of natural hazards or grave accidents as such. Whatever risk is there is related to the implementation of algorithms and the time constraints.
Risk Risk Level
L/M/H Likelihood of Event Mitigation Strategy
1. Incompletion of a particular module or not achieving a goal within the allotted time span.
M Less likely The modules,next to the current one which has taken up extra time is carried out with extra efficiency to reduce time consumption as much as possible
2. Loss of data due to certain issues H Less likely Even if this is less likely to happen, the risk itself is extremely critical since loss of data because of any reason will cause us to start working from the initial level which is undesired. So DATA BACKUP is must to help in case of unwanted data loss.
3. Meeting with an accident causing a lot of physical damage. M Less Likely In such a case there is no mitigation strategy. Only patience will help and once we are back to normal we should work harder to achieve the goals within given time span.
4. Damage to device such as laptop, PC making it unable for us to work on it L Less likely In this case, one of the mitigation strategies is to keep a backup system which is too expensive. Hence one can only wait for the system to be repaired and then resume working.
Fig 6: Risk Assessment Table
Literature survey
Software testing
“Testing is the process of executing a program with the intent of finding errors.”
Testing identifies the faults, whose removal increases the potential reliability of the software as well as its quality. Therefore, testing basically is the measurement of software’s quality. We measure the quality of a software by testing the factors such as correctness, reliability, reusability, testability and so on. Software Testing consists of series of activities carried out methodologically to help certify a particular software product. These activities (stages) constitute the Software Testing Life Cycle (STLC). After the end of each phase of the STLC, we get a set of deliverables (which will be seen in the portion describing the STLC). These deliverables are nothing but a set of documents produced at the end of each phase.
Testing Fundamentals:
a. Verification
Verification refers to “the set of activities that ensure that software correctly implements a specific function.” According to Boehm, verification answers the question“Are we building the product right”?
b. Validation
Validation refers to “a different set of activities that ensures that the software that has been built is traceable to customer requirements”. According to Boehm, validation answers the question“Are we building the right product”?
OBJECTIVES OF TESTING:
• Testing is a process of executing a program with intent of finding an error or making a judgment about its quality.
• A good test is one that has a high probability of finding an error that hasn’t been discovered as of yet.
• A successful test is one that uncovers or finds an undiscovered error.
• Testing should also suggest modifications if required, thus adding value and improving the entire procedure.
• The objective is to design tests that systematically uncover different classes of errors without consuming too much of time and effort.
• Demonstrating that the software application appears to be working as required by the specification.
• Meeting performance requirements.
• Software reliability and software quality based on the data collected during testing
SOFTWARE TESTING LIFE CYCLE:
Fig 7: Software Testing Life Cycle
The steps included in the software testing life cycle are as follows:
1. Requirement Analysis
In this phase, “the team analyzes and studies the requirements, and then has brain storming sessions with other teams and try to find out if the requirements are testable or not. This phase helps to identify the scope of the testing. If any feature is not really testable, one should communicate it during this phase so that the mitigation strategy can be planned”.
Activities:
• Identify the types of tests that need to be performed.
• Gather details about the priorities and focus.
• Preparation of the Requirement Traceability Matrix (RTM).
• Identify the details about the test environment.
• AFA (Automation feasibility analysis) is carried out if required.
Deliverables:
• RTM
• Automation feasibility report. (if AFA is carried out)
2. Test Planning
“Also known as the Test Strategy phase, in this phase, the effort and cost are estimated for the project and the TP (Test Plan) is prepared and finalized”.
Activities:
• Write down and prepare the test strategy document for different types of testing.
• Selection of test tool
• Estimation of test effort and cost.
• Determining the roles and responsibilities
• Resource planning
Deliverables:
• Test plan /strategy document.
• EED ( The effort estimation document).
3. Test Case Development
“This phase basically includes the making, verification and rework of the generated test cases & test scripts. Test data is spotted out and is checked and reworked simultaneously”.
Activities:
• Design test cases, automation scripts
• Review and baseline test cases and scripts
• Create the test data (In case of an available Test Environment)
Deliverables/Documents:
• The test cases/scripts
• The test data
4. Test Environment Setup
This phase might be carried out simultaneously with the Development Stage and in this phase the officials set up the conditions under which the work product is meant to be tested.
Activities:
• Understand the environment set-up required and then, prepare the same.
• Setup the test Environment as well as the test data.
• Carry out smoke test on the environment build.
Deliverables:
• Environment prepared in this phase.
• Results of the smoke test.
5. Test Execution
While this phase goes on the test team will performs the testing based on the TP (test plans) and the TCs prepared initially. Bugs or errors will be reported back to the development team for correction and retesting will be performed.
Activities:
• Execute the tests as per the plan
• Document the test results
• Keep record of any defects in case of failed cases
• Map the defects to the test cases present in the Matrix created earlier.
• Retesting of the defect fixes so as to avoid any errors in the initial piece of code.
• Find out the defects to closure.
Deliverables:
• The whole RTM along with the execution status.
• Test cases updated with the obtained results.
• Reports regarding the defects.
6. Test Cycle Closure
Testing team will gather together to discuss and analyse testing artifacts to identify strategies that are meant to be implemented in future the basic idea of which is to eliminate the process hold-ups so as to avoid any issues in the future which is capable of adversely affecting the product obtained.
Activities:
• Evaluate CCC (cycle completion criteria) which depends on Time, Cost, and Coverage etc.
• Develop the test metrics based on the criteria given above.
• Document what has been learnt so far while this project was being carried out.
• Prepare the TCR (Test closure report).
• Reporting of the quality of the product obtained to the stakeholders/customers.
• Test result analysis to find out the distribution of the defects.
Deliverables:
• Test metrics.
• TCR (Test Closure Report).
TYPES OF SOFTWARE TESTING:
a. Black box testing – “Believes in abstraction. The internal system design is not considered in this. Rather the tests are solely based on the requirements and its functionality”.
b. White box testing – “A.k.a glass box testing, this type of testing depends solely on knowledge of the internal logic of the software’s code. Internal software and code working should be known well before hand for WBT. Tests are mostly based on different parameters for coverage such as branches, paths, statements, conditions”.
c. Unit testing – “Testing of individual software components or modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. May require developing test driver modules or test harnesses”.
d. Integration testing – “Testing of integrated modules to verify combined functionality after integration. Modules are typically code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems”.
e. System testing – “The whole system is tested as per the requirements. Black-box testing that is mostly based on overall requirements specifications, covers nearly all of the combined parts of a particular system”.
f. Regression testing – “This testing deals with testing the application as a whole after the introduction of certain modifications/changes in any module of the system. The difficulty arises because it’s hard to cover the whole system as a result of which automation tools are used”.
g. Acceptance testing –“Acceptance testing is conducted by the users/customers and sometimes the stakeholders may be involved in this process as well. The goal of acceptance testing is to build strong confidence in the system i.e. enhance the acceptability of the system. Acceptance testing is most often focused on a validation type testing”.