Testing in Agile based methodologies


Testing in Agile Framework

In contrast to traditional SDLCs, in Agile, testing life cycle is squeezed from few months to few weeks and each cycle begins and ends in the same iteration. Iteration is like a sprint in terms of Scrum. Test planning is done at the beginning of each iteration during iteration planning meeting. Testing team needs to work in close collaboration with the development team and Product owner. During iteration, development team produces frequent fully or partially unit test builds for testing.

Though the objective of testing is same, the challenges of testing in Agile are different as in the traditional SDLC. However, in many cases, Testing professionals still try to fit the traditional testing approach, techniques and measurements in agile. This blogs talks about role of test engineers and what testing they should consider during the testing in agile..

In 2001, Agile manifesto was introduced to software industry to provide a framework to build software faster in turbulent business environment. Over the time various SDLCs such as Scrum, XP etc. complying with the Agile Manifesto, got matured and various success stories are backing up the success of agile methodologies.

Software is developed in iterations. Each iteration, agile team is focused on design, code, and test a small set of requirements. Agile team delivers a potentially shippable software i.e. implementation of small chunk of the requirements at the end of each iteration. Iteration sizes from 2 weeks to 6 weeks. Test engineers involves from sizing the user stories to confirming their correct implementations.

Sizing of User Stories

During User Stories Sizing workshop, agile team relatively sizes each user story against a specific user story size. Usually test engineers don’t participate or participate passively. Test engineers needs to compare the testing effort they may need to put in to test user story and accordingly share their estimates. Usually, development and testing teams estimates matches in terms of relativity. However there might be cases where development efforts are less than testing and vice versa. E.g. test engineers needs to put more efforts if they need to test a user story (As an Admin I want to see yearly reports containing monthly sums so that I can compare which month has larger sums). For this user story, testing team needs to create test data for across the years, months, and days.

Unit Testing

Unit testing ensures the developer are writing right code correctly to implement the desired functionality. It also catches cases which black box test engineers may miss and reduces the cost of the bug by finding them early. Test Driven Development (TDD), Behavior Driven Development (BDD), and Acceptance Test Driven Development (ATDD), are getting popular among developer community. They discover the bugs at unit level as well as at integration points. In cases where following TDD/BDD/ATDD is not possible, still Unit testing and integrated testing should not be compromised and implemented. To ensure the frequent and efficient unit/integration testing, continuous integration tools must be used.

Acceptance Testing

It is recommend that each iteration’s delivery should be designed, coded and Tested in the same iteration. To capture the requirements, User stories are being used by the agile practitioners. A user story contains the purpose, business value as well as completion criteria. This completion criteria also known as Done criteria, becomes the basis of testing and accepting if a user story is implemented as expected.

Test engineers needs to ensure that each user stories has agreed upon completion criteria. They are supposed to write acceptance test cases to ensure user stories meets the acceptance criteria.

Acceptance test cases should be written against each completion criteria and these test cases should be executed frequently and once more after code is freeze and added to the potentially shippable product.

Functional Testing

Though Acceptance testing cases confirms that user story is implemented as expected, but it cannot be the replacement of testing of end to end flows, alternate flows, negative cases etc.

As the new user stories are implemented and integrated with the existing software, new and alternate flows gets introduced, test data need increases, and additional user roles/persona comes into the picture. Functional test cases will ensure that these additional testing cases are captured and executed to ensure the correct integration between new and existing functionalities. In agile however it is not recommended to write exhaustive test cases as most of the test cases’ life will end with the end of iteration. Few of the functional test cases will be selected for automated regression suite. Hence implementing the test case optimization techniques such as Orthogonal array, Equivalence class partitioning, workflow based & risk based testing, plays pivotal role. More the test cases, more the overhead of the managing the test cases.

Automated Regression

Automate as early as possible and automate as much as possible, are the mantra of testing in agile. Software is developed incrementally and by every passing iteration, the regression suite gets larger while the Iteration size remain same. Gradually testing team starts spending more time on regression instead of Exploratory testing and testing of new user story implementation. As a thumb rule all validation tests and acceptance tests should be automated. However in cases where test automation is not possible, it is better to have a hardening iteration after every 4-5 iteration. During the hardening iteration, testing team focuses on bug regression and retesting, while development team addressing the bugs from the backlog.

Exploratory Testing

Agile is for faster development by increasing productivity and reducing waste & rework. Test cases though try to catch most of the cases, ends up with test cases for happy paths and known negative cases. As we discussed the need of optimized test cases to reduce the test case counts, still test case based testing cannot substitute what and how human mind observe and react while using the software. Here comes the Exploratory testing to unearth more issues in shortest time. It helps in finding scenarios and defects which with the help of test cases difficult to even imagine at the time of test case writing.

Testing team needs to work closely with the product owner to understand the end user. To ensure that exploratory testing does not go unguided, testing team develops user Roles, Persona and Extreme characters and then imitate them.

Measuring Testing Progress

Though working in Agile, project managers still judge Test engineers’ productivity by measuring the test cases written and executed in a specific period. Testing progress is measured by measuring executed and pending test cases. Measurement for testing progress should be Burn down charts. And to measure the quality of testing, Bug escape rate should be used. To check the stability of User stories, map Bugs with the user stories.

Defect Management

Defects in user stories implementation should be addressed in the same iteration. Still there are bugs that could not be fixed because of time constraints, priorities, resource crunch, ambiguous confirmation criteria, afterthoughts cases etc. Over the time unattended bugs gets accumulated and dealing with them becomes a challenge. Some team prefers to create a bug backlog and adds the bugs that could not be addressed in the recently completed iteration. With the help of Product owner they prioritize the bugs and fix them. This approach increase the overhead of maintaining two backlogs.

Another approach is to consider the open bugs as user stories and add them to Product backlog. This approach is least preferred as it increases the size of product backlog and product owner has to put more efforts to prioritize the backlog. Teams need to spend more time on estimations and planning. Also product owner resists as they have to accept bugs as user stories which he did not created but outcome of incorrect implementation.

During iteration planning, agile teams which consider only User Stories during the Iteration planning, have to struggle later to find time to fix and regress the bugs. Hence it is better to have 10-20% of Iteration length for bug fixing and regression. To reduce the bug fixing effort, Team selects the bugs which were found in recently complete iteration first as the code and constraints are still fresh in their mind.

Test engineers, next to product owner, usually has the better understanding of the system, they can identify the potential bugs which can be converted in to User Stories. Then these user stories can be added to Product backlog for sizing and priority.

Non Functional Testing

Team needs to break the user stories in to multiple stories to keep the functional and non-functional requirements separate. It helps in tracking both functional and nonfunctional needs. Testing of Nonfunctional needs such as security, performance, accessibility, high availability, reliability, and usability requires different skill set and expertise. Agile team first focuses on functional user stories of themes. Then team picks the nonfunctional user stories and involves the nonfunctional testing experts. Though functional test engineers remain in the team in all iterations, specialized test engineers are involved on need basis.

Buddy testing

In XP, two developers work on the same user story, one developer is writing code while another developer generating tests to generate quality code. In similar way, associate a test engineer with a developer. Whenever, developer develops and integrates his code, he invites the test engineers to test. Test engineers performs exploratory testing and explains the purpose of various tests. Developers consolidate the bugs found and fix them later. This approach is good as it helps in bringing the development and testing team on same page, unearth ambiguous requirements, increase collaboration and pulls down the project cost. Gradually developers also starts understanding the testing techniques and starts testing their code that consequently reduces bug count and increase productivity of entire team.

Role of Test Engineers

Keeping in mind the changing paradigm, writing lots of test cases, create big test suites for various type of testing, establishing various traceability matrices, filing defects will not bring the value in Agile. Teams are small and team members work in coloration in time boxed environment.

Functional Test engineers needs to have manual and automation expertise besides good knowledge of database. As the iterations are time boxed, they must be expert in exploratory testing and use skills to optimize test data and test cases. In absence of Product Owner, Testing professional has to play role of product owner. Automating regression suites is need of the hour to save reduce the test cycle in iterations. Team is working closely with product owner to understand user stories, hence the test engineers must have good communication skills and analytical skill.


What do you think about testing in Agile, please do share.


Want to save Testing Time: Go for Orthogonal Array

The Orthogonal Array testing technique is a statistical way of reducing the test cases by removing redundant test conditions. A software system works on large number of interactions and integrations. It is very difficult to find out the erring interaction/integrations which the It is observed that major source of the bugs are interactions and integrations.

The bugs in the software are usually caused by a specific Condition independent of other conditions in the system. A condition is a set of conditions or variables, configurations, and logical linking of variables.

Think about testing a simple system, which has four variables and each variable can have 3 inputs. We need 81 cases (3x3x3x3) to test each pair of this system. If we use the orthogonal array techniques, we will have only 9 cases to test the system where all the pair-wise combination will be validated. It will unearth the combination which is causing the failure in the system.

In the above case, Orthogonal Array techniques has reduced the test condition by 89%, i.e. now with 11% of existing test cases all the variable pairs are validated. Interestingly, it will will provide 100% coverage of pair combinations, 33% of three way combinations, 11% of four way combinations. Further to reduce the test condition, orthogonal technique can be used on three way and four way combinations.

Use of Orthogonal array techniques will ensure

  • All the pair-wise combinations of the selected variables are validated.
  • An effective test set containing with fewer test cases
  • Lowers the cost of testing as test cycles are shorter

For one of the client, which is Leader in the Video communications, orthogonal array techniques reduced the test cases from 8171 to 2614 i.e. reduced by 68%. Test cycle efforts were reduced from 164 to 55 person days.

9 ways to Reduce Software Development Cost

We know that project cost can be reduced if there is low number of Bugs/Changes as well as if these bugs/changes can be detected early in S/W Development Life Cycle. This topic talks about how we can reduce the number of bugs passed to testing team from development team.
During the development, one should not surprise seeing the bugs. They are byproduct of entire development cycle. Bugs cause rework which consequently increases the cost of the project as well as schedule.  This topic is about save rework and reduces cost of the software.

Though bug free system cannot be developed in a multimember team but the number of bugs found during the testing and production can be reduced by detecting the bugs early, before releasing the Build to testing team.

Adding the below mentioned activities in your development and testing cycle will definitely help in reducing the project cost as well as help in quality delivery. You may get an impression that most of the activities are suggested for developers. Let me clarify that as they are the people

  1. Right Process and Right Tools: Implement the right lightweight processes since beginning of the software development. Anything which has to be done repetitively, automate it.
  2. Clarify Assumptions: For all functionality, if there are any assumptions are made, clarify them with client, test engineers, Business analyst and other stake holders.Actor (Development and Testing Team)
    Why: Impact of declination of an assumption later will cause rework in either design, coding, testing or in all phases.
  3. During the implementation, if you are considering any behavior as implicit, obvious, or understood, then better to document/communicate it and verify with stakeholders.
    Actor (Development Team/Test Engineers)
    Why: It will help in developing what is required by customer rather than investing time in development of features which is luxury to have but not required.
  4. Test Engineers should develop test cases component wise and release them to Development team asap for Review (Design phase is recommended).
    Actor (Testing Team/Development Team)
    Why: Developer understands what test engineer is going to test beforehand including special cases. Also another advantage is that most of the test cases are User’s action on system; hence the usability and interface problems can be detected during design and coding.
  5. Each developer should Review the test cases related to his component. Test engineers should add missing test cases as detected by development team.
    Actor (Development Team/Testing Team)
    Why: To learn more about the functionality. Also it will expose the understanding gaps between Development and Testing team. This activity needs more time to be allocated to development team. Remember cost of bug is low in early phases of development.
  6. Unit Testing: It will help in identifying the issues, which may be missed by Blackbox test engineers.
    Actor (Development Team/Testing Team (white box testing experts))
  7. Testing team should develop a set of test cases called ‘Development Smoke Test cases’. Development team should execute the ‘Development Smoke Test cases’ before releasing build to Testing Team.
    Actor (Development Team/Testing Team)
    Why: There are cases where build gets rejected by QA in their smoke testing because of broken major features. Development/Testing team has to do rework.
  8. Find the Test cases effectiveness. How?, check how many bugs can be mapped to test cases. If the effectiveness is <80%, review the test cases development approach of testing team.
    Actor (Testing Team)
    WHY: All stakeholders are relying on testing team’s test cases to detect the bugs. Find technique to improve the testing effectiveness. (See my blogs on this topic).
  9. Configuration Testing and Load Testing should be done asap as later detection of performance issues does not leave enough time to do something substantive.

How to Save Cost of Testing by Tests Reusability

Reduce Investment planned for Testing by Tests Reusability

In software test life cycle (STLC), among various phases, test case development is most critical and consumes considerable portion of the time, allocated for testing. While working on various projects, my target was to write optimized number of effective test cases.  During each product/project I tested, I used to feel that I have executed the similar tests in the earlier assignments. I am sure you might also have observed, that you are sometime writing the test cases, which you have already written for some other project/product.

Let me explain with an example of Login, whether you are in Finance, health care or CRM domain, test engineer has to write almost similar cases with few more addition or deletion. Also, as a Supervisor, you might have observed that on many occasions your team missed potential test cases; reason could be the time crunch, skill, experience ….. Usually Test case effectiveness is dependent on individual’s skills.  Supervisor faces bigger challenges on estimating the count of test cases to be written as it will provide the basis for test execution time. Also they face challenge to ensure their team is not missing potential test cases as well as spending effort to write quality test cases instead of redundant and exhaustive test cases. I experienced that my various teams used to find many potential test cases during the test execution and add them to the test suite later.

How is the idea if a Team

  • Can estimate testing efforts close to reality
  • Saves investments by shortening the Testing life cycle
  • Instead of investing efforts in writing test cases from scratch, choosing test cases from an organized test cases repository
  • Can verify the completeness of their Tests by checking them against predefined set of cases
  • Can shorten the Test cases Review cycle
  • Ensure the completeness of Test suite
  • deliver quality product by ensuring no missing test cases

Now, how to implement the Test reusability and ensure the completeness of test cases.  I am writing on these topic too. Will publish them soon.

Selenium-WebDriver Code to Write in an excel file

// Before executing this program, create an XLS file, testExcel.xls in D drive


import java.io.FileOutputStream;

import java.io.IOException;

import jxl.Workbook;

import jxl.read.biff.BiffException;

import jxl.write.WritableWorkbook;

import jxl.write.WritableSheet;

import jxl.write.Label;

import jxl.write.WriteException;

public class Sample{

public static void main(String[] args) throws IOException, WriteException, BiffException{

//Declare the name of the Excel file

FileOutputStream exlFileName= new FileOutputStream(“D:\\testExcel.xls”);

//making the instance of a Writable Excel workbook and giving it a Name

WritableWorkbook exlWorkBook = Workbook.createWorkbook(exlFileName);

//Creating Writable worksheets in the writable workbook

WritableSheet exlWorkSheet = exlWorkBook.createSheet(“sheet num 0”, 0);

//Creating content for a cell in the excel workbook

Label cellContent = new Label(0,0,”at 0,0 the Content is : good”);

//Adding a cell and inserting content in the cell.


//Confirm the writing in the excel i.e. Write in the excel workbook.


//Close the workbook


}//eof main

}//eof class

Quality Test cases results in quality Deliveries

Test Cases development is one of the most important activities in the Testing life cycle. Test cases development strategy changes depending upon the type of project, schedule, and input domain. Another challenge for the test engineers to write the effective test cases. First step during the test cases development is to find type of users. Choose those who are going to test the Target Test Item (TTI) using test data

While developing test cases, if we keep in mind the following items, the rework on test cases will be lesser and quality of test cases will be much better.

  • Understand the requirement well. How please refer my blog on Requirement understanding.
  • Find out the type of application and its module:
  • Input Centric application/module: where user has to enter lots of data
  • Querying centric application/module: User has to raise query, such as Reporting modules.
  • Domain Understanding: Learn about domain, it will help in finding gaps.
  • Make control flow on paper:

¨       These graphs need not to follow any conventions. They are only three components in the diagram. One directional line (arrow) to show the direction of control movement, Boxes to show Outputs, line between two boxes used for describing step or input data. We are drawing our understanding on paper. They are just showing the movement of Control from one step to another, information being generated at each step, data inputted at each step.

  • Scenarios development:

¨       Now with the help of control flow, draw scenarios. One scenario is a set/group of test cases, which are verifyin an aspect of System under test. Such as for a scenario “Invalid Login not allowed”, following are test cases,

  • Invalid User name and Valid Password will not be allowed to login.
  • Invalid User name and InValid Password will not be allowed to login.
  • Valid User name and InValid Password will not be allowed to login.

¨       Once the scenarios are ready, develop test cases. You may add/delete/modify the Scenarios, as your test cases development progresses as understanding of system, may require the re-organization.

Test cases development sequence: Develop the test cases in following sequence:

1. One Path Test cases.

Set of Test cases which will cover the functionalities of complete flow, such, from Login -> Searching an Item -> viewing its details -> Add to Shopping Cart -> Make Payment-> Log Out.

2. Alternate Path Test Cases

If you have written test cases for a flow of functionality, then find out the alternate way of performing same task.

3. Integration Test Cases:

Test cases for functionalities, where outcome of one functionality, is used by other. Such as After placing an order, the Order information is saved in database. Another functionality is try to fetch this information for print, query, modification.

4. Boundry test cases:

5. Write test cases for each input box.

6. Test cases to check the database length and length of data in input controls are same.

7. Use Orthogonal Array to reduce the Test cases and test data count.

Database Testing Checklist for test engineers

Some time back I thought to create a checklist for database testing to ensure we are covering every aspect in Testing. With the help of my colleagues, Neha and Ankit,we prepared this list. Add one more column to this list for the result of each check.


Database Testing Checklist


Check Point


Data Integrity


Is the complete data in the database is stored in tables


Is the data well logically organized


Is the data stored in tables is correct


Is there any unnecessary data present


Is the data present in the correct table


Is the data present in correct field within the table


Is the data stored correct with respect to Front End updated data


Is LTRIM and RTRIM performed on data before inserting data into database




Field Validations


Is ‘Allow Null’ condition removed at database level for mandatory fields on UI


Is ‘Null’ as value not allowed in database


Is Field not mandatory while allowing NULL values on that field


Is the Field length specified on UI same as field length specified in table to store same element from UI into database.


Is the length of each field of sufficient size


Is the Data type of each field is as per specifications


Does all similar fields have same names across tables


Is there any computed field in the Database






Is required Primary key constraints are created on the Tables


Is required Foreign key constraints are created on the Tables


Are valid references are done for foreign key


Is Data type of Primary key and the corresponding foreign key same in two tables


Does Primary key’s ‘Allow Null’ condition not allowed


Does Foreign key contains only not NULL values




Stored Procedures/ Functions


Is proper coding conventions followed


Is proper handling done for exceptions


Are all conditions/loops covered by input data


Does TRIM is applied when data is fetched from Database


Does executing the Stored Procedure manually gives the correct result


Does executing the Stored Procedure manually updates the table fields as expected


Does execution of the Stored Procedure fires the required triggers


Are all the Stored Procedures/Functions used by the application (i.e. no unused stored procedures present)


Does Stored procedures have ‘Allow Null’ condition checked at data base level


Are all the Stored Procedures and Functions successfully executed when Database is blank






Is proper coding conventions followed in Triggers


Are the triggers executed for the respective DML transactions


Does the trigger updates the data correctly once executed






Are required Clustered indexes created on the tables


Are required Non Clustered indexes created on the tables






Are the transactions done correct


Is the data committed if the transaction is successfully executed


Is the data rollbacked if the transaction is not executed successfully


Is the data rollbacked if the transaction is not executed successfully and multiple Databases are invlolved in the transaction


Are all the transactions executed by using TRANSACTION keyword






Is the data secured from unauthorized access


Are different user roles created with different permissions


Do all the users have access on Database






Does Database perform as expected (within expected time) when query is executed for less number of records


Does Database perform as expected (within expected time) when query is executed for large number of records


Does Database perform as expected (within expected time) when multiple users access same data


Is Performance Profiling done


Is Performance Benchmarking done


Is Query Execution Plan created


Is Database testing done when server is behind Load Balancer


Is Database normalized






Are the log events added in database for all login events


Is it verified in SQL Profiler that queries are executed only in Stored Procedures (i.e. no direct visible query in Profiler)


Does scheduled jobs execute timely


Is Test Data Dev Tool available




SQL Injection


Is the Query parameterized


Does URL contain any query details




Backup and Recovery


Is timely backup of Database taken