Web services/API testing, go open source

Are you paying for the web services/api testing tool? Then this post is for you.

Few weeks back, during a discussion, Client shared they have around 1800+ web services tests and automated 1100+ using (I will not name the tool J) and they are planning to buy more licenses to automate more.  I advised him to consider open source tools over licensed as my team successfully automating and already delivered 1250+ functional tests with Open source technology (java, selenium, testNG, maven, Jenkins, extent reports etc.). After the demo, client got confidence and we have started automating new ones as well as migrating existing API tests. Plan is to migrate all existing test in another 4 months.

Let me talk about this framework, which you may also like to build and use. We have developed and have a stable REST Assured API based automation framework built with Java, where you keep all your data in excel, use testNG to drive the parallel execution to save time. Any licensed tool provides you, the feature of sending the request, perform the assertions on response, control the execution, execute test in parallel and keep the test data & configuration separate from your tests. Complexity increases when your services require some kind of authentication using Tokens or Certificates and do the validation in databases. Another challenge is when your api tests are fetching data from previous test’s Response.

Below is the high-level overview of the framework for your reference.

RestAssuredFramework

You would like to go through the below comparison with licensed tools.

RestAssuredAPI_Vs_LicensedToolComparision

In case you want to share your opinion or experience, please share your opinion in comments.

Is it possible to Automate Accessibility Testing???

Accessibility- Visual Disability  This blog discuss the challenges to Automate the accessibility testing of Web applications made accessible  for people having Vision related issues; such as blindness and low vision etc.

   Starting with a brief about Accessibility, I will discuss the challenges we faced, and then will end with the
solution.

As per W3C, Web accessibility means that people with disabilities can use the Web. More specifically, Web accessibility means that people with disabilities can perceive, understand, navigate, and interact with the Web, and that they can contribute to the Web.

W3C started an initiative; Web Accessibility Initiative (WAI), to lead the Web to its full potential to be accessible, enabling people with disabilities to participate equally on the Web. Web Accessibility testing is to validate that website is accessible to people with various level of disabilities.

Businesses are making their websites accessible to avoid legal issues, expend the business (approx 1 trillion $ market) and remove inequality among people with various level of abilities. Tesco invested £35,000 to make their website accessible and generated £1.5 million in a year from online sales to disabled people in Europe.

Broadly disabilities can be grouped under Sensory (Vision, Hearing), Physical (Hand movement, paralysis etc), and Cognitive (dyslexia, slow processing of information etc) disabilities.

For people with Visionary problems such as low vision or blindness, there are some assistive screen reading tools; such as JAWS, NVDA etc.  These tools read the web content; end user hears and accordingly with the help of Keyboard can interact with the website. Tab key, Arrow keys, Enter Key, Shift, CTRL, ALT, and Space bar are most used keys for navigation.

Testing the Website for vision accessibility is a two steps process;  Step 1, Use free tools where you provide the URL of your website and the tool generates a report showing the how accessible is the website. Take appropriate action. Step 2, Manual test engineers imitate the blind users, hear the web content for correctness, and test the navigation and functionalities using keyboard.

Hearing the content and then verifying what you see on the screen, repetitively is monotonous and boring tasks for manual test engineers.  Disorientation leaves space for missing vision accessibility issues during regression testing. Keeping this in mind, since few days I along with my colleague, trying to automate, accessibility testing.

The foremost challenge was we could not find over web, if somebody has tried to automate Screen readers.  Then next biggest challenge was how to verify if the Screen readers are reading the content right. Another technical challenge is that Screen reader tools are not accepting the Keyboard Shortcut inputs sent by various paid/open source tool such as QTP, SilkTest, Test Complete, Selenium, AutoIT, Robot Api etc.  These Keyboard shortcuts help the disabled people to navigate and use the functionality of the Web page.

The solution; we created an Object Repository which contains all the objects, their IDs, specific attribute which JAWS reads, and expected content. We were sure that if the right content is set in the right property of an object, JAWS is going to read it correctly. We also found during our R&D, that JAWS reads the ARIA labels first. So in cases where the development is at initial stage, I would recommend to ensure that development team is entering content in the ARIA labels associated with an object. These contents will be read by JAWS when user moves control on the object.

What is ARIA (http://en.wikipedia.org/wiki/WAI-ARIA) WAI-ARIA describes how to add semantics and other metadata to HTML content in order to make user interface controls and dynamic content more accessible.)

In our case, the test website is already developed, so instead of asking development team to add ARIA labels for all objects, we collected all objects, and their content in specific attributes, which JAWS is reading. This way we created our object repository and automated Screen Readers. For navigation, currently we are using the Tab key, Arrow keys, Enter Key, and Space bar.  With these keys we are able to check all objects, content for JAWS, and functionalities.

I am looking forward to hear from you if you have suggestions and queries.

Planning to Outsource Testing: OSTC or OSDC

Outsourcing testing services to outside vendors (we call them extended partners these days) is not a new phenomenon. There are many Offshore Software Testing companies (OSTC) exclusively focusing on testing services, such as AppLabs (acquired by CSC) , NeuSoft etc. Sensing the potential of exponential growth, major Offshore Software Development Companies (OSDC)  are also offering, testing services such as Impetus, Globallogic, Symphony, and Persistent etc.

It is understood that outsourcing the testing will speed up the testing, improve quality, reduce project cost, and lets you focus on business. But when it comes to selecting an outsourced business partner, it becomes difficult to make a choice between OSTC and OSDC offering testing services.

OSDC should be prefered if you have vision to outsource the complete or partial software development in future. There are instances where few customers outsourced the Development and Testing contracts to OSDC and OSTC companies respectively. Usually this happens when a customer has already outsourced testing to some OSTC and now planning to oursource the product development.  Viceversa, if OSDC does not have appropriate capability to provide the testing, customer involves OSTC for testing services. Though OSDC-OSTC collaboration model is also successful, however in such cases ownership and coordination become a big challenge.

Only deterent to go with OSDC is they are more focussed in Software development , hence you need to check their References, Testing infrastructure, Talent pool, Ramp-up capacity, Testing  and Tools expertise, Attrition of testing talent. It is good to look if they have a Testing Center of excellance.

OSTC are usually keeping watch on current Software testing trends and usually acquire required Talent. They invest a lot in research and has right mindset as well as sytem to carry out quality testing.  OSDC has Testing Center of Excellance to serve while in cases of OSTC, the entire company it self become Testing center of Excellence.

Do you think that OSDC should be prefered over OSTC because if required they have software development capabilities. Looking forward to hear your opinions.

Want to save Testing Time: Go for Orthogonal Array

The Orthogonal Array testing technique is a statistical way of reducing the test cases by removing redundant test conditions. A software system works on large number of interactions and integrations. It is very difficult to find out the erring interaction/integrations which the It is observed that major source of the bugs are interactions and integrations.

The bugs in the software are usually caused by a specific Condition independent of other conditions in the system. A condition is a set of conditions or variables, configurations, and logical linking of variables.

Think about testing a simple system, which has four variables and each variable can have 3 inputs. We need 81 cases (3x3x3x3) to test each pair of this system. If we use the orthogonal array techniques, we will have only 9 cases to test the system where all the pair-wise combination will be validated. It will unearth the combination which is causing the failure in the system.

In the above case, Orthogonal Array techniques has reduced the test condition by 89%, i.e. now with 11% of existing test cases all the variable pairs are validated. Interestingly, it will will provide 100% coverage of pair combinations, 33% of three way combinations, 11% of four way combinations. Further to reduce the test condition, orthogonal technique can be used on three way and four way combinations.

Use of Orthogonal array techniques will ensure

  • All the pair-wise combinations of the selected variables are validated.
  • An effective test set containing with fewer test cases
  • Lowers the cost of testing as test cycles are shorter

For one of the client, which is Leader in the Video communications, orthogonal array techniques reduced the test cases from 8171 to 2614 i.e. reduced by 68%. Test cycle efforts were reduced from 164 to 55 person days.

How to Save Cost of Testing by Tests Reusability

Reduce Investment planned for Testing by Tests Reusability

In software test life cycle (STLC), among various phases, test case development is most critical and consumes considerable portion of the time, allocated for testing. While working on various projects, my target was to write optimized number of effective test cases.  During each product/project I tested, I used to feel that I have executed the similar tests in the earlier assignments. I am sure you might also have observed, that you are sometime writing the test cases, which you have already written for some other project/product.

Let me explain with an example of Login, whether you are in Finance, health care or CRM domain, test engineer has to write almost similar cases with few more addition or deletion. Also, as a Supervisor, you might have observed that on many occasions your team missed potential test cases; reason could be the time crunch, skill, experience ….. Usually Test case effectiveness is dependent on individual’s skills.  Supervisor faces bigger challenges on estimating the count of test cases to be written as it will provide the basis for test execution time. Also they face challenge to ensure their team is not missing potential test cases as well as spending effort to write quality test cases instead of redundant and exhaustive test cases. I experienced that my various teams used to find many potential test cases during the test execution and add them to the test suite later.

How is the idea if a Team

  • Can estimate testing efforts close to reality
  • Saves investments by shortening the Testing life cycle
  • Instead of investing efforts in writing test cases from scratch, choosing test cases from an organized test cases repository
  • Can verify the completeness of their Tests by checking them against predefined set of cases
  • Can shorten the Test cases Review cycle
  • Ensure the completeness of Test suite
  • deliver quality product by ensuring no missing test cases

Now, how to implement the Test reusability and ensure the completeness of test cases.  I am writing on these topic too. Will publish them soon.

Selenium-WebDriver Code to Write in an excel file

// Before executing this program, create an XLS file, testExcel.xls in D drive

 

import java.io.FileOutputStream;

import java.io.IOException;

import jxl.Workbook;

import jxl.read.biff.BiffException;

import jxl.write.WritableWorkbook;

import jxl.write.WritableSheet;

import jxl.write.Label;

import jxl.write.WriteException;

public class Sample{

public static void main(String[] args) throws IOException, WriteException, BiffException{

//Declare the name of the Excel file

FileOutputStream exlFileName= new FileOutputStream(“D:\\testExcel.xls”);

//making the instance of a Writable Excel workbook and giving it a Name

WritableWorkbook exlWorkBook = Workbook.createWorkbook(exlFileName);

//Creating Writable worksheets in the writable workbook

WritableSheet exlWorkSheet = exlWorkBook.createSheet(“sheet num 0”, 0);

//Creating content for a cell in the excel workbook

Label cellContent = new Label(0,0,”at 0,0 the Content is : good”);

//Adding a cell and inserting content in the cell.

exlWorkSheet.addCell(cellContent);

//Confirm the writing in the excel i.e. Write in the excel workbook.

exlWorkBook.write();

//Close the workbook

exlWorkBook.close();

}//eof main

}//eof class

Quality Test cases results in quality Deliveries

Test Cases development is one of the most important activities in the Testing life cycle. Test cases development strategy changes depending upon the type of project, schedule, and input domain. Another challenge for the test engineers to write the effective test cases. First step during the test cases development is to find type of users. Choose those who are going to test the Target Test Item (TTI) using test data

While developing test cases, if we keep in mind the following items, the rework on test cases will be lesser and quality of test cases will be much better.

  • Understand the requirement well. How please refer my blog on Requirement understanding.
  • Find out the type of application and its module:
  • Input Centric application/module: where user has to enter lots of data
  • Querying centric application/module: User has to raise query, such as Reporting modules.
  • Domain Understanding: Learn about domain, it will help in finding gaps.
  • Make control flow on paper:

¨       These graphs need not to follow any conventions. They are only three components in the diagram. One directional line (arrow) to show the direction of control movement, Boxes to show Outputs, line between two boxes used for describing step or input data. We are drawing our understanding on paper. They are just showing the movement of Control from one step to another, information being generated at each step, data inputted at each step.

  • Scenarios development:

¨       Now with the help of control flow, draw scenarios. One scenario is a set/group of test cases, which are verifyin an aspect of System under test. Such as for a scenario “Invalid Login not allowed”, following are test cases,

  • Invalid User name and Valid Password will not be allowed to login.
  • Invalid User name and InValid Password will not be allowed to login.
  • Valid User name and InValid Password will not be allowed to login.

¨       Once the scenarios are ready, develop test cases. You may add/delete/modify the Scenarios, as your test cases development progresses as understanding of system, may require the re-organization.

Test cases development sequence: Develop the test cases in following sequence:

1. One Path Test cases.

Set of Test cases which will cover the functionalities of complete flow, such, from Login -> Searching an Item -> viewing its details -> Add to Shopping Cart -> Make Payment-> Log Out.

2. Alternate Path Test Cases

If you have written test cases for a flow of functionality, then find out the alternate way of performing same task.

3. Integration Test Cases:

Test cases for functionalities, where outcome of one functionality, is used by other. Such as After placing an order, the Order information is saved in database. Another functionality is try to fetch this information for print, query, modification.

4. Boundry test cases:

5. Write test cases for each input box.

6. Test cases to check the database length and length of data in input controls are same.

7. Use Orthogonal Array to reduce the Test cases and test data count.