HP ALM frequent crashes and how we are dealing with it

HP-ALM is one of the tools which is being used by large companies to maintain their testing life cycle. I consider it as a reliable tool and it helps in managing testing well, however recently i have had a bad experience with ALM and one of my team lost almost 200 person days productivity.

One of my team was using HP-ALM 11.5 for BPT approach to automate tests for a leading US based telecom company. We were using HP-ALM for managing our test cases, test data, components, flows, scripts, automated test cases etc. Initially it went well when team was busy in creating reusable components and pulling them into test cases. When my engineers started executing tests from test lab, we were troubled by many issues.

-UFT is not getting launched from ALM;

-Components get crashed when trying to run test cases;

-Components and data sheets get vanished from ALM without leaving a trace

-Test set were executed half and then UFT getting crashed

-Components used to get locked due to crashes

We had to re-develop/write the components, after every crash when we could not find way to find them back. Data sheets have to be reloaded and then again the mapping tasks have to be done. Team gradually getting frustrated with the rework and despite our efforts, we were not able to deliver what we committed. We lost a three week sprint with no delivery.

While team was struggling with ALM, our ALM Admin was looking into these issues with HP team trying to find the root cause. They run few utilities, but we did not get any respite.

However in mean time we also explored our infrastructure. We realized that ALM server is holding 7 big projects under one ALM project with more than 52GB of data. You could imagine the size that there were 0.14 million test cases which got executed and produced 0.30 million test runs.

To address these issues, we are now proceeding for below activities:

  1. Clean up to remove unwanted records (test data, test cases, runs, user details etc). To improve the performance by reducing the load on server
  2. Segregating existing projects to have one ALM project for each development project

In case you are also facing such issue, you may like to know how currently we are dealing with it:

  1. We take backup every day in the morning, which helps in recovering components and datasheets in event of loss.
  2. Team is working in two shifts to put load as low as possible
  3. Avoid working on same components concurrently
  4. Reducing frequent checkin/out

 

HP ALM is a good tool for automation testers as it saves time to develop and maintain the frameworks as well as provide management, reports to measure the quality of product being built. If you have gone through the similar experience with ALM and able to resolve, please do write, it will be a great help. I will update this blog once our cleaning activity is complete or if we find some solution.

Testing in Agile based methodologies

https://mmathpal.wordpress.com/

Testing in Agile Framework

In contrast to traditional SDLCs, in Agile, testing life cycle is squeezed from few months to few weeks and each cycle begins and ends in the same iteration. Iteration is like a sprint in terms of Scrum. Test planning is done at the beginning of each iteration during iteration planning meeting. Testing team needs to work in close collaboration with the development team and Product owner. During iteration, development team produces frequent fully or partially unit test builds for testing.

Though the objective of testing is same, the challenges of testing in Agile are different as in the traditional SDLC. However, in many cases, Testing professionals still try to fit the traditional testing approach, techniques and measurements in agile. This blogs talks about role of test engineers and what testing they should consider during the testing in agile..

In 2001, Agile manifesto was introduced to software industry to provide a framework to build software faster in turbulent business environment. Over the time various SDLCs such as Scrum, XP etc. complying with the Agile Manifesto, got matured and various success stories are backing up the success of agile methodologies.

Software is developed in iterations. Each iteration, agile team is focused on design, code, and test a small set of requirements. Agile team delivers a potentially shippable software i.e. implementation of small chunk of the requirements at the end of each iteration. Iteration sizes from 2 weeks to 6 weeks. Test engineers involves from sizing the user stories to confirming their correct implementations.

Sizing of User Stories

During User Stories Sizing workshop, agile team relatively sizes each user story against a specific user story size. Usually test engineers don’t participate or participate passively. Test engineers needs to compare the testing effort they may need to put in to test user story and accordingly share their estimates. Usually, development and testing teams estimates matches in terms of relativity. However there might be cases where development efforts are less than testing and vice versa. E.g. test engineers needs to put more efforts if they need to test a user story (As an Admin I want to see yearly reports containing monthly sums so that I can compare which month has larger sums). For this user story, testing team needs to create test data for across the years, months, and days.

Unit Testing

Unit testing ensures the developer are writing right code correctly to implement the desired functionality. It also catches cases which black box test engineers may miss and reduces the cost of the bug by finding them early. Test Driven Development (TDD), Behavior Driven Development (BDD), and Acceptance Test Driven Development (ATDD), are getting popular among developer community. They discover the bugs at unit level as well as at integration points. In cases where following TDD/BDD/ATDD is not possible, still Unit testing and integrated testing should not be compromised and implemented. To ensure the frequent and efficient unit/integration testing, continuous integration tools must be used.

Acceptance Testing

It is recommend that each iteration’s delivery should be designed, coded and Tested in the same iteration. To capture the requirements, User stories are being used by the agile practitioners. A user story contains the purpose, business value as well as completion criteria. This completion criteria also known as Done criteria, becomes the basis of testing and accepting if a user story is implemented as expected.

Test engineers needs to ensure that each user stories has agreed upon completion criteria. They are supposed to write acceptance test cases to ensure user stories meets the acceptance criteria.

Acceptance test cases should be written against each completion criteria and these test cases should be executed frequently and once more after code is freeze and added to the potentially shippable product.

Functional Testing

Though Acceptance testing cases confirms that user story is implemented as expected, but it cannot be the replacement of testing of end to end flows, alternate flows, negative cases etc.

As the new user stories are implemented and integrated with the existing software, new and alternate flows gets introduced, test data need increases, and additional user roles/persona comes into the picture. Functional test cases will ensure that these additional testing cases are captured and executed to ensure the correct integration between new and existing functionalities. In agile however it is not recommended to write exhaustive test cases as most of the test cases’ life will end with the end of iteration. Few of the functional test cases will be selected for automated regression suite. Hence implementing the test case optimization techniques such as Orthogonal array, Equivalence class partitioning, workflow based & risk based testing, plays pivotal role. More the test cases, more the overhead of the managing the test cases.

Automated Regression

Automate as early as possible and automate as much as possible, are the mantra of testing in agile. Software is developed incrementally and by every passing iteration, the regression suite gets larger while the Iteration size remain same. Gradually testing team starts spending more time on regression instead of Exploratory testing and testing of new user story implementation. As a thumb rule all validation tests and acceptance tests should be automated. However in cases where test automation is not possible, it is better to have a hardening iteration after every 4-5 iteration. During the hardening iteration, testing team focuses on bug regression and retesting, while development team addressing the bugs from the backlog.

Exploratory Testing

Agile is for faster development by increasing productivity and reducing waste & rework. Test cases though try to catch most of the cases, ends up with test cases for happy paths and known negative cases. As we discussed the need of optimized test cases to reduce the test case counts, still test case based testing cannot substitute what and how human mind observe and react while using the software. Here comes the Exploratory testing to unearth more issues in shortest time. It helps in finding scenarios and defects which with the help of test cases difficult to even imagine at the time of test case writing.

Testing team needs to work closely with the product owner to understand the end user. To ensure that exploratory testing does not go unguided, testing team develops user Roles, Persona and Extreme characters and then imitate them.

Measuring Testing Progress

Though working in Agile, project managers still judge Test engineers’ productivity by measuring the test cases written and executed in a specific period. Testing progress is measured by measuring executed and pending test cases. Measurement for testing progress should be Burn down charts. And to measure the quality of testing, Bug escape rate should be used. To check the stability of User stories, map Bugs with the user stories.

Defect Management

Defects in user stories implementation should be addressed in the same iteration. Still there are bugs that could not be fixed because of time constraints, priorities, resource crunch, ambiguous confirmation criteria, afterthoughts cases etc. Over the time unattended bugs gets accumulated and dealing with them becomes a challenge. Some team prefers to create a bug backlog and adds the bugs that could not be addressed in the recently completed iteration. With the help of Product owner they prioritize the bugs and fix them. This approach increase the overhead of maintaining two backlogs.

Another approach is to consider the open bugs as user stories and add them to Product backlog. This approach is least preferred as it increases the size of product backlog and product owner has to put more efforts to prioritize the backlog. Teams need to spend more time on estimations and planning. Also product owner resists as they have to accept bugs as user stories which he did not created but outcome of incorrect implementation.

During iteration planning, agile teams which consider only User Stories during the Iteration planning, have to struggle later to find time to fix and regress the bugs. Hence it is better to have 10-20% of Iteration length for bug fixing and regression. To reduce the bug fixing effort, Team selects the bugs which were found in recently complete iteration first as the code and constraints are still fresh in their mind.

Test engineers, next to product owner, usually has the better understanding of the system, they can identify the potential bugs which can be converted in to User Stories. Then these user stories can be added to Product backlog for sizing and priority.

Non Functional Testing

Team needs to break the user stories in to multiple stories to keep the functional and non-functional requirements separate. It helps in tracking both functional and nonfunctional needs. Testing of Nonfunctional needs such as security, performance, accessibility, high availability, reliability, and usability requires different skill set and expertise. Agile team first focuses on functional user stories of themes. Then team picks the nonfunctional user stories and involves the nonfunctional testing experts. Though functional test engineers remain in the team in all iterations, specialized test engineers are involved on need basis.

Buddy testing

In XP, two developers work on the same user story, one developer is writing code while another developer generating tests to generate quality code. In similar way, associate a test engineer with a developer. Whenever, developer develops and integrates his code, he invites the test engineers to test. Test engineers performs exploratory testing and explains the purpose of various tests. Developers consolidate the bugs found and fix them later. This approach is good as it helps in bringing the development and testing team on same page, unearth ambiguous requirements, increase collaboration and pulls down the project cost. Gradually developers also starts understanding the testing techniques and starts testing their code that consequently reduces bug count and increase productivity of entire team.

Role of Test Engineers

Keeping in mind the changing paradigm, writing lots of test cases, create big test suites for various type of testing, establishing various traceability matrices, filing defects will not bring the value in Agile. Teams are small and team members work in coloration in time boxed environment.

Functional Test engineers needs to have manual and automation expertise besides good knowledge of database. As the iterations are time boxed, they must be expert in exploratory testing and use skills to optimize test data and test cases. In absence of Product Owner, Testing professional has to play role of product owner. Automating regression suites is need of the hour to save reduce the test cycle in iterations. Team is working closely with product owner to understand user stories, hence the test engineers must have good communication skills and analytical skill.

 

What do you think about testing in Agile, please do share.

44 seconds, to check if you are an Agile team

Does having Daily stand up, Planning, and Retrospective meeting ceremonies, make a Team agile.

Just spent few seconds to respond 11 quick check points, to find out if you are an Agile team. If your response is NO for any question, then you may like to reconsider your claim of being Agile.

#

Quick Check Items

Your Response (Yes/No)

1 Do the team members have direct access to the Product owner?  
2 Do the Team members know the team’s velocity?  
3 Do the Team member estimate the User Stories without influence of the supervisors?  
4 Do the Team members help each other?  
5 Do the Team members respect viewpoint and skills of other team members?  
6 Do the Team members have courage to accept the mistakes without fear?  
7 Does the Iteration (sprint) size fall below 6 weeks?  
8 Does the Team improve and tracks the action items determined in the Retrospective meeting  
9 Does the Team use various Burn down charts?  
10 Is the Team empowered to push back the change in the scope during the iteration?  
11 Are Items in Product backlog prioritized by their business value?  

These 11 questions look at the existing process from different perspective to evaluate the agility. Leave your comments if you have experience in working with Agile teams.

15 point tests for Browser Compatiability

This blog may be useful for you if  you have encountered that your team: is getting Production bug reproducible on specific browser(s) has missed important Browser Compatibility test cases is new to Browser compatibility has to ensure testing coverage on … Continue reading

Is it possible to Automate Accessibility Testing???

Accessibility- Visual Disability  This blog discuss the challenges to Automate the accessibility testing of Web applications made accessible  for people having Vision related issues; such as blindness and low vision etc.

   Starting with a brief about Accessibility, I will discuss the challenges we faced, and then will end with the
solution.

As per W3C, Web accessibility means that people with disabilities can use the Web. More specifically, Web accessibility means that people with disabilities can perceive, understand, navigate, and interact with the Web, and that they can contribute to the Web.

W3C started an initiative; Web Accessibility Initiative (WAI), to lead the Web to its full potential to be accessible, enabling people with disabilities to participate equally on the Web. Web Accessibility testing is to validate that website is accessible to people with various level of disabilities.

Businesses are making their websites accessible to avoid legal issues, expend the business (approx 1 trillion $ market) and remove inequality among people with various level of abilities. Tesco invested £35,000 to make their website accessible and generated £1.5 million in a year from online sales to disabled people in Europe.

Broadly disabilities can be grouped under Sensory (Vision, Hearing), Physical (Hand movement, paralysis etc), and Cognitive (dyslexia, slow processing of information etc) disabilities.

For people with Visionary problems such as low vision or blindness, there are some assistive screen reading tools; such as JAWS, NVDA etc.  These tools read the web content; end user hears and accordingly with the help of Keyboard can interact with the website. Tab key, Arrow keys, Enter Key, Shift, CTRL, ALT, and Space bar are most used keys for navigation.

Testing the Website for vision accessibility is a two steps process;  Step 1, Use free tools where you provide the URL of your website and the tool generates a report showing the how accessible is the website. Take appropriate action. Step 2, Manual test engineers imitate the blind users, hear the web content for correctness, and test the navigation and functionalities using keyboard.

Hearing the content and then verifying what you see on the screen, repetitively is monotonous and boring tasks for manual test engineers.  Disorientation leaves space for missing vision accessibility issues during regression testing. Keeping this in mind, since few days I along with my colleague, trying to automate, accessibility testing.

The foremost challenge was we could not find over web, if somebody has tried to automate Screen readers.  Then next biggest challenge was how to verify if the Screen readers are reading the content right. Another technical challenge is that Screen reader tools are not accepting the Keyboard Shortcut inputs sent by various paid/open source tool such as QTP, SilkTest, Test Complete, Selenium, AutoIT, Robot Api etc.  These Keyboard shortcuts help the disabled people to navigate and use the functionality of the Web page.

The solution; we created an Object Repository which contains all the objects, their IDs, specific attribute which JAWS reads, and expected content. We were sure that if the right content is set in the right property of an object, JAWS is going to read it correctly. We also found during our R&D, that JAWS reads the ARIA labels first. So in cases where the development is at initial stage, I would recommend to ensure that development team is entering content in the ARIA labels associated with an object. These contents will be read by JAWS when user moves control on the object.

What is ARIA (http://en.wikipedia.org/wiki/WAI-ARIA) WAI-ARIA describes how to add semantics and other metadata to HTML content in order to make user interface controls and dynamic content more accessible.)

In our case, the test website is already developed, so instead of asking development team to add ARIA labels for all objects, we collected all objects, and their content in specific attributes, which JAWS is reading. This way we created our object repository and automated Screen Readers. For navigation, currently we are using the Tab key, Arrow keys, Enter Key, and Space bar.  With these keys we are able to check all objects, content for JAWS, and functionalities.

I am looking forward to hear from you if you have suggestions and queries.

Want to save Testing Time: Go for Orthogonal Array

The Orthogonal Array testing technique is a statistical way of reducing the test cases by removing redundant test conditions. A software system works on large number of interactions and integrations. It is very difficult to find out the erring interaction/integrations which the It is observed that major source of the bugs are interactions and integrations.

The bugs in the software are usually caused by a specific Condition independent of other conditions in the system. A condition is a set of conditions or variables, configurations, and logical linking of variables.

Think about testing a simple system, which has four variables and each variable can have 3 inputs. We need 81 cases (3x3x3x3) to test each pair of this system. If we use the orthogonal array techniques, we will have only 9 cases to test the system where all the pair-wise combination will be validated. It will unearth the combination which is causing the failure in the system.

In the above case, Orthogonal Array techniques has reduced the test condition by 89%, i.e. now with 11% of existing test cases all the variable pairs are validated. Interestingly, it will will provide 100% coverage of pair combinations, 33% of three way combinations, 11% of four way combinations. Further to reduce the test condition, orthogonal technique can be used on three way and four way combinations.

Use of Orthogonal array techniques will ensure

  • All the pair-wise combinations of the selected variables are validated.
  • An effective test set containing with fewer test cases
  • Lowers the cost of testing as test cycles are shorter

For one of the client, which is Leader in the Video communications, orthogonal array techniques reduced the test cases from 8171 to 2614 i.e. reduced by 68%. Test cycle efforts were reduced from 164 to 55 person days.

Defect Density and Its Use

Why Defect Density

If you want to:

  • take decision whether a product is ready for release/shipping.
  • predict the count of remaining bugs.
  • estimate the testing and rework due to bugs
  • identify the areas having more bugs
  • determine if the enough testing is done

Defect Density and Formula:

Defect Density is the calculated by dividing the Valid Bugs identified in a specific duration, by the size of the release.

The formula is simple, Defect Density= Defect Count / Size of the release

Size of the release, can be measured in terms of Line of Code (LoC), which is very popular. However, it is better to size the projects using Function Point, Use Case Point, Size of iterations/sprints etc.

It is based on the concept that if you have historical data, then you can predict the count of the bugs in a release. Then based on the difference of Expected Bug count and actual bug count, one can take decision.

Example

This example calculates the Defect Density using KLoC

Release # Lines of Code Defect Count
Pre release
Defect Count
Post release
Defect
Density
1.0 1,00,000 650 50 7.00
2.0 95,000 500 75 6.05
3.0 1,10,000 600 60 6.00
4.0′ 1,20,000 590 55 5.38
5.0 1,50,000 700 50 5.00
6.0 1,90,000 540 55 3.13
7.0 2,25,000 780 35 3.62
8.0 3,00,000 890 20 3.03
9.0 2,50,000 300 20 1.28

The Defect Density in above product is in range 3 to 6. Hence if we consider release 9, then you may like to spend more time on testing as the defect density is yet to go near expected density. In other words, there is still scope of finding 430 to 1180 bugs.

Once you close the testing of Release 9.0, then you will updated the above table, and it may possible that expected Defect Density dips/surges further.

Following Graph is the example, of Defect Density using size (Function/Use point).

DefectDensity