Test Strategy – Service Level Management Questions

Business Objective and Success Factor Related

  1. What are the business objectives?
  2. What are the critical success factors for this project?
  3. Have you defined SLA’s?
  4. What are the entry and exit criteria for each of the Testing phase?
  5. What is the % of Regression Test bed that can be reused?
  6. What % of the test cases has been automated?
  7. What deliverables do you normally expect from us?

Test Basis Related

  1. Do you have well defined requirements document?
  2. Do you have functional specification document?
  3. Are the design documents are available? If yes, can you share them?
  4. Can you share the architecture document?
  5. Are interface specification documents available?
  6. Do you maintain traceability matrix?
  7. Can you share the available test cases?
  8. Do you have the regression test bed?

Knowledge Transfer and Training Related

  1. We expect the Client to provide us knowledge transfer/hands-on on the tools/product. Please validate our assumption.
  2. What are the different tools used for: Test Management, Defect Management, Automation etc ?
  3. Can you share the licenses of the 3rd party tools?
  4. As we don’t have the competency on XXX product/tool, can you bear the cost of Training?
  5. Are there any special training required for this project?
  6. What is the License period of the software?

Metrics Related

  1. Do you need Metrics?
  2. What are the different Metrics you capture?
  3. What the Metrics captured for measuring productivity?

Status Reporting, Monitoring and Control Related

  1. What are the metrics that are expected from Sr. Level Management?
  2. Do you publish Daily, Weekly and Monthly Reports?
  3. Do you have any dashboards for reporting?
  4. How many defects are uncovered during production?
  5. Please provide the point of contact for Escalation?

Warranty and Support Related

  1. What is the Warranty period?
  2. Do you have annual maintenance Contract?
  3. Can you share the trouble Tickets data for the past one yeas?
  4. What are the support hours?

Test Strategy – Portfolio related questions

It is good to make a portfolio of the project that will help in finding clients SLA to their customers, availability of the Artifacts, test basis etc.

Test Basis Related

  1. Do you have the test bases? Is so, what are those?
  2. Could you please share with us your previous test basis, which might include test cases, test results, bug reports and so on and so forth?
  3. Could you please share with us the architecture and Design documents?

Product Related- Non-Functional

Could you please share with us?

  1. How stable is the product?
  2. How frequently the product crashes?
  3. What are the MTBF and MTTF?
  4. Specific patterns of the product crashes, if any?
  5. How frequently the application undergoes changes?
  6. Is there any Change Management in the process?
  7. Do you foresee this product be retiring in the couple of years to adapt the new technologies?
  8. Do you have any migration plan?
  9. Do you have identified the new technologies and platforms?
  10. On what time zones the application will be running?
  11. Do you need multilingual support?
  12. You have any planned security measures?
  13. What are the security protocols this product will use?
  14. If you need role based security levels?
  15. If you have any pain areas in the current look and feel of the system?
  16. If the current system in intuitive by nature?
  17. If you want your system to be portable?

SLA, IPR, Support and Impact of Failure Related

Could you please share with us?

  1. Your support SLAs with your customers?
  2. The sensitivity aspects of your customer artifacts?
  3. Is there any IP involved in any of your customer or your artifacts?
  4. Is the application data intensive?
  5. Will you provide us the data generator tools or we need to create this on our own.
  6. What are the platforms you would like to support?
  7. What is the impact of failure of this product on your business and on your end customer?
  8. What are the liabilities in cases of failure of this product?
  9. What the logistic support required and how they will be achieved?

Back up and Disaster Recovery Related

  1. Do you need daily/weekly /Monthly backups?
  2. Do you have a disaster Recovery plan in place?

Status Reporting and Vendor Interaction Related

  1. If you have any reporting and meeting protocols in place for example Daily status calls over VC or telephone etc.
  2. The protocol involved in interacting with your other vendors?

Test Strategy – Process Understanding Questions

In cases, where client is already following some processes to manage various phases, then it becomes very critical for the Test Managers to understand them. These process related questions helps in finding the existing processes, their maturity, shortcomings, limitations etc. It helps Test Manager to decide Test Approach, Test Deliverables, etc.

Testing Model/Process

  1. RUP/WaterFall
  2. Agile Scrum

Requirement Management Process:

  1. Do you have any standard template for Requirement Specification?
  2. Who will be the points of contact?
  3. Are there any Key Agreement points for Requirement?
  4. What are the Test basis documents that you would provide?
  5. Do you have any Requirements Traceability Metrics?

Test Management Process

  1. Would you please describe the Model you use for Testing?
  2. How is it aligned with the Development life cycle?
  3. What are the different phases of testing?
  4. What are the Metrics to be provided?

Environment Management

  1. Would you please describe the Platform?
  2. What is the automation framework and license dependency?
  3. Who will provide the test bed and supports?
  4. Is the system to be test is the stand alone system or Depends on the External system?
  5. Where do we receive inputs to the system?
  6. Where we need to supply the output to?

Release Management

  1. What are the entry and Exit criteria for the Release process?
  2. Are there any intermediate releases and what is the Frequency of the releases?
  3. What are the deliverables as part of each release?

Change Management

  1. Who are the stake holders for the change?
  2. What is the CR intimation procedure?

Configuration management

  1. What is the configuration management tool to be used?
  2. Who is responsible for the Configuration Management?
  3. Who is responsible for the build?
  4. Is build use any Automation tool? Is so, what about licensing and use?

Defect Management

  1. What is the Service Level Agreement (SLA) for defect turnaround time?
  2. What is the tool be to used for defect management?
  3. What are the criteria for Root Cause Analysis (RCA)?

Test Strategy – Requirement Elicitation Questions

There questions will help in finding the data/information about the Target project such as functional/nonfunctional priorities, potential risks. These questions basically helps in finding the types of testing to be performed.

Scope:

  1. Do you have any kind of special Non-Functional Requirement? (Performance, Security, Usability, Localization, Browser/OS Compatibility)
  2. What are the applications/Modules that you need to be tested?
  3. Do you have any interoperability requirements?
  4. What are the upstream/downstream processes and are they also in the scope?
  5. Is there any migration activity for which testing is to be performed / planned?

Risks:

  1. Are experiencing any network connectivity issues?
  2. Do you have a need for the protecting the sensitive data from leakage?
  3. Do you have issues with expertise availability in Testing/Process/Technology area?
  4. What is the average experience of the developers engaged in the Project?

Objectives:

  1. What are your requirements with respect to the application availability and response time?
  2. What is the maximum permissible number of Defects (Critical/major) ini the application in production?

Constraints:

  1. Do you have any limitations with respect to the Network bandwidth?
  2. Do you have any budget constraints with respect to the Tools usage?
  3. Do you have any budget constraints with respect to Resources?
  4. Do you have any constraints while performing Remote Testing?

Application Size:

  1. What is the application size in terms of the number of the function points?

Dev Effort / Size:

  1. What is the effort that was put into the Development activities for the application?
  2. What was the size of the development team?

Current Project Status:

  1. At what stage is the current application with respect to SDLC /Maintenance?
  2. What is the status of the various test Artifacts and test base documents?
  3. Are the requirements in the form of Requirements Models or in the form of an RS documents?

Testing Team:

  1. What is the current testing team size?
  2. What is the Team composition with respect to skills/experience?

Automation:

  1. Do you have any plans for automation?
  2. What are the existing automation artifacts available? (Scripts, Design documents, Execution/ Coverage reports)
  3. Do you have any specific vendor preference for the automation?

Process maturity:

  1. What is the current maturity level of Development Process (CMM/ISO Standards)
  2. What is the current maturity level of Testing Process (CMM/ISO Standards)

Pain Areas:

  1. Do you have any issues with respect to the Application Downtime/ Instability
  2. Are you experiencing any application Performance Issue?
  3. Are you experiencing any Compatibility issues across platforms / OS /Browsers?
  4. Do you have any concerns with respect to Security, Reliability, Usability ratings?

Domain:

  1. What is the domain area in which the application operates?
  2. What is the Service line for the application within the domain? (Mission critical, Real Time app, Embedded systems, Internet app, client / Server app, Mainframe.)

Testing Challenges:

  1. Do you have any challenges with respect to the Lack of Technology / Tool Expertise within your current team?
  2. Do you face issues with respect to thigh cost of Subject Matter Experts?
  3. Do you have any challenges with respect to the lack of expertise in testing/ Processes/ Methodologies?
  4. Do you face frequent Delays in deliverables from the development team?
  5. Do you have pressing needs for the Reducing Testing Lifecycles time?
  6. Do you have the need to go in for Frequent Change Requests?
  7. Is there lack of Clarity in the Requirements?
  8. Is the application having stability issues?
  9. Is your team facing Attrition problems?

User Base:

  1. How many users does your system have?
  2. What is the experience level of the users?
  3. Where are the Users located? (Timezones)
  4. Are you facing problems supporting varying time zones? (With respect to the distributed applications, batch processes)

Platform – Hosting, Hardware/Software:

  1. Does the Test Environment simulate the production environment? If not, please provide the actual configuration of both the environments?
  2. What is the platform your application is hosted in? (Unix/Linux/Solaris/Windows/Others)?
  3. What is the Configuration of the servers?
  4. What is the network speed?
  5. What is the typical Desktop Configuration used?
  6. What is the Database size?
  7. What is the Application server used? (Weblogix, Websphare, Apache Web server, IIS)
  8. What is the database used? (Oracle, Sybase, SQL Server, DB2)
  9. What is the Technology Used? (java/J2EE, .NET, SAP, SOA)
  10. What is the Middleware used? (Tibco, MQ Series)
  11. Do you have any legacy Systems and are there any plans for the porting the same?
  12. What are the Development tools used? (Eclipse, Netbeans)
  13. What are the Testing tools used? (Open source, Licensed, name of the vendor)
  14. Who will be bearing the Licensing costs?

Others:

  1. Are there any regulation compliance specific requirements? E.g HIPPA, SOX
  2. Are there clustering, Load Balancing mechanisms in place?
  3. What is the current application maintenance cost with respect to the technical support and environment setup?

Test Strategy – Develop Test Strategy

A test strategy can be developed for a project/product which can be in any phase such as Requirement, Coding, Testing or Maintenance phase. Test Strategy usually is developed at the initial stage of the “Testing”.  There could also be scenarios where you get a project for testing whose development is over and the client is outsourcing only “Testing”

Regardless of the phase, we have to start with some basic activities to develop test strategy, i.e. understanding the business issues, requirements, processes, working environment, development and testing practices, and expectations of the customer. Once we understand them well, we have the basic blocks, and now we have to put them in place.

Now how to understand the business better and quicker?

The best way is to come up with questions, as many as possible. And then try to find out their answers with the help of documents, SME, Knowledgebase, History; Client’s appointed Point of Contacts, Internet, Proof of Concepts etc.

I have prepared a long list of questions, which I have divided in to four categories for better readability and learnability.  These question are helpful in getting the information, which will be used to prepare Test Strategy.

The categories are:

1. Test Related Requirement Elicitation

2. Test Related Process Elicitation

3. Test Related Portfolio Management

4. Test Related Service Level Management.

The list of questions to be asked under each of above mentioned categories are given in different Blog posts. Please go through them and add values by adding questions and suggestions. I will keep them updated based on your suggestions.

Test Strategy and Test Plan; Difference

Test Strategy and Test Plan are two different documents which are developed for different audience with different objective.

Test Strategy defines the overall approach that will be taken by the Test Team when delivering testing services to client. It helps various stakeholders to understand the testing activities, roles and responsibilities, processes and practices to be used during Testing of a project/Product.

Test strategy conveys:

  1. The Business of the Test Project
  2. Test Deliverable
  3. Standards to be followed
  4. Automation tools
  5. Testing Measurements and matrices
  6. Risk identification and Mitigation
  7. Defect Tracking process
  8. Change Management process
  9. Configuration management
  10. Roles and Responsibilities
  11. Communication and status reporting process
  12. Training Plans
# Test Strategy Test Plan
1 Owner PM Test Lead
2 For Management/Team Team
3 Input Business Requirement Specification document SRS,
Use Cases
4 Modification
Frequency
Low Frequent
5 Project /Product Size Large / Medium Small / Medium / Large
6 Scope Entire Project/Product Entire ProjectProduct (SMALL)
Phase (Unit / Integration / System Testing)