10 points you need to know to Sign off automated tests

10 points you need to know to Sign off automated tests
Like Tweet Pin it Share Share Email

The failure rate of test automation implementation project is higher due to several factors, we shall publish a separate article on it, but for the scope of this article you will know in detail about what you need to take care as part of Script Sign Off process.

Following these can help you in delivering good and stable test automation assets that can give smoother execution cycles whenever needed.

1. Scripts to execute seamlessly irrespective of url / instance as long as the functionality is unchanged for the application.

As part of the automation script development process, automation engineers would be asked to use either a Dev or QA instance of the application under test, i.e. the business application, and in reality these scripts are executed against different instances for the same business application. There will be many places in the script where the dependencies could be based on the url or reference to the application under test, for example:

  • Launching of application under test.
  • Clicking on a Link
  • Clicking on an Image
  • etc…

It is always good to check if your scripts are running seamlessly without changing the automation script.

How to validate?

Ask your counter parts to execute the scripts against different instances or urls and the scripts should be executed seamlessly without any surprises of failure or abrupt termination of script execution.

Executable on Different Instances

2. Scripts to iterate business transaction validation for different sets of test data.

The value of test automation is highly realised only when the automated test case can be executed for different sets of test data without any change in the automation script, in other words the scripts are to be
parameterized for test data, i.e. you can supplly different sets of data to execute the same business transaction and validate the desired check points based on the test data provided.

One other important point here is to make sure, the test data variation can limit only to the business transaction that has been automated as part of the script, but not for the new business flows that can emerge due
to test data variations.

Note: There is always a debate that the scripts need to handle multiple business flows, but we recommend you not to implement it, as it leads to different other challenges in reality which would be published in a separate article by us, as it does not fall under the scope of current article.

How to validate?

So, once the script is ready for execution, ask your counter part to provide multiple sets of test data and execute the scripts to make sure it runs seamlessly. In case of any issues, script / framework need to skip the current test data iteration and continue to execute the automation script for the remaining test data iterations.

Test Data Iterations

3. Partial automation of test data generation.

As part of test data parameterization of the script, there are cases where test data needs to be dynamic or unique values, script or the framework should be handling it properly, so that the same script with same test data would be able to run for multiple times.

Dynamic test data:

  • Where the test data could be a current date or 30 days from current date or 30 days before current date etc.
  • Where the test data is based on a calculation.
  • Where the test data is generated in the initial stages of the script and the same needs to be used in different parts of the script, for example creating an employee might generate an employee id, and this id needs to be used as input in the same script at different place of business transaction like, adding bank details to the employee.
    and similar other cases.

How to validate?

Based on the script, ask your counter part as what is the format to be given in test data to handle above cases, provide test data and ask for execution of scripts. On completion of script execution, review the test results to understand if everything is as expected.

Dynamic Data

Identify the automation scripts where there is need to propagate generated data to be captured from application under test and will be used in the further steps of script, execute these kind of scripts and make sure nothing fails.

Unique test data:
This is the case where some of the fields in the application accept only unique data, for example: contact number for an employee should always be unique, so when this field is parameterized, the script or framework should take care of it.

How to validate?

Ask your counterpart for the format to be followed to have unique data being used as test data, once provided, execute the script for same test data multiple times and make sure that script executes fine without any issues. Review the test results to see if the unique test data is generated properly and if the test results are as expected.

Unique Data

4. User friendly test results.

In case of any script execution failures, you should be able to understand where exactly the script failed in the business transaction flow.

Better to have screenshots as part of test result, which will help to understand if the script executed properly or not, especially in the cases when script fails.

How to validate?

Ask to share all the test results in a shared location and review all the test results and understand if they are easy to understand and be able to identify the validation points easily and clearly.

Review if the dynamically generated values are encapsulated as part of the result file, i.e. for example: booking an item in an e-commerce website, should capture the booking id, which can be used as reference for review or later part of the script.

5. Scripts to validate all the check points required for a business transaction or setup.

Make sure all the checkpoints / validations, which you want to incorporate as part of the automation script are incorporated as part of the script.

How to validate?

You can ask to execute the script and review the test result generated having the list of all check points being validated.

6. Functional Pre-requisites to execute a script.

This is one of the most challenging and dangerous this to be taken care of, the scripts are developed by automation engineers which performs a business transaction on the application under test, but there is always a hidden factor that, in order automate the script there will always be some functional pre-requisites which they might have done manually before starting the script development or stabilisation process.

It is generally known to the business users, but this might become challenging if the number of scripts as part of the project are huge, so it always better to have collaboration between the business user and automation engineer so that these things are properly documented and is used during the execution of these scripts. Most of the times these are given least priority but once we start executing the scripts multiple times for different test cycles it becomes more handy and useful.

For example: The test case is to search for an item during the discount period and place the order in an e-commerce website, during the development of test case, the functional user or automation engineer might have enabled discount for the time duration and completed the test case automation. But later when they want to re execute this script at a different time period, script might fail on some of the validations related to discount. So this functional configuration need to be enabled in the application under test and then the script needs to be executed. There are several ways to handle this and that topic of discussion is subjective to the scope of current article.

How to go about it?

Have a collaborative effort in documenting all these at the time of script development and keep updating regularly whenever additional points to be considered.

7. Test Data preparation documentation.

This is one of the most time consuming and challening aspect where utmost care need to be taken, generally when you want different people to execute this scripts or if there is scope to add more test data iterations or need to run for different sets of test data in every cycle, it is huge and combursome activity for you to understand what test data need to be given for this field?

For example: one of the test case is to check if discounted items are getting 20% deduction on the selling price, now when developing or stabilising the automation script, functional user might have shared few items for which discount is enabled. Now the challenge comes when you want to run this for few more test data iterations, you have to first provide valid test data and then you would be able to execute the script successfully. As long as the same user was asked to execute the script and assuming the user remembers how to be pick the test data from application under test it is always easy for him / her to execute by sourcing new test data. Which otherwise of a new person is always challenging and needs more time to provide the test data than the actual execution of the test case.

How to go about it?

For each test case, ask your business user or test automation user to document as what are all the fields in the test data sheet and are the ways of sourcing the test data to execute with new sets of test data. For example in the above case could be, have a database query documented to retrieve an item for which discount is enabled or it could be detailed steps that are to be performed on application UI to get the desired item name.

8. Scripts perform the business transaction / setup as per the intended test case.

As a general practice or approach, as a business user you either share an existing manual test case or provides KT on the business transactions to be tested. Once the automated scripts are developed, it is always better to review the automated script if it does what it is ought to do, this can be done by viewing a live script execution / a recorded video of script execution.

But the down side is, you need to spend more time to review every script by watching the live execution / recorded video, as an alternative, you can adopt following phase wise approach:

How to go about it?

  • Phase 1 ( for the first n scripts of project ) – Ask for live executions, the count n is determined based on the project size, script size and duration of the project.
  • Phase 2 – Ask for recorded videos, this is optional based on the confidence you gain in previous phase.
  • Phase 3 – Ask for execution results, but make sure the results have screenshots and the result format is easy to understand.

This approach will speed up the review process and test automation project implementation without compromising on the quality.

9. Gain trust on your test scripts:

Quite frequently we hear script ran successful last time and not now, need to debug and get back, though this creates mistrust on the script and you lose confidence whether to use this for actual test cycles? Though, this cannot be eleminated completely, you can do the following to make sure script is stable and can be trusted.

How to validate?

Ask your vendors / in house team members to run the same script for 5 times at different timings (i.e. not consecutive) and share the test results. If the script passes for 4 times and above you yourself will gain the trust, otherwise there is still scope to improve script stability and your counterpart need to work on.

10. Scripts are seamlessly shareable and integrate able:

As you are aware scripts are generally developed with a test automation framework in place, though the test automation frameworks are quite useful for multiple reasons they come up with some pre-requisites and conditions to execute the scripts. One of them could be, scripts need to are to be arranged in a specific folder structure, dependent assets to be placed in a specific location and only then the scripts are executable.

You should have thorough understanding of the pre-requisites to execute a test case on your own if you intend to execute them without support from vendor or in house team. Most of the times you develop automation scripts and utilize them for multiple executions in later point of time, so unless you document you will not be able to remember all the important points to execute on your own.

How to validate?

Make sure the framework or tool or platform used, allows you to easily execute scripts on your own without any hurdles of first setting up the framework and other dependencies.

In case your test automation platform can trigger the executions on multiple configured machines, its the responsibility of the platform owner to configure the machine to be able to execute on it.


Comments (7)

  • Most of the times we get concerns from the customer that the dynamic data is not handled specially with respect to dates. It has been very nicely described in this article all the points that needs to be taken care, these things can also be put in as a checklist to deliver to the client along with you script to get an easy sign off.

    • Author

      Thanks for sharing your views, I am glad that you liked this article about what are the things to be considered when you need to review and sign off scripts and at the same works as a checklist for the automation testing people to deliver their scripts

    • Author

      And yes, these can be made as a checklist to your test automation team to get an easy sign off. this would make life easy for both the parties

  • Very Nice article Srini, Given lot of information.

  • Thanks for sharing you knowledge in terms of blog. Its really helpful & i will create check-list for my automation deliverable things, so can able to deliver more qualitative work.

    • Author

      I am really happy that you are using this as a checklist for your project works, I would be more than happy to hear back from you any suggestions and additions.

      Please feel free to do so.


Leave a Reply

Your email address will not be published. Required fields are marked *