Rational Functional Tester: Test Automation Architecture

This post is part 2 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 3) Creating Automated Tests, (Part 4) Reviewing Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Identify Custom Recognition Mechanisms
IBM Rational Functional Tester provides a Test Object Map that contains a list of every Test Object that is interacted with during the recording process.  Each object is assigned a unique name so that it can be referenced from within the automated test script.  For the majority of tests, the Test Object Map provides a sufficient mechanism for recognising and interacting with the Test Objects within the application.  However, there are some occasions when it is not desirable to use the Test Object Map and a custom recognition method is written to find and retrieve the necessary Test Object.  Review the parts of the application you intend to automate and identify where a custom recognition method may be necessary.  Often this activity will need to be accompanied by an investigation of how IBM Rational Functional Tester interacts with the project’s application and how Test Objects are stored within the Test Object Map.

For each custom recognition method that is identified, add its description to the project’s test automation architecture document.  As a minimum, express the custom recognition method in terms of its signature and an example of when the method can be utilised to benefit the tester.  It is likely that this section will evolve over the life of the project as new custom recognition needs are identified.  As this occurs, ensure that you keep the test automation architecture document up-to-date.

Identify Custom Verification Mechanisms
IBM Rational Functional Tester provides three distinct methods for verifying the state of the application under test.  They are Object Data, Object Property and Object Existence.  For the majority of tests, the relevant verification points can be implemented using one of the standard verification methods.   However, there are some occasions when the test cannot be reliably interpreted as a pass or fail and consequently a custom verification mechanism must be produced to obtain an accurate result.  Review the parts of the application you intend to automate and identify where a custom verification method may be required.  Often this activity will need to be accompanied by an investigation of how IBM Rational Functional Tester interacts with the project’s application and how Test Objects can be interrogated (using the three standard verification methods) to extract the application’s state.

For each custom verification point that is identified, add its description to the project’s test automation architecture document.  As a minimum, express the custom verification method in terms of its signature and an example of when the method can be utilised to benefit the tester.  It is likely that this section will evolve over the life of the project as new custom verification needs are identified.  As this occurs, ensure that you keep the test automation architecture document up-to-date.

Identify Test Bed Components
The key to successful automated testing is a well-defined and maintained test environment.  Combined with the state of the application (typically defined by the information that it contains within the application database) this is known as the test bed.  Before each automated test cycle begins, it is important for the test team to reset the test bed to a known state to avoid misleading results and false failures.

Investigate the application under test and identify the different components that together form the test bed.  Some of the test bed components will remain static during testing (typically, the operating system), whilst other test bed components will change as a result of the automated testing.  Document each test bed component and note which components are dynamic and which are static.  For each dynamic test bed component detail how it will be manually reset to a known state before automated test execution can begin.  This information is typically stored in the project’s test automation architecture document.

Capture Test Bed Automation Needs
Resetting a test bed before test automation begins can be a time consuming activity.  As your familiarity with IBM Rational Functional Tester increases you should consider how you can automate resetting the test bed (potentially to a variety of different configurations).  The complexity of automating such a task will vary greatly depending on the application under test and the test team’s ability to acquire a self-contained test bed.  Ensure that the benefits received by automating the resetting of the test bed outweigh the cost of developing the necessary code within IBM Rational Functional Tester.  For each dynamic test bed component that you decide to automate the resetting of, detail how it will be achieved using IBM Rational Functional Tester.  This information is typically stored in the project’s test automation architecture document.

Define Failure Recovery Routines
One of the distinct benefits of automated testing is the ability to verify the quality of an application without any human interaction.  On occasions (due to test script faults, defects within the application, and incorrect test bed configurations, to mention a few) the automated tests will become unsynchronised with the application under test.  That is to say, the automated test is expecting to interact with a specific test object, but that test object cannot be found as the application is displaying another window or has closed / frozen due to a serious application crash.  In summary the application under test is not in the same state that the automated test is expecting.  If a synchronisation error occurs it will typically leave any unexecuted automated tests incapable of running.  To minimise the number of affected tests, the test team should consider incorporating some form of failure recovery into their automation testing solution.  The sophistication of a failure recovery routine will depend on the ability to control the test bed using IBM Rational Functional Tester.  It is recommended that a test team start with simple failure recovery routines, such as those used to handle unexpected active windows.  As the test team’s experience of IBM Rational Functional Tester increases, they should consider additional failure recovery routines, such as those that are capable of completely resetting the test bed to a known baseline.  As the test team identifies failure recovery routines they should be documented within the project’s test automation architecture document.  As a minimum, each failure recovery routine should be described in terms of the criteria (or signal) that will trigger the routine, the test bed components affected and the state to which the test bed components will be returned.

Define Test Data Structure and Storage Needs
One of the powers of automated testing is the ability to perform the same scenario multiple times, with different variants of test data.  To take advantage of this power, the test data must be stored in a maintainable format, which can be easily accessed by both the test team and the automated test script.  As a rule of thumb, large volumes of hard-coded values should be avoided and the test data should be stored independently of the automated test script.  The possibilities for independent data storage are endless, but a common set of options exist, including IBM Rational Datapools, Excel spreadsheets and external databases.  Investigate which test data structure and type of storage best fits the test team’s needs.  Document the location of the test data, its structure and any special routines required to access the test data in the project’s test automation architecture document.

Define Package Structure
As you automated testing solution grows it will quickly expand to a large number of elements, including various test scripts, datapools and verification points.  To avoid your automated testing solution becoming unmanageable and difficult to maintain, it is a good idea to agree a package structure before your begin the majority of your automating testing development.  The structure of a project’s packages will be determined by their specific needs.  However, it is typical to subdivide test scripts into separate packages that represent functional areas, application feature or use-case.  A separate temporary ‘scratch’ package to keep ideas for new techniques and approaches can also be useful.  A project’s package structure should be recorded within the test automation architecture document.

Agree Naming Conventions
Maintaining an automated testing solution can be significantly aided by using a common set of naming conventions throughout the entire test team.  Naming conventions will also increase readability and help other testers read your automated test scripts as if they were their own.  Many organisation own naming conventions that apply to other area of the software development lifecycle.  Where appropriate, the test team may also use these conventions.  For elements that are specific to the test discipline, such as verification points, various test scripts, and datapools, the test team should define their own naming conventions and record them in the project’s test automation architecture document.

Identify Common-Code Owner and Policy
Reusing common-code within an automated testing solution can dramatically reduce development time and increase maintainability by providing a single point fix.  IBM Rational Functional Tester supports the use of common-code through a Helper Super Class that is inherited by each test scripts.  As each test script within a Functional Tester Project inherits from the Super Helper Class any changes made to the Super Helper Class may affect the way in which each test script performs its tests.  Therefore any changes made to the Super Helper Class (that is to say, the common-code) should be controlled.  The formality of this management will vary depending on the size and complexity of the project and the associated automated testing solution.  Look within your project team and identify a person or group of people that will act as the common-code owner.  It will be the common-code owner’s responsibility to review and agree any changes and additional to the Super Helper Class.  Within the project’s test automation architecture document identify the project’s common-code owner and how they will handle requests to alter the Super Helper Class.  Also describe how requests to the common-code are to be submitted and managed.

Rational Functional Tester: Creating Automated Tests

This post is part 3 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 4) Reviewing Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Prioritise Tests to Automate
An automated testing solution is most effectively developed using an iterative approach.  By focusing on automating the tests that are most difficult to execute manually and those that will add the greatest value to the project team, a test team can achieve 80% of the benefits of automated testing through 20% of the effect.  By developing iteratively the test team can also focus on a subset of the tests to be automated and more easy mange the change and quality of the automated testing solution. Review the list of tests to be automated and rank each test based upon its ease to be automated, its complexity to be executed manually and the quality risk associated with the part of the application it will validate.  Record the results of this activity so that it can be refined during later automation development iterations.

Select Top <X> Tests to Automate
Review the list of prioritised tests to automate and identify which tests will be automated during the next automated development iteration.  Whilst it would be ideal to only automate those tests that ranked highest on our list, in reality a range of high and low priority tests typically need to be automated during each automated development iteration.  This selection is often driven by the dependencies between the tests.  It may be necessary to automate a low priority test to support the execution of a one or more high priority tests.  Avoid including too many tests within an automated development iteration.  Clearly record the tests you plan to automate in this automated development iteration so that the test team has a precise definition of the scope and know when to stop and review.

Assign Automation Activities
Based upon the agreed scope of the automated development iteration, record which tester will be responsible for analysing, designing and developing each automated test script.

Setup Test Environment
Before each automated test script is created, ensure that the test environment is set to a predefined state.  By knowing the state of the test environment when the automated test script is created it will allow the test team to produce more reliable results when using the automated test script to validate future release of the application.  Typically this will involve some form of basic resetting of the operating system, restoration of underlying databases to the require state, and the setup of any peripheral devices, such as printers. While some tasks can be performed automatically, some aspects typically require human attention.

Set Recording Options
Before each automated test script is created, ensure that IBM Functional Tester is correct configured to support both the application under test and its specific domain (such as .NET, Java or Web).  If IBM Functional Tester is not configured to work with the specific domain then its ability to interact with the application under test will be greatly reduced.

Record Test Script
The easiest way to initially create the majority of automated test scripts is by using IBM Functional Tester’s record facility.  This will automatically amend new Test Objects to the Test Object Map and provide a basic script that can be later refined to become more resilient to change and easier to maintain.  Using the project’s naming conventions, create a new Automated Test Script and begin performing the steps to reproduce the test.  Once complete, add additional comments to the Automated Test Script to increase readability.  As a rule of thumb, if you are automating an existing manual test script, each step within the manual test script should become a comment in the automated test script.

Refine Test Script
Many automated test scripts will benefit from being manually refined after they have been initially recorded.  This process is performed for a variety of reasons and can be as simple as reformatting the automated test script code or removing unnecessary action precision (such as removing specific coordinates when they are not required).  Other refinements may include the introduction of a datapool or a custom recognition mechanism.

Replay Test Script (Against the Same Application)
Regardless of whether an automated test script was refined after recorded or not, the test should always be executed against the same version of the application used to initially develop it.  Using this approach, a test team can confidently identify any failures or error produced by the automated test script as issues with the test itself and not the application.  If the test passes successfully then no further action needs to be taken.  If the test fails then the ‘Debug Script’ activity should be performed.

Debug Script
When an automated test script fails against the release of software used to initially create the test then three possible actions can follow.  (1) A major issue exist with the automated test script and the easiest way to fix the problem is to start again and rerecord the test.  (2) A minor issue exists with the automated test script and a small manual refinement will fix the problem.  (3) The source of the problem is unknown or the fix for the problem is complex and unclear.  Using the test log and the debugging facilities offered by IBM Functional Tester discover in which category (1, 2 or 3) the problem belongs.  Take the necessary action to fix the problem or if the problem cannot be easily fixed performed activity ‘Capture Script Issues’.

Capture Script Issues
To keep momentum within a given automated development iteration, if a significant problem is discovered that has no immediate fix or requires significant custom code to be developed then leave the automated test script in a known condition and record the script issue in a convenient location.  As a minimum, each script issue should contain the name of the script(s) that contains the problem, the severity of the script issue in terms of how it will affect automating the remainder of the application, a headline that briefly describes the problem, and finally a detailed description of the problem.

Rational Functional Tester: Reviewing Automated Tests

This post is part 4 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 3) Creating Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Review New Scripts
At the end of each automated development iteration the test team must verify the quality of the automated test scripts that they have produced.  The formality and thoroughness of this activity will depend on the scale and critically of the automated testing solution being produced.  Typically, the larger and more important the automated testing solution is, the greater the effort that should be applied to this activity.  When a test team is new to automated testing the most effective way of reviewing automated test scripts is to perform peer-reviews with the support of automated test script checklists.  A checklist is a simple list of questions specifically chosen to help the reviewer think about different aspects of an automated test script and whether it is conforms to the agreed approach to automated test script development.

Analyse Change & Issue Log
At the end of each automated development iteration the test team (specifically the common-code owner) must review the outstanding changes and issues associated with the automated testing solution.  After removing any duplicates, the changes and issues should be analysed in terms of their importance to the automated testing solution and the amount of effort required making the change or investigate and fix the issue.  Any changes made to the automated testing solution should be completed before the next automated development iteration is scoped.

Refine Test Automation Architecture
As a result of making any changes that were agreed during the ‘Analyse Change & Issue Log’ activity it may be necessary to refine the test automation architecture document to include alterations made at the architectural level.  This may include, for example, any new custom recognition or verification mechanisms identified to fix a test script issue.

Rational Functional Tester: Using and Maintaining Automated Tests

This post is part 5 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 3) Creating Automated Tests and (Part 4) Reviewing Automated Tests.

Review Release Notes for New Build
Before the automated testing solution is used to verify the quality of the application under test the test team should review the release notes associated with the specific build or release.  This should identify both the changes that have been made to the application and any unfixed defects that were discovered during previous test cycles.  By having this information accessible to the test team, any known defects discovered by the automated tests can be ignored.

Setup Test Environment
Before any automated tests are executed the test team must reset the test environment to a know state.  This may be performed manually or may be automated as part of the automated testing solution.

Run the Automation
Automated tests can be executed at anytime of day or night.  If the automated test scripts are to be run overnight the test team must consider how the execution will be scheduled.  Typically, the simplest approach is to utilise the IBM Functional Tester command line interface and the scheduler built into the local operating system.  Regardless of whether the automated tests are executed manually or scheduled, the test team must consider how the automation will affect other test environment stakeholders, such as other test teams.

Analyse Failures
Once all of the automated test scripts have been executed the test team must analyse the test log and review any test failures.  A test script failure can fall into three different categories.  (1) The test script failed because the test environment was not reset correctly and consequently the application under test was not in the expected state.  (2) The test script failed because the test script itself contained an error.  (3) The test script failed because there is an error in the application, which must be investigated.  As the test team becomes more experienced in automated testing they will appreciate that the greater the amount of information and failure details contained within the test log, the easier this task is to perform.

Mark for Redevelopment
If a test script fails and the test team identifies the problem to be an error in the automated test script, the test script should be marked for redevelopment but not immediately fixed.  Unless fixing an error within a test script is necessary to complete the remainder of the test cycle, any fixes to the automated testing solution should be made within the subsequent automated development iteration and the application under test should be validated using the original manual test script.  Following this approach will allow the test team to remain focused on the primary objective or the test cycle (verifying the quality of the application) rather than becoming preoccupied with fixing error in the test scripts.

Log Defect
If a test script fails and the test team categorises the failure as a failure in the application under test then a defect log  should be entered into the project’s defect tracking database.  An a minimum each defect log should contain a brief headline that summarises the defect, a detailed description that explains how to recreate the defect in the application and a severity rating that provides an indication as to how important the defect is for the project team to fix.

Produce Test Evaluation Summary
At the end of any test cycle, the test team should create a test evaluation summary that describes the quality of the application.  Test evaluation summaries will vary from project to project, however a common set of topic exist.  (1) A subjective assessment – A brief description of the perceived application quality.  (2) Risks, Issues and Blockers – Anything that is preventing the test team from performing the testing effort.  (3) Progress against the Plan – Number of test executed, passed and failed.  (4) Defect Analysis – Metrics related to defect trends, defect densities and defect ages.  (5) Coverage Analysis – How the tests performed relate to project requirements, project risks and application code.

Rational Functional Tester: Planning Test Automation

This post is part 1 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 2) Test Automation Architecture, (Part 3) Creating Automated Tests, (Part 4) Reviewing Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Capture Project Background
Start the automation development effort by understanding the project background.  Pay particular attention to the way that the application is currently tested and how the testing process relates to other disciplines within the software development lifecycle, such as requirements, coding, and configuration and change management.  Review any existing process documentation and consider augmenting the process with additional information that describes how test automation will be incorporated.

Scope the Automation Effort
Many automation development efforts fail to meet project expectations due to poorly managed scope and a lack of clarity and visibility around the motivation for adopting automated testing.  Spend some time analysing the problem behind the motivation to ensure test automation is the correct solution.  Communicate the expected benefits of automated testing in terms of the problems it will solve.

Sometimes the tests to be automated to already exist in terms of manual test scripts.  Before the automation development effort begins, it is important to agree what will be automated, if any.  This can be defined using a variety of methods.  Some project teams find it easier to express the scope of the test automation in terms of existing manual test scripts, simply by providing a list of existing manual script names or IDs.  Other project teams find it easier to express the scope of the test automation by subdividing the application into different areas of feature-set and identifying which areas will be automated.  Either way, it is a good idea to agree what will be automated before starting the automation development effort.

Collate and Refine Manual Test Scripts
If a collection of manual test scripts already exist, there is a possibility that over time they have grown organically and become unsynchronised with the application requirements.  Review the existing manual test scripts against the application’s requirements and remove any duplicate or redundant tests.  Ensure that the purpose of the test is unambiguous.  Furthermore, ensure that any data and information exchanged between the tester and the application is well understood.

To support maintainability and reduce intellectual complexity, an automated test should be atomic and avoid unnecessarily long sequences of user actions and verification points.  To maintain a one-to-one mapping between manual test scripts (which should be kept up-to-date, even when the test automation exists) it may be necessary to divide a single manual test script into one or more smaller tests.  Provide any new tests with a unique name or ID and document the test steps and verification points appropriately.

Identify Test Suites
It can be useful to arrange the refined manual test script into a number of test suites.  A single test suite should focus on a specific area of the application and state the sequence in which the containing tests are executed.  By cleverly sequencing tests within a test suite, different tests can support each other.  That is to say, an early test in a test suite can manipulate the application into a particular state that can be used as the foundation for a subsequent test in the test suite.  Test suites can also be used to manage the scope of the automation development effort and be used to focus the test team on a specific subset of the tests to be automated.

SQS Conference Slides Available to Download

For those of you that didn’t manage to attend my presentation on how much detail to include when writing test cases (entitled “A thinking framework for context driven test documentation”) you can download the slides from the IJI website here.

IBM Rational Quality Manager Webinar – 7th Nov 2008

I was recently asked to deliver a webinar on IBM Rational Quality Manager (RQM), as part of the free winter webinar series (click here to register) from Ivar Jacobson International.

I’m still working on the detailed agenda, but you can expect to see a series of RQM demonstrations within the context on a light-weight testing framework.  For me, the testing framework is just as important as the tool, as if you haven’t agreed how testing is going to work on your project then you haven’t got much chance of making it better by introducing a tool.

The testing framework I’m going to use focuses on the essential aspects of testing and anyone attending should easily be able to see the similarities with their own way of working.  If it’s a bit unclear how you go about testing, then not only will this webinar give you an introduction to RQM, but it should also give you a good place to start thinking about your own testing process. Two for the price of one! ( not a bad offer, if it wasn’t already free )

Click here to register.

“The Tester” – Call for Articles

Last week I started a new role as Editor for “The Tester” – An online magazine published every 3 months that focuses on software testing and quality assurance. You can view past editions here.

 

In a typical edition, you will find:

important announcements from the testing community;

programme information and dates for future SIGiST Conferences;

articles covering everything from case-studies to cutting edge research.

 

And it is the articles section where anybody from the world of testing can submit material to be included. If you have a testing story you would like to share, a test technique you would like to evangelise or testing research you would like to publish, then this is the place to do it.

 

What’s in it for me, you may ask? “The Tester” is read by a wide audience from testers and test managers to consultants and trainers, so there will be plenty of people interested in your material – You may even find someone to collaborate with in the future. You will also have the chance to present your material at a future SIGiST conferences and talk face-to-face with your peers in the industry.  “The Tester” recently won the British Computer Society (BCS) Best Specialist Group Magazine award, so you will also be able to tell your friends, colleagues and future employers that you have been published in a award winning magazine!

 

So don’t hold back, get working on your material today. If you’re not sure whether your material is ready for a wider audience, feel free to send me what you have or a brief abstract / introduction and we can work from there.  My email address (displayed as a picture) is…

Software & Systems Quality Conference Highlights

I’ve just got back from two good days at the UK Software & Systems Quality Conference.  After speaking myself (you can download the slides from here) on Monday morning I then had the rest of the conference to catch up with old friends and listen to what the other speakers had to say.

There was a large number of talks this year related to outsourcing and centralising testing.  Many of which were companies sharing their own experiences, which suggests that this is a trend already well underway and being driven by IT departments rather than outsourcing or process improvement organisations.  The best centralising testing talk that I attending was delivered by Mark Mitton of Lloyds TSB.  He’d clearly delivered a huge number of benefits since centralising testing at Lloyds TSB, but still has plenty of ideas for improvements in the future.  The best outsourcing talk I attending, without doubt, was delivered by Andrew Brammer, who is Head of Systems Development at Allen & Overy.  For those of you that don’t know, Allen and Overy are an international legal practice and whilst their testing experiences are in their infancy (something that Andrew quite happily acknowledged), it hasn’t stopped them creating an extremely effective outsourcing framework based upon risk and reward.  I guess that suggests that if you want to outsource testing, its better to be a law-guru than a testing-guru

Richard Noble’s keynote speech on the second morning was also extremely good.  It’s interesting that many of the reasons he gave for being successful closely align to many of the practices that are promoted in software development today.  Similar to how Lean has made the transition into software development, maybe Richard could create his own software methodology too – Supersonic Software Development

Finally, thanks must go to Julie Wells and the rest of the conference team for organising the event.  Everything run to time and most importantly, the food and drink was superb.