SIGiST Testing Conference – December 2009

It’s surprising how quickly time flys.  It feels like just the other day that I was sitting in the audience of the December 2008 SIGiST conference and here I am giving an update of December 2009!

In my opinion, the best talk of the day was by Steven Ramsay who spoke about overcoming various obstacles to set up a testing practice at LinkLaters.  What I liked about Steve’s talk was that it was a description from the front-line, worts and all.  Steve suggested at the beginning that he was feeling nervous, but it didn’t show.  When it comes to creating the programme for the 2010 conferences (11th March, 29th June, 16th September and 8th December), I’ll certainly be voting for him to come back and give us an update.

Another good talk was by Martin Gijsen.  Martin spoke about how to make use of a Domain Specific Test Language (an advanced form of keyword driven testing) to increase the maintainability of automated tests.  I use a similar technique for my own automation so it was nice to listen to somebody who shares my ideas.

For me, my attention now shifts to the publication of the March edition of The Tester.  I’m always on the lookout for good articles.  If you have an article you believe would be suitable, feel free to email me.

Goodbye Ivar Jacobson. Hello Express Software Ltd.

As many people are already aware, I have left Ivar Jacobson International to offer my services (and the services of a few close associates) to organisations on a freelance / contract basis.

We will trade under the name of Express Software Ltd and I am pleased to say that work is already successfully underway at our first client – a niche finance company that builds software to monitor the health of defined benefit pension schemes by performing sensitivity analysis on the underlying assets and liabilities.

With my new Head of Marketing hat firmly on, I’ve updated my About Page to showcase the organisations that have paid for my services in the past.  It currently stands at a respectable 20, with some impressive boasts across a number of different market sectors.

Finally, I couldn’t write this post without at least one shameless tout for new business.  If you have worked with me in the past and would like to work with me again, or like the sound of the ideas in this blog and would consider a contract resource to help with their introduction, please feel free to contact me on my new email address (shown below as an image to avoid the spam).

Software & Systems Quality Conference – London, 5 October 2009

For the last couple of weeks I have been preparing my slides for the Software & Systems Quality Conference that is due to take place in London on the 5th of October.  I’m scheduled to deliver a presentation called “A picture is worth a thousand tests” and whilst the abstract (below) doesn’t suggest as much, the talk was inspired by 3 things.  My use of visual images within the training courses I run and two commonly asked questions I receive when out and about at other companies.  “Can you make me into an agile tester?” and “Do you have a template?”

I’ll explain more on the day and try to write a follow up post with more information as a reference.

If you’re interested in attending, I have included links below to the registration page and program overview.

http://www.sqs-conferences.com/uk/program/program_1st.htm

https://www.sqs.de/sqs_conferences/registration_uk.htm

 

Abstract: A picture is worth a thousand tests

People have been using visual images to effectively exchange ideas for thousands of years, often opting for a single picture instead of several paragraphs of written text. From the ancient Egyptians and their enchanting hieroglyphics to modern day astrologers and their charts of the solar system, images have been widely used to preserve and communicate information in almost every area of life – apart from manual software testing.

 
When it comes to manual software testing, almost every test is captured as a series of written steps. At a glance, one test looks much like another and tests are routinely read from beginning to end to understand their purpose. What’s alarming is when two tests from the same area of an application are studied in detail, similarities are often found at a deeper level, with duplicate steps and instructions being commonplace. In summary, tests preserved and communicated using tables of written text are often time consuming to create, execute and maintain. Considering these factors, it’s not surprising that many of the brilliant test ideas conceived by testers never find their way into the project’s official testware.

 
This presentation demonstrates how using visual images can allow a tester to provide meaningful feedback faster – a key measure in today’s agile environment. Based on examples, the first part of the presentation discusses the advantages of using images to preserve and communicate tests, including how they can support a more agile way of working based upon reduced preparation and maintenance costs. The second part of the presentation discusses the practical challenges that a team must overcome when using such an approach and the changes in mentality a team must make in order to deliver the desired benefit of meaningful feedback faster.

2009 UK Testing Events Published in “The Tester”

After a couple of weeks of hard work, I’m please to say that the Spring 2009 edition of The Tester is now public and available on the British Computer Society (BCS) website, here.

A big thank you to all of the contributors, especially the 5 authors (Rhiannon Thomas, Michael Bolton, Adrian, O’Leary, Pradeep Govindasamy and Chris Whelan) who provided superb articles for this edition.

In addition to the articles, something that is well worth a look is the new testing events calendar. I can remember starting out in testing and not knowing were I could go to socialise with other testers – other than on-line. That was many years ago, but the same problem still exists. Where do testers go to get an overview of the year’s testing events? Yesterday I would have to say I don’t know. Today I am pleased to say that this information can be found in The Tester!

Software & Systems Quality Conference – Dublin, 4 March 2009

In a little over a month I will be visiting Ireland to speak at the Irish Software & Systems Quality conference, in Dublin. It will be the first time I have visited Ireland, so hopefully I’ll be able to do a bit of sightseeing whilst I’m there.

My talk is about how much documentation we need as testers. You can find the abstract here.

If you’re interested in coming along you can register here.

As a speaker I can get a limited number of delegates a 20% discount. Contact me if you’re interested.

The Testing V-Model Catch-22

I rarely meet a tester that has never heard of the V-Model. It’s one of the parts of the software testing body of knowledge that every tester seems to know about, even if it is just by name. I’ve added a picture that summaries the model below (Source: Wikipedia).

I’ve seen the V-Model used in two different ways. One approach is to use the V-Model as an extension to a waterfall software development lifecycle, where the “V” is perform just once. The other approach is to use the V-Model as an extension to an iterative software development lifecycle where lots of mini-”V”s are performed, one per iteration.

Regardless of how you apply the V-Model (just once or iteratively) the prescribed sequence of testing levels on the right-and-side of the “V” (Unit Testing -> Integration Testing -> System Testing -> Acceptance Testing) can encourage a project to work in a way that may not be in their best interest. This is easiest to explain if we just look at two of the testing levels, System and Acceptance. I’ve added definitions (taken from the ISTQB Glossary) for each term below.

System Testing: The process of testing an integrated system to verify that it meets specified requirements.

Acceptance Testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

My own interpretation of these terms and how they relate to the V-Model is that a group of testers perform system testing to checks that the application doesn’t contain any bugs, before one or more users perform acceptance testing to check they like the application and it’s going to truly help them accomplish one or more tasks quicker, more accurately, with more confidence, etc.

This always sounds sensible to me until I remember how much time most people spend system testing, how much users love to change their mind and how changing anything tends to introduce more bugs and the need for more testing. In a fairly extreme example, but a team could write some software, system test it to make sure it’s free of bugs, get a group of users to acceptance test it, who ask for 95% of the system to be changed… And then we’re back to square one!

This would, of course, be much more of a tear-jerking moment if you had chosen only to follow the “V” once in a waterfall style, compared to if you were performing many mini-”V” in an iterative pattern. However, even with many mini-”V” the team could still have wasted precious time and money system testing something that will never see the light of day.

So here’s the Catch-22 with the V-Model… Nobody wants to put a buggy piece of software in front of their users, but who wants to spend precious time and money system testings a feature that may be changed or removed?

If you interpret the V-Model as all system testing must be completed before acceptance testing can be started then the project is likely to run into problems. Either the application will be changed and the tester may feel like their original work was for nothing, or the application will remain unchanged and the users may feel like they didn’t get the solution they wanted. Neither of which are good.

Like with many Catch-22 situations, a compromise can be found if the rules of the game are loosened. The problem stems for trying to complete system testing and gold-plating the application before allowing users to perform acceptance testing. This is never going to work as the team will always end up changing something (even if it’s just something small) which need to be system tested again.

For me, the solution is not to try to finish system testing before acceptance testing begins, but instead perform as much system testing as necessary (no more, no less) to be confident that the application is of sufficient quality to be acceptance tested by the user(s). Once the application is then stable in terms of its features and how those features are realised the team can then focus on completing any outstanding system testing necessary to be confident that the application is of sufficient quality to release.

This means that a team should get into the habit of thinking about two different thresholds when it comes to an application being fit-for-purpose. (1) Is it fit-for-acceptance testing, and then (2) Is it fit-for-release.

Rational Functional Tester: Test Automation Architecture

This post is part 2 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 3) Creating Automated Tests, (Part 4) Reviewing Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Identify Custom Recognition Mechanisms
IBM Rational Functional Tester provides a Test Object Map that contains a list of every Test Object that is interacted with during the recording process.  Each object is assigned a unique name so that it can be referenced from within the automated test script.  For the majority of tests, the Test Object Map provides a sufficient mechanism for recognising and interacting with the Test Objects within the application.  However, there are some occasions when it is not desirable to use the Test Object Map and a custom recognition method is written to find and retrieve the necessary Test Object.  Review the parts of the application you intend to automate and identify where a custom recognition method may be necessary.  Often this activity will need to be accompanied by an investigation of how IBM Rational Functional Tester interacts with the project’s application and how Test Objects are stored within the Test Object Map.

For each custom recognition method that is identified, add its description to the project’s test automation architecture document.  As a minimum, express the custom recognition method in terms of its signature and an example of when the method can be utilised to benefit the tester.  It is likely that this section will evolve over the life of the project as new custom recognition needs are identified.  As this occurs, ensure that you keep the test automation architecture document up-to-date.

Identify Custom Verification Mechanisms
IBM Rational Functional Tester provides three distinct methods for verifying the state of the application under test.  They are Object Data, Object Property and Object Existence.  For the majority of tests, the relevant verification points can be implemented using one of the standard verification methods.   However, there are some occasions when the test cannot be reliably interpreted as a pass or fail and consequently a custom verification mechanism must be produced to obtain an accurate result.  Review the parts of the application you intend to automate and identify where a custom verification method may be required.  Often this activity will need to be accompanied by an investigation of how IBM Rational Functional Tester interacts with the project’s application and how Test Objects can be interrogated (using the three standard verification methods) to extract the application’s state.

For each custom verification point that is identified, add its description to the project’s test automation architecture document.  As a minimum, express the custom verification method in terms of its signature and an example of when the method can be utilised to benefit the tester.  It is likely that this section will evolve over the life of the project as new custom verification needs are identified.  As this occurs, ensure that you keep the test automation architecture document up-to-date.

Identify Test Bed Components
The key to successful automated testing is a well-defined and maintained test environment.  Combined with the state of the application (typically defined by the information that it contains within the application database) this is known as the test bed.  Before each automated test cycle begins, it is important for the test team to reset the test bed to a known state to avoid misleading results and false failures.

Investigate the application under test and identify the different components that together form the test bed.  Some of the test bed components will remain static during testing (typically, the operating system), whilst other test bed components will change as a result of the automated testing.  Document each test bed component and note which components are dynamic and which are static.  For each dynamic test bed component detail how it will be manually reset to a known state before automated test execution can begin.  This information is typically stored in the project’s test automation architecture document.

Capture Test Bed Automation Needs
Resetting a test bed before test automation begins can be a time consuming activity.  As your familiarity with IBM Rational Functional Tester increases you should consider how you can automate resetting the test bed (potentially to a variety of different configurations).  The complexity of automating such a task will vary greatly depending on the application under test and the test team’s ability to acquire a self-contained test bed.  Ensure that the benefits received by automating the resetting of the test bed outweigh the cost of developing the necessary code within IBM Rational Functional Tester.  For each dynamic test bed component that you decide to automate the resetting of, detail how it will be achieved using IBM Rational Functional Tester.  This information is typically stored in the project’s test automation architecture document.

Define Failure Recovery Routines
One of the distinct benefits of automated testing is the ability to verify the quality of an application without any human interaction.  On occasions (due to test script faults, defects within the application, and incorrect test bed configurations, to mention a few) the automated tests will become unsynchronised with the application under test.  That is to say, the automated test is expecting to interact with a specific test object, but that test object cannot be found as the application is displaying another window or has closed / frozen due to a serious application crash.  In summary the application under test is not in the same state that the automated test is expecting.  If a synchronisation error occurs it will typically leave any unexecuted automated tests incapable of running.  To minimise the number of affected tests, the test team should consider incorporating some form of failure recovery into their automation testing solution.  The sophistication of a failure recovery routine will depend on the ability to control the test bed using IBM Rational Functional Tester.  It is recommended that a test team start with simple failure recovery routines, such as those used to handle unexpected active windows.  As the test team’s experience of IBM Rational Functional Tester increases, they should consider additional failure recovery routines, such as those that are capable of completely resetting the test bed to a known baseline.  As the test team identifies failure recovery routines they should be documented within the project’s test automation architecture document.  As a minimum, each failure recovery routine should be described in terms of the criteria (or signal) that will trigger the routine, the test bed components affected and the state to which the test bed components will be returned.

Define Test Data Structure and Storage Needs
One of the powers of automated testing is the ability to perform the same scenario multiple times, with different variants of test data.  To take advantage of this power, the test data must be stored in a maintainable format, which can be easily accessed by both the test team and the automated test script.  As a rule of thumb, large volumes of hard-coded values should be avoided and the test data should be stored independently of the automated test script.  The possibilities for independent data storage are endless, but a common set of options exist, including IBM Rational Datapools, Excel spreadsheets and external databases.  Investigate which test data structure and type of storage best fits the test team’s needs.  Document the location of the test data, its structure and any special routines required to access the test data in the project’s test automation architecture document.

Define Package Structure
As you automated testing solution grows it will quickly expand to a large number of elements, including various test scripts, datapools and verification points.  To avoid your automated testing solution becoming unmanageable and difficult to maintain, it is a good idea to agree a package structure before your begin the majority of your automating testing development.  The structure of a project’s packages will be determined by their specific needs.  However, it is typical to subdivide test scripts into separate packages that represent functional areas, application feature or use-case.  A separate temporary ‘scratch’ package to keep ideas for new techniques and approaches can also be useful.  A project’s package structure should be recorded within the test automation architecture document.

Agree Naming Conventions
Maintaining an automated testing solution can be significantly aided by using a common set of naming conventions throughout the entire test team.  Naming conventions will also increase readability and help other testers read your automated test scripts as if they were their own.  Many organisation own naming conventions that apply to other area of the software development lifecycle.  Where appropriate, the test team may also use these conventions.  For elements that are specific to the test discipline, such as verification points, various test scripts, and datapools, the test team should define their own naming conventions and record them in the project’s test automation architecture document.

Identify Common-Code Owner and Policy
Reusing common-code within an automated testing solution can dramatically reduce development time and increase maintainability by providing a single point fix.  IBM Rational Functional Tester supports the use of common-code through a Helper Super Class that is inherited by each test scripts.  As each test script within a Functional Tester Project inherits from the Super Helper Class any changes made to the Super Helper Class may affect the way in which each test script performs its tests.  Therefore any changes made to the Super Helper Class (that is to say, the common-code) should be controlled.  The formality of this management will vary depending on the size and complexity of the project and the associated automated testing solution.  Look within your project team and identify a person or group of people that will act as the common-code owner.  It will be the common-code owner’s responsibility to review and agree any changes and additional to the Super Helper Class.  Within the project’s test automation architecture document identify the project’s common-code owner and how they will handle requests to alter the Super Helper Class.  Also describe how requests to the common-code are to be submitted and managed.

Rational Functional Tester: Creating Automated Tests

This post is part 3 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 4) Reviewing Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Prioritise Tests to Automate
An automated testing solution is most effectively developed using an iterative approach.  By focusing on automating the tests that are most difficult to execute manually and those that will add the greatest value to the project team, a test team can achieve 80% of the benefits of automated testing through 20% of the effect.  By developing iteratively the test team can also focus on a subset of the tests to be automated and more easy mange the change and quality of the automated testing solution. Review the list of tests to be automated and rank each test based upon its ease to be automated, its complexity to be executed manually and the quality risk associated with the part of the application it will validate.  Record the results of this activity so that it can be refined during later automation development iterations.

Select Top <X> Tests to Automate
Review the list of prioritised tests to automate and identify which tests will be automated during the next automated development iteration.  Whilst it would be ideal to only automate those tests that ranked highest on our list, in reality a range of high and low priority tests typically need to be automated during each automated development iteration.  This selection is often driven by the dependencies between the tests.  It may be necessary to automate a low priority test to support the execution of a one or more high priority tests.  Avoid including too many tests within an automated development iteration.  Clearly record the tests you plan to automate in this automated development iteration so that the test team has a precise definition of the scope and know when to stop and review.

Assign Automation Activities
Based upon the agreed scope of the automated development iteration, record which tester will be responsible for analysing, designing and developing each automated test script.

Setup Test Environment
Before each automated test script is created, ensure that the test environment is set to a predefined state.  By knowing the state of the test environment when the automated test script is created it will allow the test team to produce more reliable results when using the automated test script to validate future release of the application.  Typically this will involve some form of basic resetting of the operating system, restoration of underlying databases to the require state, and the setup of any peripheral devices, such as printers. While some tasks can be performed automatically, some aspects typically require human attention.

Set Recording Options
Before each automated test script is created, ensure that IBM Functional Tester is correct configured to support both the application under test and its specific domain (such as .NET, Java or Web).  If IBM Functional Tester is not configured to work with the specific domain then its ability to interact with the application under test will be greatly reduced.

Record Test Script
The easiest way to initially create the majority of automated test scripts is by using IBM Functional Tester’s record facility.  This will automatically amend new Test Objects to the Test Object Map and provide a basic script that can be later refined to become more resilient to change and easier to maintain.  Using the project’s naming conventions, create a new Automated Test Script and begin performing the steps to reproduce the test.  Once complete, add additional comments to the Automated Test Script to increase readability.  As a rule of thumb, if you are automating an existing manual test script, each step within the manual test script should become a comment in the automated test script.

Refine Test Script
Many automated test scripts will benefit from being manually refined after they have been initially recorded.  This process is performed for a variety of reasons and can be as simple as reformatting the automated test script code or removing unnecessary action precision (such as removing specific coordinates when they are not required).  Other refinements may include the introduction of a datapool or a custom recognition mechanism.

Replay Test Script (Against the Same Application)
Regardless of whether an automated test script was refined after recorded or not, the test should always be executed against the same version of the application used to initially develop it.  Using this approach, a test team can confidently identify any failures or error produced by the automated test script as issues with the test itself and not the application.  If the test passes successfully then no further action needs to be taken.  If the test fails then the ‘Debug Script’ activity should be performed.

Debug Script
When an automated test script fails against the release of software used to initially create the test then three possible actions can follow.  (1) A major issue exist with the automated test script and the easiest way to fix the problem is to start again and rerecord the test.  (2) A minor issue exists with the automated test script and a small manual refinement will fix the problem.  (3) The source of the problem is unknown or the fix for the problem is complex and unclear.  Using the test log and the debugging facilities offered by IBM Functional Tester discover in which category (1, 2 or 3) the problem belongs.  Take the necessary action to fix the problem or if the problem cannot be easily fixed performed activity ‘Capture Script Issues’.

Capture Script Issues
To keep momentum within a given automated development iteration, if a significant problem is discovered that has no immediate fix or requires significant custom code to be developed then leave the automated test script in a known condition and record the script issue in a convenient location.  As a minimum, each script issue should contain the name of the script(s) that contains the problem, the severity of the script issue in terms of how it will affect automating the remainder of the application, a headline that briefly describes the problem, and finally a detailed description of the problem.

Rational Functional Tester: Reviewing Automated Tests

This post is part 4 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 3) Creating Automated Tests and (Part 5) Using and Maintaining Automated Tests.

Review New Scripts
At the end of each automated development iteration the test team must verify the quality of the automated test scripts that they have produced.  The formality and thoroughness of this activity will depend on the scale and critically of the automated testing solution being produced.  Typically, the larger and more important the automated testing solution is, the greater the effort that should be applied to this activity.  When a test team is new to automated testing the most effective way of reviewing automated test scripts is to perform peer-reviews with the support of automated test script checklists.  A checklist is a simple list of questions specifically chosen to help the reviewer think about different aspects of an automated test script and whether it is conforms to the agreed approach to automated test script development.

Analyse Change & Issue Log
At the end of each automated development iteration the test team (specifically the common-code owner) must review the outstanding changes and issues associated with the automated testing solution.  After removing any duplicates, the changes and issues should be analysed in terms of their importance to the automated testing solution and the amount of effort required making the change or investigate and fix the issue.  Any changes made to the automated testing solution should be completed before the next automated development iteration is scoped.

Refine Test Automation Architecture
As a result of making any changes that were agreed during the ‘Analyse Change & Issue Log’ activity it may be necessary to refine the test automation architecture document to include alterations made at the architectural level.  This may include, for example, any new custom recognition or verification mechanisms identified to fix a test script issue.

Rational Functional Tester: Using and Maintaining Automated Tests

This post is part 5 of a series of 5 posts about test automation with Rational Functional Tester.  I originally created the material as a white paper in 2004 when I was coaching a project that was new to GUI-level automation and also to RFT.  I recently discovered it during a clean up and whilst some on my ideas have changed over the past 4 years, I’m happy that most of it still holds up as good advice.  Maybe one day I’ll update it.

The other posts in this series are (Part 1) Planning Test Automation, (Part 2) Test Automation Architecture, (Part 3) Creating Automated Tests and (Part 4) Reviewing Automated Tests.

Review Release Notes for New Build
Before the automated testing solution is used to verify the quality of the application under test the test team should review the release notes associated with the specific build or release.  This should identify both the changes that have been made to the application and any unfixed defects that were discovered during previous test cycles.  By having this information accessible to the test team, any known defects discovered by the automated tests can be ignored.

Setup Test Environment
Before any automated tests are executed the test team must reset the test environment to a know state.  This may be performed manually or may be automated as part of the automated testing solution.

Run the Automation
Automated tests can be executed at anytime of day or night.  If the automated test scripts are to be run overnight the test team must consider how the execution will be scheduled.  Typically, the simplest approach is to utilise the IBM Functional Tester command line interface and the scheduler built into the local operating system.  Regardless of whether the automated tests are executed manually or scheduled, the test team must consider how the automation will affect other test environment stakeholders, such as other test teams.

Analyse Failures
Once all of the automated test scripts have been executed the test team must analyse the test log and review any test failures.  A test script failure can fall into three different categories.  (1) The test script failed because the test environment was not reset correctly and consequently the application under test was not in the expected state.  (2) The test script failed because the test script itself contained an error.  (3) The test script failed because there is an error in the application, which must be investigated.  As the test team becomes more experienced in automated testing they will appreciate that the greater the amount of information and failure details contained within the test log, the easier this task is to perform.

Mark for Redevelopment
If a test script fails and the test team identifies the problem to be an error in the automated test script, the test script should be marked for redevelopment but not immediately fixed.  Unless fixing an error within a test script is necessary to complete the remainder of the test cycle, any fixes to the automated testing solution should be made within the subsequent automated development iteration and the application under test should be validated using the original manual test script.  Following this approach will allow the test team to remain focused on the primary objective or the test cycle (verifying the quality of the application) rather than becoming preoccupied with fixing error in the test scripts.

Log Defect
If a test script fails and the test team categorises the failure as a failure in the application under test then a defect log  should be entered into the project’s defect tracking database.  An a minimum each defect log should contain a brief headline that summarises the defect, a detailed description that explains how to recreate the defect in the application and a severity rating that provides an indication as to how important the defect is for the project team to fix.

Produce Test Evaluation Summary
At the end of any test cycle, the test team should create a test evaluation summary that describes the quality of the application.  Test evaluation summaries will vary from project to project, however a common set of topic exist.  (1) A subjective assessment – A brief description of the perceived application quality.  (2) Risks, Issues and Blockers – Anything that is preventing the test team from performing the testing effort.  (3) Progress against the Plan – Number of test executed, passed and failed.  (4) Defect Analysis – Metrics related to defect trends, defect densities and defect ages.  (5) Coverage Analysis – How the tests performed relate to project requirements, project risks and application code.