So why when I Google ‘Agile Test Lead’ does it return so many hits (369,000 to be exact). I haven’t invented it as part of writing this post and it’s obviously more pervasive than a few old-school software houses clinging onto their roots whilst also trying to embrace agile principles. So what is an ‘Agile Test Lead’, if I was hiring one, what would I expect them to do and is it a role that will stand the test of time?

They test

I would always expect an Agile Test Lead to roll their sleeves up, get their hands dirty and contribute towards the release like every other team member. This includes testing during every sprint, on a daily basis, not just when the team is busy. If you are familiar with the analogy of the chicken and the pig that discuss opening a restaurant¹, I believe that every member of the team should see the Agile Test Lead as one of the pigs, not a chicken.

They demystify

With the 10^th anniversary of agile software development behind us², you may be surprised to hear that I believe many agile practices remain a mystery to people, testers included. The mind shift for a tester that has never worked in an agile environment before can be significant, but this challenge is not restricted to those without experience. Due to the different practices that exist between one agile team and another and the different interpretations that teams make, even testers with previous agile experience can be left confused when they join a new agile team.

I believe a good Agile Test Lead should be able to help demystify agile testing practices, not just in terms of regurgitating what it says in the books (people can read the books for themselves), but through explaining why specific practices have been chosen and illustrating how those practices have been tailored for that particular project, paying careful attention to any deviating from the textbox definition.

It could be argued that the team as a whole should share the responsibility of guiding new members. I couldn’t agree more – this should never be a single person’s responsibility. But with so many overlapping agile practices related to testing I also find it useful to have an experienced voice that can help guide new testers as they join the team (and any other member of the team for that matter).

If you are curious about whether your project would benefit from such specialist knowledge, try this simple test. Ask a random member of your team if they can explain the differences between Test Driven Development, Behaviour Driven Development, Agile Acceptance Testing and Specification by Example. If you’re feeling really brave, go a step further and ask them how testing fits into each of these practices and which tools they would shortlist to support their implementation.

I rarely meet people that can answer these questions, but I frequently meet testers who want these questions answered. It is for this reason that if I were to hire an Agile Test Lead I would want them to be able to answer these questions, plus many other questions related to testing in an agile environment.

They innovate

Agile software development processes, such as XP and Scrum, have challenged the testing community to think about how we test.

As an example, there are few agile projects that can point to a hierarchy of test documentation that cascades from a Test Policy, to a Test Strategy, to a Master Test Plan to multiple Phase Test Plans. Why? Because, historically these documents have taken several days (sometimes several weeks) to produce, by which point the typical agile project is in their second, third or even fourth sprint. Consequently, most teams conclude that this style of test planning is not compatible with agile software development.

Another example is performance testing. It is common for teams to use a dedicated performance testing expert (sometimes in-house, sometimes external) to understand where their application fails to meet expectations. Combine the long lead time often associated with this style of resourcing with other restrictive factors often associated with performance testing (such as intermittently available test environments and limited test tool licenses) and you can see why performance testing can also be difficult to integrate into an agile project.

There are countless other testing practices that are at odds with agile software development, but many of them were never designed to be used in an agile environment. It is difficult to dismiss decades of tradition, especially if it is engrained in an organisation’s culture, but this is exactly what a good Agile Test Lead must be able to do. When a team is new to agile software development, few things can be more troublesome than trying to wrap traditional practices in sprints and iterations. It’s a tough job, but a good Agile Test Lead should know, and be able to communicate, which testing practices should be left in the past and which should be embraced in the present.

Unfortunately, this task isn’t simply a case of dismissing old practices and picking new practices from a predefined list. Such as list doesn’t exist and even it did, it wouldn’t contain all the practices that a team needs to be successful. It is for this reason that a good Agile Test Lead must recognise the need for innovation, be able to innovate themselves, and most importantly, champion and support innovation within the team related to testing. The most successful agile testing efforts I have seen have taken popular practices, tweaked them a little, before finally surrounded them with their own, home-grown practices. Unfortunately, many people still believe that testing is the same today as it has been for the last 20 years, but this has never been further from the truth. A good Agile Test Lead must not only recognise this personally, but also help others appreciate the innovative solutions that are often required to test in an agile environment.

They improve

One of the things I like about agile software development is the collaborative practices. Group bug hunts, pairing testers with developers and fleshing out stories as a team all help reduce the blame-game and improve the team as a whole³ by breaking down silos⁴.

But life is always a compromise and whilst working as a team can improve many aspects of software development it isn’t a silver bullet. In fact, it could be argued that spreading testers throughout a team (rather than having a dedicated test team) introduces its own set of problems, especially as teams grow or become geographically distributed.

One such problem is related to sharing knowledge between testers, improving testing practices and optimising the way that testers work. With this in mind, I believe that a good Agile Test Lead should be able to take a step back and look across every aspect of testing, identify areas of weakness and help facilitate their improvement, often by facilitating the personal development of one or more testers. This doesn’t need to be a time consuming task. It can be as simple as getting Tester A to talk to Tester B, arranging for Tester C to run an informal lunchtime learning session or opening budget holder’s eye to the need for training for Tester D.

Once a team is up and running with agile, I believe it is the tiny micro improvements that help a team reach hyper productivity⁵. What can seem like a small inconvenience can significantly affect a team’s velocity if they are left to fester. It is for this reason that I believe a good Agile Test Lead must be able to identify opportunities for improvements, however large or small. Below is a list of sample questions I would expect a Good Agile Test lead to ask, should the situation arise.

- Is there a consistent way that testers install the latest build on their machine or is each release individually installed by hand?

- Is there a reason why some testers are using JUnit, whilst others are using TestNG as their execution framework for automated tests?

- Where does each tester store their test automation code, how often is it run, and most importantly, how often does it break and need to be repaired / maintenance?

Are Agile Test Leads here to stay?

I frequently meet people who refer to themselves as an Agile Test Lead, many of which fulfil a role similar to the one that I describe above. Was their role part of the original agile blueprint? – Certainty not. But I have no doubt that for many teams this is an essential role.

Being an Agile Test Lead is undoubtedly a practical role, but with subtle connotations of coaching too. I believe it is this coaching element that makes many Agile Test Leads tremendously valuable to the projects they serve, often more so than external agile test consultants who lack the day-to-day exposure necessary to understand a team’s subtle needs.

Are Agile Test Leads here to stay? – I think so, at least in the short to medium term. Agile software development has hit the mainstream, but many projects still have a long way to go. I have no doubt that a good Agile Test Lead can help a team progress towards a way of working that delivers the benefits associated with agile software development. Once a team has reached this milestone, maybe the need for the role becomes less important, but with so much change happening every day in the world of software development, who knows what he future will bring.

References

1 – http://en.wikipedia.org/wiki/The_Chicken_and_the_Pig

A Wikipedia page explaining the chicken and pig analogy

2 – http://blog.mountaingoatsoftware.com/reflections-on-the-10-years-since-the-agile-manifesto

A blog by Mike Cohn, reflecting on the 10 years since the agile manifesto

3 – http://www.allaboutagile.com/lean-principle-7-optimise-the-whole/

A blog by Kelly Waters in which he describes the problems associated with sub-optimisation

4 – http://testobsessed.com/blog/2010/07/19/why-test-automation-costs-too-much/

A blog by Elisabeth Hendrickson in which she explains why silos can dramatically increase the cost of test automation

5 – http://www.theagileengineer.com/public/Home/Entries/2010/5/22_Hyper_Productive_Agile_Webinar_files/HyperProductiveAgile.pptx

A presentation by Ryan Shriver introducing hyper-productivity

Risk Based Testing

In its heyday, risk-based testing¹ was the buzz-word of the industry; every tester was talking about it and every project had it as part of their testing strategy (even if it was only given lip-service in a superficial strategy document somewhere).

Today, it is all about being lean² and agile³ – testing early, testing often and having a zero tolerance to defects. It’s a good way of working, but have the guiding principles associated with agile software development made us lose track of one of the most important facts of software testing? That ensuring quality by testing everything is impossible. That even at its best, testing is a risk reduction exercise and different projects have wildly different tolerances to risk.

Many of the Scrum teams I join follow a delivery pattern that is made up of a number of internal releases, followed by a final public release. In terms of timeframes, this could mean an internal release every 2 weeks over the course of 4 months (for example) before a final public release at the end of that 4 month period. When it comes to planning the testing effort, this delivery pattern introduces an extra dimension that is rarely discussed on agile projects of this nature. That dimension is time.

In situations such as these, our first option is to ignore the total amount of time available to test (typically multiple sprints’ worth) and pretend that the software produced each sprint will be publically released. By working in this way we force ourselves to verify every aspect of quality during each sprint and to the best of our ability keep the risk of unknown bugs to a minimum.

Our second option is to investigate this extra dimension and postpone testing some aspects of quality until a later sprint, consciously accepting the level of risk associated with that decision.

You’re eyes are not deceiving you; I did just hint at an approach to testing on an agile project whereby every aspect of quality isn’t tested every sprint. I’ve had this conversation with people face-to-face so I can imagine what some of you may be thinking.

“You’re transitioning from a waterfall model and you’re currently going through the ‘w-agile’ phase” ⁴

“You’re not taking advantage of test automation properly” ⁵

“You need to get testers involved earlier in the lifecycle”

The majority of comments I receive suggest that teams who delay some of their testing until a later sprint are suffering from a process failure that should be fixed by adopting a new way of working. But does every team that delays testing need to be fixed? Or are they already working successfully by deliberately applying a risk-based approach to testing based upon their available time, tools, skills and project’s tolerance to risk? Have they, quite rightly, adapted their testing to their context⁶?

What follows is an explanation of risk-based browser-compatibility testing on a Scrum project. I must stress that there are countless ways of applying risk-based testing, each with their own subtle twists and nuances. The approach below is one I have found useful in the past. I hope you find it useful in creating your own unique way of working.

Estimating Risk

At the heart of risk-based testing is the idea that we cannot test everything in every possible way and consequently some things must be tested less thoroughly and/or frequently than others. With this idea in mind, which browsers should be near the top of our testing to-do list and which browsers can be left untested until a later sprint, tested less frequently or tested less thoroughly?

To answer this question, we need to decide how to estimate the level of risk associated with each browser. The majority of techniques used to estimate risk are based upon two parameters. Wikipedia⁷ nicely combines these two parameters into the formula below. By introducing this formula, I am able to replace “risk” (a fairly vague and ambiguous concept) with two scales upon which I can more easily rank one browser against another.

Risk = (probability of the accident occurring) x (expected loss in case of the accident)

When it comes to estimating the “expected loss” caused by accidentally releasing a bug into the wild I like to use the popularity of a browser. I use this indicative measure because I believe a bug that presents itself in a high usage browser (such as IE8) is worse than a bug of similar priority that presents itself in a low usage browser (such as IE5.5). I also like this measure because of the correlation that typically exists between the number of users who enjoy a successful visit to a website and its reason for existing (direct product sales, lead generation or positive brand awareness, for example).

Whilst an important indicator, I rarely find a browser’s popularity is enough to prioritise it against its peers, especially when working with multiple browsers with similar usage profiles. As an example, think back a couple of years to when IE6, IE7 and IE8 each held a similar market share (~10%-15% in late 2009). Now image that you had been asked to test an AJAX intensive website with an extremely rich and interactive UI. If you were going to get a free cup of coffee for every P1 browser compatibility bug you found, but only had 30 minutes to test, where would you start? I’d choose IE6 every time and I don’t know many testers who wouldn’t (other than those who don’t like free coffee!). It is for this reason that I believe the “probability” of a browser compatibility bug existing is also an important part of prioritising the testing of one browser over another.

Unlike “expected loss”, I like to consider a variety of factors when estimating the “probability” of a bug being discovered in a website whilst using a particular browser. Examples include the age of a browser, the default browser used by the team and any similarities between the browser I am currently estimating and the list of browsers already marked as a high priority or already tested. My reasons for these choices are listed below.

I think about the age of a browser because both older browsers and cutting edge browsers often work in ways that are less frequently considered whilst designing a website.

I think about the default browser(s) used by the team because people will often stumble across (and fix) compatibility issues with their own browser, reducing the chance of finding compatibility issues with that browser during dedicated browser compatibility testing.

I also think about the similarities between the browser I am evaluating and the list of browsers that I have already deemed to be high priority or have already tested in a previous sprint(s). Why? Because if I have decided to thoroughly test Firefox 3.6.13, I am unlikely to find additional bugs if I test Firefox 3.6.14, compared to if I tested a completely different browser, potentially on an entirely different operating system, such as Safari 5 on a Mac OS.

Armed with this approach to estimating the potential impact caused by a browser compatibility bug and the probability of such a bug existing, I can rank the different browsers I want to consider for testing. At this stage, my preference is to use “high”, “medium” and “low” categories rather than absolute values. The table below contains example estimates for each browser. I must stress that such a table is specific to a particular project. To highlight this point, consider the range of browsers a website may encounter if it was built for public consumption versus another website built for internal use within a large corporate organisation. Before even starting to prioritise one browser against another, these two projects are likely to have very different lists of browsers that they intend to support.

I sometimes find it useful to show the same information graphically to help me explain a particular test strategy to senior management or other members of the team. The same information as in the table above is also represented graphically below. I like this style of presentation due to the colours. If you have one or more browsers in the red or amber area then this is probably where you should turn your thoughts first. Browsers in the green area can then follow, as desired.

Sprint Planning and Release Planning

Now comes the tricky part. Based upon their associated risk, how often should each browser be tested and to what level of thoroughness? As I suggested at the beginning of this post, if a website will not be publically released until several sprints into the future, we have a greater number of options available.

At one end of the spectrum are projects that set aside resources every sprint to test every browser for incompatibility bugs, ranging from the blatantly obvious to single pixel misalignments. This is an extremely risk averse, but also very costly approach to understanding what browser compatibility bugs exist. Someone related to this project is likely to have high expectations when it comes to browser compatibility.

At the other end of the spectrum are projects that do not perform any browser compatibility testing until the final sprint before a public release, at which point their testing is limited to traversing one or two paths using the top three browsers, whilst scanning for obvious visual anomalies. These projects are likely to have a high tolerance to risk (whether they have consciously chosen to accept it is another matter).

Most of the teams I work with find themselves somewhere in the middle, but typically not before a healthy series of arguments and counter arguments take place in favour of different testing profiles, each with their own level of risk. Below is an example of how I like to represent this information as a consensus is thrashed out within the team. In my experience, low-tech works best, so grab a white board and some markers and talk through the various options that are compatible with your available time, tools, skills and project’s tolerance to risk.

Finally, remember that whilst this kind of testing profile deliberately spans multiple sprints, it is not intended to be fixed in stone during sprint 1 and religiously adhered to until the website is externally released. Truth be told, your original estimates are likely to be wrong, so be prepared to tweak the priority associated with each browser from one sprint to the next.

References

1 – An article written by James Bach on risk based testing.

http://www.satisfice.com/articles/hrbt.pdf

2 – A blog post by Alan Richardson (a.k.a. “the Evil Tester”) in which he describes how to apply one of the principles of lean (avoiding waste) to testing.

http://www.eviltester.com/index.php/2008/02/20/some-notes-on-software-testing-waste-my-lean-perspective/

3 – A good overview of agile testing by Elisabeth Hendrickson.

http://testobsessed.com/wp-content/uploads/2011/04/AgileTestingOverview.pdf

4a – A blog post by Rachel Davies in which she describes some of the signs that indicate a team could be following ‘w-agile’.

http://agilecoach.typepad.com/agile-coaching/2010/03/agile-or-wagile.html

4b – Another blog post about ‘w-agile’, this time by Rebecca Porterfield on the ThoughtWorks community site. In this post Rebecca takes the position that there is both ‘good’ and ‘bad’ ‘w-agile’. A belief I share based upon my own agile transformation experiences.

http://community.thoughtworks.com/posts/ffc85995c8

5 – A blog post by Mike Cohn in which he describes an approach to test automation that is frequently used on agile projects. Notice, similar to other test automation models proposed for agile projects, the focus is on ‘functional’ test automation. How are the non-functional aspects tested?  If ‘manually’ is the answer, this is where risk based testing can help.

http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-pyramid.

6 – An introduction to context-driven testing, at the heart of which is the principle that there are no testing best practices.

http://www.context-driven-testing.com

7 – The Wikipedia page for ‘Risk’.

http://en.wikipedia.org/wiki/Risk#Mathematical_formulations

When it comes to adoption, BDD can lead to some interesting discussions due its wide reaching influence. As a practice, BDD can help agree the scope with the customer, act as requirements for developers and help testers verify the solution. It is this cross cutting nature that can leave multiple members of the team thinking it is their practice to champion and adopt. Individuals from any corner of the team could easily find themselves at a conference (supposedly focused on their specialism) listening to a talk on BDD, in some instances blissfully unaware that other members of their team have received a similar presentation on BDD, just with a subtly different spin.

So if we focus solely on the test creation part of BDD, who is responsible for creating the automated test to prove that a feature has been built to the agreed specification?  My opinion is that both developers and testers have their part to play, each with their own unique skills to contribute, but to date I haven’t seen a truly integrated effort between a group of developers and a group of testers to implement BDD, at least not one where everyone helps both define and verify specifications. What follows below is a way of working that I plan to try when I get the chance. If you have tried something similar I would love to hear from you.

Regardless of who ultimately creates the automated test, every BDD cycle needs to start with a specification. A simple example is included below.

In order to keep my financial information secure

As a registered user

I want to be asked for a username and password when I visit my online banking website

With scenario – “Incorrect password”

Given a registered user

When they enter an incorrect password

Then they should be informed that they have not been granted access

Once we have a specification, we need to represent that specification in a tool that can ultimately make it “executable”. I’ve opted to use StoryQ⁴ (shown below is an abstract class I have written to represent my specification).

    [TestClass]
    abstract public class UserLoginSpec
    {
        [TestMethod]
        public void LoginAsAnInvalidUser()
        {
            new Story("User Login")
            .InOrderTo("Keep my personal financial information secure")
            .AsA("Registered User")
            .IWant("To be asked for a username and password when I visit my online banking website")
            .WithScenario("Incorect password")
            .Given(ARegisteredUser)
            .When(TheyEnterAnIncorrectPassword)
            .Then(TheyShouldSeeAnErrorMessageStatingTheyHaveNotBeenGrantedAccess)
            .Execute();
        }

        abstract public void ARegisteredUser();
        abstract public void TheyEnterAnIncorrectPassword();
        abstract public void TheyShouldSeeAnErrorMessageStatingTheyHaveNotBeenGrantedAccess();
    }

Code Sample 1 – UserLogin Specification

Anyone who has used StoryQ, or any other BBD framework for that matter, will know that we now need to write some methods to wire our specification to our solution. For arguments sake, let’s assume we’re creating a Microsoft ASP.NET MVC application. Who would be the best person in the team to implement these methods – a developer or a tester?

In this situation, I can’t help thinking, why not both? Why not let developers utilise their expertise to implement our specification methods in a way that interacts with our solution below the UI (mocking where necessary), whilst leaving the “testers” to use their expertise to implement our specification methods in a way that interacts with our solution above the UI? Both sets of methods could implement the same specification, so the project only has to maintain one version of what the application should do (a single version of the truth).

I’ve included below an example of what this would look like in code. Notice how both test classes inherit from the same specification class and override each method to create their own version of the test (above and below the UI). I’ve used the sample ASP.NET MVC application as the basis for my example. In case you’re wondering, the above UI version of the test is using the Selenium³ API to provide UI level automation.

    public class UserLogin_BelowUI : UserLoginSpec
    {
        AccountController accountController;
        LogOnModel logOnModel;

        public override void ARegisteredUser()
        {
            logOnModel = new LogOnModel()
            {
                UserName = "someUser",
                Password = "badPassword",
                RememberMe = false
            };
        }

        public override void TheyEnterAnIncorrectPassword()
        {
            accountController = new AccountController()
            {
                FormsService = new MockFormsAuthenticationService(),
                MembershipService = new MockMembershipService()
            };

            accountController.LogOn(logOnModel, null);
        }

        public override void TheyShouldSeeAnErrorMessageStatingTheyHaveNotBeenGrantedAccess()
        {
            Assert.AreEqual("incorrect password", accountController.ModelState[""].Errors[0].ErrorMessage);
        }
    }

Code Sample 2 – UserLogin Below UI Test

    public class UserLogin_AboveUI : UserLoginSpec
    {
        UserData testUserData;

        public override void ARegisteredUser()
        {
            testUserData = TestUserFactory.GetUserWithIncorrectPassword();
        }

        public override void TheyEnterAnIncorrectPassword()
        {
            browser.Click("link=Log On");

            browser.Type("UserName", testUserData.Username);
            browser.Type("Password", testUserData.Password);

            browser.Click("//input[@value='Log On']");
        }

        public override void TheyShouldSeeAnErrorMessageStatingTheyHaveNotBeenGrantedAccess()
        {
            Assert.IsTrue(browser.IsTextPresent("incorrect password"));
        }
    }

Code Sample 3 – UserLogin Above UI Test

.

UserLogin Above UI Test and UserLogin Below UI Test in Visual Studio Test Explorer

But is this duplication? Will a project benefit from having a test that checks a specification below the UI and another test that checks the same specification above the UI?  The table below contains examples of different quality related questions a project could ask themselves and the degree to which each type of test can help answer that question (the table is incomplete; many more examples could be added). I have colour coded the cells in the table to show my preferred type of test for a particular question (those coloured green being my preference). As you can see, tests that are good at answering one question are not so good, or completely incapable, of answering others. This is why I believe a joint effort between testers and developers to check each specification above and below the UI is worth considering.

There is, of course, nothing to say that every specification must be tested both above and below the UI. If a particular specification doesn’t lend itself to being tested either above or below the UI, the team can make the conscious choice to create a single test in this instance, possibly adding an attribute to the other test class (or specification class) to signal that a test has been consciously skipped rather than forgotten. On the topic of tracking and reporting, a team could also use some simple reflection to keep track of what specifications were being checked above the UI, below the UI or both. With something like this is place, any member of the team could add a specification and those responsible for creating tests would quickly realise there was a gap to fill. Even the results of exploratory testing or good old fashion bug hunts could be incorporated in this way.

There is also nothing to say that a UI-level test has to do everything above the UI. If the “Given” part of the test is particularly difficult to reproduce above the UI then that part of the test could be performed below the UI. For example, a test could directly manipulate the database to setup a particular system state, before performing an action and checking the expected result above the UI.

References

1. http://antonymarcano.com/blog/2011/05/updated-lessons-learned-in-close-quarters-battle/

The link above is to an article written by Antony Marcano in which he describes a multi-skilled team.

2 http://www.bcs.org/upload/pdf/tester-dec10.pdf

The link above is to an article written by Gojko Adzic in the December 2010 edition of The Tester in which he describes the on-going work to consolidate the practices and the language around BDD (or whatever it ends up being called!)

3 http://storyq.codeplex.com

More information about StoryQ

4 http://seleniumhq.org

More information about Selenium

I like automated tests.  They’re typically quick to run and assuming they acquire the correct actual and expected result they rarely fail to produce an accurate comparison.  Don’t get me wrong, I’m no fanatic, but I have seen enough success to know that the right tool, in the right tester’s hand, can be a valuable addition to a project.

That said, I can see how test automation can fall out of favour within a project or organisation.  Tools have changed significantly over the years, however, for many projects their tool’s knack to do just about anything ironically remains its greatest strength and weakness.  Even if a project decides to use one of the latest tools for their particular technology, such as Selenium or Watir for GUI-level web testing, it is easy to erode the expected benefits if either tool is used inefficiently (something that is easier to do than you may think).

As I’m sure many of you already know, automated tests are small pieces of software that test (typically) larger pieces of software.  This makes them great for comparing one thing to another, assuming the comparison isn’t subjective.  Provide two numbers and an automated test will tell you whether they are equal.  Provide two pieces of text and an automated test will tell you whether they are the same.  Provide two pictures and an automated test will even tell you whether they contain the same pixels.  Bottom line – if an automated test gets the opportunity to compare two things we would otherwise compare by hand we can feel relatively confident that the final outcome (pass or fail) is the correct one.

So why can so many of us relate to the feeling I am about to describe?  You know, that feeling you sometimes get when you start to run your automation, and as the results are produced, not knowing whether it is the software you are testing that is failing or a problem with your automated tests.  For me, one of the goals of testing is to help a team feel confident that the software we are building is ready for a given event (such as a demo, a release or maybe just the next phase of the project).  On occasion, I find this desire for greater confidence to be in direct conflict with test automation.  This typically occurs if the automation is “flaky” and consequently I end up feeling less confident in my results compared to if I had performed the tests manually.  Have you ever tried making others feel confident, when you’re not feeling confident yourself?  It’s not easy.  In fact, easy or not, passing on false hope (or despair) is not the sort of habit you want to embrace.

At first glace, the root cause of this uncertainty can be difficult to track down, often leading to the conclusion that all automation is bad.  For me this is not true.  There are just a few approaches to automation that whilst in the right situation can prove useful, more often than not tend to cause the team an unnecessary headache.  One of these approaches is calculating expected results on the fly.  Another is including unnecessary navigation.

All example code that follows is a mix of pseudo code and Selenium C#.

Calculating expected results on the fly

Imagine for a moment that we are testing an online dispatch system for a logistics company.  Each morning the system receives a CSV file containing a list of items to dispatch that day.  There is a simple business rule that we want to test.  If an item weighs more than 50kg then mark the item as a two-man lift, otherwise mark the item as a one-man lift.  This sounds easy enough so we create an automated test to pass a file containing fictional items to the dispatch system and check each item has been labelled correctly on the website.

public void CheckWeightCategory_Version1()
{
    // For each item in the file, check it has been labelled correctly
    foreach (Item currentItem in testFile)
    {
        // Navigate to the item tracking page on the website
        selenium.Open("/ItemTrackingPage.html");

       // Get the weight category for the current item
        string acutalWeightCategory = selenium.GetText("weightCategory" + currentItem.ID);

        // Calculate the expected result
        string expectedWeightCategory;

        // If an item weighs more than 50kg then mark the item as a two-man lift,
        // otherwise mark the item as a one-man lift
        if (currentItem.weight > 50)
        {
            expectedWeightCategory = "two-man lift";
        }
        else
        {
            expectedWeightCategory = "one-man lift";
        }

        // Compare actual to expected result
        Assert.AreEqual(acutalWeightCategory, expectedWeightCategory);
    }
}

So that we can calculate the expected result of the fly, we have coded the if-then-else business rule in a similar way to the dispatch system itself.  On the up side this means that we can test as many input files as we like without having to worry about the expected result, however, we now have a mixture of code to prepare the expected result and code to perform the test, all intermingled together.  As the example stands at present, this doesn’t feel like a huge problem, but we are standing at the top of a very slippery slope.  Consider the following;

Q: What happens if the business rule is made more complex?

A: We can add some more code to represent it.

Q: What happens if a particular item is missing a weight?

A: We can add some more code to trap it.

Adding more code is fine in principal, but it can have an unpleasant side effect.  Below is an updated example based upon the two questions above.  Notice we now have eight lines of code related to calculating the expected result and only four lines of code that relate to performing the test itself.  In essence, we have created a test with three times as many lines of code than is absolutely necessary to perform the test.  By embedding the business rule, we have also made (or are at least are well on our way towards making) the code for our automated test as complicated as the system we are testing.  And this is one reason I can personally lose confidence in automation.  Crudely speaking, when the odds of finding a bug are equally divided between the system we are testing and our own test automation code we must think carefully about the return on investment we are likely to receive.

public void CheckWeightCategory_Version2()
{
    // For each item in the file, check it has been labelled correctly
    foreach (Item currentItem in testFile)
    {
        // Navigate to the item tracking page on the website
        selenium.Open("/ItemTrackingPage.html");

        // Get the weight category for the current item
        string acutalWeightCategory = selenium.GetText("weightCategory" + currentItem.ID);

       // Calculate the expected result
        string expectedWeightCategory;

        // If the customer wishes to collect the item from the warehouse,
        // the weight category should be marked as N/A
        if (currentItem.customerCollect == true)
        {
            expectedWeightCategory = "N/A";
        }
        else
        {
            // If the item does not have a weight, fail the test
            if (customer.weight != null)
            {
                // If an item weighs more than 50kg then mark the item as a two-man lift,
                // otherwise mark the item as a one-man lift
                if (currentItem.weight > 50)
                {
                    expectedWeightCategory = "two-man lift";
                }
                else
                {
                    expectedWeightCategory = "one-man lift";
                }
            }
            else
            {
                Assert.Fail("No weight available to calculate expected result");
            }
        }

       // Compare actual to expected result
        Assert.AreEqual(acutalWeightCategory, expectedWeightCategory);
    }
}

That said, any automated test that performs a comparison needs to know the expected results.  I can’t argue against this fact, but my recommendation would be to avoid calculating any expected results automatically, especially if that automatic calculation takes place as part of the test itself.  Like all aspects of testing there are countless ways of achieving a particular task and maintaining expected results is no exception.  If you believe that automatically calculating expected results is for you then I would at least consider separating the code that calculates the expected results from the code that performs the actual test.  And I don’t just mean splitting the code into two different methods that are executed one after the other by a single piece of code we call the test.  I’m talking about a much harder divide where one piece of code saves the expected results to a known location that can be read (maybe even edited) by a member of the team, before a second piece of code takes the persisted values and performs the comparison.  Not only does this keep a clean divide between the two pieces of code, it also allows for easier debugging based on human inspection (and where necessary manipulation) of the expected result before they are used as part of the test.

Another option is to work out any expected results “by hand” in the same way as if we were performing the test manually.  Once we have the expected results (assuming they are a sensible format) we can use them as the basis for both manual and automated tests.  I can tell what some of you are thinking, “my aim is to automate the majority of my tests and do very little testing manually”.  That may be a great aim for your project, but I guarantee that you will end up performing a great deal of manual testing, even if you do not call it by that name.  Whilst debugging automated tests (both as part of their creation and later as part of their maintenance) sometimes the best approach is to step through the lines of code, placing various break points and watching the value of variables change.  That said, on other occasions it’s far easier to perform the test manually to double check that the application is behaving as we expect, often proving both the quality of the website and the quality of the automation at the same time.  It can also be easier to explain a bug to somebody when you can demonstrate it manually rather than attempting to commentate on a screen that is racing past at one-hundred miles per hour.

Unnecessary Navigation

Let us now take a look at another potential destroyer of automated tests, unnecessary navigation.  Imagine for a moment that we are testing an online book store.  The customer is keen to ensure that every book is displayed with the correct title, description and price and has provided this data for a sample of 100 books.  To automatically check our sample books we could write something similar to the code below.  As part of its creation we would need to give it a meaningful name, such as CheckBookTitleDescriptionAndPrice.

public void CheckBookTitleDescriptionAndPrice_Version1()
{
    foreach (Book currentBook in testFile)
    {
        // Open the homepage
        selenium.Open("/HomePage.html");
        selenium.WaitForPageToLoad("30000");

        // Login
        selenium.Click("link=Login");
        selenium.WaitForPageToLoad("30000");
        selenium.Type("TxtUserName", "TestAccount1");
        selenium.Type("TxtPassword", "TestPassword1");
        selenium.Click("Submit");
        selenium.WaitForPageToLoad("30000");

        // Search for the book we want to check
        selenium.Click("link=Search");
        selenium.WaitForPageToLoad("30000");
        selenium.Type("TxtSearchTerm", currentBook.Title);
        selenium.Click("Submit");
        selenium.WaitForPageToLoad("30000");

       // Open the Detailed Information page for the book we want to check<br />
        selenium.Click(currentBook.ID);
        selenium.WaitForPageToLoad("30000");

        // Check that the Detailed Information page has loaded
        Assert.AreEqual("Detailed Information page", selenium.GetTitle());
        
        // Compare the information on the Detailed Information page to our expected results
        string actualBookTitle = selenium.GetText("BookTitle" + currentItem.ID);
        Assert.AreEqual(actualBookTitle, currentBook.Title);
        
        string actualBookDescription = selenium.GetText("BookDescription" + currentItem.ID);
        Assert.AreEqual(actualBookDescription, currentBook.Description);
        
        string actualBookPrice = selenium.GetText("BookPrice" + currentItem.ID);
        Assert.AreEqual(actualBookPrice, currentBook.Price);
    }
}

This code is fine in principle, but just how much of it is actually related to the things we want to check?  Do we really need to login to the system, start this test from the homepage and use the search functionality to find the relevant book?  Probably not is the answer.  But what is the problem with having these actions in the test, they’re surely not causing any harm and you never know we may stumble across a bug that we weren’t expecting to find by mistake?

Whilst this sounds nice in principle, for me, there are two problems with the philosophy of if I’m lucky I might find some extra bugs by chance, for free! Even though this sounds great of the surface, automated tests that stumble across a bug (other than the one(s) they were deliberately trying to detect) can take a considerable time to diagnose as we ponder over whether it is the website that is broken, or dare I say it, the test itself.  The frustration, however, does not stop there.  If the homepage, search or login happened to be broken then there is a strong possibility that any test that uses those features as part of their navigation will fail to reach the part of the system we are trying to test.  Not ideal if we need a quick assessment of the website’s quality as we prepare for a release later that day.  At this point in time, our automated tests are next to useless, as all they tell us (and this may not even be directly) is that the homepage is broken.  It is also misleading for a test entitled CheckBookTitleDescriptionAndPrice to fail even though there is nothing potentially wrong with anything the title suggests.

Fortunately the solution is easy.  We put this (in my opinion, unnecessary) navigation in by choice, but if it’s surplus to requirements why not leave it out or replace it with something more succinct?  We could rewrite our test to look something similar to the code below.  Notice how we have reduced the amount of unnecessary navigation by replacing much of our previous example with a single, parameterised deep link.

public void CheckBookTitleDescriptionAndPrice_Version2()
{
    foreach (Book currentBook in testFile)
    {
        // Navigate directly to the Detailed Information Page for the book we want to check
        selenium.Open("/DetailedInformationPage?bookId" + currentBook.ID);
        selenium.WaitForPageToLoad("30000");

        // Check that the Detailed Information page has loaded
        Assert.AreEqual("Detailed Information page", selenium.GetTitle());

        // Compare the information on the Detailed Information page to our expected results
        string actualBookTitle = selenium.GetText("BookTitle" + currentItem.ID);
        Assert.AreEqual(actualBookTitle, currentBook.Title);

        string actualBookDescription = selenium.GetText("BookDescription" + currentItem.ID);
        Assert.AreEqual(actualBookDescription, currentBook.Description);

        string actualBookPrice = selenium.GetText("BookPrice" + currentItem.ID);
        Assert.AreEqual(actualBookPrice, currentBook.Price);
    }
}

Summary

I am not suggesting that calculating expected results on the fly or tests with large amounts of navigation should never be used, but I do believe they should be treated with caution.   Both approaches leave tests less focused, harder to maintain and open to a variety of false failures.  As I mentioned at the beginning, I like automated tests.  They’re typically quick to run and assuming they acquire the correct actual and expected result they rarely failed to produce an accurate comparison.  And here lies the problem.  Over time, calculating expected results of the fly can reduce the probability of a test acquiring the correct expected results and unnecessary navigation can make it much harder for a test to capture the actual result.  If we keep both of these things to a minimum, we provide each test with a much greater chance of getting to the part is does best – the comparison.

To accompany the tool, IBM Press have recently published a book entitled “Software Test Engineering with IBM Rational Functional Tester”.  The authors have left no stone unturned in their feature by feature discussion of the tool’s abilities, which undoubtedly makes it worthy of its tagline – “The Definitive Resource”.

As an experienced user I found myself skipping pages in the early chapters, but it’s not long before the book arrives at the juicer topics, including handling unsupported objects, testing specialised applications (SAP, Siebel, Flex, etc) and using RFT within a Linux environment.

For those new to RFT I would follow the authors’ advice and not attempt a cover to cover read.  The first few chapters will provide a new user with everything they need to know to get started and be productive.  The remainder of the 600+ pages can then be dipped into as necessary.

My favourite part of the book is the script samples that are scattered throughout every chapter.  Like any automated testing tool, sometimes the best solution is to hand-craft all or part of your script.  It is at this point that the average tester (myself included) typically turns to Google with the hope of finding a similar example.  I can remember spending hours looking for information on how best to query a database, connect to excel and extract data from a PDF.  All of these topics are covered in the book with examples, along with many others.

Your local bookshop / website should have no problem getting you a copy if you’re interested.  For those members of the BCS Special Interest Group in Software Testing (SiGIST), you can also borrow a copy for free from the SiGIST Library.

In my opinion, the best talk of the day was by Steven Ramsay who spoke about overcoming various obstacles to set up a testing practice at LinkLaters.  What I liked about Steve’s talk was that it was a description from the front-line, worts and all.  Steve suggested at the beginning that he was feeling nervous, but it didn’t show.  When it comes to creating the programme for the 2010 conferences (11th March, 29th June, 16th September and 8th December), I’ll certainly be voting for him to come back and give us an update.

Another good talk was by Martin Gijsen.  Martin spoke about how to make use of a Domain Specific Test Language (an advanced form of keyword driven testing) to increase the maintainability of automated tests.  I use a similar technique for my own automation so it was nice to listen to somebody who shares my ideas.

For me, my attention now shifts to the publication of the March edition of The Tester.  I’m always on the lookout for good articles.  If you have an article you believe would be suitable, feel free to email me.

We will trade under the name of Express Software Ltd and I am pleased to say that work is already successfully underway at our first client – a niche finance company that builds software to monitor the health of defined benefit pension schemes by performing sensitivity analysis on the underlying assets and liabilities.

With my new Head of Marketing hat firmly on, I’ve updated my About Page to showcase the organisations that have paid for my services in the past.  It currently stands at a respectable 20, with some impressive boasts across a number of different market sectors.

Finally, I couldn’t write this post without at least one shameless tout for new business.  If you have worked with me in the past and would like to work with me again, or like the sound of the ideas in this blog and would consider a contract resource to help with their introduction, please feel free to contact me on my new email address (shown below as an image to avoid the spam).

I’ll explain more on the day and try to write a follow up post with more information as a reference.

If you’re interested in attending, I have included links below to the registration page and program overview.

http://www.sqs-conferences.com/uk/program/program_1st.htm

https://www.sqs.de/sqs_conferences/registration_uk.htm

Abstract: A picture is worth a thousand tests

People have been using visual images to effectively exchange ideas for thousands of years, often opting for a single picture instead of several paragraphs of written text. From the ancient Egyptians and their enchanting hieroglyphics to modern day astrologers and their charts of the solar system, images have been widely used to preserve and communicate information in almost every area of life – apart from manual software testing.

 
When it comes to manual software testing, almost every test is captured as a series of written steps. At a glance, one test looks much like another and tests are routinely read from beginning to end to understand their purpose. What’s alarming is when two tests from the same area of an application are studied in detail, similarities are often found at a deeper level, with duplicate steps and instructions being commonplace. In summary, tests preserved and communicated using tables of written text are often time consuming to create, execute and maintain. Considering these factors, it’s not surprising that many of the brilliant test ideas conceived by testers never find their way into the project’s official testware.

 
This presentation demonstrates how using visual images can allow a tester to provide meaningful feedback faster – a key measure in today’s agile environment. Based on examples, the first part of the presentation discusses the advantages of using images to preserve and communicate tests, including how they can support a more agile way of working based upon reduced preparation and maintenance costs. The second part of the presentation discusses the practical challenges that a team must overcome when using such an approach and the changes in mentality a team must make in order to deliver the desired benefit of meaningful feedback faster.

A big thank you to all of the contributors, especially the 5 authors (Rhiannon Thomas, Michael Bolton, Adrian, O’Leary, Pradeep Govindasamy and Chris Whelan) who provided superb articles for this edition.

In addition to the articles, something that is well worth a look is the new testing events calendar. I can remember starting out in testing and not knowing were I could go to socialise with other testers – other than on-line. That was many years ago, but the same problem still exists. Where do testers go to get an overview of the year’s testing events? Yesterday I would have to say I don’t know. Today I am pleased to say that this information can be found in The Tester!

My talk is about how much documentation we need as testers. You can find the abstract here.

If you’re interested in coming along you can register here.

As a speaker I can get a limited number of delegates a 20% discount. Contact me if you’re interested.