Matt Archer's Blog

  • New Book

    I'm working on a new book. View the home page for the 'Tips for manual testers working in an agile environment' book
  • Enter your email address to receive notifications of new posts by email.

  • Subscribe

However you document your manual tests, don’t repeat yourself (D.R.Y.)

Posted by Matt Archer on January 3, 2013

This post is part of the tips for manual testers working in an agile environment series. A series of posts inspired by the topics covered in the Techniques for Agile Manual Testers course that is currently available to take in London (via the Ministry of Testing) and in Copenhagen (via PrettyGoodTesting).

For years, programmers (at least the good ones) have known that when one piece of code looks very similar to another piece of code then this typically isn’t a good sign. If something needs changing in the first piece of code, then the same change will almost certainly need to be made in the second piece of code too.

The more duplication that exists, the greater the chance that a mistake will be made during the update, the update will only be made in some places or changes will be rejected or ignored due to the daunting amount of effort that often comes with such duplication.

Unsurprisingly, when the description for one manual test looks very similar to the description for another then this typically isn’t a good sign either. It generally isn’t a good sign for any tester, but for a tester who is working in a agile team where “responding to change” is valued over “following a plan”, duplication can quickly spell trouble.

So next time you find yourself reaching for the copy & paste icons, consider the duplication that you may be about to introduce into your manual tests. And more importantly, consider the future maintenance cost to you and the team.

If you have a comment or question about this particular tip, please do not hesitate to Leave a Reply.  A complete list of tips is listed below.


7 Responses to “However you document your manual tests, don’t repeat yourself (D.R.Y.)”

  1. Dani said

    — MA: Hi Dani, some interesting thoughts. Let me try to pick apart your comment and offer a reply.

    I must dissagre with the general warning. In my opinion we better be awer of reusabe nature of owr testing and mostly when talking automation.

    — MA: I am not talking about test automation here, where tests are typically expressed as code, but instead I am solely talking about manual testing, where the most common approach to describe a test is the written word. When reading manual tests that other people have written, I often find a lot of duplication, for example, I often find duplicate explanations related to setting up the environment, common pre-conditions and starting or accessing the software. Imagine you include an explanation on how to access the software you are testing at the beginning of every test; that is a lot of tests to update should that access procedure ever change. It is this type of duplication that I am warning against because I believe the maintenance it causes is a real agility killer.

    When attempting to explain the nature of the reusability concept – one can always argue “so, what you say is we have done it all before? Do we have nothing new to contribute?” It is almost right; most of the new contribution lays in the context and interpretation. Let’s analyze our day to day testing activities; how much of every action, operation, thinking, doing – is actually uniquely new? When we design a new test case for a certain application we actually use our memory and past experience, rearranging all our old knowledge into a new pattern – forming new test case that ought to answer the newly issue we are testing. So conceptually we are reusing.

    — MA: An important part of test design is combining together different sources of information (including your examples of memories and past experiences) to identify different patterns of use to help us explore the software we are testing. How my DRY argument relates to this process is as follows. If you imagine a series of different patterns (tests) and want to keep a record of them, do this in a way that stresses the differences between those patterns (the bits that make each pattern unique and likely to find a bug that another pattern wouldn’t) and de-focus the supporting information that is often better stored in another location (or not recorded at all). I say this because it is often this supporting information that creates much of the duplication, which in turn leads to significant maintenance and a slowing of the team’s delivery.

    My question here will be how much of reuse are we doing in regards to testing artifacts? And how much are we aware of the reusable nature of our work when we design a testing artifact? Our day to day manual testing working flow is built out of: context -> concept –> build –> use. I ask myself how much of my attention/awareness aims to the reuse issue.

    • Dani said

      Hello Matt
      I do agree with most of your comments. I do resent when people cat and paste. In order to avoid it we better reinvent our inventories buildup and infrastructure, enabling to reuse our old testing artifacts while enhancing the ability to mingle all newly gathered information with all past experience and artifacts. (Must we repeat manually our login when we could have created an item to perform it again and again?)

  2. –MA: Recently, Mario Gonzalez Macedo wrote a response to this post. His original post, plus my comments in reply are below.

    I recently stumbled across a blog post from Matt Archer in which he misuses the old adage DRY (don’t repeat yourself).

    See why applying DRY in testing is a dumb idea and why duplication is good, especially in agile environments.

    How to misuse a principle

    Duplication is certainly a smell that every developer needs to be careful about. It is very well known that code duplication results in maintenance issues that increase the potential for bugs throughout a SUT.

    But this principle is only for code.

    –MA: Very few principles (if you treat them as that, principles) can only be applied to one thing and one thing only. To suggest that a principle is (and only ever will be) applicable to a particular domain goes against how most new ideas and principles are created, which is to take inspiration from other ideas and principles. You could argue that when an existing principle is applied to a different domain that a new name for that principle is required, but I generally try to avoid giving things a new name just because I have applied my own interpretation. That said, I was reading Gojko Adzic’s new book a few days ago about impact mapping and in the introduction he explains why the term impact mapping was specifically chosen to differentiate it from InUse effect maps (another technique from which he has taken inspiration) because people had commented that he was applying the effect map technique incorrectly. Do I need a new name here? I don’t think so, at least not at the moment.

    Having experience as both a developer and a tester, I can assure you that the DRY principle cannot be blindly applied to testing.

    –MA: Based upon similar experience, I can also assure people that the DRY principle cannot be blindly applied to testing. Blindly applying a principle to anything is rarely a good idea. I hope I didn’t suggest a blind or one-size-fits-all application of DRY to testing, but I’ll re-read the post to make sure.

    In fact, following the DRY principle for tests actually introduces churn and increases the work required to execute them.

    Why? Because it breaks their encapsulation. Let me illustrate with an example.

    Suppose that your team decides to minimize the repetition of pre-conditions, as Matt specifically suggests.

    You create a test case with certain pre-conditions and, for the next subsequent applicable test cases, you simply place a pointer to the first test case by means of a link or a “Look in test case 1 for the pre-conditions”.

    What is wrong with this?

    Answer: if there is a change to the pre-conditions in the first test, then all the tests pointing to it have thus changed as well.
    Is this what you intended? In general, I bet that you just skewed the accuracy and validity of all subsequent tests.

    What must happen now is that you have to go through each test case and verify that they are still accurate and valid. This is something you would not have to do if each test case contained its own pre-conditions.

    This, my dear readers, is one of many example of churn caused by misusing a principle. In this case, applying the development-specific principle of DRY to testing.

    –MA: The approach you propose (of pointing one test to another) is not something I typically advocate as a refactoring pattern for manual tests with duplicate pre-conditions. When I mentioned pre-conditions I had a few things in mind. I have expanded on one of them below.

    –MA: Something that makes applying DRY to manual tests different to code is that in manual testing we have an addition place where we can refactor information. That location is the person performing the testing. When I see the same pre-condition repeated a large number of times it prompts me to question why it is explicitly stated in so many places. Is it really an important part of that test or is it the result of unconsidered copy-and-pasting that is now adding noise to the description of a particular test rather than helping emphasise what makes it unique in comparison to other tests? The pre-condition may be valid, but then there are an infinite number of other valid pre-condition that we chose not to explicitly mention, like “the computer has power”. So when I encounter duplication in pre-conditions I am not immediately looking to move that information to a common location to reduce maintenance, I will also consider removing it entirely (or reducing it pervasive use) to aid clarity.

    Case study

    Several years ago a test team I was working with at IBM attempted to follow the same logic and decided to apply the DRY principle to new test cases pertaining to the auditability functionality of our SUT.

    The lead tester, a very bright fellow, created one master test case that every other test case would use for common pre-conditions, steps, charts and other necessary material.

    Additionally, the master test case had hyperlink targets to different areas within it depending on different conditions in the SUT that were expected to be tested.

    After successfully writing several tests conforming to and using the master test case, the team was ecstatic with their low level of duplication and apparent efficiency in their work.

    Our managers and our test leader were certain they would get their end-of-year bonuses for it was them who so intelligently suggested avoiding duplication… at all costs.

    Several more dozens of test cases later, the team began to notice our test management tool struggle with the spaghetti of nested cross-links required to reach the different sections of the master test case.

    The TMT was clearly not intended for this.

    Gradually, the master test case became enormous as it accommodated new conditions, steps, et cetera.

    The team also found it harder and harder to keep track of where to place the links in the master test case and where certain information was located. It became a common exercise to linearly scan the master test just to know where the information was.

    And then, the unthinkable happened.

    As it tends to occur in testing for some jolly old reason, a variation of a test that required an ever so slightly different step that was not covered yet by the horribly gigantic master test case, spawned out of thin air.

    Panic ensued. Of course, several meetings were held.

    Here was the dilemma: if the master test case changed, then several dozen other tests would change too but would be incorrect. What to do?!

    I will admit that it was a bit hilarious seeing the team go through such nonsense, but the real pain set in when even I had to attend such silly meetings to provide input.

    –MA: You open your post by saying that duplication is a smell. I couldn’t agree more. Duplication doesn’t necessary mean there is a problem, but instead that you should think about whether that duplication may cause your team pain, now or in the future. In software development we tend to address pain-causing smells by applying patterns, but for every smell there are not only patterns but also anti-patterns (gotchas) that lure people in with the promise of perfection and then turn out to be problematic in reality. Your case study is an example of such anti-patterns (of over-refactoring / over-generalising / avoiding duplication at all costs).

    –MA: Of course, one bad example (or implementation of an anti-pattern) does not refute a principle for an entire domain. There are programmers out that that have been burnt by over-refactoring their code, myself included, but that doesn’t mean every programmer should ignore the principle. If I began a story by saying “several years ago, I met a very bright programmer who decided to create one class from which every other class would inherit [pause]” I can image the coy smiles that would appear in the audience as people began to think, “yep, I tried something similar once, I’ll think twice before I refactor like that again”. The point that I want to stress is that few of those programmers with smiles on their face will have come to the conclusion that duplication is no longer a smell or refactoring is bad, but refactoring in that specific way can cause more harm than good. The same applies to refactoring how we represent our manual tests. Just because there are bad ways of attempting it, doesn’t mean there aren’t good ways too.

    Duplication helps you stay agile

    The moral of the story is: keep your test cases self-contained.

    Test cases are not like code. In my experience as a tester, it is not the case at all that duplication in tests increases errors in their execution.

    –MA: My experience is different. I have found that more duplication* often leads to more time being spent on maintenance and when that maintenance cost becomes too high, rather than perform the maintenance and update the tests to reflect reality, the team opts to follow the spirit of the tests because the actual description is out of sync with the software they are testing. I have no problem with testers following the spirit of a test (I like to encourage testers to explore), but for whatever reason, if a team has decided that they need to represent their manual tests in a more prescriptive way, any synchronisation issues (differences) between the tests and how the software actually works is likely to lead to problems in test execution. If it doesn’t cause a problem for the team this points towards another question, which is, did your team even need that information in the first place?

    –MA: Addressing my star (*) above… a team’s maintenance cost is not only influenced by the amount of duplication, but also the frequency with which updates need to be made too. This means that two tests that look almost identical can cause little pain if they rarely need updating. Conversely, repeating a really small piece of information like an IP address in every test may leave you wishing you’d stayed in bed on the day that IP address changes or they change your test environment from a single node to two.

    Having each test case contain its own information allows for easier maintenance because you only have to perform the change once.

    –MA: Having each test contain its own information may have its benefits, but in my experience, completely encapsulating tests and easy maintenance rarely come together. If you have duplicated information across multiple tests and a change needs to be made, then that change has to be made in every test it is duplicated, not just once.

    Not only that, but because test cases are generally a write-once, read-many kind of artifact, you want to keep all necessary information as close to where you need it for fast reading and execution.

    –MA: There are ways in which manual tests can be represented so that they are not tightly coupled to the software and to a degree are insulated from changes in the software itself. That said, we cannot completely decouple our tests and rapidly changing software (as often experienced in agile environments) will in turn always lead to rapidly changing tests. This makes representing our manual tests in a way that makes them quick and easy to update an important part of manual testing in an agile environment. Reducing duplication (when it causes pain) I have found to be a key part.

    Refactoring test cases is a very dumb idea for the very simple reason that it makes you work harder to get to information that could easily have been included where it was needed: in the test itself.

    –MA: When talking about manual testing it is easy to forget the human factor and the fact that what we write down is just a guide (a memory aid) for the person performing the testing. Over time, what we have written down gets used in different ways. Some information may need to be continually referenced each and every time we perform a test (for example, a complex, multi-dimensional set of test data with specific characteristics), but there is other information that can be learnt once and remembered. Differentiating between these different types of information is an important part of refactoring manual tests. If you mercilessly refactor information that needs to be continually referenced then your point is valid, but if you refactor the information that can be learnt once and remembered, then you created a much leaner representation of your manual tests that is not only easier to maintain but also often easier to use as a guide for testing too.

    –MA: There is such a thing as too much information. As an experiment, find a tester that is experienced with a piece of software and ask them to perform 10 tests that have been expressed as single-line statements. Once they have finished, ask the same tester to perform a different 10 tests that have been fully scripted (UI steps, explicit test data, a full set of pre-requisite instructions, the lot). You are likely to find that the second set of tests take longer to perform because of the amount of noise the tester has to negotiate to get to the essence of the test. There is an argument that this type of experiment only works with “experienced” testers, which is a fair criticism. That said, what I frequently find people overlook is the time it takes to move a new tester to the status often referred to as “experienced”. I tend to find this takes place over a much shorter timeframe than people expect / believe.

    This is why having self-contained test cases can only increase your flexibility in fast-changing environments and where Agile is used.

    There is no flexibility afforded by DRY for test cases. It is an illusion. Duplication in tests is not a thing to avoid or be afraid of.

    What you should be afraid of is using the wrong principle on the wrong situation, as Matt advocates.

    Wait! What ended happening in the case study?

    Well, by the 3rd meeting and against several better judgements, our test leader decided to have someone add that little variation to the master test case and to annotate all other affected test cases with the following note:

    “Not applicable for this test case.”

    I reckon he considered it a moment of true agility and efficiency that would have made the Poppendiecks proud.

    Sadly, he did not get his raise.

    –MA: Thanks for your response Mario. I still plan to advocate DRY to manual testers as a principle for detecting smells in the way that they represent their manual tests, but your post has definitely given me some new food for thought. I especially like the anti-pattern in your case study. If I ever use it, I’ll make sure to reference you as the source. Thanks again, Matt.

  3. I think the lesson from the previous comments s to apply DRY intelligently. In my organization DRY makes a lot of sense since we often try to follow a typical usage flow, which allows us to visit a number of situations that require a significant setup. Some of our offshore partners were trained to create test plans where each test could be executed independently (the tool they were using at the time required it). In some cases this meant elaborate preambles for each otherwise relatively simple test. This inflated test plan size and test times. It took a while to break them of the habit.

    –MA: Thanks Sean, yes, I agree that intelligently applying DRY to how we represent out manual tests is the key.

    –MA: My experiences sound similar to yours. As Mario and I were discussing, it’s unwise to make your target removing all duplication at any cost, especially if that duplication is not causing you any pain (I’d categorise that as a DRY anti-pattern), but there are occasions like the one you mention above where applying DRY can help you keep simple tests just that; simple!

    On the other hand, if a bug was found it is good practice to confirm the minimum setup required to trigger it.

    –MA: I agree with your thinking around bugs. I’m never one to say never, but from memory, I’ve never intentionally applied the DRY principle to bug descriptions before. Now, some intelligently applied generalisation, that’s another story, but a story that will have to wait for another post. Thanks again for you comment, Matt.

  4. Absolutely one should avoid duplication.

    Copy and paste are the biggest threat for better development of a product because more we use copy and paste more we are spreading same kind of error or syntax error all over the product and once we start updating code to gear-up with certain enhancement we face same error everywhere and this cause update of certain module a very tedious task to accomplish

    –MA: Thanks Dwarika, I once read somewhere that any time you copy and paste your code, you are probably breaking a design pattern. We’re a little short of design pattern in testing, so we often have to think a bit harder about how the resulting duplication may come back to haunt us in the future. That said, I believe there are some principles that we can borrow from coding (and other disciplines) and apply them to how we represent our manual tests. For me, DRY is an example of such a principle.

    • Thank you Matt,

      Absolutely we should avoid similar steps and pre-requisite steps in writing test cases but what if dependency of one test case is on another. In this case we need to make repetition to execute the dependent test case na. So I think we should start following some mapping based test case designed techniques that would keep telling you that you have to execute this one after executing this test case without repeating same steps

      –MA: Thanks Dwarika, I like your idea of having something to tell people that you have to perform one test before another (if you want to use a preceding test to help set up some test data or a particular state for a later test). My only word of warning would be related to the amount of effort it took to maintain such a map; I could see it becoming quite a chore if it wasn’t done efficiently. You may also like to try letting people work out the order of tests themselves (without even creating a map). An element of random test ordering is likely to uncover a broader range of bugs and it will also help testers prove (or disprove) any undocumented assumptions they have about how the software they are testing flows.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s