Matt Archer's Blog

  • New Book

    I'm working on a new book. View the home page for the 'Tips for manual testers working in an agile environment' book
  • Enter your email address to receive notifications of new posts by email.

  • Subscribe

The Testing V-Model Catch-22

Posted by Matt Archer on December 8, 2008

I rarely meet a tester that has never heard of the V-Model. It’s one of the parts of the software testing body of knowledge that every tester seems to know about, even if it is just by name. I’ve added a picture that summaries the model below (Source: Wikipedia).

I’ve seen the V-Model used in two different ways. One approach is to use the V-Model as an extension to a waterfall software development lifecycle, where the “V” is perform just once. The other approach is to use the V-Model as an extension to an iterative software development lifecycle where lots of mini-“V”s are performed, one per iteration.

Regardless of how you apply the V-Model (just once or iteratively) the prescribed sequence of testing levels on the right-and-side of the “V” (Unit Testing -> Integration Testing -> System Testing -> Acceptance Testing) can encourage a project to work in a way that may not be in their best interest. This is easiest to explain if we just look at two of the testing levels, System and Acceptance. I’ve added definitions (taken from the ISTQB Glossary) for each term below.

System Testing: The process of testing an integrated system to verify that it meets specified requirements.

Acceptance Testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

My own interpretation of these terms and how they relate to the V-Model is that a group of testers perform system testing to checks that the application doesn’t contain any bugs, before one or more users perform acceptance testing to check they like the application and it’s going to truly help them accomplish one or more tasks quicker, more accurately, with more confidence, etc.

This always sounds sensible to me until I remember how much time most people spend system testing, how much users love to change their mind and how changing anything tends to introduce more bugs and the need for more testing. In a fairly extreme example, but a team could write some software, system test it to make sure it’s free of bugs, get a group of users to acceptance test it, who ask for 95% of the system to be changed… And then we’re back to square one!

This would, of course, be much more of a tear-jerking moment if you had chosen only to follow the “V” once in a waterfall style, compared to if you were performing many mini-“V” in an iterative pattern. However, even with many mini-“V” the team could still have wasted precious time and money system testing something that will never see the light of day.

So here’s the Catch-22 with the V-Model… Nobody wants to put a buggy piece of software in front of their users, but who wants to spend precious time and money system testings a feature that may be changed or removed?

If you interpret the V-Model as all system testing must be completed before acceptance testing can be started then the project is likely to run into problems. Either the application will be changed and the tester may feel like their original work was for nothing, or the application will remain unchanged and the users may feel like they didn’t get the solution they wanted. Neither of which are good.

Like with many Catch-22 situations, a compromise can be found if the rules of the game are loosened. The problem stems for trying to complete system testing and gold-plating the application before allowing users to perform acceptance testing. This is never going to work as the team will always end up changing something (even if it’s just something small) which need to be system tested again.

For me, the solution is not to try to finish system testing before acceptance testing begins, but instead perform as much system testing as necessary (no more, no less) to be confident that the application is of sufficient quality to be acceptance tested by the user(s). Once the application is then stable in terms of its features and how those features are realised the team can then focus on completing any outstanding system testing necessary to be confident that the application is of sufficient quality to release.

This means that a team should get into the habit of thinking about two different thresholds when it comes to an application being fit-for-purpose. (1) Is it fit-for-acceptance testing, and then (2) Is it fit-for-release.

 

5 Responses to “The Testing V-Model Catch-22”

  1. Gaurav Pandey said

    Hi

    I am not sure if I understood the Catch 22 situation of the V-Model!

    Nobody wants to put a buggy piece of software in front of their users, but who wants to spend precious time and money system testings a feature that may be changed or removed?
    >> The V-Model is meant for testers (at all stages) not for users!! Since the role of the tester is to find these bugs, we may expect a buggy software.
    Yes, as an user I would not like to see a buggy software.

    If you interpret the V-Model as all system testing must be completed before acceptance testing can be started then the project is likely to run into problems.
    >> Yes, “all” system testing must be completed before acceptance testing.
    >> What does “ALL” mean here? Before the requirements changed? or after the requirements changed?
    If the requirement does change, then we start a smaller V-Model (either formally or informally). This smaller V-Model normally does not include Acceptance Testing.

    Either the application will be changed and the tester may feel like their original work was for nothing, or the application will remain unchanged and the users may feel like they didn’t get the solution they wanted. Neither of which are good.

    Like with many Catch-22 situations, a compromise can be found if the rules of the game are loosened. The problem stems for trying to complete system testing and gold-plating the application before allowing users to perform acceptance testing.
    >>Here, I would like to refer to the difference between a Unit and a System. Depending on the scope of the application, a system can become a unit.
    Similarly, if scope screep is occuring, then it makes sense to execute multiple small V-Models.

    This is never going to work as the team will always end up changing something (even if it’s just something small) which need to be system tested again.
    >> We may consider implementing the use of a Change Control Board. (with technical experts being a part). The CCB would analyze the change and the impact (this includes testing too)
    Risk Based Testing also assists us testing these changes.

    I feel the V-Model is a very good model. Yes, it has its disadvantages, however like most standards (aka guidelines) in our industry, the V-Model is flexible and may be modified as per each organizations/projects need.

    Where did I go wrong in understaning your post?

    • Thanks for your comment Gaurav,

      If you let me know what disadvantages you see with the V-Model I can tell you whether I agree. You never know, our views may be much closer than you think.

      Hopefully, your response will also give me a better understanding of your testing values and provide me with a better foundation to respond to your initial comments.

      Cheers,

      Matt

      • Gaurav Pandey said

        Hi Matt,

        Regret the delay in responding – I would subscribe to the comments too 🙂

        I am not sure if I understood on which disadvantages you would like to me mention. You have already mentioned the disadvantages – I had attempted to provide “workarounds” to these disadvantages.

        Considering the V-Model is an extension of the waterfall model(very rigid in nature), the evil is observed in V-Model too – Phase blockage or phase paralysis would be observed.
        Another aspect is that the maturity of the testing process has to be well established to ensure that V-Model is successfull.

        Regards
        Gaurav Pandey

      • Hi Gaurav,

        When I asked what disadvantages you see with the V-model, I was referring to the paragraph towards the end of your original post where you commented that you “feel the V-Model is a very good model. Yes, it has its disadvantages…” I wasn’t looking for a specific answer, I was just curious to know what was on your mind when you wrote the word “disadvantages” in that sentence.

        To answer your original question, “where did I go wrong in understanding your post”, I can see a few places where differences in opinions could have lead to a misunderstanding, but let me focus on what I believe is the most significant.

        I notice that you believe the “V-Model is meant for testers (at all stages)”. If by this you mean that testers perform unit testing, integration testing, system testing and acceptance testing then this is not an interpretation I have come across before. The most common mapping I see is that developers are responsible for unit testing, testers are responsible for system testing and users / customers are responsible for acceptance testing (integration testing is a grey area, so I wont go into details here). I appreciate that the titles I have used here are open to just as much interpretation as the V-model itself, so to phrase it another way, I would be surprised to see the same person, or group of people, perform every level of testing, from unit to acceptance, on the same project. Why? Because it is rare within our industry to find a person with the necessary skills and knowledge to successfully perform what are uniquely different pieces of testing.

        If you are happy to acknowledge that acceptance testing may be performed by a customer and/or user then this is where I believe a software development team can experience a catch-22 situation. If the customer and/or user of the software I am helping to build has never seen it before (and we have never built software for them in the past) then we want to make a good first impression, however, the reason we want to give them access is to understand (ideally as soon as possible) why they would not accept it. If my team was to treat system testing and acceptance testing as atomic activities that cannot be split, do not overlap and were performed only once (as most graphical representation of the V-Model suggest) then we would have to make a choice. Would we rather make a good first impression or would we rather have customer / user feedback early. This represents a catch-22 situation in my mind. As I suggested in my original post, for me, the solution is not to try to finish system testing before acceptance testing begins, but instead perform as much system testing as necessary (no more, no less) to be confident that the application is of sufficient quality to be acceptance tested by the customer and/or user(s).

  2. […] The Testing V-Model Catch-22 « Matt Archer’s BlogZephyr fits to v model test also. In v model QA methodology all testing organized according to the process being followed and aligned with the Stages. It maintains rigor in the testing process and can handle multiple testing cycles for System … ; Zephyr as applied to V-Model. … Books on Testing… […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s