Tuesday, January 25, 2011

Evolution of Automated Testing

In the Agile world, and increasingly, in software development in general, automated testing is a key ingredient in delivering quality software. It's one of the many reasons why Agile has become so successful and is a component to any long-term, sustainable effort. Automated testing blurs the lines between the QA and the Dev. Both parties need to work together to produce a solid piece of software.

If your code has been developed in a "non-Agile" way, meaning code that is not easily testable, we need to work on paying back that technical debt. (Technical debt is anything in your code that wasn’t done cleanly. You know it’s there, and eventually you’ll have to pay it back. Unfortunately the interest rates are usually obscene!) There are many ideas on ways to do this, including Defect Driven Testing (find a bug, write a test).

Today I don’t want to discuss writing testable code for new features, but rather how to deal with our current technical debt. How do we move our code (and developers!) from a manual testing process to a more automated testing mindset?


clip_image003 clip_image001 clip_image005

How do we get there? Iteratively, of course!

Always use an iterative, phased in approach when introducing automated testing to your development team. Remember, the goal isn’t to fix everything overnight and make sure everything is being done The Right Way. That approach often creates so much churn that the team gives up and reverts to old habits. Our goal is create lasting, sustainable change.

First, let's define different umbrellas of automated testing:

  • Manual - tests which cannot be automated, it requires human interaction and a human eye.
  • GUI - tests that will mimic the user clicking buttons in the UI to perform end to end testing
  • Functional - tests that can provide end to end functionality testing that occurs right below the user interface level.
  • Integration – tests requiring an external source such as a database.
  • Unit - tests that are all on the front end, no external sources required but which test a small amount of code.


    Phase 1


    With legacy code, we all start with manual testing. A QC person manually tests the same test case over and over again throughout the entire system through different releases. The goal is to move off of the manual boat and onto a more automated boat!


    Phase 2


    While most of our tests are still manual, an introduction to automated GUI testing has begun. Typically, in legacy code, the business logic and the user interface is tightly coupled. Because of this, GUI tests are the only automated way to provide a full, end to end test of the system. GUI tests mimic the end users actions on the screen. It's as if a person is there performing actions on the screen. While GUI tests are slow, brittle, and prone to error, this is not the ideal way to automate tests but is the biggest bang for the buck to start off with that typically requires very little, if any, code refactoring.


    Phase 3


    Introduce a new form of testing, the integration test. This layer of testing is not designed to be end to end as a GUI test does, it does test smaller chunks of code but are more reliable than a GUI test and can be ran much more frequently. Starting out, integration tests can be hooked up to a Continuous Integration (CI) server to alert the developers if they have broken a build and get it fixed sooner rather than later.

    This is the first major paradigm shift that developers need to learn is to how to write code that can be easily testable. This type of testing will more than likely require code refactoring in order to create independent tests.

    In this phase developers need to start thinking about how to loosely couple their code in order to further test it. Ultimately we want to move into a very dumb User Interface.


    Phase 4


    Unit testing is the fastest, most reliable type of test but tests the smallest amount of code. With unit testing, we create mock objects, run the data through a method or a small group of methods, and test the result. These tests are fast because they do not require connections to external sources like a database or a remote service.

    More code refactoring will be required in this phase. This phase will further introduce concepts such as Dependency Injection and IOC (Inversion of Control) in order to "fake out" data in order to isolate the specific method(s) we are testing for.

    In the end, because unit tests test the smallest chunk of code and are the most reliable type of test, we will have more of these kinds of tests than any other.


    Phase 5


    Phase 5 is a refactoring/building stage. Armed with the knowledge of unit and integration testing and experience in refactoring code to write these tests, we need to build up our test suite. This phase will focus on paying back the technical debt that has accumulated over time and begin structuring our code in a testable manner.

    This phase will begin to introduce design patterns such as Model-View-Present (MVP) for Windows Development or even Model-View-Controller (MVC) for web development. Begin introducing how to develop and refactor code in this design patterns. Writing/refactoring code in a design pattern enforces a separation of concerns between the user interface and the business logic. This will allow us to test end to end, functional requirements, effectively testing as much code as possible without doing a GUI test.


    Phase 6


    Now that we have refactored code in a much more testable manner, we need to introduce Functional Testing. The idea behind functional testing is that we test functional requirements from a business perspective. Having our code in the MVP design pattern will allow us to test functionality below the user interface. This will allow for faster, more reliable tests than GUI tests and it keeps our tests running even when the GUI is changed.

    The end goal is solid software. How does test automation get us there? It takes time, and there is some risk getting there, but if we take short, iterative steps, it can be done! Be sure that when developers are writing tests that a member of QA pairs with them (or at least reviews each test). They’ll ensure the developers write valuable tests. Over time you’ll find the developers learn what the QA team members need tested, and the QA team will start to learn what types of scenarios are easiest to automate. This will result in even faster and more effective test creation in the future!



    1. At minimum, create different projects for different types of tests. Create an integration test project, unit test project, and a GUI test project. If you have a large number of GUI tests, you may want to consider having that as a separate solution and have those broken down. One advantage of breaking tests up into multiple projects; you decide which tests can run when on a Continuous Integration server.

    2. Run unit and integration tests as part of the build on the CI server. This will guarantee developers are notified in a timely manner if they have broken code. The sooner broken code (and the test) is found, the sooner it can be fixed, and the cheaper it is to fix!