The problem with QA

As far as Java goes, we've solved most of our testing problems. We built enough infrastructure to make unit testing easy and effective, even when complex mock objects are involved. Fitnesse provides acceptance testing, which doubles as functional tests. While we haven't had time to put resources into JMeter testing, we've got the infrastructure set up and ready to go (hopefully we'll eventually be able to afford additional headcount to build out a full performance test suite).

For our client apps, unit testing is done in a similar fashion to Java. Mocking is generally pretty easy and we've had no trouble putting the test suites into our automated build process. Coverage tests help identify risks we expose when we forget to follow TDD practices.

While we've done some unit testing for Javascript in the past, it's been pretty minimal. One developer on our team really likes working with Javascipt, and he's put effort into improving our testing capabilities. Unfortunately, we still have issues with testing events (mainly due to the inability to run multiple threads in our tests). We still haven't attempted to automate the tests within our build, but I'm confident we won't have any issues with finding a solution.

To really test our entire application stack, we need more infrastructure. We're using Selenium in conjunction with Fitnesse to write tests, but we're far from a sustainable solution. We're able to run multiple systems with the exactly the same software as the production systems (config changes, but no code changes - mocks are not allowed).

But keeping this running has been an incredible challenge. Besides having to solve the technical challenges, we'll never be able to automate our entire testing infrastructure. Because we have multiple systems involved, we can't (nor want) systems to be automatically deployed. We don't want our front-end tests to break just because we're making changes to how the web server works. Sure, once we think the change is "complete", then we want tests to show us where we've missed things. Having multiple systems deployed on every code change would just lead to tests always failing without indicating which ones are serious and which are trivial. This makes the tests useless as developers quickly learn to ignore the results.

We also have the issue of running tests across multiple machines. Even though Selenium provides the ability to control other machines, just keeping those machines running is a challenge. While VM's may reduce our need for hardware (and space), they certainly don't reduce our need for admin assistance.

The real issue with QA is that, like code, it takes people and time to provide a high quality solution. If you don't have people to build and maintain your test suite, your code will suffer. On the other hand, if all of you developers are forced to become QA implementers, your code will suffer (at least as far as scheduled delivery is concerned). I believe building software without balancing QA staff with development staff leads to technical debt that's risky and dangerous.