Testing/QA
Systematically and properly testing a product is an art form. Throughout my career I have specialized in testing products, code, and data.
QA Experience
I have been deeply involved with QA and have either performed and/or managed QA for most of the jobs I have had.
- At Silicon Graphics
-
I designed, coded, and implemented an automated testing system. This system tested a wide variety of both valid and purposefully incorrect data. There were dozens of custom functions to simulated a wide variety of both valid and problematic circumstances. It also tested all needed API calls.
- I worked on two different projects to visual database data. This helped us to both analyze the data and to find and fix problems in large data sets. One project used Perl, HTML, and SQL queries. The other was based on Business Objects for visualizing database data.
- I project managed improving the sales order download process and related data. This project largely focused on clearly identifying different types of problem data and auto sending reports of that data to the responsible group. This required that I work with various groups so they would take responsibility for and provide valid data. The project resulted in a massive improvement in the quality of sales order data.
- I project managed people doing QA for a wide variety of project over many years. I did extensive exploratory, acceptance, and integration testing, as well as writing of unit tests.
-
- I wrote unit tests at The Human Creativity Project. I did extensive exploratory testing. I experimented with Selenium and decided it a powerful tool, but not yet mission-critical.
- I did exploratory QA testing at The Image Maker
- At Industrial Light and Magic I designed and wrote a Nagios style tool to gather problem information from the 100+ SGI and Sun machines I was in charge of. This exercised my programming and design skills. Give that the programs primary purpose was to find and identify problems, I consider it very related to QA.
- I started my QA work doing full time QA at a startup called Atherton Technology.
QA Philosophy
Below are some of my thoughts on testing.
Hit the "low hanging fruit"
Consider the ROI (Return On Investment) for testing. First hit the types of testing that have the “best bang for the buck”, especially when building a budget conscious product. It's probably okay to initially avoid testing methods that offer less short term benefit. Other types of testing are important to add, but can potentially be added later.
On the other hand, skilled exploratory testing is a must. Programmers should also be sure to add unit tests as they build the product.
Bug tracking
There are a lot of bug tracking systems out there. All of them have strengths and weaknesses. Unless you have some pressing reason, you should use one of the common bug trackers, ideally one that several folks on the team are familiar with. Most projects do not need an especially complex bug tracking system. The Wikipedia comparison list of bug tracking systems is useful, but it does not cover which have the most adoption. In order to see how popular given bug tracking systems are, look at free bug tracking systems as well as a list that includes commercial ones. Check the google rank for a bug tracker you're considering. You only want to use ones with adoption. Once a given bug tracking system starts being used, changing to a new system quickly becomes untenable. Choose a bug tracking system with care, and then stick with it.
Testing methods
Here are some of the things that I found to be important regarding the different testing methods. I embrace automated testing, but I tend to emphasize manual testing a bit more than some other QA experts.
Unit TestS
Each unit test tests a specific functionality. Unit tests are positively a good idea. Ideally, unit tests should be added as a program is written initially. Adding unit tests after the fact is also fully doable, but is a bit more work. It tends to be much more efficient to have the original programmer add the unit tests, rather than having somebody else add them later.
exploratory Manual testing
This is simply a matter of poking around and trying to find where there are problems. From my experience, skilled exploratory testing returns great results for a reasonably low cost. The tester should know what areas of the product have changed most recently. Then it's a matter of following intuition, testing the basic functionality to assure works, and specifically trying to find flaws in the places that feel most likely. Regardless of what automated testing is in place, exploratory testing should always be high on the menu. When testing this way, it's important to keep I and I
Scripted manual testing
After learning where the problems tend to reside through exploratory testing, it's worth making a list of tests that can be performed manually. This should test basic functionality and areas that tend to be problematic. When new features are added, those features should be added to the testing list. For any particular test, specify how important it is. For minor releases, for example, only the most important tests might be done. For major releases, more thoroughness is necessary. Scripted testing can typically be done by reasonably cost-effective offshore labor or by folks on the team who are currently less critical.
Automated testing
An over-dependence on automated testing is dangerous. Your automated test has no instinct for trouble, and it can’t tell you when something happens that it was not designed for. Automated testing is certainly important to implement, but it's not worth going overboard initially. For mature products, it may be worth going deeply into developing automated tests.
Acceptance testing
It's important to have a set of basic tests in place, before considering putting the product into production. This can be a combination of automated tests and a specific set of scripted manual tests. If there are manual acceptance tests, they should ideally be kept to a minimum.
Integration testing
This testing is for verifying that various components (modules or functions) integrate well with each other. As programs get more and more complex this becomes increasingly important. The more clear and modular the design, the easier this is to implement well. If the specifications are written to a notably low level, integration tests can be written using the specs as reference.
System Integration Testing
This is similar to integration testing, but at a higher level. Having clear APIs is immensely beneficial for a large-scale program. The more systems there are, the more important it is to have tests to assure that the different systems communicate end interoperate with each other properly.
Regression Testing
When something breaks in a specific place, even after it's been fixed it's more likely to break in that same place again later. It's good to make a list of tests, both automated and manual, that cover areas that have been problematic in the past.
Testing browser compatibility
There's a lot that could be said here, but I'll keep this short. First off, never do a major release of a web product without testing it on the main browsers. But that doesn't mean that all web projects should support all browsers. When picking which browsers to support, it's vital to look at the adoption number. Again, the goal is not to support all versions of all browsers. The goal is to pick the best more reasonable and realistic compromise and to have a planned out failure scenario for the browsers you don't support.
Some basic questions to consider: Is your site JavaScript heavy? Does your HTML validate? Do you have a setup for testing with a wide variety of web platforms? Who are your customers? Do they use Windows, Mac, or possibly Linux desktop? Are they using current browsers or old ones? Use Google analytics and common sense to help find the answers to most of these questions.
If your code works for the current version of firefox, it will likely also work for Chrome, Safari, IE7, and IE8. But do take the time to test. IE6 tends to be extremely problematic. You can vastly increase your development effort (a bad thing) and make your CSS uglier in an attempt to make your site fully IE6 compatible. Depending on what you're building, it may be worth considering not fully supporting IE6.
Contact Julian
Get in touch and chat about your company or project.
Julian Cash 415.738.9385 julian@HireThisGeek.com or see the contact page.