Focused and powerful error monitoring for web, mobile, and server apps.

William Starling

Subscribe to William Starling: eMailAlertsEmail Alerts
Get William Starling: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Article

Test all you want, it will never be enough

Software teams increasingly face pressure to deliver quality software while staying ahead of the competition.

Anytime you ask a software developer if their app has been tested enough and they will tell you “it hasn’t.” Often they feel compelled to justify this lack of testing, but the reality is that it’s the norm. So, the question is what should we do about it?

Problems with testing

Software teams increasingly face pressure to deliver quality software while staying ahead of the competition. Releasing buggy software will quickly drive users away, but if you lack the features your competitors provide and you won’t be competitive will struggle to attract new users.


Traditionally in software testing it’s thought that the earlier a bug is found, the cheaper it is to fix. If a developer finds a bug shortly after writing the code,it can be fixed quickly; if the bug is found in production it needs to be diagnosed, fixed and the software re-verified and released which can be an expensive, drawn out process.


Whether you are performing unit testing, integration testing or system testing, a balance must be struck between the cost and the value testing provides.


UNIT TESTING

Unit testing breaks code into small, logical units and then tests that functionality in isolation from the rest of the code. It is typically performed by the developer and  verifies that code is working as  expected, as well as testing boundary cases which are difficult to produce in later stages of testing.


Many software teams prioritize writing a lot of unit tests as catching bugs early can save time and money later on. However, these tests are time consuming to write, often difficult to understand and maintain, and are too low level to catch logical bugs across the codebase.

INTEGRATION TESTING

Individual modules of an application can be integrated together and tested as a group - this is integration testing. At this level entire features can be tested in context and logical bugs discovered. Integration tests can be brittle when code is refactored and can take a lot of effort to fix. As with unit tests, they are expensive to write and are not well suited to finding edge cases.

SYSTEM TESTING

System testing is performed once all the software has been written. The aim is to ensure that the complete system behaves as expected, is usable, stable and secure. This involves a lot of manual ad-hoc testing alongside automated testing, which makes it difficult to perform consistent testing on every release. This makes it easier for bugs to slip through the cracks.

Striking a balance

Testing is time consuming and software is complex. No matter how much testing you do there will always be bugs that find their way into your releases. So what is the best testing strategy?


A lot of software teams get wrapped up measuring how much testing is being done; this can lead to unhelpful testing strategies like “100% line coverage in unit testing”. Even if good coverage is achieved, you may  test for the wrong behaviour and the tests themselves can be buggy or not actually verifying intended outcomes.


To devise a useful testing strategy that improves the quality of your code, you first have to answer the question “what does my testing need to achieve?”. The answer to this question can depend on the context (for example a bug in your self-driving car is a lot more concerning than a bug in a game), but typically the answer is that you want few bugs experienced by the end users and no serious defects that could affect security or integrity.


The following are some things to consider when creating a testing strategy.

FOCUS ON WHAT’S IMPORTANT

More tests does not always equal fewer bugs, but targeting testing efforts in the right place can make a real difference to code quality. Here are some examples of where testing can make a difference:


  • Complex logic — Bugs in areas where the logic is very complex or convoluted can be subtle and hard to diagnose, with regressions being hard to spot. These areas should be tested more exhaustively.

  • Publicly accessible interfaces — Ensure that interfaces which can be reached publicly do only what they are intended to do. This includes verifying that requests are appropriately validated to avoid interfaces returning more information than intended or modifying data in unintended ways.

  • High impact code — Focus testing in areas that have the greatest impact on your users. A feature used by a small percentage of users may be less important to test than core functionality.

DESIGN FOR DAMAGE CONTROL

Inevitably bugs will happen, but when they do you want to control the impact. This can be done at the code level, for example handling exceptions where appropriate, or at the design level, for example using transactions when modifying databases. Thinking about what happens when things go wrong can help reduce the severity of bugs when they do occur.

MONITOR BUGS

Once your software is released into the wild, how do you know if your testing strategy is working and how can you monitor the experience of your users? Error monitoring (such as Bugsnag) allows you to see errors that are occurring and make decisions about what needs to be fixed based on the impact a particular bug is having. You can focus on fixing the edge cases that actually matter in production, which can be difficult to discover during testing.


Error monitoring can be used to get an early warning when things are going wrong, which means you can either fix things before they become too disruptive to users or even roll back a release if the impact appears significant.


Bugs in software are inevitable, but being able to get immediate feedback on how your software is performing post-release is a valuable last line of defense in an effective testing strategy.

Final thoughts

 

Software is a complicated, collaborative exercise, and trying to produce it with as few defects as possible is time consuming and costly. You will never have enough tests to catch everything that could possibly go wrong, but with the right testing strategy and design you will be able to produce good quality software - without being bogged down by writing and maintaining too many tests.


More Stories By William Starling

I love solving problems, building software in Java (been doing it for nine years), and a big Linux fan. I'm also interested in entomophagy, the practice of eating insects.