TestSmells in a Nutshell

(Test)Code Smells

• Conditional Test Logic

  • ?: conditionals and loops in test-code
  • leads to: Buggy/Obscure/Fragile Tests, maybe unrepeatable/non-deterministic tests, complex code, hard to understand
  • fixed by: refactoring tests, Guard Assertions, delegation, Mock Object, Doubles, Stubs

• Hard to test code

  • caused by: highly coupled code, asynchronous code
  • leads to: manual testing

• Hardcoded Test Data

  • ?: magical/suspect numbers and strings
  • leads to: Obscure Tests, hard to extract behavior/documentation
  • fixed by: fixtures, creation mthods

• Obscure Test

  • ?: long test, much logic
  • caused by: not spending enough time in designing tests, Eager Tests, General/Shared Fixture, Mystery Guests (external resources), Verbose Test,
  • leads to: Buggy/Fragile Tests, increased maintenance costs, debugging due bad bug localization
  • fixed by: splitting up the test into smaller units, delegation

• Test Code Duplication

  • caused by: copy-past, not reusing code or bad test design
  • leads to: Obscure/Fragile Tests, increased maintenance costs
  • fixed by: Custom Assertions, Verification Method, generic reusable code, test inheritance, Expected Objects - using objects instead of fields, delegation

• Test Logic in Production

  • caused by: Test Hooks, code for tests only, test dependencies, equality pollution
  • leads to: hidden bugs
  • fixed by: special subclasses exposing behavior, utilities, clean up organisation/modularisation, custom assertions

Behavior Smells

• Assertion Roulette

  • ?: which assertion caused the failure -> localization
  • caused by: Eager Tests,
  • leads to: debugging
  • fixed by: splitting up the test

• Erratic Test

  • ?: determinism, sometimes tests pass, sometimes not
  • caused by: Interacting Tests/Suites, Shared Fixtures, execution order of tests, Lonely Test, Resource Leakage/Optimism, Unrepeatable Test
  • fixed by: fresh fixtures

• Fragile Test

• Frequent Debugging

  • caused by: lack of defect localization
  • leads to: manual testing, long debugging sessions
  • fixed by: test automation, small tests

• Manual Intervention

  • ?: test need user interaction/input
  • caused by: manual fixtures/setup, event injection
  • leads to: unrepeatable tests
  • fixed by: simulate human interaction

• Slow Tests

  • caused by: Obscure Tests, only high level tests, shared Fixtures,
  • leads to: decreased productivity
  • fixed by: split up in smaller parts

Sensitivity Smells

• Behavior Sensitivity

  • ?: test A suddenly fails after B was modified
  • caused by: introducing a new feature, bug 'fixed', setup modified

• Context Sensitivity

  • ?: test A suddenly fails after the state of some other components has been changed
  • caused by: time/date dependencies, dependencies on other components
  • leads to: non-deterministic tests

• Data Sensitivity

  • ?: test A suddenly fails after the data has been changed
  • caused by: data require new functionality, new data, data removed
  • leads to: increased effort in maintaining tests/data

• Interface Sensitivity

  • ?: test A suddenly fails after the interface of the SUT has been modifies
  • caused by: changed application/API
  • leads to: no refactorings ('don't touch it')
  • fixed by: encapsulate the API/SUT behavior

Project Smells

• Buggy Tests

  • ?: tests fail although they shouldn't and vice versa
  • caused by: Fragile/Flexible/Obscure Tests, hard to test code, too much logic in the test, source changes too fast
  • leads to: Fragile Tests and high maintenance costs
  • fixed by: write test first (->requirements), refactor legacy code, learn testing

• Developers not writing Tests

  • caused by: missing time, schedule, hard to test code, wrong automation strategy
  • leads to: Fragile/Obscure Tests, manual debugging/testing

• High Test Maintenance cost

  • ?: it takes long to introduce new behavior
  • caused by: Fragile/Obscure Test, spending much time with debugging, manual testing, badly test strategy, missing test automation
  • fixed by: refactoring tests before continuing

• Production Bugs

  • ?: too many tests fail, many bugs found in source
  • caused by: untested code, unfrequently run tests, not enough tests, lost/missing tests

• Productivity Smell

  • caused by: frequent debugging, missing automation, slow tests