TestSmells in a Nutshell
(Test)Code Smells
-
Conditional Test Logic
- ?: conditionals and loops in test-code
- leads to: Buggy/Obscure/Fragile Tests, maybe unrepeatable/non-deterministic tests, complex code, hard to understand
- fixed by: refactoring tests, Guard Assertions, delegation, Mock Object, Doubles, Stubs
-
Hard to test code
- caused by: highly coupled code, asynchronous code
- leads to: manual testing
-
Hardcoded Test Data
- ?: magical/suspect numbers and strings
- leads to: Obscure Tests, hard to extract behavior/documentation
- fixed by: fixtures, creation mthods
-
Obscure Test
- ?: long test, much logic
- caused by: not spending enough time in designing tests, Eager Tests, General/Shared Fixture, Mystery Guests (external resources), Verbose Test,
- leads to: Buggy/Fragile Tests, increased maintenance costs, debugging due bad bug localization
- fixed by: splitting up the test into smaller units, delegation
-
Test Code Duplication
- caused by: copy-past, not reusing code or bad test design
- leads to: Obscure/Fragile Tests, increased maintenance costs
- fixed by: Custom Assertions, Verification Method, generic reusable code, test inheritance, Expected Objects - using objects instead of fields, delegation
-
Test Logic in Production
- caused by: Test Hooks, code for tests only, test dependencies, equality pollution
- leads to: hidden bugs
- fixed by: special subclasses exposing behavior, utilities, clean up organisation/modularisation, custom assertions
Behavior Smells
-
Assertion Roulette
- ?: which assertion caused the failure -> localization
- caused by: Eager Tests,
- leads to: debugging
- fixed by: splitting up the test
-
Erratic Test
- ?: determinism, sometimes tests pass, sometimes not
- caused by: Interacting Tests/Suites, Shared Fixtures, execution order of tests, Lonely Test, Resource Leakage/Optimism, Unrepeatable Test
- fixed by: fresh fixtures
-
Fragile Test
-
Frequent Debugging
- caused by: lack of defect localization
- leads to: manual testing, long debugging sessions
- fixed by: test automation, small tests
-
Manual Intervention
- ?: test need user interaction/input
- caused by: manual fixtures/setup, event injection
- leads to: unrepeatable tests
- fixed by: simulate human interaction
-
Slow Tests
- caused by: Obscure Tests, only high level tests, shared Fixtures,
- leads to: decreased productivity
- fixed by: split up in smaller parts
Sensitivity Smells
-
Behavior Sensitivity
- ?: test A suddenly fails after B was modified
- caused by: introducing a new feature, bug 'fixed', setup modified
-
Context Sensitivity
- ?: test A suddenly fails after the state of some other components has been changed
- caused by: time/date dependencies, dependencies on other components
- leads to: non-deterministic tests
-
Data Sensitivity
- ?: test A suddenly fails after the data has been changed
- caused by: data require new functionality, new data, data removed
- leads to: increased effort in maintaining tests/data
-
Interface Sensitivity
- ?: test A suddenly fails after the interface of the SUT has been modifies
- caused by: changed application/API
- leads to: no refactorings ('don't touch it')
- fixed by: encapsulate the API/SUT behavior
Project Smells
-
Buggy Tests
- ?: tests fail although they shouldn't and vice versa
- caused by: Fragile/Flexible/Obscure Tests, hard to test code, too much logic in the test, source changes too fast
- leads to: Fragile Tests and high maintenance costs
- fixed by: write test first (->requirements), refactor legacy code, learn testing
-
Developers not writing Tests
- caused by: missing time, schedule, hard to test code, wrong automation strategy
- leads to: Fragile/Obscure Tests, manual debugging/testing
-
High Test Maintenance cost
- ?: it takes long to introduce new behavior
- caused by: Fragile/Obscure Test, spending much time with debugging, manual testing, badly test strategy, missing test automation
- fixed by: refactoring tests before continuing
-
Production Bugs
- ?: too many tests fail, many bugs found in source
- caused by: untested code, unfrequently run tests, not enough tests, lost/missing tests
-
Productivity Smell
- caused by: frequent debugging, missing automation, slow tests