Simple Statistics

updated: October 14, 2006

• TestSmells/TestLint still runs too slow -> optimize run time

• Aconcagua (supposed to be very "good") - collecting results of all rules, 1 afternoon ;( ;( —> smells and interesting things:

  • some untested behaviors sent/used
  • some behaviors untested (no coverage, no tests)
  • some behaviors implemented, but never/nowhere used
  • some behaviors only used in tests but not elsewhere (this is no problem for a system extension)
  • lots of cascaded test assertions (complexity, debug-ability)
  • a few "too" long tests (eager)
  • tremendous amount of magic literals (encapsulation break)
  • some dangerous literal conversions and comparisions (encapsulation break)
  • some non-effictive object comparisions (e.g. 1 = 1)
  • 1 hidden-shared fixture (shared fixture, but no (correct) setup/teardown)
  • no strict testing conventions (naming, .......)
  • lots of enumerations in tests (complexity, debug-ability)
  • most parts probably not done by TDD (fuzzy)

• Network (well ... see Squeak Graveyard) - collecting results of all (except 1) rules, <5 min —> smells and interesting things:

  • only a very few tests, poor coverage, most behaviors untested
  • one early returning test (e.g. "^self" instead of running assertions)
  • some enumerations (complexity, debug-ability)
  • lots of equivalent tests ( -> partial ordering / coverage)
  • 2 hidden-shared fixtures (shared fixture, but no (correct) setup/teardown)
  • tremendous amount of literal conversions and comparisions (-> can be expected as Network is quite lowlevel, encapsulation break)
  • some logic operations in tests (ifTrue / ifFalse / ..., repeatability)
  • tremendous amount of magic literals ( -> can be expected as Network is quite lowlevel, encapsulation break)
  • rather complex and long tests with lots of comments (complexity, readability, documetation)
  • 1 outcommented test code, but wouldn't break if comment removed - lucky ;)
  • poor names for test selectors (e.g. test1, test2, test3, ...)
  • much behavior only used for tests
  • not done in TDD (fuzzy)

• MSE-Exporter - collecting results of all (except 1) rule —> smells and interesting things:

  • some enumerations (complexity, debug-ability)
  • some behaviors only used in tests but not elsewhere (this is no problem for a system extension)
  • 1 non-effictive object comparisions (e.g. 1 = 1)
  • many literal conversion/comparisions (encapsulation break)
  • many magic literals
  • some test-code put into comments
  • 1 outcommented test that would fail on comment removal - UPS ! -> MSEExportTest>#testClassCategory
  • 3 very long tests (size and runtime)
  • 1 test with too many comments (more comments than actual test code)
  • 2 classes with bad selector names (complexity, readability, documetation)
  • some behaviors implemented, but never/nowhere used
  • some untested behaviors sent/used
  • some behaviors untested (no coverage, no tests)

• Intelligent Games

  • tremendous amount of magic literals ( -> expected as part of the "game")
  • some enumerations (complexity, debug-ability)
  • some behaviors implemented, but never/nowhere used
  • some untested behaviors sent/used
  • some behaviors untested (no coverage, no tests)

• Nebraska

  • only a very very few tests > lots of behaviors untested (no coverage, no tests)
  • some behaviors only used in tests but not elsewhere (this is no problem for a system extension)
  • some magic literals
  • 1 organization convention ignored
  • 2 bad names for testselectors
  • ? none of the fixtures could not be analysed -> instrumentation error ?

• Files / Filesystem

  • many tests returning "too early" (->not all assertsions/statements executed -> ^self/^nil/.... )
  • many logic operators (-> droping assertions -> ifTrue: / ifFalse: / ...)
  • many magic literal (expected)
  • 2 mystery guests (access to an external file)
  • tests contain rather complex than simple assertions
  • 3 tests with outcommented code
  • organization convention ignored (almost everywhere, development was a bit sloppy ?)
  • lot of comments in tests, 1 really really really big one (6 lines)
  • one testclass does only have a teardown (that doesn't do what it is supposed to do) -> ok, but not really nice -> sideffects of tests

• ICal

  • 1 method overrides the default perform-behavior of sunit
  • some test-methods with outcommented code
  • some test-methods with outcommented assertsions that would fail on comment removal - UPS !!!
  • 1 testmethod is completely outcommented
  • many enumerations (complexity, debug-ability)
  • many missing tests
  • at least 1 testclass having hidden shared fixtures (shared fixture, but no (correct) setup/teardown)
  • 1 class with a fixture not used enough (mostly unused)
  • many literal conversions and comparisions
  • tremendous amount of magic literals ( expected)
  • 4 testclasses with "too many" instance variables, only 2 of them have a too large fixture
  • many testmethods not using the shared fixture (unbalanced use)
  • 2x organization convention ignored
  • 1 test-method with no assertions at all
  • some compex assertions (nested, cascaded, block, ...)
  • (unsure) 1 shared fixture instead on fresh fixtures
  • many bad test-selector names
  • unmanaged variables instead of using a fixture
  • some behavior that is nowhere use nor tested
  • 2 shared fixtures with variables/behavior never used/accessed by any test-method

• RoelTyper

  • some long testcases / manual tests -> no test selectors
  • some enumerations (complexity, debug-ability)
  • many missing tests / bad coverage
  • 1 hidden-shared fixture (shared fixture, but no (correct) setup/teardown)
  • some logic operators (-> droping assertions -> ifTrue: / ifFalse: / ...)
  • tremendous amount of magic literals
  • many bad test-selector names
  • organization convention ignored

• Cryptography

  • quite difficult to analyse as 'Cryptography' is not just one package but in fact a bundle of seperate packages - maybe a set for analysis would be appropriate, tests are not together with the code ...

  • average coverage in generel: ASN (0%), Core (27%), DES (59%), DSA (23%), ELG (88%), MD4 (84%), MD5 (90%), RC4 (75%), RSA (32%), RiJ (51%), SHA (58%), SHA256 (94%), SSL (0%), X509 (0%), Base (62%)
  • only a very very few very abstract tests (~30 for such a big package !, bad: complexity, debug-ability, bug-tracking)
  • all tests have magic literals (expected)
  • some very long tests (runtime and code)
  • 1 test with literal conversion (encapsulation break)
  • many enumerations (complexity, debug-ability)
  • test-organization convention ignored
  • many untested behaviors

• Tsunami

  • medium-good coverage
  • some literal conversions
  • organization convention ignored
  • more abstract tests than primitive ones, inly a few tests

• OpenDocument

  • some enumerations
  • tests with too many referenced / coupling
  • many literal conversions
  • rather complex tests with a lot of control logics
  • tremendous amount of magic literals
  • some very long tests
  • organization convention ignored
  • many tests have too many/large comments
  • some tests with outcommented code and assertsions
  • 1 test with outcommented assertions that would fail on comment removal - UPS
  • bad test selector names
  • much untested behavior (sent)