Case Studies

updated: November 8, 2006

Packages (from SqueakSource)

coverage subject coverage in % time (1) # methods (2) # test-cases (2) Notes
Aconcagua 65 % 1:00 800 400
Chronos ? % install of environment meesed up the image, selectors missing, ...
Cryptography (core only) 27 % 4:00 77 31
Cryptography (complete) ~40 % (3) 5:00 1680 31 #encrypt is mostly fully-covered, but #decrypt not, no tests for SSL?
Eg 10% 0:20 365 30
Files 15 % 1:00 900 83
Flash-Support 5 % 0:10 120 2
Geography 5 % 0:10 120 2
Goods 1 % 0:10 338 12 all tests have errors -> no db; instrumenting class-extension-methods causes Christo to crash the image —> CHECK !!!
Graphics 15 % 20:00 2250
Herd Behavior 56 % 0:15 10 4
HexConverter 26 % 0:10 23 3
ICal 25 % (F) 1:00 1155 124
IntelligentGames 65 % 1:00 125 83
Java 38 % 5:00 1654 12 tests run quite slow
Magritte (model only) 56 % 1:30 850 1600
Math Quaternion 55% 0:10 56 4
Metaspin ? % ? 98 14 tests run extremely slow with instrumentation
MethodCategoryNotifications 35% 0:05 27 7
Mondrian 16 % 0:05 440 6
Monticello ? % - 1600 115 (don’t run)
MSE-Exporter 56 % 0:30 120 28 1 error, 2 failures
Nebraska 5 % 0:30 720 4
Network-Standard 15 % 0:40 1400 100
Network-IRC 18 % (F) 1:00 606 14 only tests for profile and server
NetworkHTML 1 % 0:10 450 1
ObjectAsMethodWrappers 35 % 0:20 35 2
OB-Standard 50 % 5:00 540 230
Omnibrowser 60 % 6:00 1000 200
OpenDocument 50 % : 460 40 no "primitive" tests
PatRecLecture 55 % 0:05 40 1
RefactoringEngine 30 % 30:00 2200 300
RoelTyper 45 % 0:10 170 70
Saphir 32 % 0:10 149 2 problems while loading package, some tests might omitted (?)
Scheme 53 % 0:10 200 16
Smallwiki-Kernel 20 % 10:00 2320 250
Syslog 61 % 0:05 77 10
TestSmells (core only) 66% 3:00 269 383
Tsunami 75 % 0:20 50 20
Yaxo, XML Parser Tools no tests ? - - -

(1) aproximated, on a Powerbook G4 1.21 (2003) using ByteSurgeon-Adaptor (2) approximated (3) this package is composed by multiple subpackages, coverage for the super-package is calculated by averaging the coverage value of the subpackages (F) failures while running tests

Results (see above) ...

  • accessors and smaller utility methods are mostly well-covered, however some accessors don’t have coverage at all -> they’re not used in the class implementation (variables are accessed directly)
  • very small methods (up to 2 lines) mostly don’t have their own tests and/or have no coverage
  • core behavior is less-well tested, probably due higher test complexity
  • complex or very abstract behavior is badly tested or not at all
  • coverage is mostly missing for very small classes (max of 10 methods)
  • most coverage-items have either no or 25% to 65%, only a few have more than 85%, even less have 100%
  • class side behavior is mostly untested/uncovered, except instance creation methods (implicitly covered)
Last changed by admin on 21 April 2009