The following describes known problems with the current release of the
DaCapo benchmark suite. We only list problems that we believe are
intrinsic to the suite. JVM-specific problems are not listed here.
- The 'tradebeans', 'tradesoap' and 'tomcat' all use network connections very inefficiently (using netstat we have observed more than 64,000 open connections when running trade). We believe that these problems are inherent to the workload (not our harness), so we have elected to leave them as-is.
- Some of the benchmarks do not build with a 1.6 jre (all will run on 1.6 JVMs). These limitations are intrinsic to the benchmarks---we have elected to work around those limitations and require 1.5 jres at build time. This only affects those who use the source distribution.
- Benchmark validation currently just uses a checksum of various outputs (including stdout, stderr, and in some cases output files). Ideally the validation might compare against a correct version of the output and produce a diff on failure. We were unable to add such support to the 9.12 release, but will consider it for subsequent releases. As always, contributions are welcome!