The benchmarks of some Common LISP implementations

1. Preface

   PRODATA recently had performed benchmarking for a number of different Common LISP implementations. You can see the results below.    Most benchmarks are based on the well-known Gabriel benchmarks. Some additional benchmarks were taken from the CMU CL source code tree (src/benchmarks), which include Richard's benchmark (an operating system simulation) and modified version of cascor1.lisp (cascade correlation learning algorithm). You can find this modified version here.

   All benchmarks have been performed using following hardware, having all three OS'es (Windows, Linux and FreeBSD) installed with multiboot.

• Model - HP Omnibook XE3 laptop
• CPU - Intel Celeron (Coppermine) 650 MHz
• RAM - 128 MB
• ACPI - disabled
• HDD - internal IBM - 10 GB
• Video - S3 Savage IX / TFT panel (isn't very important for tests)

   Some notes and conclusions about these benchmarking are placed at the end of the page. Please see the results:

2. Benchmarking results

3. Notes on benchmarking process

   All benchmarks where performed in a 'quiet' environment, which means that almost all daemons on Linux and FreeBSD had been stopped and no graphical environment was running. Also the amount of physical memory was big enough for running LISP processes without page faults and swapping.

   All open-source Lisp distributions (all, except Allegro CL) either had been found preinstalled, or installed from legal source (RPM's and 'ports') in both Linux and FreeBSD distributions. The version numbers of all LISPs on both platforms are chosen the same. Due to strange GCL test results on FreeBSD (much slower, than on Linux) it had been recompiled from the sources, but nothing significantly had changed.

   (*) The cascor1.lisp benchmark was finishing every time with different 'epoch' counts on various LISPs, so the benchmark result was calculated as a mean value for one epoch. Even more, it has to be pointed out, every latest epoch calculation took slightly longer time to complete, so this benchmark has not to be treated as very strict. Also, this benchmark was slightly modified in order to run on all LISPs by working around with floating point arithmetic underflows and setting up all arithmetic strictly to float type (slowing benchmark results by this, though).

   (**) The Windows version of CLISP was extremely slow. We think that it is because CLISP uses the virtual machine concept. The other possible reason - it runs in MSDOS window, which is scheduling rather slowly under MS Windows.

   (***) The Corman Lisp 'stak' benchmark took about 5 minutes and then had hung. There is nothing we can tell more, except that there was intensive garbage collecting during the test. We can suppose, that this probably had happened because of unregistered version being used.
Also, the Corman Lisp benchmarks were performed in compiled form only, because all LISP source files were compiled on-the-fly by it, - so there was no such possibility to test non-compiled interpretation.

   Each benchmark was run in non-compiled (interpreted) and compiled variants, because we also like to inspect the LISP behaviour as a dynamic language (even when the code is compiled, you still can use the constructs like (eval) and lambda expressions, which are dynamically interpreted). These test results show, that not all LISP implementations are very good in that.

   Very important note: These benchmarks are NOT Linux vs. FreeBSD performance tests. They are only cross-LISPs tests. In this particular case Linux performed some better. For those who want to know where FreeBSD performs better - please read more about FreeBSD and the cases when FreeBSD really outperforms Linux. It's only to say here - one of them is when memory is tight, and/or load is high, and this is not the case we had here when we've tested LISPs.

4. Conclusion

   Despite some important and unique features of other LISP inplementations we can state here, that the clear winner between LISPs on UNIX platforms is CMU CL. Even its speed of interpretation is very close to winners in this category. But CMU CL has one serious disadvantage - it has no MS Windows implementation, nor it has a good graphics support (there are GTK bindings for Linux, though). So if we chose to write a cross-platform graphical application, the Allegro CL with its Common Graphic Interface would be a better alternative. CLISP and GCL have their own benefits in some cases, especially for writing applications, which are designed to run on UNIX-like platforms only.

   Disclaimer: We are aware, that there are lies, damn lies, statistics and benchmarks. The benchmarks published here, are proposed for your own use (and risk) only.

   Comments and corrections are welcome. Please visit the PRODATA contacts page for contact information.