[Openmcl-devel] Performance benchmark results

Gary Byers gb at clozure.com
Mon Apr 22 18:53:00 PDT 2002


This is certainly interesting; thanks for doing this.

A few of these numbers don't look right to me (sadly, a few others
look All Too Right.)  I wasn't quite sure what was going on at first;
as brilliant and clever as sbcl/ppc's backend is, it's a little hard
to believe that FACTORIAL should be more than 3x faster than OpenMCL's
(OpenMCL stoleuses the same bignum representation as CMUCL/
SBCL, and OpenMCL's bignumXfixnum multiply routine is written in
assembler and not obviously inefficient.)

I finally realized that the value of the "lisp-heap-gc-threshold"
in OpenMCL is perhaps a little snug, considering how much consing
was happening in repeated invocation of some of these benchmarks.
In OpenMCL >= 0.10, the "lisp-heap-gc-threshold" is roughly the
number of bytes that (generally) should be consed between full
GC invocations; there's a similar parameter in SBCL/CMUCL called
something like "*AUTO-GC-TRIGGER*".

In SBCL, the parameter's set to 20MB or so; in OpenMCL, it defaults
to 4MB.  As I said, this might be a little on the snug side (though
I'm not sure that improving benchmark performance is a compelling
reason to raise OpenMCL's default threshold.)

Not surprisingly, doing
? (set-lisp-heap-gc-threshold (ash 2 20))
and
? (use-lisp-heap-gc-threshold)

before running the benchmarks in OpenMCL seems to have a dramatic
effect on their overall execution time.  (Sadly, making the gc threshold
larger doesn't make the compiler any smarter about avoiding unnecessary
floating-point consing, so some things - like FFT - are still slow
in OpenMCL.)

(It would be more convenient in many cases to have a way of setting the
gc threshold and putting it in effect from the command line.)

I don't know enough about CLISP and its GC to know whether or not there's
a similar parameter (or how to set such a parameter to a similar 20MB
value.)

I find it surprising that a bytecoded implementation is often over an
order of magnitude faster than either of the native implementations on
most of the BIGNUM/* benchmarks; I suspect some mixture of Black Magic,
clever algorithms, and (possibly) a little excessive zeal on the part
of CLISP's compiler.  (It seemed that some calls to "side-effect-free"
arithmetic functions were getting optimized out because their results
weren't used.)  I may have been mistaken about the excessive zeal or
may have misinterpreted the disassembled bytecode I looked at; I'll
try to look again when I'm more awake.  Whether that's a factor or not,
I'm sure that there's room for improvement in both OpenMCL and SBCL.

Gary Byers
gb at clozure.com

On Wed, 17 Apr 2002, Eric Marsden wrote:

> Hi,
>
> Here are the results of running some performance bencharks comparing
> SBCL 0.7.2, OpenMCL 0.10 and CLISP CVS, on a G3 running Linux. The
> reference time is in seconds, and the two other columns are a
> multiple, so a value of 0.5 means two times as fast.
>
>
> ,---- 400MHz G3, 1024KB L2 cache, 192 MB RAM --
> |
> | Benchmark                 Reference  CLISP   SBCL
> | --------------------------------------------------
> | BOYER                          7.15   3.57   0.36
> | BROWSE                         2.42   2.51   0.61
> | DDerviv                        4.00   2.62   0.29
> | Deriv                          6.91   2.83   0.28
> | DESTRUCTIVE                    3.31   4.42   0.69
> | DIV2-TEST-1                    5.06   3.26   0.28
> | DIV2-TEST-2                    2.54   3.27   0.26
> | FFT                           16.63   1.27   0.05
> | FRPOLY/FIXNUM                  2.89   4.29   0.40
> | FRPOLY/BIGNUM                  5.75   2.22   0.46
> | FRPOLY/FLOAT                   3.53   2.43   0.34
> | PUZZLE                        19.53   2.37   1.06
> | TAK                            2.08  12.04   1.45
> | CTAK                           3.21   6.04   0.76
> | TRTAK                          2.08  12.05   1.45
> | TAKL                           6.94   5.36   0.67
> | STAK                           2.86   7.01   0.94
> | FPRINT                         5.38   1.33   1.75
> | TRAVERSE                       7.81  10.37   1.20
> | TRIANGLE                       7.59  11.73   1.04
> | RICHARDS                       2.37  17.46   0.72
> | FACTORIAL                      2.30   1.82   0.30
> | FIB                            0.45  12.87   1.44
> | BIGNUM/ELEM-100-1000           2.33   0.17   0.99
> | BIGNUM/ELEM-1000-100           9.50   0.18   0.61
> | BIGNUM/ELEM-10000-1           18.52   0.09   0.33
> | BIGNUM/PARI-100-10             1.88   0.10   1.85
> | BIGNUM/PARI-200-5             17.84   0.05   3.35
> | HASH-STRINGS                   5.60   1.49  14.50
> | HASH-INTEGERS                  2.12   1.28   0.72
> | BOEHM-GC                      58.63   1.82   0.22
> | DEFLATE                        4.53   4.78   1.36
> | CLOS/defclass                  0.75   4.76  14.77
> | CLOS/defmethod                 0.31   2.87 150.39
> | CLOS/instantiate              31.41   0.83   0.54
> | CLOS/methodcalls              29.87   1.34   0.50
> | CLOS/method+after             12.61   1.69   6.09
> | CLOS/complex-methods           2.71  -1.00   1.55
> | 1D-ARRAYS                      8.36   5.42   1.49
> | BITVECTORS                    11.30   0.36   0.29
> | FILL-STRINGS                  32.03   0.75   2.84
> | fill-strings/adjustable       35.86   1.87   2.78
> |
> | Reference implementation: OpenMCL Version (Beta: Linux) 0.10.1
> | Impl CLISP: CLISP 2.28 (released 2002-03-03) (built 3227525942) (memory 3227526439)
> | Impl SBCL : SBCL 0.7.2
> `----
>
> --
> Eric Marsden                          <URL:http://www.laas.fr/~emarsden/>
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel
>
>


_______________________________________________
Openmcl-devel mailing list
Openmcl-devel at clozure.com
http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel




More information about the Openmcl-devel mailing list