[Openmcl-devel] Performance benchmark results
Gary Byers
gb at clozure.com
Mon Apr 22 18:53:00 PDT 2002
This is certainly interesting; thanks for doing this.
A few of these numbers don't look right to me (sadly, a few others
look All Too Right.) I wasn't quite sure what was going on at first;
as brilliant and clever as sbcl/ppc's backend is, it's a little hard
to believe that FACTORIAL should be more than 3x faster than OpenMCL's
(OpenMCL stoleuses the same bignum representation as CMUCL/
SBCL, and OpenMCL's bignumXfixnum multiply routine is written in
assembler and not obviously inefficient.)
I finally realized that the value of the "lisp-heap-gc-threshold"
in OpenMCL is perhaps a little snug, considering how much consing
was happening in repeated invocation of some of these benchmarks.
In OpenMCL >= 0.10, the "lisp-heap-gc-threshold" is roughly the
number of bytes that (generally) should be consed between full
GC invocations; there's a similar parameter in SBCL/CMUCL called
something like "*AUTO-GC-TRIGGER*".
In SBCL, the parameter's set to 20MB or so; in OpenMCL, it defaults
to 4MB. As I said, this might be a little on the snug side (though
I'm not sure that improving benchmark performance is a compelling
reason to raise OpenMCL's default threshold.)
Not surprisingly, doing
? (set-lisp-heap-gc-threshold (ash 2 20))
and
? (use-lisp-heap-gc-threshold)
before running the benchmarks in OpenMCL seems to have a dramatic
effect on their overall execution time. (Sadly, making the gc threshold
larger doesn't make the compiler any smarter about avoiding unnecessary
floating-point consing, so some things - like FFT - are still slow
in OpenMCL.)
(It would be more convenient in many cases to have a way of setting the
gc threshold and putting it in effect from the command line.)
I don't know enough about CLISP and its GC to know whether or not there's
a similar parameter (or how to set such a parameter to a similar 20MB
value.)
I find it surprising that a bytecoded implementation is often over an
order of magnitude faster than either of the native implementations on
most of the BIGNUM/* benchmarks; I suspect some mixture of Black Magic,
clever algorithms, and (possibly) a little excessive zeal on the part
of CLISP's compiler. (It seemed that some calls to "side-effect-free"
arithmetic functions were getting optimized out because their results
weren't used.) I may have been mistaken about the excessive zeal or
may have misinterpreted the disassembled bytecode I looked at; I'll
try to look again when I'm more awake. Whether that's a factor or not,
I'm sure that there's room for improvement in both OpenMCL and SBCL.
Gary Byers
gb at clozure.com
On Wed, 17 Apr 2002, Eric Marsden wrote:
> Hi,
>
> Here are the results of running some performance bencharks comparing
> SBCL 0.7.2, OpenMCL 0.10 and CLISP CVS, on a G3 running Linux. The
> reference time is in seconds, and the two other columns are a
> multiple, so a value of 0.5 means two times as fast.
>
>
> ,---- 400MHz G3, 1024KB L2 cache, 192 MB RAM --
> |
> | Benchmark Reference CLISP SBCL
> | --------------------------------------------------
> | BOYER 7.15 3.57 0.36
> | BROWSE 2.42 2.51 0.61
> | DDerviv 4.00 2.62 0.29
> | Deriv 6.91 2.83 0.28
> | DESTRUCTIVE 3.31 4.42 0.69
> | DIV2-TEST-1 5.06 3.26 0.28
> | DIV2-TEST-2 2.54 3.27 0.26
> | FFT 16.63 1.27 0.05
> | FRPOLY/FIXNUM 2.89 4.29 0.40
> | FRPOLY/BIGNUM 5.75 2.22 0.46
> | FRPOLY/FLOAT 3.53 2.43 0.34
> | PUZZLE 19.53 2.37 1.06
> | TAK 2.08 12.04 1.45
> | CTAK 3.21 6.04 0.76
> | TRTAK 2.08 12.05 1.45
> | TAKL 6.94 5.36 0.67
> | STAK 2.86 7.01 0.94
> | FPRINT 5.38 1.33 1.75
> | TRAVERSE 7.81 10.37 1.20
> | TRIANGLE 7.59 11.73 1.04
> | RICHARDS 2.37 17.46 0.72
> | FACTORIAL 2.30 1.82 0.30
> | FIB 0.45 12.87 1.44
> | BIGNUM/ELEM-100-1000 2.33 0.17 0.99
> | BIGNUM/ELEM-1000-100 9.50 0.18 0.61
> | BIGNUM/ELEM-10000-1 18.52 0.09 0.33
> | BIGNUM/PARI-100-10 1.88 0.10 1.85
> | BIGNUM/PARI-200-5 17.84 0.05 3.35
> | HASH-STRINGS 5.60 1.49 14.50
> | HASH-INTEGERS 2.12 1.28 0.72
> | BOEHM-GC 58.63 1.82 0.22
> | DEFLATE 4.53 4.78 1.36
> | CLOS/defclass 0.75 4.76 14.77
> | CLOS/defmethod 0.31 2.87 150.39
> | CLOS/instantiate 31.41 0.83 0.54
> | CLOS/methodcalls 29.87 1.34 0.50
> | CLOS/method+after 12.61 1.69 6.09
> | CLOS/complex-methods 2.71 -1.00 1.55
> | 1D-ARRAYS 8.36 5.42 1.49
> | BITVECTORS 11.30 0.36 0.29
> | FILL-STRINGS 32.03 0.75 2.84
> | fill-strings/adjustable 35.86 1.87 2.78
> |
> | Reference implementation: OpenMCL Version (Beta: Linux) 0.10.1
> | Impl CLISP: CLISP 2.28 (released 2002-03-03) (built 3227525942) (memory 3227526439)
> | Impl SBCL : SBCL 0.7.2
> `----
>
> --
> Eric Marsden <URL:http://www.laas.fr/~emarsden/>
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel
>
>
_______________________________________________
Openmcl-devel mailing list
Openmcl-devel at clozure.com
http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel
More information about the Openmcl-devel
mailing list