[Openmcl-devel] help on diagnosing an OpenMCL crash

Wed Jun 23 10:10:17 PDT 2004

>> We're running OpenMCL 0.14.2-p1 under OS X 10.3.4 on several different
>> machines. We have a fairly large Lisp tick-based simulation / data
>> collection application and have run into a problem recently when 
>> trying
>> to run the simulation for long periods of time. Note that this runs
>> successfully under MCL. Under OpenMCL, one of the following behaviors
>> occurs:
>>
>> (). OpenMCL just quits and returns to the command prompt at the 
>> terminal
>> (). OpenMCL quits with a bus error
>>
> When it quits with a bus error, what -exactly- does the message say ?

There is no crash log. When it quits, it can say either

Bus Error
Trace/BPT trap

and it says nothing more than this.

> In most cases, a thread-level exception handler should catch the
> exception that would eventually lead to a SIGBUS signal.  The
> exceptions (as in "exceptions to the rule ...") that I can think
> of involve:
>
> 1) foreign threads that have never called back into lisp
> 2) the thread that handles exceptions for all other threads
> 3) a thread running in the environment immediately after #_fork()
>    and before #_exec()
>
> (2) handles exceptions by pushing an exception frame on the faulting
>     thread's stack and then making it call some code in the lisp
>     kernel.  In theory, this can get a SIGBUS if pushing the frame
>     on a suspended thread's stack access unmapped memory; it's not
>     clear how this could happen unless the suspended thread had
>     severely overflowed its stack and this had gone undetected.
>
> That possibility is probably more likely than the others, but it
> doesn't sound very likely ... I'd be interested in seeing the
> crash dump if you can find it, but it's not clear that it'd
> tell us -why- the thread's stack pointer is in or near an unmapped
> page, if that's indeed what the problem is.

We are executing only Lisp code -- no FFI or anything else; also, it's 
also single process.

One additional clue is that we tried running GC ourselves (every 
hundred ticks). When we do this, OpenMCL does not crash.

Here is some memory data from a run without "forced" GC'ing:

The lisp-heap-gc-threshold in openmcl is (ccl::lisp-heap-gc-threshold) 
=> 16777216.  But watch what happens.  This behavior starts around 
:tick 2977 - openmcl appears to be operating below the gc threshold for 
several ticks, then gc'ing. After ~500 ticks of this it operates below 
the gc threshold until the freebytes bottom out a zero.  Then operates 
for a few more ticks until it finally performs (I think obviously) a gc 
at :tick 3782. From there it is all down hill.  I don't see another gc 
and openmcl operates below the lisp-heap-gc-threshold.

:tick 3713 :usedbytes 19136512  :freebytes 1179648 :heap-size(K) 726886
:tick 3714 :usedbytes 19267584  :freebytes 1048576 :heap-size(K) 727071
:tick 3715 :usedbytes 19464192  :freebytes 851968 :heap-size(K) 727256
:tick 3716 :usedbytes 19660800  :freebytes 655360 :heap-size(K) 727442
:tick 3717 :usedbytes 19857408  :freebytes 458752 :heap-size(K) 727639
:tick 3718 :usedbytes 20054016  :freebytes 262144 :heap-size(K) 727831
:tick 3719 :usedbytes 19202048  :freebytes 1114112 :heap-size(K) 728032
:tick 3720 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 728259
:tick 3721 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 728446
:tick 3722 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 728635
:tick 3723 :usedbytes 19988480  :freebytes 327680 :heap-size(K) 728820
:tick 3724 :usedbytes 19202048  :freebytes 1114112 :heap-size(K) 729046
:tick 3725 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 729243
:tick 3726 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 729424
:tick 3727 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 729600
:tick 3728 :usedbytes 19988480  :freebytes 327680 :heap-size(K) 729785
:tick 3729 :usedbytes 19267584  :freebytes 1048576 :heap-size(K) 729964
:tick 3730 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 730146
:tick 3731 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 730364
:tick 3732 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 730546
:tick 3733 :usedbytes 19988480  :freebytes 327680 :heap-size(K) 730731
:tick 3734 :usedbytes 20185088  :freebytes 131072 :heap-size(K) 730913
:tick 3735 :usedbytes 19333120  :freebytes 983040 :heap-size(K) 731098
:tick 3736 :usedbytes 19529728  :freebytes 786432 :heap-size(K) 731282
:tick 3737 :usedbytes 19726336  :freebytes 589824 :heap-size(K) 731463
:tick 3738 :usedbytes 19922944  :freebytes 393216 :heap-size(K) 731645
:tick 3739 :usedbytes 20119552  :freebytes 196608 :heap-size(K) 731823
:tick 3741 :usedbytes 19464192  :freebytes 851968 :heap-size(K) 732190
:tick 3742 :usedbytes 19660800  :freebytes 655360 :heap-size(K) 732376
:tick 3743 :usedbytes 19857408  :freebytes 458752 :heap-size(K) 732559
:tick 3744 :usedbytes 20054016  :freebytes 262144 :heap-size(K) 732738
:tick 3745 :usedbytes 19333120  :freebytes 983040 :heap-size(K) 732924
:tick 3746 :usedbytes 19464192  :freebytes 851968 :heap-size(K) 733142
:tick 3747 :usedbytes 19660800  :freebytes 655360 :heap-size(K) 733320
:tick 3748 :usedbytes 19857408  :freebytes 458752 :heap-size(K) 733502
:tick 3749 :usedbytes 20054016  :freebytes 262144 :heap-size(K) 733683
:tick 3750 :usedbytes 20185088  :freebytes 131072 :heap-size(K) 733860
:tick 3751 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 734046
:tick 3752 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 734239
:tick 3753 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 734426
:tick 3754 :usedbytes 19988480  :freebytes 327680 :heap-size(K) 734613
:tick 3755 :usedbytes 20185088  :freebytes 131072 :heap-size(K) 734808
:tick 3756 :usedbytes 19333120  :freebytes 983040 :heap-size(K) 734997
:tick 3757 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 735220
:tick 3758 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 735408
:tick 3759 :usedbytes 20054016  :freebytes 262144 :heap-size(K) 735653
:tick 3760 :usedbytes 20250624  :freebytes 65536 :heap-size(K) 735891
:tick 3761 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 736068
:tick 3762 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 736246
:tick 3763 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 736468
:tick 3764 :usedbytes 19988480  :freebytes 327680 :heap-size(K) 736648
:tick 3765 :usedbytes 20250624  :freebytes 65536 :heap-size(K) 736860
:tick 3766 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 737028
:tick 3767 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 737227
:tick 3768 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 737392
:tick 3769 :usedbytes 19922944  :freebytes 393216 :heap-size(K) 737555
:tick 3770 :usedbytes 20119552  :freebytes 196608 :heap-size(K) 737722
:tick 3771 :usedbytes 20250624  :freebytes 65536 :heap-size(K) 737883
:tick 3772 :usedbytes 19398656  :freebytes 917504 :heap-size(K) 738047
:tick 3773 :usedbytes 19529728  :freebytes 786432 :heap-size(K) 738216
:tick 3774 :usedbytes 19726336  :freebytes 589824 :heap-size(K) 738385
:tick 3775 :usedbytes 19922944  :freebytes 393216 :heap-size(K) 738586
:tick 3776 :usedbytes 20119552  :freebytes 196608 :heap-size(K) 738760
:tick 3777 :usedbytes 20316160  :freebytes 0 :heap-size(K) 738938
:tick 3778 :usedbytes 19595264  :freebytes 720896 :heap-size(K) 739187
:tick 3779 :usedbytes 19791872  :freebytes 524288 :heap-size(K) 739375
:tick 3780 :usedbytes 19922944  :freebytes 393216 :heap-size(K) 739538
:tick 3781 :usedbytes 20119552  :freebytes 196608 :heap-size(K) 739697
:tick 3782 :usedbytes 20250624  :freebytes 65536 :heap-size(K) 739860
:tick 3783 :usedbytes 12189696  :freebytes 16711680 :heap-size(K) 740025
:tick 3784 :usedbytes 12386304  :freebytes 16515072 :heap-size(K) 740190

HTH,
-- 
Gary Warren King, Lab Manager
EKSL East, University of Massachusetts * 413 577 0176

[The Bush administration’s global war on terrorism may have set the 
United States] "on a course of open-ended and gratuitous conflict with 
states and non-state entities that pose no serious threat to the United 
States."
-- Jeffrey Record,  Strategic Studies Institute of the Army War College