<html><head></head><body dir="auto"><div><br></div><div>Did you try it using the :size argument to make-hash-table, with the estimated number of entries in the file system?</div><div><br></div><div><br>Best Regards,<div><br></div><div> Dave</div><div><br></div></div><div><br>On Jan 16, 2013, at 6:31 PM, George Khouri <<a href="mailto:gk1@four-four.com">gk1@four-four.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<font size="-1">Friends,<br>
In a program that traverse<font size="-1">s</font> a file system </font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">of
~125000 entri<font size="-1">es </font></font></font>and
adds the pathnames as keys to an <font size="-1">#'equal</font><font size="-1"> hash table<font size="-1">,</font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"> ccl
hangs in CCL::LOCK-FREE-PUTHASH with no progress after <font size="-1">~14000 entries </font>- seems like it's just
thrashing. </font></font>No error is thrown and I
can interrupt it with control-C. <br>
<font size="-1">If I add the pathnames to <font size="-1">a<font size="-1">n #'eql hash tab<font size="-1">le <font size="-1">ccl has no problem with the same dir<font size="-1">ectory<font size="-1"><font size="-1">. <br>
<font size="-1">Is this expected behavior<font size="-1"><font size="-1">? </font> <br>
<br>
<font size="-1">I<font size="-1"> tried the
same thing</font> in SBCL which works with
ei<font size="-1">ther an eql or <font size="-1">an equal hash table. <font size="-1">BTW, the eql version runs in
<font size="-1">~6 seconds in SBCL,
compared to <font size="-1">~70
seconds in CCL<font size="-1">, th<font size="-1">ough I realize this
speed difference could be a
memory issue and not one of
algorithms. I<font size="-1">
know nothing about SB<font size="-1">CL<font size="-1">.</font></font></font><br>
<br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">I'm running ccl 1.8 on linux <font size="-1">3.2.0-33-generic Ubuntu 12.04 on a core
i5 thinkpad, 4GB. <br>
</font></font></font><br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">It
doesn't "appear" that I'm running low on
memory, but the program behaves that way with
the equal hash table, and t<font size="-1">hough
I've tried <font size="-1">the various ccl
command-line options to increas<font size="-1">e sta<font size="-1">ck and
heap allocations<font size="-1">, as
well as ulimit stack size,</font>
I've had no change in results. </font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">I<font size="-1">'ve tried various val<font size="-1">u<font size="-1">es for
init<font size="-1">ial hash
table size, <font size="-1">reha<font size="-1">sh size and
threshold but noth<font size="-1">ing seems to
have any <font size="-1">ben<font size="-1">eficial
effect. The system
monitor doesn't show
any <font size="-1">disk
swapping going on.</font><br>
</font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">Is
there some memory parameter I'm
missing that I need to change? Is
there a problem related to having 4
cpu cores? </font></font></font><br>
<br>
Here's the typ<font size="-1">ical </font>output <font size="-1">when
I interrupt <font size="-1">the seemingly hung program - the
hash<font size="-1"> table won't grow beyond this point</font></font>:</font><br>
<br>
Welcome to Clozure Common Lisp Version 1.8-r15286M (LinuxX8664)!<br>
? (traverse-filesystem "<font size="-1"><font size="-1">/home/gkhouri</font>/"</font>)<br>
................^C<br>
> Break: interrupt signal<br>
> While executing: CCL::LOCK-FREE-PUTHASH, in process
listener(1).<br>
...<br>
1 > (:f 0)<br>
(7FB9FE90FF80) : 0 (LOCK-FREE-PUTHASH
#P"/tmp/gkhouri-test/Documents/Music/M Stal Tunes.csv"
#<HASH-TABLE :TEST EQUAL size 13672/13673 #x302000753F9D> T)
613<br>
(CCL::KEY CCL::HASH CCL::VALUE)<br>
CCL::KEY: #P"/tmp/gkhouri-test/Documents/Music/M Stal
Tunes.csv"<br>
CCL::HASH: #<HASH-TABLE :TEST EQUAL size 13672/13673
#x302000753F9D><br>
CCL::VALUE: T<br>
<br>
1 > (room)<br>
Approximately 17,432,576 bytes of memory can be allocated <br>
before the next full GC is triggered. <br>
<br>
Total Size Free
Used<br>
Lisp Heap: 41025536 (40064K) 17432576 (17024K) 23592960
(23040K)<br>
Stacks: 11072816 (10813K) 11055872 (10797K) 16944
(17K)<br>
Static: 20304144 (19828K) 0 (0K) 20304144
(19828K)<br>
376792.870 MB reserved for heap expansion.<br>
<br>
<br>
If I create the equal hash table with :lock-free set to nil, <font size="-1"> </font>(make-hash-table :test #'equal :lock-free
nil), ccl presumabl<font size="-1">y uses an older hashing
algorithm and </font>I don't think I have the problem storing
the path<font size="-1">names</font>, BUT after about 15,000
entries, things slow down drastically, and unacceptably,
presumably from rehashing<font size="-1"> </font>(?), and I
killed the prog<font size="-1">ram after about 60,000 entries
because it was taking so long</font>. <font size="-1">With <font size="-1">:lock-free</font></font> enable<font size="-1">d, I
never get past ~14<font size="-1">K entries. </font></font><br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><br>
T<font size="-1">hanks for any info.<br>
<font size="-1">George Khouri</font><br>
</font> </font><br>
<br>
<br>
<br>
</font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Openmcl-devel mailing list</span><br><span><a href="mailto:Openmcl-devel@clozure.com">Openmcl-devel@clozure.com</a></span><br><span><a href="http://clozure.com/mailman/listinfo/openmcl-devel">http://clozure.com/mailman/listinfo/openmcl-devel</a></span><br></div></blockquote></body></html>