<html>
<head>
</head>
<body text="#000000" bgcolor="#FFFFFF">
I did but there appears to be some limit to the initial size of a
hash table in ccl. I tried :size 90000 but when I interrupted the
program the initial size of the table was only about 15k. I admit
that l0-hash.lisp is a little too much brain drain for me to try to
understand (in a reasonable amount of time) everything that's going
on with hashing so I thought I'd ask.<br>
Thanks,<br>
George<br>
<br>
<div class="moz-cite-prefix">On 01/16/2013 04:07 PM, Dave Cooper
wrote:<br>
</div>
<blockquote cite="mid:CA8806A0-82D9-4293-8525-D79916150261@genworks.com" type="cite">
<div><br>
</div>
<div>Did you try it using the :size argument to make-hash-table,
with the estimated number of entries in the file system?</div>
<div><br>
</div>
<div><br>
Best Regards,
<div><br>
</div>
<div> Dave</div>
<div><br>
</div>
</div>
<div><br>
On Jan 16, 2013, at 6:31 PM, George Khouri <<a moz-do-not-send="true" href="mailto:gk1@four-four.com">gk1@four-four.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div> <font size="-1">Friends,<br>
In a program that traverse<font size="-1">s</font> a file
system </font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">of
~125000 entri<font size="-1">es </font></font></font>and
adds the pathnames as keys to an <font size="-1">#'equal</font><font size="-1"> hash table<font size="-1">,</font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"> ccl hangs in
CCL::LOCK-FREE-PUTHASH with no progress
after <font size="-1">~14000 entries </font>-
seems like it's just thrashing. </font></font>No
error is thrown and I can interrupt it with
control-C. <br>
<font size="-1">If I add the pathnames to <font size="-1">a<font size="-1">n #'eql hash
tab<font size="-1">le <font size="-1">ccl
has no problem with the same dir<font size="-1">ectory<font size="-1"><font size="-1">. <br>
<font size="-1">Is this
expected behavior<font size="-1"><font size="-1">? </font>
<br>
<br>
<font size="-1">I<font size="-1"> tried the
same thing</font> in
SBCL which works with ei<font size="-1">ther an eql
or <font size="-1">an
equal hash table. <font size="-1">BTW, the
eql version runs
in <font size="-1">~6
seconds in SBCL,
compared to <font size="-1">~70
seconds in CCL<font size="-1">, th<font size="-1">ough
I realize this
speed
difference
could be a
memory issue
and not one of
algorithms. I<font size="-1">
know nothing
about SB<font size="-1">CL<font size="-1">.</font></font></font><br>
<br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">I'm
running ccl
1.8 on linux <font size="-1">3.2.0-33-generic
Ubuntu 12.04
on a core i5
thinkpad, 4GB.
<br>
</font></font></font><br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">It
doesn't
"appear" that
I'm running
low on memory,
but the
program
behaves that
way with the
equal hash
table, and t<font size="-1">hough
I've tried <font size="-1">the
various ccl
command-line
options to
increas<font size="-1">e
sta<font size="-1">ck
and heap
allocations<font size="-1">, as
well as ulimit
stack size,</font>
I've had no
change in
results. </font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">I<font size="-1">'ve
tried various
val<font size="-1">u<font size="-1">es
for init<font size="-1">ial
hash table
size, <font size="-1">reha<font size="-1">sh
size and
threshold but
noth<font size="-1">ing
seems to have
any <font size="-1">ben<font size="-1">eficial
effect. The
system monitor
doesn't show
any <font size="-1">disk
swapping going
on.</font><br>
</font></font><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1">Is
there some
memory
parameter I'm
missing that I
need to
change? Is
there a
problem
related to
having 4 cpu
cores? </font></font></font><br>
<br>
Here's the typ<font size="-1">ical
</font>output
<font size="-1">when
I interrupt <font size="-1">the
seemingly hung
program - the
hash<font size="-1">
table won't
grow beyond
this point</font></font>:</font><br>
<br>
Welcome to
Clozure Common
Lisp Version
1.8-r15286M
(LinuxX8664)!<br>
?
(traverse-filesystem
"<font size="-1"><font size="-1">/home/gkhouri</font>/"</font>)<br>
................^C<br>
> Break:
interrupt
signal<br>
> While
executing:
CCL::LOCK-FREE-PUTHASH,
in process
listener(1).<br>
...<br>
1 > (:f 0)<br>
(7FB9FE90FF80)
: 0
(LOCK-FREE-PUTHASH
#P"/tmp/gkhouri-test/Documents/Music/M
Stal
Tunes.csv"
#<HASH-TABLE
:TEST EQUAL
size
13672/13673
#x302000753F9D>
T) 613<br>
(CCL::KEY
CCL::HASH
CCL::VALUE)<br>
CCL::KEY:
#P"/tmp/gkhouri-test/Documents/Music/M
Stal
Tunes.csv"<br>
CCL::HASH:
#<HASH-TABLE
:TEST EQUAL
size
13672/13673
#x302000753F9D><br>
CCL::VALUE:
T<br>
<br>
1 > (room)<br>
Approximately
17,432,576
bytes of
memory can be
allocated <br>
before the
next full GC
is triggered.
<br>
<br>
Total
Size
Free
Used<br>
Lisp
Heap:
41025536
(40064K)
17432576
(17024K)
23592960
(23040K)<br>
Stacks:
11072816
(10813K)
11055872
(10797K)
16944 (17K)<br>
Static:
20304144
(19828K)
0 (0K)
20304144
(19828K)<br>
376792.870 MB
reserved for
heap
expansion.<br>
<br>
<br>
If I create
the equal hash
table with
:lock-free set
to nil, <font size="-1"> </font>(make-hash-table
:test #'equal
:lock-free
nil), ccl
presumabl<font size="-1">y
uses an older
hashing
algorithm and
</font>I don't
think I have
the problem
storing the
path<font size="-1">names</font>,
BUT after
about 15,000
entries,
things slow
down
drastically,
and
unacceptably,
presumably
from rehashing<font size="-1"> </font>(?),
and I killed
the prog<font size="-1">ram
after about
60,000 entries
because it was
taking so long</font>.
<font size="-1">With
<font size="-1">:lock-free</font></font>
enable<font size="-1">d, I
never get past
~14<font size="-1">K
entries. </font></font><br>
<font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><br>
T<font size="-1">hanks
for any info.<br>
<font size="-1">George
Khouri</font><br>
</font> </font><br>
<br>
<br>
<br>
</font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>Openmcl-devel mailing list</span><br>
<span><a moz-do-not-send="true" href="mailto:Openmcl-devel@clozure.com">Openmcl-devel@clozure.com</a></span><br>
<span><a moz-do-not-send="true" href="http://clozure.com/mailman/listinfo/openmcl-devel">http://clozure.com/mailman/listinfo/openmcl-devel</a></span><br>
</div>
</blockquote>
</blockquote>
<br>
</body>
</html>