[Openmcl-devel] Speeding up I/O

[Tim Bradshaw]

On 23 Jul 2009, at 21:00, Ron Garret wrote:

> I'm trying to crunch a semi-large file (~500MB) and just reading the
> data via (while (read-line f nil nil)) take about seven times longer
> than it does in, say, Python (73 seconds versus 12).  Is there some
> low-level I/O primitive that I can call that bypasses all the high-
> level stream stuff that would speed this up?  %ioblock-unencoded-
> character-read-vector maybe?  Or %ioblock-tyi?  Can I call these
> directly without screwing things up?


READ-SEQUENCE & WRITE-SEQUENCE are your friends here.  On a Mac laptop  
with enough memory (4G) but an ordinary laptop disk, I can write a  
500M file in about 11 seconds and read it in about 8 (if uncached by  
the FS) or 2 (if cached by the FS), reading 1k blocks of (UNSIGNED- 
BYTE 8)s. In both cases I wrote or read the same sequence multiple  
times so that does not include consing overhead for sequences.  It  
doesn't actually take much longer to read the whole file into a single  
large array though.

I guess reading characters would be slower because of the encoding/ 
decoding overhead.

--tim