[Openmcl-devel] FINISH-OUTPUT/fsync slowness

[Waldek Hebisch]

R. Matthew Emerson wrote:
> 
> On Nov 8, 2014, at 11:27 AM, Waldek Hebisch <hebisch at math.uni.wroc.pl> =
> wrote:
> 
> > IMHO decision to call fsync from FINISH-OUTPUT is unfortunate.
> > ANSI wording for FORCE-OUTPUT say that it merely starts writing
> > out buffer but do not wait for write to finish.  In particular
> > FORCE-OUTPUT is allowed to use asynchronous I/O.  Which
> > means that when Lisp program wants to pass data in a file
> > to another program it should call FINISH-OUTPUT as FORCE-OUTPUT
> > may be still doing its work at time when another program wants
> > to read the data.
> 
> CCL has called fsync on finish-output for a long time (10+ years).
> Some years ago, there was a lot of discussion about how expensive
> fsync was on ext3 filesystems;  I don't know what the current state
> of things is.

>From fsync man page:

       The  fsync()  implementations in older kernels and lesser used filesys-
       tems does not know how to flush  disk  caches.   In  these  cases  disk
       caches  need  to  be disabled using hdparm(8) or sdparm(8) to guarantee
       safe operation.

So, under old kernel fsync would merely copy data to disc cache.
Now data goes to disc (as it should).  Concerning cost: with
FINISH-OUTPUT removed FriCAS build needs 6m1.013s, using
FINISH-OUTPUT on new kernel 42m32.979s.  Using old kernel cost
of FINISH-OUTPUT did not stand out, build times were comparable
to current cost without FINISH-OUTPUT.  So with new kernels
FINISH-OUTPUT is significantly more expensive, of order 10-100 times.
Basically on consumer hard discs disc operation will take of
order 10ms.  If whole write cache is flushed this may be
worse.  With multimegabyte disc cache and fast controller
10 to 100 thousends of "disc" operations per second is possible
with moderate actual disc traffic so on old kernels fsync did not
look so bad (but did not really work).  Linux can easily create
tens of thousends of processes per second.  So doing fsync on
files passed to spawned processes would kill performance.

> The spec says, "finish-output attempts to ensure that any buffered
> output sent to output-stream has reached its destination, and then
> returns."  That sounds a lot like fsync to me.

Depends what is meant by destination.  For me destination is OS.
Namely Lisp handles data to OS and in general can not know what
happens next.  Also, this is what is normally needed.  And this
is what other languages are doing.  AFAICS currently ecl, gcl
and sbcl are sending data to OS, without fsync.  clisp is
using fsync quite liberally, even if program preforms no
FINISH-OUTPUT or FORCE-OUTPUT.

Let me add that it is not my goal to push Closure CL to
abandon fsync -- I have a workaround so I can live with current
behaviour.  Rather, I think that it worth knowing how expensive
fsync is.

-- 
                              Waldek Hebisch
hebisch at math.uni.wroc.pl