[Openmcl-devel] Transfer the contents of a file to a tcp stream

[Gary Byers]

On Thu, 5 May 2005, [ISO-8859-1] Christian Nybø wrote:

>
> On May 3, 2005, at 23:22, Gary Byers wrote:
>
>> Hmm.  I was just about to recommend cheating (using sendfile) if it's
>> available (it's available on Linux, but doesn't seem to be there on
>> Darwin/OSX).
>
> From googling sendfile mac os x, I got the impression that sendfile (or 
> transmitfile as it is called in NT)  is not yet present in mac os x.
>
> I downloaded apache and had a look at writev_it_all in server/core.c, but I'm 
> unfamiliar with reading production code in C, so I don't know whether apache 
> does anything particular when sendfile is absent.

#_writev (and #_readv) are like #_write and #_read (respectively), but
read and write multiple buffers in a single system call.  There's still
some address translation and copying between user space and the kernel
going on, but presumably the cost of this is amortized somewhat.

>
>> You can cheat a bit
>> by doing something like:
>> 
>> (defun copy-file-to-socket (file-stream socket-stream)
>>   (let* ((file-fd (ccl::stream-device file-stream :input))
>>          (socket-fd (ccl::stream-devices socket-stream :output))
>> 	 (bufsize 8192)) ; arbitrary
>>     (force-output socket-stream) ; flush any buffered output
>>     (%stack-block ((buffer bufsize))
>>     ;; You could just say (#_lseek ...) here, but Linux's
>>     ;; #_lseek may have difficulty with large file offsets.
>>     (ccl::fd-lseek file-fd 0 #$SEEK_SET)
>>     (loop
>>       (let* ((nread (#_read file-fd buf bufsize)))
>>         (cond ((zerop nread) (return))
>>               ((minusp nread)(error ...))
>>               (t (let* ((nwritten (#_write socket-fd buf nread)))
>>                    (when (< nwritten nread)
>>                           ;;; Handle partial writes, errors
>>                          )))))))))
>> 
>> That's an artist's conception and may be a little buggy, but using
>> a single buffer and bypassing most of the buffered stream overhead
>> might get things -closer- to Apache's performance.
>
> nwritten was set to -1 quite often, so I added a loop that would try until 
> #_write returned the right number of bytes.  That's probably not the right 
> way, is it?
>
> (defun copy-file-to-socket (file-stream socket-stream)
> (declare (optimize (speed 3) (safety 0)))
>  (let* ((file-fd (ccl::stream-device file-stream :input))
>         (socket-fd (ccl::stream-device socket-stream :output))
> 	 (bufsize 2048))		; arbitrary
>    (force-output socket-stream)	; flush any buffered output
>    (ccl::%stack-block ((buf bufsize));; %stack-block is not visible without 
> the package prefix (chr)
>      ;; You could just say (#_lseek ...) here, but Linux's
>      ;; #_lseek may have difficulty with large file offsets.
>      (ccl::fd-lseek file-fd 0 #$SEEK_SET)
>       (loop
> 	  (let* ((nread (#_read file-fd buf bufsize)))
> 	    (cond ((zerop nread) (return))
> 		  ((minusp nread) (error "Read error: ~A." (ccl::%strerror 
> (ccl::%get-errno))))
> 		  (t (loop (let* ((nwritten (#_write socket-fd buf nread)))
> 			     (when (= nwritten nread) (return)) ;; try again 
> till we get it right

I think that it's possible for nwritten to be positive (but less than nread)
here.

It's certainly possible that the buffer size also affects things.  An
8Kb buffer might be optimal for sequentially reading from a file, but
may be too large to be efficiently/atomically written to a socket.

Maybe something like:

(let* ((bufsize 8192))
   (ccl::%stack-block ((buf bufsize)
                       (iovec (* 4 (ccl::record-length :iovec))))
     ;; Set up each element of the iovec array to point at 1/4 of the
     ;; buffer.
     (dotimes (i 4)
       (with-macptrs ((p (%inc-ptr iovec (* i (ccl::record-length :iovec)))))
         (setf (pref p :iovec.iov_base) (%inc-ptr buf (* i 2048))
               (pref p :iovec.iov_len) 2048)))
     ...
     (loop
       (let* ((nread (#_read file-fd buf bufsize)))
         ((cond ((= nread bufsize)
                 ;; Got a full buffer, write to socket via #_writev
                 (#_writev socket-fd iovec 4))
                ((> nread 0)
                 ;; Got less than a full buffer.  Either adjust
                 ;; the count of iovecs/the iov_len of the last one,
                 ;; or just do a single #_write as before.
                )
                ((= nread 0) (return))
                (t (error ...))))))))

> 			     )))))))))
>
> I got a speedup from using file descriptors:
>
> First, how apache performs on a 20 megabyte file on the loopback
> interface, with 10 repeated downloads.  I collected the reported MB/s
> speed from wget, and looked at the average.
>
> (avg 7.50 6.56 12.69 6.75 7.01 10.27 6.94 8.27 14.74 7.75) 8.84 MB/s
>
> my own code:
>
> (avg 5.45 10.78 5.73 4.90 1.44 2.82 6.05 8.00 5.88 10.22) 6.12 MB/s

This may be too small a sample to draw many conclusions from, but it
looks like a few very bad transfer rates (1.44 !) may have skewed
things significantly.  I'd expect this case to be a little slower
than the others, but it's hard to see why there'd be as much variance
as there seems to be.

>
> Next, how the code with file descriptors, suggested by Gary Byers, performs:
>
> (avg 6.06 7.23 8.68 7.31 6.92 9.92 7.86 5.95 11.03 7.19) 7.81 MB/s
> -- 
> chr
>
>