[Openmcl-devel] Transfer the contents of a file to a tcp stream

[Christian Nybø]

On May 3, 2005, at 23:22, Gary Byers wrote:

> Hmm.  I was just about to recommend cheating (using sendfile) if it's
> available (it's available on Linux, but doesn't seem to be there on
> Darwin/OSX).

 From googling sendfile mac os x, I got the impression that sendfile (or 
transmitfile as it is called in NT)  is not yet present in mac os x.

I downloaded apache and had a look at writev_it_all in server/core.c, 
but I'm unfamiliar with reading production code in C, so I don't know 
whether apache does anything particular when sendfile is absent.

> You can cheat a bit
> by doing something like:
>
> (defun copy-file-to-socket (file-stream socket-stream)
>   (let* ((file-fd (ccl::stream-device file-stream :input))
>          (socket-fd (ccl::stream-devices socket-stream :output))
> 	 (bufsize 8192)) ; arbitrary
>     (force-output socket-stream) ; flush any buffered output
>     (%stack-block ((buffer bufsize))
>     ;; You could just say (#_lseek ...) here, but Linux's
>     ;; #_lseek may have difficulty with large file offsets.
>     (ccl::fd-lseek file-fd 0 #$SEEK_SET)
>     (loop
>       (let* ((nread (#_read file-fd buf bufsize)))
>         (cond ((zerop nread) (return))
>               ((minusp nread)(error ...))
>               (t (let* ((nwritten (#_write socket-fd buf nread)))
>                    (when (< nwritten nread)
>                           ;;; Handle partial writes, errors
>                          )))))))))
>
> That's an artist's conception and may be a little buggy, but using
> a single buffer and bypassing most of the buffered stream overhead
> might get things -closer- to Apache's performance.

nwritten was set to -1 quite often, so I added a loop that would try 
until #_write returned the right number of bytes.  That's probably not 
the right way, is it?

(defun copy-file-to-socket (file-stream socket-stream)
  (declare (optimize (speed 3) (safety 0)))
   (let* ((file-fd (ccl::stream-device file-stream :input))
          (socket-fd (ccl::stream-device socket-stream :output))
	 (bufsize 2048))		; arbitrary
     (force-output socket-stream)	; flush any buffered output
     (ccl::%stack-block ((buf bufsize));; %stack-block is not visible 
without the package prefix (chr)
       ;; You could just say (#_lseek ...) here, but Linux's
       ;; #_lseek may have difficulty with large file offsets.
       (ccl::fd-lseek file-fd 0 #$SEEK_SET)
        (loop
	  (let* ((nread (#_read file-fd buf bufsize)))
	    (cond ((zerop nread) (return))
		  ((minusp nread) (error "Read error: ~A." (ccl::%strerror 
(ccl::%get-errno))))
		  (t (loop (let* ((nwritten (#_write socket-fd buf nread)))
			     (when (= nwritten nread) (return)) ;; try again till we get it 
right
			     )))))))))

I got a speedup from using file descriptors:

First, how apache performs on a 20 megabyte file on the loopback
interface, with 10 repeated downloads.  I collected the reported MB/s
speed from wget, and looked at the average.

(avg 7.50 6.56 12.69 6.75 7.01 10.27 6.94 8.27 14.74 7.75) 8.84 MB/s

my own code:

(avg 5.45 10.78 5.73 4.90 1.44 2.82 6.05 8.00 5.88 10.22) 6.12 MB/s

Next, how the code with file descriptors, suggested by Gary Byers, 
performs:

(avg 6.06 7.23 8.68 7.31 6.92 9.92 7.86 5.95 11.03 7.19) 7.81 MB/s
-- 
chr