[Openmcl-devel] peek-char advances the file-position

Pascal J. Bourguignon pjb at informatimago.com
Tue Apr 3 21:40:50 PDT 2012


Using com.informatimago.common-lisp.lisp-text.source-text:source-read
(which calls com.informatimago.common-lisp.lisp-reader.reader::read-1
which implements the lisp reader algorithm  using cl:read-char,
cl:peek-char an cl:unread-char and provided reader macros), with calls
to file-position and print added before and after peek-char:

(defun source-read (&optional input-stream
                    (eof-error-p t) (eof-value nil)
                    (recursive-p nil) (preserve-whitespace-p nil))
  (let ((*read-suppress* nil)           ; we do want the source! 
        (*readtable* *source-readtable*)
        (*stream* input-stream)
        (*file*  (ignore-errors (pathname input-stream))))
    (print `(before ,'peek-char  ,(file-position input-stream)))
    (unless preserve-whitespace-p 
      (print `(peek-char --> ,(peek-char t input-stream nil nil t))))
    (let ((*start* (file-position input-stream)))
      (print `(after ,'peek-char ,*start*))
      ;; (read input-stream eof-error-p eof-value recursive-p)
      ;; We want to allow all-dots tokens.
      (com.informatimago.common-lisp.lisp-reader.reader::read-1
       input-stream eof-error-p eof-value
       recursive-p preserve-whitespace-p nil t))))

The pattern of I/O is that of recursive reads: each reader macro is
replaced by a reader macro that reads either with read-char, read-line
or recursive calls to source-read above, and that calls the following
read-string-between-file-positions with *start* and (file-position
stream).  Sexps are therefore read twice for each level of parenthesis.

(defun read-string-between-file-positions (stream start end)
  "
PRE:    (eq 'character (stream-element-type stream)) 
        and START and END are file positions of this STREAM.
RETURN: A string containing the characters read between the START 
        and END file positions in the STREAM.
POST:   (= end (file-position stream))
"
  (let ((buffer (make-array (- end start) :element-type 'character
                            :fill-pointer 0)))
    (file-position stream start)
    (loop
       :while (< (file-position stream) end)
       :do (vector-push (read-char stream) buffer)
       ;; We could use copy-seq to return a simple-string,
       ;; but it's not worth it.
       :finally (unless (= (file-position stream) end)
                  (warn "While reading beetween file positions, ~
                         reached a different file position: ~A < ~A"
                        end (file-position stream)))
       (return buffer))))


cl-user> (let ((source "/home/pjb/test-file-position-read-sequence.lisp"))
           (with-open-file (stream source)
             (loop
               :for sexp = (source-read stream nil stream)
               :until (eq sexp stream)
               :collect sexp)))

gives:

(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 0) 
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\;) 
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 1) 
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 37) 
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\() 
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 38) 
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 39) 
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\i) 
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 39) 
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 50) 
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\:) 
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 50) 
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 58) 
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\() 
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 61) 

on Version 1.7-dev-r14788M-trunk  (LinuxX8664)

while it gives:

(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 0) 
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\;) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 0) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 37) 
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\() 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 37) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 38) 
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\i) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 38) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 49) 
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\:) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 49) 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 58) 
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\() 
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 60) 

on the other implementations.


I can't understand why PEEK-CHAR changes the FILE-POSITION.  And I
cannot reduce it to a small test case (small test cases seem to work
well).

Calling (trace cl:read-char cl:unread-char cl:peek-char
cl:file-position) doesn't seem too wise (ccl hangs after a few I/O).  It
looks like (trace cl:read-char) is not effective too.  However we reach
the first call to PEEK-CHAR:

unintern> (let ((source "/home/pjb/test-file-position-read-sequence.lisp"))
            (with-open-file (stream source)
              (loop
                :for sexp = (source-read stream nil stream)
                :until (eq sexp stream)
                :collect sexp)))
0> Calling (COMMON-LISP:UNREAD-CHAR #\e #<STRING-INPUT-STREAM  #x30201A19B69D>) 
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\l #<STRING-INPUT-STREAM  #x30201A19B69D>) 
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\r #<STRING-INPUT-STREAM  #x30201A19B69D>) 
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\) #<STRING-INPUT-STREAM  #x30201A19B69D>) 
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (unread-char #\) #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\f #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\u #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\c #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream  #x30201A112DBD>) 
<0 unread-char returned nil
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 0

(com.informatimago.common-lisp.lisp-text.source-text::before "COMMON-LISP" peek-char 0) 
0> Calling (peek-char t #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> nil nil t) 
<0 peek-char returned #\;

(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\;) 
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 1

(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 1) 
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> nil nil t) 
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 37
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> 1) 
<0 file-position returned 1
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 1
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 2
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 3
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 4
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\ 
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 5
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\-
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 file-position returned 6
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>) 
<0 read-char returned #\*

[hung]


Any idea what may be wrong?

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.




More information about the Openmcl-devel mailing list