[Openmcl-devel] peek-char advances the file-position
Pascal J. Bourguignon
pjb at informatimago.com
Tue Apr 3 21:40:50 PDT 2012
Using com.informatimago.common-lisp.lisp-text.source-text:source-read
(which calls com.informatimago.common-lisp.lisp-reader.reader::read-1
which implements the lisp reader algorithm using cl:read-char,
cl:peek-char an cl:unread-char and provided reader macros), with calls
to file-position and print added before and after peek-char:
(defun source-read (&optional input-stream
(eof-error-p t) (eof-value nil)
(recursive-p nil) (preserve-whitespace-p nil))
(let ((*read-suppress* nil) ; we do want the source!
(*readtable* *source-readtable*)
(*stream* input-stream)
(*file* (ignore-errors (pathname input-stream))))
(print `(before ,'peek-char ,(file-position input-stream)))
(unless preserve-whitespace-p
(print `(peek-char --> ,(peek-char t input-stream nil nil t))))
(let ((*start* (file-position input-stream)))
(print `(after ,'peek-char ,*start*))
;; (read input-stream eof-error-p eof-value recursive-p)
;; We want to allow all-dots tokens.
(com.informatimago.common-lisp.lisp-reader.reader::read-1
input-stream eof-error-p eof-value
recursive-p preserve-whitespace-p nil t))))
The pattern of I/O is that of recursive reads: each reader macro is
replaced by a reader macro that reads either with read-char, read-line
or recursive calls to source-read above, and that calls the following
read-string-between-file-positions with *start* and (file-position
stream). Sexps are therefore read twice for each level of parenthesis.
(defun read-string-between-file-positions (stream start end)
"
PRE: (eq 'character (stream-element-type stream))
and START and END are file positions of this STREAM.
RETURN: A string containing the characters read between the START
and END file positions in the STREAM.
POST: (= end (file-position stream))
"
(let ((buffer (make-array (- end start) :element-type 'character
:fill-pointer 0)))
(file-position stream start)
(loop
:while (< (file-position stream) end)
:do (vector-push (read-char stream) buffer)
;; We could use copy-seq to return a simple-string,
;; but it's not worth it.
:finally (unless (= (file-position stream) end)
(warn "While reading beetween file positions, ~
reached a different file position: ~A < ~A"
end (file-position stream)))
(return buffer))))
cl-user> (let ((source "/home/pjb/test-file-position-read-sequence.lisp"))
(with-open-file (stream source)
(loop
:for sexp = (source-read stream nil stream)
:until (eq sexp stream)
:collect sexp)))
gives:
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 0)
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\;)
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 1)
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 37)
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\()
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 38)
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 39)
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\i)
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 39)
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 50)
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\:)
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 50)
(com.informatimago.common-lisp.lisp-text.source-text::before peek-char 58)
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\()
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 61)
on Version 1.7-dev-r14788M-trunk (LinuxX8664)
while it gives:
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 0)
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\;)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 0)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 37)
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\()
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 37)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 38)
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\i)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 38)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 49)
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\:)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 49)
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::BEFORE PEEK-CHAR 58)
(PEEK-CHAR COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::--> #\()
(COM.INFORMATIMAGO.COMMON-LISP.LISP-TEXT.SOURCE-TEXT::AFTER PEEK-CHAR 60)
on the other implementations.
I can't understand why PEEK-CHAR changes the FILE-POSITION. And I
cannot reduce it to a small test case (small test cases seem to work
well).
Calling (trace cl:read-char cl:unread-char cl:peek-char
cl:file-position) doesn't seem too wise (ccl hangs after a few I/O). It
looks like (trace cl:read-char) is not effective too. However we reach
the first call to PEEK-CHAR:
unintern> (let ((source "/home/pjb/test-file-position-read-sequence.lisp"))
(with-open-file (stream source)
(loop
:for sexp = (source-read stream nil stream)
:until (eq sexp stream)
:collect sexp)))
0> Calling (COMMON-LISP:UNREAD-CHAR #\e #<STRING-INPUT-STREAM #x30201A19B69D>)
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\l #<STRING-INPUT-STREAM #x30201A19B69D>)
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\r #<STRING-INPUT-STREAM #x30201A19B69D>)
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (COMMON-LISP:UNREAD-CHAR #\) #<STRING-INPUT-STREAM #x30201A19B69D>)
<0 COMMON-LISP:UNREAD-CHAR returned NIL
0> Calling (unread-char #\) #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\f #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\u #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\c #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (unread-char #\) #<string-input-stream #x30201A112DBD>)
<0 unread-char returned nil
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 0
(com.informatimago.common-lisp.lisp-text.source-text::before "COMMON-LISP" peek-char 0)
0> Calling (peek-char t #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> nil nil t)
<0 peek-char returned #\;
(peek-char com.informatimago.common-lisp.lisp-text.source-text::--> #\;)
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 1
(com.informatimago.common-lisp.lisp-text.source-text::after peek-char 1)
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> nil nil t)
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 37
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD> 1)
<0 file-position returned 1
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 1
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 2
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 3
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\;
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 4
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 5
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\-
0> Calling (file-position #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 file-position returned 6
0> Calling (read-char #<basic-file-character-input-stream ("/home/pjb/test-file-position-read-sequence.lisp"/6 utf-8) #x30201A0FA8FD>)
<0 read-char returned #\*
[hung]
Any idea what may be wrong?
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
More information about the Openmcl-devel
mailing list