[Openmcl-devel] %ioblock-read-u16-encoded-char

[Takehiko Abe]

The test form in %ioblock-read-u16-encoded-char:

(< 1st-unit
   (ioblock-literal-char-code-limit ioblock))

returns T for UTF-16 surrogate values. So,
ioblock-decode-input-function will not be called when it should be.

(defun %ioblock-read-u16-encoded-char (ioblock)
  (declare (optimize (speed 3) (safety 0)))
  (let* ((ch (ioblock-untyi-char ioblock)))
    (if ch
      (prog1 ch
        (setf (ioblock-untyi-char ioblock) nil))
      (let* ((1st-unit (%ioblock-read-u16-code-unit ioblock)))
        (if (eq 1st-unit :eof)
          1st-unit
          (locally
              (declare (type (unsigned-byte 16) 1st-unit))
            (if (< 1st-unit
                   (the (mod #x110000) (ioblock-literal-char-code-limit
ioblock)))
              (code-char 1st-unit)
              (funcall (ioblock-decode-input-function ioblock)
                       1st-unit
                       #'%ioblock-read-u16-code-unit
                       ioblock))))))))

A possible fix is to remove the test form and always call
ioblock-decode-input-function. Although I am not sure if this is the
right fix, it works at least for now because
%ioblock-read-u16-encoded-char is used only by utf-16 and ucs-2
currently -- they both have ioblock-literal-char-code-limit set to
#x10000. (I am not sure if the test will ever be necessary.)

After the fix:

(with-open-file (out "home:utf-16.txt" :external-format :utf-16
                     :direction :output
                     :if-does-not-exist :create)
  (write-char #\U+2000B out))

(with-open-file (in "home:utf-16.txt" :external-format :utf-16)
  (read-char in))
--> #\U+2000B  ;; was nil

;; ucs-2
(with-open-file (in "home:utf-16.txt" :external-format :ucs-2)
  (read-char in))
--> #\Replacement_Character ;; as expected


regards,
T.