[Openmcl-devel] Unicode UTF-8 problems with UFFI on Clozure CL

[Sergey Makeev]

Hello Victor,

Maybe you need to set up uffi:*default-foreign-encoding* variable?

I've had the similar problems and solved it not very good, but for me it
works:

1) First of all I set ccl:*default-file-character-encoding*
and ccl:*default-socket-character-encoding* to :utf-8

2)And then I patched uffi soures in quicklisp directory

i18n.lisp:
;;i've changed function  foreign-encoded-octet-count because of it cl-mysql
returs shrinked strings in non latin-1 encodings:

(defun foreign-encoded-octet-count (str &key encoding)
  "Returns the octets required to represent the string when passed to a ~
foreign function."
  (declare (ignorable encoding))
  ;; AllegroCL 8-bit, CCL, and Lispworks give correct value without
converting
  ;; to external-format. AllegroCL 16-bit, SBCL, and CLISP requires
conversion
  ;; with external-format
  #+(or (and allegro ics) (and sbcl sb-unicode) (and clisp i18n))
  (length (string-to-octets str :encoding
                            (or encoding *default-foreign-encoding*
                                ccl:*default-file-character-encoding*
:utf-8)))
  #-(or (and allegro ics) (and sbcl sb-unicode) (and clisp i18n))
  (length str)
  #+(and openmcl openmcl-unicode-strings)
  (ccl:string-size-in-octets str :external-format (ccl:make-external-format
:character-encoding (or encoding *default-foreign-encoding*
ccl:*default-file-character-encoding* :utf-8)))

strings.lisp
;;i've changed macro with-cstring, not to use ccl:with-cstrs but use
ccl:with-encoded-cstrs :
(defmacro with-cstring ((cstring lisp-string) &body body)
  #+(or cmu sbcl scl lispworks)
  `(let ((,cstring ,lisp-string)) , at body)
  #+allegro
  (let ((acl-native (gensym))
        (stored-lisp-string (gensym)))
    `(let ((,stored-lisp-string ,lisp-string))
       (excl:with-native-string (,acl-native ,stored-lisp-string)
         (let ((,cstring (if ,stored-lisp-string ,acl-native 0)))
           , at body))))
  #+(or openmcl digitool)
  (let ((stored-lisp-string (gensym)))
    `(let ((,stored-lisp-string ,lisp-string))
       (if (stringp ,stored-lisp-string)
           (ccl:with-encoded-cstrs
              (or *default-foreign-encoding*
                  ccl:*default-file-character-encoding*
                  :utf-8) ((,cstring ,stored-lisp-string))
             , at body)
           (let ((,cstring +null-cstring-pointer+))
             , at body))))
  )

Additionally i've defined encoding for still widely used cp1251, but I
think it's offtopic.

Best regards, Sergey.




2013/10/16 Victor <bobbie at ua.fm>

> Hi!
>
> I've stuck into a problem with unicode strings handling in the
> UFFI. At some point UFFI corrupts UTF-8 encoded string (I guess that
> this is a problem in uffi:convert-to-foreign-string function):
>
> (uffi:convert-from-foreign-**string (uffi:convert-to-foreign-**string
> "abc"))
> "abc"
> ? (uffi:convert-from-foreign-**string (uffi:convert-to-foreign-**string
> "абв"))
> "^Z^Z^Z"
>
> Does anybody else have this problem? Are there any workarounds ready
> to be used or possible fixes?
>
> I am working on Ubuntu (but saw this problem reproduce on Windows as well).
> Clozure Common Lisp Version 1.9-r15756
>
> Thanks,
> Victor
> ______________________________**_________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/**listinfo/openmcl-devel<http://clozure.com/mailman/listinfo/openmcl-devel>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20131017/f9d33f4a/attachment.htm>