[Openmcl-devel] encoding/decoding clozure strings to/from UTF-8

[Ron Garret]

You’re working way too hard.  Character streams automatically do work for you.  All character streams in CCL have a parameter that sets their character encoding, and the default is UTF-8.  So unless you want a different encoding, you should just be able to output characters to a stream and the Right Thing will happen.  If it doesn’t, then either your default encoding has changed somehow, or you’ve found a bug in CCL.

On Dec 30, 2014, at 11:35 AM, Mark Klein <m_klein at MIT.EDU> wrote:

> Karsten, Matthew,
> 
> Thanks for the pointers, that makes sense. My next challenge is to change a bunch of web page generation methods so that they output UTF-8. I wrote a little macro that I think will do the job. Am I on the right track, can it be made substantially faster or reliable somehow? 
> 
> (defmacro c2utf8 (&rest body)
>  "Takes everything body sends to stream, and converts it to UTF-8"
>  `(let ((list (encode-string-to-octets (with-output-to-string (stream) , at body) :external-format :utf-8)))
>     (loop for c across list do (write-char (code-char c) stream))))
> 
> It might be better, for example, if I could somehow redefine the stream so the UTF-32 text that goes in comes out as UTF-8?
> 
>  Thanks!
> 
> 	Mark
> 
>> On Dec 30, 2014, at 2:13 PM, Karsten Poeck <karsten.poeck at gmail.com> wrote:
>> 
>> On 30.12.14 19:46, R. Matthew Emerson wrote:
>>> 
>>>> On Dec 30, 2014, at 1:19 PM, Mark Klein <m_klein at MIT.EDU> wrote:
>>>> 
>>>> 
>>>> I have Clozure 1.10 running on a Mac (OS 10.10.1). I'd like to be able to encode strings to and from UTF-8 (to handle multi-language text in a web application I am developing). Can someone give me some pointers on how I can do that?
>>> 
>>> http://ccl.clozure.com/docs/ccl.html#encoding-and-decoding-strings
>>> 
>>> Basically, use (encode-string-to-octets string :external-format :utf-8) to encode a lisp string into a vector of octets containing UTF-8-encoded data, and use (decode-string-from-octets vector :external-format :utf-8) to create a lisp string from a vector of octets containing UTF-8-encoded data.
>> 
>> Since Matthew already answered the original question, another example with 1,2 and 3 byte encoding in utf-8 from http://en.wikipedia.org/wiki/UTF-8
>> 
>> (map 'vector #'(lambda(byte)(write-to-string byte :base 16)) (encode-string-to-octets "$¢€" :external-format :utf-8 :use-byte-order-mark nil))
>> 
>> #("24" "C2" "A2" "E2" "82" "AC")
>> 
>> To read an utf-8 encoded file use
>> (defun read-utf-8-file (file)
>> (with-open-file (stream file :direction :input :external-format :utf-8)
>> .......)
>> 
>> regards
>> 
>> Karsten
>> 
>> 
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> https://lists.clozure.com/mailman/listinfo/openmcl-devel
> 
> -------------------------------
> Mark Klein
> http://cci.mit.edu/klein
> 
> Principal Research Scientist
> Center for Collective Intelligence
> Massachusetts Institute of Technology
> 
> Visiting Researcher
> Dynamic and Distributed Information Systems Group
> University of Zurich
> 
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> https://lists.clozure.com/mailman/listinfo/openmcl-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20141230/e24140b7/attachment.bin>