[Openmcl-devel] Slime's utf-8-unix

[Ben Hyde]

On Oct 27, 2006, at 12:20 AM, Gary Byers wrote:
> On Thu, 26 Oct 2006, Ben Hyde wrote:
>> Here's how I got openmcl's unicode support working with slime and
>> xemacs.
>>
>> In my lisp init I modified the how it should fire up openmcl.
>>
>> (setf slime-lisp-implementations
>>       '((openmcl
>> 	 ("/Users/bhyde/bin/openmcl" "--terminal-encoding" "utf-8")
>> 	 :coding-system utf-8-unix)
>> 	(sbcl
>> 	 ("sbcl")
>> 	 :coding-system utf-8-unix)))
>>
>> I also define a m-x command:
>>
>> (defun openmcl () (interactive)
>>   (let ((slime-default-lisp 'openmcl)) (slime)))
>>
>> Then I taught swank that how to deal with swank's utf-8-unix coding-
>> system by replacing accept-connection's implementation in swank-
>> openmcl.lisp:
>>
>
> Another approach might be to change the call to MAKE-SOCKET in the  
> version
> of SWANK:CREATE-SOCKET in swank-openmcl.lisp (adding :EXTERNAL- 
> FORMAT :UTF-8
> to the arglist.)  That'd save having to bind the special variables,  
> but
> since the rest of SLIME apparently thinks that it's reasonable to  
> specify
> :EXTERNAL-FORMAT at accept time it might be better to make  
> CCL:ACCEPT-CONNECTION
> accept an :EXTERNAL-FORMAT argument.
>
> It's not really clear that that'd be otherwise useful; I tend to  
> think that
> you'd want all streams created from the same listening socket to  
> use the same
> external-format (at least initially).  (If you thought  
> that :EXTERNAL-FORMAT
> should be specified at accept time, how about other arguments that  
> apply to
> the stream socket, like :KEEPALIVE ?)  Perhaps there's some  
> rationale for
> SWANK doing things this way, but it seems wrong to the extent that  
> I've thought
> about it.

In the HTTP world the character set is very volatile.  Coming from  
that world my first attempt was to slam the encoding just after the  
accept.  At this moment I'm actually a bit unclear on how to do that  
in a safe and reliable manner.

Those files, swank-*.lisp, are certainly lesson in how much variation  
there about how to handle this.  If you want to maximize the number  
of libraries that are rapidly brought into openmcl then following a  
market leader's design might be the best option.  At least then you  
can blame the tangle of design choices on them.

>> (defimplementation accept-connection (socket
>>                                       &key external-format buffering
>> timeout)
>>   (declare (ignore buffering timeout))
>>   (let ((ef (or external-format :iso-latin-1-unix)))
>>     (cond
>>       ((eq external-format :utf-8-unix)
>>        (let ((ccl:*default-socket-character-encoding* :utf-8)
>>              (ccl:*default-line-termination* :unix))
>>          (ccl:accept-connection socket :wait t)))
>>       (t
>>        (assert (eq ef :iso-latin-1-unix))
>>        (ccl:accept-connection socket :wait t)))))
>
> One (very dumb, very simple) way of testing that the connection is  
> using utf-8
> is to try something like:
>
> whatever-the-lisp-prompt-is> (format t "~a~%" #\Plus-Minus_Sign)
>
> If you see a glyph comprised of plus sign over a minus sign, you're  
> winning (most
> fonts - on OSX at least - seem to contain a glyph for that character.)

On OSX, if you enable the menu bar widget for "input menu" in the  
international control panel then you get easy access to the character  
pallet; where you can find all manner of amusing unicode character  
codes quickly and easily.  And, if the gods are smiling on you, you  
can cut and paste to test round trip unicode.

Another technique is to fetch a page or feed from the country of your  
choice.

> If you see something else (and you're sure that the terminal/Emacs  
> is using utf-8),
> then it's likely that the lisp is not using utf-8 ...
>
>> I haven't tested this much, and presumably somebody who actually is
>> fluent in swank and openmcl external coding would have done something
>> different.
>
> It'd probably be good - once some dust settles - to make utf-8 the  
> default;
> getting Slime to support it would settle a lot of that dust.

Absolutely!  Unicode support in the emacs world is just as tangled as  
it is in the lisp world.  Dust would come out of both sides.

  - ben