[Openmcl-devel] Solaris x86-64 and %gs

[Gary Byers]

I installed a recent OpenSolaris build over the weekend, so I might
be able to answer this sort of thing by peeking at Solaris man pages/
headers/small code snippets ...

On Mon, 7 Jan 2008, Chris Curtis wrote:

> (Replying to myself with some additional info...)
>
> From looking at lisp-kernel/thread-manager.c, it looks like
> setup_tcr_extra_segment doesn't ever get called unless both HAVE_TLS
> and X8664 are defined (in new_tcr), which may or may not be the case
> for Solaris. So on the one hand, it looks like it *should* be possible
> to run without HAVE_TLS and just get the tcr chain calloc'd. Is this
> code path actually valid on any current platforms?

The tcr gets calloc'ed on Darwin.

There may be some additional Darwin-specific nonsense having to do
with the fact that the TCR's address is used as a Mach exeption
port identifier, which means/meant that it has to fit in 32 bits.)

There's a separate issue related to how we get %gs to point to the
tcr in each thread; we may have to allocate a segment descriptor
(if there's a way to do that) and make %gs point to it, or we
may be able to find something like the Linux/FreeBSD functions
that do that at a higher level.  The message that Matt cited
made it sound like someone involved in Solaris development
wanted to provide some simple way of doing this, but as Matt
noted it wasn't clear if/how this was to be exposed to the
user.

Since it's a little simpler to let the toolchain/runtime
deal with thread-local-storage than it is to call the
pthread library functions, it seems simplest to define HAVE_TLS.
(A few months ago when trying to track down a bug which seemed
to cause tls problems on Linux, we built a Linux version with
HAVE_TLS disabled.  As I recall, that meant that we had to
make a few slight changes to ensure that the %gs register
was set even if HAVE_TLS was unset: they really are separate
issues, but it's (incidentally) the case that "there's a reasonable
way to get %gs to point to the tcr" whenever HAVE_TLS is true.
As it turned out, the bug caused similar problems whether the
runtime was managing TLS or we were ...)


>
> OTOH, all the asm macros refer to %rcontext (= %gs) so that doesn't
> seem quite likely.

In lisp, most code should refer to it as :rcontext, and references to
:rcontext should look up the actual register in the target backend
structure.  (This was intended to allow us to use %fs for the TCR
on Darwin, which uses %gs for threads data ... there didn't seem
to be a way of doing that on x86-64, so we wound up overloading
%gs instead.)  It seems that Solaris follows the standard ABI, so
we just need to figure out how.

If we did have to make rcontext be something other than %gs (e.g.,
%fs), then on the kernel side we'd have to define it conditionally
in a couple of places ...

BTW, another Solaris issue (it'd be good to get to the point where
this is the big issue) has to do with GCC versions: some versions
of GCC 3.x seem to have problems compiling the kernel source (especially
the GC) correctly.  Disabling -O2 sometimes makes it better; I don't
know whether the issue has to do with some code in the lisp/gc sources
depending on non-portable C behavior in some subtle way or whether
it's an outright bug in GCC 3.x.

Recent OpenSolaris builds seem to ship with a fairly old GCC; there's
a newer one at blastwave.org, but it seems to be 32-bit only.

>
> Deep knowlege helpful here. ;-)
>
> --chris
>
>
> On Jan 7, 2008, at 12:18 PM, Chris Curtis wrote:
>
>> Well, after a fair bit of digging I've made a little bit of headway on
>> this issue. It feels like 2 steps forward, (- 2 epsilon) steps back.
>>
>> The good news is that the post you found here seems to mean we
>> probably don't have quite the same problem as on Darwin.
>> DARWIN_GS_HACK swaps the pthread %gs data with the CCL tcr, and on
>> Solaris x86-64 libpthread uses %fs instead as per the ABI (supposedly
>> leaving %gs alone).
>>
>> Interestingly, I can do "mov %0,%%gs" with inline assembly, but only
>> as long as the value I'm trying to set is [0-3]. Any other value
>> segfaults. (Same with %fs, BTW.)
>>
>> FWIW, the sbcl runtime (on Solaris x86) sets %fs directly, but only
>> after setting up a new LDT block via SI86DSCR. It then saves the LDT
>> selector in its pthread_specific block.
>>
>> I'm continuing to dig... any thoughts or suggestions would be greatly
>> appreciated. :-)
>>
>> --chris
>>
>>
>> On Jan 3, 2008, at 3:07 PM, R. Matthew Emerson wrote:
>>
>>> I stumbled across this:
>>> http://blogs.sun.com/tpm/entry/solaris_10_on_x64_processors3
>>>
>>> [begin excerpt]
>>>
>>> Threads and Selectors
>>>
>>> In previous releases of Solaris, the 32-bit threads library used the
>>> %gs selector to allow each LWP in a process to refer to a private
>>> LDT entry to provide the per-thread state manipulated by the
>>> internals of the thread library.  Each LWP gets a different %gs
>>> value that selects a different LDT entry; each LDT entry is
>>> initialized to point at per-thread state.  On LWP context switch,
>>> the kernel loads the per-process LDT register to virtualize all this
>>> data to the process.  Workable, yes, but the obvious inefficiency
>>> here was requiring every process to have at least one extra locked-
>>> down page to contain a minimal LDT.  More serious, was the implied
>>> upper bound of 8192 LWPs per process (derived from the hardware
>>> limit on LDT entries).
>>>
>>> For the amd64 port, following the draft ABI document, we needed to
>>> use the %fs selector for the analogous purpose in 64-bit processes
>>> too.  On the 64-bit kernel, we wanted to use the FSBASE and GSBASE
>>> MSRs to virtualize the addresses that a specific magic %fs and magic
>>> %gs select, and we obviously wanted to use a similar technique on 32-
>>> bit applications, and on the 32-bit kernel too.  We did this by
>>> defining specific %fs and %gs values that point into the GDT, and
>>> arranged that context switches update the corresponding underlying
>>> base address from predefined lwp-private values - either explicitly
>>> by rewriting the relevant GDT entries on the 32-bit kernel, or
>>> implicitly via theFSBASE and GSBASE MSRs on the 64-bit kernel.  The
>>> result of all this work makes the code simpler, it scales cleanly,
>>> and the resulting upper bound on the number of LWPs is derived only
>>> from available memory (modulo resource controls, obviously).
>>>
>>> [end excerpt]
>>>
>>> So it sounds like the functionality is there, it's just a question
>>> of whether/how it's exposed to user processes.
>>>
>>
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> http://clozure.com/mailman/listinfo/openmcl-devel
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>