[Openmcl-devel] M1 port: Call for funding

[Andrew Shalit]

Thank you, Gilbert, for getting this conversation, and thanks to all who have contributed ideas and offered to contribute funding. I don’t have the skills or the time to work on the port, but I’ve witnessed several ports as well as bootstrapping.

Gilbert — it is great that you have a brilliant, motivated, well-qualified hacker available to work on the port.  I hope this person’s qualifications include comfort with low-level debugging, i.e. using GDB or similar tools.  CCL is implemented relatively close to the metal; it touches things at a much lower level than SBCL, for example. That is what allows it to have native threads, and other important features. It also makes it more fragile. And it is also a different kind of demand on someone porting it, or bootstrapping a new version. That person needs to be able to look at a block of hex and recognize it as a stack frame; or see that a value is a tagged or untagged value; and generally just navigate the interior of CCL in a debugger that wasn’t designed for Lisp. I liken it to be a Linux kernel hacker. So hopefully your person is game for that. The two people in this discussion who I know are comfortable at that level (if comfort is the right word) are Matthew and Gary Palter.

The following is probable wrong in pieces, but hopefully still helpful.

CCL consists of a runtime (written in C and assembler) and an image (consisting of Lisp code and data).  When something about the runtime architecture changes, the system needs to be bootstrapped again.  That means creating a runtime and an image that both use the same new architecture. There is no step-by-step recipe for doing this. There are some tools, I believe. In outline, you need to do the equivalent of cross-compiling, but it is cross-image creation.  You do that from the old architecture, creating an image that can then be saved to disk, and used to with the new runtime.

Of course, when try do that the first time, you start up the new runtime and new image and immediately crash into GDB.  You poke around, see what is broken, hopefully figure out how to fix it, and iterate.  It’s ball of mud followed by ball of mud until you have your new Lisp. In my memory, it seems like it often took GB a couple of days to bootstrap a new version of CCL. This isn’t a port to a new CPU architecture, but something like changing the tag bits in some kind of vector or something. A relatively simple, but architectural, change.

Doing a port to a new CPU is the same process, but in spades.

I’d suggest a first valuable step would be for people to learn how to bootstrap a new version of CCL. Doing that will help you learn your way around parts of the system.

Anyway, my two cents.  Hopefully helpful.  I welcome corrections from those who know better.

> On Dec 28, 2023, at 12:18 PM, Gilbert Baumann <gilbert at bauhh.de> wrote:
> 
> Hello,
> 
> I have a brilliant hacker at hand who is motivated, well-qualified, and
> available to do the M1 port of CCL. She is an expert with garbage collectors
> and well-versed in runtime and compiler design. I am personally happy to
> contribute both time and a considerable chunk of the needed funding.
> 
> However, I ask for additional funds.
> 
> Some already have mentioned here that they would contribute. We can take
> rme's mentioned three month as an estimate of the needed work.
> 
> So please raise your hand if you're interested or send me a personal message
> by mail. Or catch me as gilberth on libera.chat, if you're on IRC. I'd like
> to coordinate this joined effort.
> 
> I hope that together we could pull this off and keep CCL alive and kicking
> on the Apple platform.
> 
> --
> Gilbert.
>