[Openmcl-devel] M1 port: Call for funding

[R. Matthew Emerson]

> On Dec 29, 2023, at 10:29 AM, Ron Garret <ron at flownet.com> wrote:
> 
> So... are you actually available now?  On what terms?  (i.e. how many hours a week?  Do you want/need to get paid?  How much?)

I think it’d be most productive to work full-time on the job as long as possible.

I would need to get paid. I’d rather talk about the sordid details in private, but I need to keep making about the same as I do now (which isn’t an outrageous sum), no matter how much I like hacking CCL.

> AFAICT you are the single most valuable resource we have in this community, as you are the only active member who has actually done a CCL port.  So as long as you're offering to help, maybe you can provide some guidance with regards to the most immediate technical problem we are facing: where do we start?  Tim McNerny and his intern Ayla Kurdak have made some progress investigating the M1 architecture and building an emulator for it, but it's still unclear (at least to me) how (or even if) that fits into an overall strategy for doing the port.
> 
> Let me make a concrete proposal: would you be willing to write up a "guide to porting CCL"?  It would consist of a description of the overall CCL architecture, how the various pieces fit together (kernel, GC, compiler, runtime, vinsns, etc.) and the design decisions that need to be made when doing a port, as well as a guide to what to do first, what to do next, etc.  Such a document would be useful not just for this port, but the next one (because if history is any guide there will be a next one, and one after that, and one after that).  If so, how long do you think it would take?  How much would you want to get paid to do it?  (BTW, if it would help, I would be more than happy to do the actual writing if you want to just do a brain-dump.  Whatever works.  We just need to get this information out of your head and onto paper somehow.  Oral tradition is not a recipe for longevity.)

This kind of thing also takes substantial time, and indeed, would probably require actually working on a port in order to do the job at all well. The process of onboarding and mentoring also requires time. There’s nothing wrong with that, but it is an investment.

Ideally, people working on an ARM64 port would be using CCL already, and be familiar with the existing ports, and maybe have already tackled a bug or two. Doing a port to a new architecture is a hard way to get started hacking on CCL, but it’s by and large what I did.

> 
> Failing that, maybe you would be willing to answer a few direct questions:
> 
> 1.  What would you recommend as the best place to start?  The kernel?  The compiler?  Some other piece of the runtime?  (Where did you start when you did the x86 port?  Was that the right thing in retrospect?)

The rough milestones I posted elsewhere in the thread sum up my views on that.

> 
> 2.  What is the best version to use as a basis for doing a port today?  The x8664 port is the most accessible, but the PPC is (I am given to understand) the most similar to the M1 architecturally.

I’d say it makes a lot of sense to look at how the PPC64 does things.

> 
> 3.  Do you think we could get any leverage out of the M1 port of SBCL?  Maybe use it for bootstrapping?

Maybe there would be some things in the assembler and disassembler that would be worth consulting, since CCL’s assembler and disassembler are going to face similar architecture-specific issues (how certain immediate operands are decoded and encoded, etc.).

Otherwise, I wouldn’t expect that an ARM64 CCL would be able to borrow much from SBCL at this level.

> 
> 4.  You have said in the past that getting hash tables working is a major milestone in producing a port.  Is that because CCL hash tables are particularly difficult to port, or because they exercise many different parts of the system?  I'm guessing the latter because the code in lib/hash.lisp just doesn't look that complicated.

The latter. A lot has to work in order to make a hash table, and many hours of single-stepping in the debugger are needed to sort out the trouble spots.


> 
> 5.  Are hash tables a pre-requisite for getting a REPL running (because you need them to build symbol tables)?  If so, would it be feasible to get a REPL running first by building a temporary symbol table implementation?  (That would be a lot easier than a general CL hash table.)

No, that wouldn’t be helpful.


> 
> 6.  One of the things that makes CCL what it is is the ObjC bridge and the IDE.  Would it be reasonable to consider porting those to SBCL?  Or are they intimately bound to the CCL architecture?

This is a good question. However, I believe that at this level, when interfacing with Objective-C and Mac UI frameworks, it is advantageous to be able to change the lisp implementation itself if needed to better support this integration. I’m thinking in particular of the need to deal with thread & UI issues, not to mention the pervasive use now of C blocks in newer Apple frameworks.

It could be that building a good Mac UI on SBCL is an approach that might pay off. I’m not in touch with SBCL developers or users, so I don’t know if Macintosh integration is an area of interest for them.

> 
> 7.  What is your reading of the tea leaves with regards to the future of ObjC on the Mac platform?  Is it even worth bothering trying to port the ObjC bridge, or should we be thinking about a Swift bridge instead?

It seems to me that the Objective-C runtime is going to be around for some time yet. Maybe at some point it will go away, and I can only hope that Apple will provide some other way for non-Apple languages to access UI libraries. Swift doesn’t have a stable ABI as far as I know, so it’s not possible yet to talk about calling Swift directly from CCL (without going through C somehow).



> 
> Thanks,
> rg
> 
> 
>> On Dec 29, 2023, at 12:23 AM, R. Matthew Emerson <rme at acm.org> wrote:
>> 
>> 
>> 
>>> On Dec 28, 2023, at 9:18 AM, Gilbert Baumann <gilbert at bauhh.de> wrote:
>>> 
>>> Hello,
>>> 
>>> I have a brilliant hacker at hand who is motivated, well-qualified, and
>>> available to do the M1 port of CCL. She is an expert with garbage collectors
>>> and well-versed in runtime and compiler design. I am personally happy to
>>> contribute both time and a considerable chunk of the needed funding.
>>> 
>>> However, I ask for additional funds.
>>> 
>>> Some already have mentioned here that they would contribute. We can take
>>> rme's mentioned three month as an estimate of the needed work.
>>> 
>>> So please raise your hand if you're interested or send me a personal message
>>> by mail. Or catch me as gilberth on libera.chat, if you're on IRC. I'd like
>>> to coordinate this joined effort.
>>> 
>>> I hope that together we could pull this off and keep CCL alive and kicking
>>> on the Apple platform.
>> 
>> I’d like to help if I can.
>> 
>> I’ve been unable to do it to date (because I’ve had to have a day job instead of work on CCL), but I would really like to be able to help find, develop, and mentor new CCL developers.
>> 
>