[Openmcl-devel] saving data to a fasl file

[Gary Byers]

On Tue, 23 Nov 2004, alex crain wrote:

> Does OpenMCL have a way to write data to a fasl file?
> I'm thinking along the lines of LispWorks DUMP-FORMS-TO-FILE
>

I think that people have written things like this for commercial
MCL in the past, but I don't remember the details.  There are a
few small differences in the structure of MCL/OpenMCL fasl files,
but they're pretty minor.

> Along those lines, are there any tools for taking apart fasl (dfsl)
> files?
> I'd just like to learn about what's in there.

The general idea (which might make reading the code in
"ccl:xdump;faslenv.lisp", "ccl:level-0;nfasload.lisp", and
"ccl:lib;nfcomp.lisp" easier to understand) is:

- a fasl file starts with a "file header", which contains version
  information and a count of the following "blocks".  There's
  typically only one "block" per fasl file, but there are ways
  to sort of store a bunch of logical files in one physical file,
  which might be easier to distribute.

- each block has a little header on the front of it, which basically
  just describes the size of the block data which follows.

- the block data is just a byte stream, which basically defines a
  simple bytecode program.  (The actual bytecodes - "fasl operators" -
  are defined in "ccl:xdump;faslenv.lisp").  The descriptions there
  are a little terse, but probably accurate.

- there are operators that are used to create a per-block "table",
  which is just a vector used to keep track of and simplify references
  to previously loaded objects.

  When a table's created (at a size specified by an operator that
  usually appears near the beginning of the block), an associated
  index is set to 0.  (This sort of allows the table to be treated
  as a stack.)

- There are currently about 50 fasl operators defined. the theoretical
  inclusive upper bound on a bytecode's value is 127.  Bytecodes can
  have their high bit set, which means "perform the operation denoted
  by the low 7 bits of the opcode, then push the result into the table."

  (One exception to the above is the opcode #xFF, which hash the
  symbolic name CCL::$FASLEND; it's used to mark the end of the block's
  bytestream.  I guess that means that 127 is really an exclusive bound.)

- The actual bytecodes are usually followd by byte-aligned data; how
  much and what kind of data is dependent on the bytecode.  The operands
  (if any) may be indices into the table, immediate, or some combination

The file name "nfasload.lisp" is so-called because this is the "new"
fasloader.  It was "new" in 1986 or so.  The format could probably
stand some modernization/overhaul; it would certainly need extensions
to deal with 64-bit architectures, etc.