[Openmcl-devel] problems building interface databases on Debian multiarch

[Gary Byers]

On Tue, 7 Aug 2012, Faheem Mitha wrote:

>
> Hi Gary,
>
> Thank you for your comments. I've thought about this some more. I
> might have a workaround to my issue. I'm sorry to trouble you further,
> but I have more comments and questions at the foot of the message,
> which are relevant, so please bear with me. Of course, someone else
> could answer. :-)
>
> On Mon, 6 Aug 2012, Gary Byers wrote:
>
>> If you add "-v" to the CFLAGS in effect in your script, you should see
>> output like:
>
>> #include <...> search starts here:
>> /usr/local/bin/../ffigen/include
>> /usr/include
>> /usr/local/include
>> End of search list.
>
>> with the default (somewhat redundant) -isystem options in effect; that's
>> one way of trying to determine where ffigen is looking for system include
>> files.
>
> Yes, I see that, thanks.
>
>> On a few Ubuntu 12.04 and 11.10 systems that I looked at (that apparently
>> also offer 'multiarch support'), /usr/include/gnu exists (in some cases as
>> a symbolic link to something like /usr/include/some-arch/gnu).  I don't
>> know why it exists for me and doesn't exist for you.
>
> There is a package called libc6-dev-amd64 which provides these symlinks. 
> However, they are not on the system by default, so this is useless. This 
> information is courtesy of Steve Langasek of Debian.

The distributed "populate.sh" script tries to parse headers that're distributed
as part of packages that aren't installed by default (libpcap-dev, libpci-dev,
a few others.)

If it was up to me (and I'm really glad that it isn't, because this is such
a time sink ...), I think that I'd just say that there's a build-time dependency
on those packages (and on anything that'll install /usr/include/gnu).


>
>> Newer versions (>= 4.4 ?  I don't know) of gcc support a -imultiarch option
>> that seems to take a relative directory name as an argument.  On a system
>> running the x8664 version of Ubuntu 12.04,
>
> I see it.
>
> gcc version 4.7.1 (Debian 4.7.1-6)
> COLLECT_GCC_OPTIONS='-m32' '-v' '-mtune=generic' '-march=i586'
> /usr/lib/gcc/i486-linux-gnu/4.7/cc1 -quiet -v -imultiarch i386-linux-gnu
> file.c -quiet -dumpbase file.c -m32 -mtune=generic -march=i586 -auxbase
> file -version -o /tmp/ccDagDSw.s
>
> This is in the multiarch enabled gcc in Debian unstable (4.7). This
> must be an internal (and new) option, it is not listed on the man
> page..
>
>> $ gcc -m32 -v file.c
>
>> shows that this option is passed as "-imultiarch i386-linux-gnu".
>> I'd guess that this is the mechanism that causes the directory
>> /usr/include/i386-linux-gnu to be searched, but this system doesn't
>> seem to have such a directory.  At first glance, this stuff doesn't
>> give the impression of being finished.
>
> I think that is just the three part triple that identifies the
> arch. It doesn't have a direct correspondence to the directories,
> which seem to vary in structure. For example, the openjdk one is
> /usr/lib/jvm/java-6-openjdk-i386/include/jni.h.
>
> The triple is arch - kernel - userland
>
> See for example http://wiki.debian.org/Multiarch/Tuples.
>
>> That's about the extent of my knowledge here (and is a ways past the
>> extent of my interest.)  I'm a little skeptical of the fact that
>> /usr/include/gnu (and possibly other subdirectories of /usr/include)
>> is just missing for you, since that would mean that compilers that
>> didn't support -imultiarch wouldn't have any way to find
>> <gcc/stubs.h>.  If a compiler was (via sheer luck or other means)
>> able to find that file, it'd find that it includes either a 32-bit
>> or 64-bit-specific file (making the notion that this sort of thing
>> needs to be squirreled away in some magic architecture-specific
>> directory sort of suspect in the first place.)
>
> I don't follow what you mean after "it'd find that it includes..."

AFAIK, /wherever/it/is/gnu/stubs.h is basically

#if targeting 32-bit
#include <gnu-stubs-32.h>
#else
#include <gnus-stubs-64.h>
#endif

which makes the process of discovering where it is somewhat anticlimactic.


>
> I also am not completely clear why those files are missing, but my
> guess is that they want files from different archs to be
> simultaneously installable, hence the wacky name. I guess they could
> put /usr/include/gnu in as a symlink, but that would mean blessing one
> particular arch as *the* arch, and they don't want to do that. The
> libc6-dev-amd64 is an optional curiousity.

And the version of Debian you're using won't allow you to compile C
code with a compiler that doesn't know how to find magic directories ?
If that's true, it seems like a poor decision.

I'm really skeptical of that.

>
>> If you use the .cdb files that Matt or I generated (whenever one of us
>> last generated them), you'll be able to access a certain set of
>> constant/type/structure/function declarations and definitions.  If you
>> generate your own set, you'll be able to access another set that's
>> likely to be extraordinarily similar.  (This stuff - especially the
>> stuff in the standard C library - changes sometimes, but much of it
>> tends to do so very slowly.)
>
> It is true that they would be the same, but Debian requires everything
> to be compiled from source. While this can sometimes be an oneroous
> burden, there are also good reasons for things, and many evils are
> avoided in so doing. At least, in my view.
>
> I think I have a workaround, but I don't know if it will break
> something.  It looks as follows.
>
> First, ffigen takes the full path to a header file. Then it searches
> the relative paths of header files within that file. It would make my
> life easier if it would just take the relative path to a file as
> argument, and then search for that. Eg. it wants

ffigen is basically just gcc, patched to write some declarations that
it's read from .h files in lispy syntax to .ffi files.

The directories that gcc (and therefore ffigen) searches in for
include files (and the order in which those directories are searched)
are controlled by command-line options.  This search isn't relative
to the including file (at least in the case of #include <file.h>; I
don't remember if #include "file.h" takes that into consideration.)

>
> ffigen /usr/include/i386-linux-gnu/gnu/stubs.h
>
> but it would be nice if it could just accept
>
> ffigen gnu/stubs.h

Nice for who ?  Why ?

>
> Hey presto! Problem solved! However, this is not the case. So, one can
> just create a temporary/dummy file which contains
>
> #include <gnu/stubs.h>

Many of the other .h files that you want to process will also try to
include <gnu/stubs.h> (directly or transitively); it isn't strictly
necessary to process this file directly (since the information that
it contains will have been processed when those other files are processed.)
If that somehow works already (I have no idea of how it would), then
just stop processing /usr/include/gnu/stubs.h directly.  If that doesn't
work, then your idea doesn't address the problem.


>
> and let ffigen search that. What could go wrong here is if the
> interface database expects the header file to be on the system during
> usage. I don't see this is true, but I'm not sure. I did notice that
> the ffi file reads in the path location of the top level header file
> (in this case it would be the temporary file), and I don't see what it
> does that for. For example, with my temporary file on unstable, using
> a temporary file, the ffi file starts with
>
> (macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) 
> "__STDC_HOSTED__" "1")
> (macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) 
> "__OBJC__" "1")
> (macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) 
> "__GNUC__" "4")
> (macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) 
> "__GNUC_MINOR__" "0")
> (macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) 
> "__GNUC_PATCHLEVEL__" "0")
>
> on squeeze, without the multiarch thing, I get
>
> (macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__STDC_HOSTED__" "1")
> (macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__OBJC__" "1")
> (macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC__" "4")
> (macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC_MINOR__" "0")
> (macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC_PATCHLEVEL__" 
> "0")
>

Nothing really cares about the filename information that gets written
to .ffi files.  There's been occasional interest in retaining the info
in .cdb files so that things like M-. could find definitions, but this
doesn't currently happen.


> So, if the temporary path name being in there is not a problem, then I
> guess this will work. Otherwise I may have to do this the hard way and
> write a top level search function myself. Which I don't really want to
> do.
>
> On a side note, I guess I don't understand exactly how FFI works. If
> you only have access to the header files, how is that enough to run
> the code?  Don't you need to know where the libraries are, too? The
> header files don't even have the complete function definitions in
> them, most of the time, just the prototypes.

That's true of programming in C in general.  Sometimes, the man page
for a function will note what library defines the function, sometimes
it won't, and it's generally much harder than it ideally should be to
find this information.  None of that has anything to do with Lisp or
with CCL or its FFI.

>
>                                                          Regards, Faheem
>
>