[Openmcl-devel] problems building interface databases on Debian multiarch

[Faheem Mitha]

Hi Gary,

Thank you for your comments. I've thought about this some more. I
might have a workaround to my issue. I'm sorry to trouble you further,
but I have more comments and questions at the foot of the message,
which are relevant, so please bear with me. Of course, someone else
could answer. :-)

On Mon, 6 Aug 2012, Gary Byers wrote:

> If you add "-v" to the CFLAGS in effect in your script, you should see
> output like:

> #include <...> search starts here:
> /usr/local/bin/../ffigen/include
> /usr/include
> /usr/local/include
> End of search list.

> with the default (somewhat redundant) -isystem options in effect; that's
> one way of trying to determine where ffigen is looking for system include
> files.

Yes, I see that, thanks.

> On a few Ubuntu 12.04 and 11.10 systems that I looked at (that apparently
> also offer 'multiarch support'), /usr/include/gnu exists (in some cases as
> a symbolic link to something like /usr/include/some-arch/gnu).  I don't
> know why it exists for me and doesn't exist for you.

There is a package called libc6-dev-amd64 which provides these symlinks. 
However, they are not on the system by default, so this is useless. This 
information is courtesy of Steve Langasek of Debian.

> Newer versions (>= 4.4 ?  I don't know) of gcc support a -imultiarch option
> that seems to take a relative directory name as an argument.  On a system
> running the x8664 version of Ubuntu 12.04,

I see it.

gcc version 4.7.1 (Debian 4.7.1-6)
COLLECT_GCC_OPTIONS='-m32' '-v' '-mtune=generic' '-march=i586'
  /usr/lib/gcc/i486-linux-gnu/4.7/cc1 -quiet -v -imultiarch i386-linux-gnu
file.c -quiet -dumpbase file.c -m32 -mtune=generic -march=i586 -auxbase
file -version -o /tmp/ccDagDSw.s

This is in the multiarch enabled gcc in Debian unstable (4.7). This
must be an internal (and new) option, it is not listed on the man
page..

> $ gcc -m32 -v file.c

> shows that this option is passed as "-imultiarch i386-linux-gnu".
> I'd guess that this is the mechanism that causes the directory
> /usr/include/i386-linux-gnu to be searched, but this system doesn't
> seem to have such a directory.  At first glance, this stuff doesn't
> give the impression of being finished.

I think that is just the three part triple that identifies the
arch. It doesn't have a direct correspondence to the directories,
which seem to vary in structure. For example, the openjdk one is
/usr/lib/jvm/java-6-openjdk-i386/include/jni.h.

The triple is arch - kernel - userland

See for example http://wiki.debian.org/Multiarch/Tuples.

> That's about the extent of my knowledge here (and is a ways past the
> extent of my interest.)  I'm a little skeptical of the fact that
> /usr/include/gnu (and possibly other subdirectories of /usr/include)
> is just missing for you, since that would mean that compilers that
> didn't support -imultiarch wouldn't have any way to find
> <gcc/stubs.h>.  If a compiler was (via sheer luck or other means)
> able to find that file, it'd find that it includes either a 32-bit
> or 64-bit-specific file (making the notion that this sort of thing
> needs to be squirreled away in some magic architecture-specific
> directory sort of suspect in the first place.)

I don't follow what you mean after "it'd find that it includes..."

I also am not completely clear why those files are missing, but my
guess is that they want files from different archs to be
simultaneously installable, hence the wacky name. I guess they could
put /usr/include/gnu in as a symlink, but that would mean blessing one
particular arch as *the* arch, and they don't want to do that. The
libc6-dev-amd64 is an optional curiousity.

> If you use the .cdb files that Matt or I generated (whenever one of us
> last generated them), you'll be able to access a certain set of
> constant/type/structure/function declarations and definitions.  If you
> generate your own set, you'll be able to access another set that's
> likely to be extraordinarily similar.  (This stuff - especially the
> stuff in the standard C library - changes sometimes, but much of it
> tends to do so very slowly.)

It is true that they would be the same, but Debian requires everything
to be compiled from source. While this can sometimes be an oneroous
burden, there are also good reasons for things, and many evils are
avoided in so doing. At least, in my view.

I think I have a workaround, but I don't know if it will break
something.  It looks as follows.

First, ffigen takes the full path to a header file. Then it searches
the relative paths of header files within that file. It would make my
life easier if it would just take the relative path to a file as
argument, and then search for that. Eg. it wants

ffigen /usr/include/i386-linux-gnu/gnu/stubs.h

but it would be nice if it could just accept

ffigen gnu/stubs.h

Hey presto! Problem solved! However, this is not the case. So, one can
just create a temporary/dummy file which contains

#include <gnu/stubs.h>

and let ffigen search that. What could go wrong here is if the
interface database expects the header file to be on the system during
usage. I don't see this is true, but I'm not sure. I did notice that
the ffi file reads in the path location of the top level header file
(in this case it would be the temporary file), and I don't see what it
does that for. For example, with my temporary file on unstable, using
a temporary file, the ffi file starts with

(macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) "__STDC_HOSTED__" "1")
(macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) "__OBJC__" "1")
(macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) "__GNUC__" "4")
(macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) "__GNUC_MINOR__" "0")
(macro ("/usr/local/src/ccl/ccl-1.8+svn15431/x86-headers/jni/C/jni.h" 1) "__GNUC_PATCHLEVEL__" "0")

on squeeze, without the multiarch thing, I get

(macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__STDC_HOSTED__" "1")
(macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__OBJC__" "1")
(macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC__" "4")
(macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC_MINOR__" "0")
(macro ("/usr/lib/jvm/java-6-openjdk/include/jni.h" 1) "__GNUC_PATCHLEVEL__" "0")

So, if the temporary path name being in there is not a problem, then I
guess this will work. Otherwise I may have to do this the hard way and
write a top level search function myself. Which I don't really want to
do.

On a side note, I guess I don't understand exactly how FFI works. If
you only have access to the header files, how is that enough to run
the code?  Don't you need to know where the libraries are, too? The
header files don't even have the complete function definitions in
them, most of the time, just the prototypes.

                                                           Regards, Faheem