[Openmcl-devel] ccl 1.11 release and GitHub

[Bill St. Clair]

I’m wondering if this is really a problem. We update CCL binaries only a
few times a year, to handle hard-to-bootstrap changes. That’s 50 megabytes
times the number of platforms on GitHub, but it’s only 50 megabytes on a
machine that runs only one platform. Is that a problem? I don’t think so.
Others?

On Mon, Feb 13, 2017 at 9:24 PM, mikel evins <mevins at me.com> wrote:

>
> > On Feb 13, 2017, at 7:57 PM, Bill St. Clair <wws at clozure.com> wrote:
> >
> > On Mon, Feb 13, 2017 at 7:26 PM, R. Matthew Emerson <rme at acm.org> wrote:
> >
> > In the Subversion world, sure, we could tag an old release, and that
> would include suitable bootstrapping binaries and interface databases.
> >
> > In the git world, a tag (as I understand it) is essentially a pointer to
> a particular commit.  It is still necessary to have suitable bootstrapping
> binaries that will work to compile the sources at that point in time.  As a
> concrete example of what I mean by this, note that a current development
> ccl (i.e., 1.12-dev) can't build a ccl from the 1.11 sources.
> >
> > ​This suggests a solution to me.
> >
> > CCL would have two GIT repositories:
> >
> > 1) Source code, tagged with releases, branched with patches for
> particular releases. Move its tag when you add a bug-fix patch to a release.
> >
> > 2) Binaries. Branched from the beginning with one branch per platform.
> Tagged to match the source tags, but with unique names for each branch:
> >
> > 1) Tags: 1.9, 1.10, 1.11
> >
> > 2) Branches: darwinx86, linuxarm, linuxppc, linuxx86, solarisx86,
> windows​
> >
> > <branch>: <tag>…
> > darwinx86: 1.9-dx86, 1.10-dx86, 1.11-dx86
> > linuxarm: 1.9-armcl, 1.10-armcl, 1.11-armcl
> > linuxppc: 1.9-pp, 1.10-pp, 1.11-pp
> > linuxx86: 1.9-lx86, 1.10-lx86, 1.11-lx86
> > solarisx86: 1.9-sx86, 1.10-sx86, 1.11-sx86
> > windows: 1.9-wx86, 1.10-wx86, 1.11-wx86
> >
> > This would allow you to put the two directories side-by-side, and, on
> the linux-like platforms, have symbolic links from the source directory
> reference the binaries in the binary directory:
> >
> > ccl/
> >   trunk/    # Source for master branch
> >     lx86cl -> ../trunk-bin/lx86cl
> >     lx86cl.image -> ../trunk/bin/lx86cl.image
> >     lx86cl64 -> ../trunk-bin/lx86cl64
> >     lx86cl64.image -> ../trunk-bin/lx86cl64.image
> >   trunk-bin/    # Binaries for master branch
> >     lx86cl
> >     lx86cl.image
> >     lx86cl64
> >     lx86cl64.image
> >
> > We could have a script to run in the source directory to create the
> symbolic links from <dir>/foo to ../<dir>-bin/foo
> >
> > One remaining issue is what to do with the header directories. We could
> either just have everybody always download all of them with the source, or
> put them with the binaries and use symbolic links from the source directory
> to get to them in the binary directory.
>
> The main problem with storing binaries in git repositories is that git's
> diff algorithm is designed for diffing text files. It doesn't get
> particularly good results in the general case on binary objects. The
> practical result is that, in effect, any time you change a binary object,
> no matter how small the change, git will store a whole new copy of the
> entire binary object.
>
> That means that repos containing binaries that change often will tend to
> grow in size very much faster than repos that contain only text.
>
> That isn't necessarily a big problem, as long as the stored binaries don't
> change very often. In fact, you should be able to do a little
> back-of-the-envelope arithmetic to determine how big a problem it would be
> to store CCL binaries in a git repo. Go over the release history for the
> past couple of years and count up the number of times each binary object
> has changed. For each binary object, multiply its size by the number of
> times it changed. Add all those together for a rough estimate of the total
> size of the repo.
>
> The common strategy for storing binaries for git repos is to store them in
> something other than git, and use a git extension that stores pointer files
> in the repo. The pointer files refer to the non-gt storage for the binary
> files, and enable git to fetch and store the appropriate binary objects
> during push and pull operations.
>
> Here's a recent discussion of the problem and the current crop of
> solutions:
>
>   https://github.com/openframeworks/openFrameworks/
> wiki/Moving-binaries-out-of-the-repo
>
> GitHub provides support for git-LFS.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20170213/7fe23d06/attachment.htm>