Distributed Compiling?

Stijn Buys writes in with this interesting question: "We are a bunch of students conected to the campus network and now we are looking for a way to distribute applications like compiler jobs to several Linux boxes. Each PC should retain it's autonomy, since these are our personal machines, but we'd really like to make them work together. I heard about things like distributed make but I haven't found something about it yet. Are there people who have experience with these kind of things and/or can give us some suggestions?" With just a quick search, I found this patch to GNU make and something called PGMAKE (dated 1994). What other options for distributed compiling are there?

duh...

[Zurk]

MOSIX (search on freshmeat..its GPLised) can allow you to migrate processes....PVM and MPI libraries can allow you to use multiple processors (altho you need to write code that calls PVM & MPI). Linux/HA might have some(?) code that asllows you to migrate stuff but thats all i know of...of course you could do a round robin rsh style make.....

pmake

[objection]

Check out pmake in your favourite distribution. If it has been set up you should be able to just replace
> make
with
>pmake

Seems this is the same as what we were using in '92-94 to compile the Sather compiler. It would distribute the compiles to idle workstations on the network and gather up the object files for linking. All completely transparent to the user - except for the significant speedup.

What kills it is screensavers that generate fractals and the like. The machine is never so busy as when it is idle!

Parallel Make

[whig]

A good test of this can be done before distributing by building with make -jX (where X is some number >1), this works whether or not you have SMP, though in the case that you do, it will actually take advantage of your multiprocessor capability. Once you can safely do parallel makes on a single machine, you should be able to extend this to a fully distributed make. (Obviously, such considerations require some thought and effort, so trying to do this for something you just need to make once is silly. But for large apps under active development, which need to be built from the base up periodically, it can be a real win.)

Re:pmake

[ÿ]

`pmake' is the usual abbreviation for BSD's make. Do you refer to that? I've never heard that it supports distributed compiles.
--
OS lover

Re:pmake

[AsmodeusB]

pmake is also the name of a linux 'make' utility which supports distributed compiles.

Solaris Sparcworks (also some comments)

[Polo]

This doesn't answer your question directly,
but solaris sparcworks has a parallel make
utility called 'dmake' that does exactly what
you want. You can even set permissions on each
system and the number of jobs allowed.

In my experience, I've found that you can't
just use parallel make on anything you find
on the net. The Makefiles have to be double
checked to make sure you have the dependencies
set up right. (using stuff like NO_PARALLEL)

For instance, I've had lots of makes break
because a link references a library member
that hasn't been built. That's kind of scary,
since if you're doing development and doing
a re-make, you might unwittingly be compiling
against an older library member. (and it's
timing-dependent with this kind of hosage -
it might work one time, but not the next)
So double check your makefiles...

good luck