=head1 NAME

makepp_speedup -- How to make makepp faster

=for vc $Id: makepp_speedup.pod,v 1.16 2013/01/20 16:54:22 pfeiffer Exp $

=head1 DESCRIPTION

So you think makepp is slow?  It has gotten noticeably faster, but granted,
it's still slow, especially if you come from GNU make.  This is because it
conscientiously checks all those things where gmake gives you a headache,
ignoring lots of dependencies (the "I think I need to gmake clean to get rid
of a mysterious bug" syndrome).  If you suspect some Perl code you added to
your makefiles might be at fault, take a look at L<perl_performance>.

But there are a few things you can do to squeeze out more speed.  Some of the
things are labeled unsafe, in the sence that you're asking makepp not to
check or do certain things, which you think are not needed.  If these things
would have been necessary, the build may not be correct.  Luckily this problem
will be temporary, however.  It will get corrected as soon as you let makepp
do all checks.

You can combine several of these tips to increase the time gain even more.


=head2 Safe Methods

=head3 Use makeppreplay

The stand-alone utility L<makeppreplay, mppr|makeppreplay> repeats things that
makepp has already done, without any overhead.

=head3 Use a Faster Perl

Within version 5.8, all are roughly the same, only 5.8.7 is a bit faster.
Tuning your Perl can also help, like not compiling it for 64 bits, which
makepp doesn't need.  For example ActiveState's build
(L<http://www.activestate.com/activeperl>) of 5.8.7 for Linux is faster than
the Perl 5.8.7 that comes with SuSE Linux 10.0.

=head3 Include as Little as Possible

Each additional file you include is doubly penalizing.  On the one hand, the
compiler must look for and at all those files.  You don't notice this so much,
because it's just a little extra per compiler call.  On the other hand makepp
must look too, to find dependencies and figure out whether they incur a
rebuild.  Then it can seem to stall, while it is digesting a lot of
dependencies at once.

An absolutely deadly variant is the project master include file, which in turn
conveniently includes anything you might need.  The result is that any header
file change leads to a full build.  Even without a change, makepp must think
about all those headers again, for every source you compile.  Just a tiny
effort, since this is cached, but thousands of files can make this staggering.

It may be cumbersome to figure out the minimal set of includes, and to cleanup
those no longer needed, but it really pays off.  If anybody knows a tool that
can identify which files get included unnecessarily, I'd be glad to mention it
here!

=head3 Build as Little as You Need

If you have a default target which makes several programs, then makepp will
have to check all their dependencies, right down to the smallest header file.
But maybe you want to test your change with only one of those programs.

Then you would call makepp with an explicit target.  The less modules or
headers all those programs have in common, the greater the benefit of not
letting makepp check them all.

Say your top level Makeppfile has this rule:

    $(phony all): proggie1 proggie2 $(only_phony_targets */**/all)

Then you would call things like

    $ makepp proggie2
    $ makepp proggie1 dir/subdir/proggie27

=head3 Use preferred makefile names

Makepp looks for makefiles (unless you specify them explicitly on the command
line or with C<load-makefile>) in the order F<RootMakeppfile>,
F<RootMakeppfile.mk>, F<Makeppfile> and F<Makeppfile.mk>, followed by the
classical makefile names.  (The F<.mk> variants are for purely suffix-based
systems.)

So, if you use F<RootMakeppfile> at the root of your build tree, and
F<Makeppfile> everywhere else, the files will be found slightly faster.
Makepp will also have a slightly smaller memory consumption (caching the fact
that the other names don't exist), which also means speed through less memory
management.

Likewise if you have a statement

    include standard

there will first be an attempt to find F<standard.makepp>, so you might as well
use that name.

=head3 Have as few rules as you need

Makepp keeps track not only of existent files, but also of any it learns to
create.  (That's why it offers reliable wildcards like F<*.o>.)  The price for
this power is a lot of management.  So, if you tell it how to create a F<.o>
from a F<.c>, that's fine, because it will happen for most if not all
candidates.

But if you tell it how to link any suffixless executable from a like named
F<.o>, that's expensive, because it will probably only happen for a small part
of them (those that contain a main function), but the basis will get laid for
all.  You have to weigh the comfort of a linker pattern rule, against the
efficiency of individual linker rules.

If you don't use any of them, you should also turn off the builtin rules with:

    makepp_no_builtin = 1

If you do use them, but, for the reasons explained above, not the builtin
linker rules, you should turn those off with:

    makepp_no_builtin_linker = 1

=head3 Put makepp extensions into a module

Makepp offers very convenient possibilities of being extended through Perl.
But if you write some functions, commands or statements in a file and include
that from dozens of makefiles, you will get dozens of copies of them all in
memory.  And they will be read dozens of times by the makepp parser, which is
a bit slower than Perl's.

In this situation it is better to L<put your own functions into a
module|makepp_extending/Putting functions into a Perl module>.

=head3 Use Repositories and/or a Build Cache

If you have several developers working on the same machine or if you change to
and fro between sets of build options, this is for you.
L<Repositories|makepp_repositories> allow you to offer a central reference
where you only need to build what is locally different.  A L<build
cache|makepp_build_cache> simply collects all produced files, and reuses them
as appropriate, with less planning needed.  The latter page also describes the
differences.

=head3 Use Sandboxes

If your build is so big that makepp is having a hard time digesting all the
information and if you can find a way of splitting it up into smaller
independent parts, L<sandboxes|makepp_sandboxes> might give you better
parallelism than the C<--jobs> option.

=head3 Don't log what you do

Makepp's logging feature is very powerful for tracking down bugs in the build
system, or for analyzing your dependencies.  Whenever you don't do these
things, you can save quite a bit of formatting and I/O with C<--no-log
--no-scan-log>.


=head2 Almost Safe Methods

=head3 Get a Headstart

The option C<--loop> (or C<--stop-before-building> or C<--stop-after-loading>
or C<--stop>) allows makepp to start its work while you are still editing.  It
will repeatedly suspend itself when it gets to the point analyzing the
dependencies.  You decide when you're ready to let it go on.  On our huge
project this saves half a minute, and that's only when we have a CPU to
ourselves.

This method has two potential drawbacks:

=over

=item *

Makeppfiles have been read by the time makepp stops.  If you edit a Makeppfile
or something from which it would have to be rebuilt, after starting makepp,
this will go unnoticed till the next time.  But this should rarely be
necessary, since makepp greatly reduces the need for Makeppfile changes.

=item *

If a target depends on a wildcard, and that would match more than when the
Makeppfile was read, makepp will not notice:

    proggie: *.o
 	$(LD) $(inputs) -o $(output)

If you add another source file, or a file from which makepp knows how to
generate a source, then C<*.o> should match the object that produces.  But, if
this file was added after starting makepp, it will not, because the wildcard
was expanded too early.

=back

In both of these cases you should kill the prestarted makepp and start it anew.

=head3 Gulliver's Travels

The option C<--gullible> tells makepp to believe that a rule changes what it
says it will, neither less nor more.  Not performing these checks can save a
few percent of makepp's CPU time.  And the Disk I/O savings is especially
welcome on network file systems.  If you do nightly full builds in an empty
directory with the C<--repository> option, but without the C<--gullible>
option, you can be fairly sure that your rule set is consistent.  Then this
option shouldn't hurt in your daytime work.


=head2 Potentially Unsafe Methods

These methods are unsafe if you give makepp the wrong hints.  But everything
will again be fine, however, as soon as you let makepp do all the checks, by
not passing it any limiting options.  For this reason I suggest using these
hints to get quick intermediate builds, and use lunchtime and nights to let
makepp do its job thoroughly.

=head3 Build as Little as Needed

This is the same tip of using explicit targets discussed under L<Build as
Little as You Need> above.  But it becomes more dangerous, if you do it
because you are sure that your change will not affect any of the other
programs.  Then they will not be built, even though it might have been
necessary.

=head3 Know Where Not to Build

The option C<--dont-build> is very powerful for speeding makepp up a lot.  If
you know one or more directories, which you are sure are unaffected by any
change you made since the last time, you can issue C<--dont-build> options for
them.  This can save makepp a lot of dependency analysis.  But it will not
build anything in those directories, even if it should have.

=head3 Know Where to Build

This is the same as L<Know where not to build>, but instead of an exclusion
list, you supply an inclusion list.  The trick is that a C<--do-build> option,
with a C<--dont-build=/> option or under a C<RootMakeppfile(.mk)> directory
without a C<--dont-build> option on a higher level directory means: build
nothing except what I tell you to.  This is what users of traditional makes
are looking for when they want to build just one directory:

    $ makepp --do-build=dir/subdir

or, if you don't have a C<RootMakeppfile(.mk)>:

    $ makepp --dont-build=/ --do-build=dir/subdir

The difference is that any default target in the top level Makeppfile,
i.e. link commands are also executed this way.  If you don't want that, you
must give an explicit target, which is automatically also marked for
C<--do-build>:

    $ makepp --do-build=dir1/subdir dir2/proggie

=head3 Know What to Build

An extreme variant is asking makepp not to build anything but what you tell it
to.  This is not so dangerous if you changed B<no include files>, only
modules, and you know which programs they go into.

Say you have only changed C<src/a.cpp> and C<src/b.cpp> and these are linked
directly into one program.  Dot is the current directory including all
subdirectories.

    $ makepp --dont-build=. src/a.o src/b.o proggie1

Or equivalently, because a C<--do-build> option, without a C<--dont-build>
option on a higher level directory implies C<--dont-build> for the root of the
build tree:

    $ makepp --do-build=src/a.o src/b.o proggie1

You can do something like the following in your Shell's $ENV file or .profile
to save typing (csh users replace '=' with ' '):

    alias mppb='makepp --do-build'
    alias mppsb='makepp --stop --do-build'

Then the last example becomes:

    $ mppb src/a.o src/b.o proggie1

=head3 Build on a RAM disk

Modern computers, especially servers, typically have a high mean time between
failure.  If this is the case for you, and you have lots of RAM to spare, you
can save the time you wait for I/O.  You should edit on a real disk, or
replicate your edits there quickly.  But the build results are reproducible,
so they can reside in RAM.  If you don't want to risk rebuilding, you can
always replicate to disk after each build or at night.  You should not do this
during the build, as you might catch partially written files, just as if the
machine had crashed.

If you have a system and/or storage unit with good caching and RAID, the gain
might not be so big.


=head1 AUTHOR

Daniel Pfeiffer <occitan@esperanto.org>