Automating the Build Process

Getting Started

Get together with your partner and clone this GitHub repository.

Review the C++ Code

cd into the build-automation-exercise folder, where you'll find a number of files. For now, let's just look at the three files

• cow.cpp — code that implements a Cow class
• cow.hpp — code that defines a Cow class
• main.cpp — a small program that uses the Cow class

At this point you're pretty used to typing the commands to compile and link C++ programs. But it's tiresome to have to remember all the right commands and options and type them in every time you want to build the program. Wouldn't it be nice if we could automate this process?

Review build-all.py

We can build the program by running the included build-all.py file, with

python3 ./build-all.py

When it's finished, run

./main

to confirm that the executable works.

Modify the Code

Next, try modifying one of the C++ source files (e.g., add a cout statement somewhere) and then rebuild the executable by running build-all.py again.

rebuild.py

We're going to throw away build-all.py and replace it with rebuild.py, which is a smarter build script that will build all the parts of a program if it hasn't been built before, but that will only recompile the files that have changed on future runs.

Look over the code for rebuild.py and run it by typing

python3 ./rebuild.py

As above, run the executable to make sure it works properly.

Modify the Code

Experiment by editing one of the C++ source files and confirm that rebuild.py recompiles and relinks only the parts that have changed.

Remove the Executable

Now try removing the main executable the script built with

rm main

and then run rebuild.py again and verify that it only does the linking step to create the executable file without recompiling any of the other code.

make.py

The problem with rebuild.py is that it's very much hard-coded to this particular program—its understanding of what files depend on other files and what commands it needs to run are baked into the Python code.

In contrast, make.py is a script that generalizes the build process. It lets us define the dependencies in a compact form that the script can use to decide what to do.

buildRules{}

The most important part of make.py is the definition of the buildRules dictionary, which encodes the build process for each component of our program.

# Unlike build.py, now we've got DATA describing our specific project, and
# generic code that understands that data.  For each file, we have two bits
# of information, the prereqs for the file (a list of files it depends on),
# and the command to run.
#
# The dictionary below maps FILENAME -> (LIST_OF_PREREQS, BUILD_COMMAND)

buildRules = {
    "main":   (('main.o', 'cow.o'),
        "clang++ -std=c++17 -o main main.o cow.o"),
    "cow.o":  (('cow.cpp', 'cow.hpp'),
        "clang++ -Wall -Wextra -pedantic -c -std=c++17 cow.cpp"),
    "main.o": (('main.cpp', 'cow.hpp'),
        "clang++ -Wall -Wextra -pedantic -c -std=c++17 main.cpp"),
    "clean":  ((),
        "rm -f *.o main"),
}

Each key in the dictionary corresponds to the name of a file we want to produce, and the value specifies both what files we need before we can produce our key file and the compilation or linker command to run to produce the file from those prerequisites.

make

Next, read over the make function:

# --- The build algorithm, a recursive function called make ---

def make(file):
    iprint(f'- Making {file}', +1)  # +1 => Indent subsequent output

    if file not in buildRules:
        # If a file doesn't have a rule, it's fine if it already exists
        if doesntExist(file):
            error(f"Couldn't make '{file}' (no rule, and doesn't it exist)!")
        iprint(f'  + Okay, no rule found, but file exists!', -1)
        return

    # Okay, the file does have a rule, get the prereqs and command to run
    (prereqs, cmd) = buildRules[file]
    iprint(f'- Found rule for {file}, prereqs: {" ".join(prereqs)}')

    # Before going further, make sure all prereqs are up to date
    # before checking this file (recursively run make on prereqs)
    for prereq in prereqs:
        make(prereq)

    # Now that prereqs are up to date, see if we need to build this file
    mustBuildReason = None
    if doesntExist(file):
        mustBuildReason = "it doesn't exist"
    else:
        for prereq in prereqs:
            if isNewer(prereq, file):
                mustBuildReason = "prereqs are newer"

    # If we do need to build the file (and have a command to run), do so.
    if mustBuildReason is not None:
        iprint(f'- Target {file} must be built because {mustBuildReason}.')
        if cmd is not None:
            iprint(f'- Building: {file}')
            run(cmd)
        iprint(f'+ Made {file}')
    else:
        iprint(f"+ Target {file} doesn't need to be rebuilt (no newer prereqs)")

    # Set indentation back to what it was before and return
    iprint(None, -1)

Run make.py

Run make.py with

python3 ./make.py

make.py is rather chatty about what it is doing and why. Read through the output that it produced explaining its actions. Then, as you did with rebuild.py, you may want to try modifying one of the C++ source files and confirm that make.py rebuilds exactly what needs to be rebuilt and no more.

The specification in buildRules also includes a rule for how to build a file called clean—have a go at figuring out what will happen when you run it.

To check your answer, run

python3 ./make.py clean

Getting to Know Makefiles — example1.mak

In 1976, after commiserating with his colleagues about the complexity of building programs, Stuart Feldman at Bell Labs wrote the first version of a C program called make that does what our make.py script does (with a few enhancements). make reads a separate file (known as a “makefile”) to get the same data that was in buildRules, so you don't have to copy a script around and modify it, and the makefile format is a bit easier to type.

Check out example1.mak; it's basically the same data, just in a different form:

# This is a simple Makefile, it exactly mirrors the data we saw in make.py
#
# Use it by running:
#     cs70-make -f example1.mak
# or
#     cs70-make -f example1.mak main
#
# (or specify any other target from the the file)

main: main.o cow.o
    clang++ -g -std=c++17 -o main main.o cow.o

cow.o: cow.cpp cow.hpp
    clang++ -g -Wall -Wextra -pedantic -c -std=c++17 cow.cpp

main.o: main.cpp cow.hpp
    clang++ -g -Wall -Wextra -pedantic -c -std=c++17 main.cpp

clean:
    rm -f *.o main

target : prerequisites
	commands

You can see how a makefile works by running

cs70-make -f example1.mak

You can also just make a specific target; for example,

cs70-make -f example1.mak clean

runs the clean target (which deletes files that were created when make ran on some other target).

Feel free to experiment with changing files and making sure that cs70-make only rebuilds what is necessary.

More make features — example2.mak

Look over example2.mak and try it out. This makefile adds macros (a.k.a. variables) to avoid having to copy and paste the same text in multiple places.

In the makefile, we can define a macro like this:

CXXFLAGS = -g -Wall -std=c++17

The variable CXXFLAGS is now bound to the text -g -Wall -std=c++17. Then, elsewhere in the makefile (basically anywhere), we can use $(CXXFLAGS) to substitute that text. Now if we want to change the compilation flags, we only have to change them in one place.

We've also added another phony target called all. This commonly provided target is most useful when we have several executables (or other components, like documentation) that we want to build as we can list all our executables as prerequisites for all.

If you don't specify a target on the command line, make defaults to building the all target's dependencies.

Advanced make features — example3.mak

Look over example3.mak and try it out. It uses some even more advanced make features that allow us to type even less code.

Automatic Variables

When writing the build commands for a rule, we often want to refer to the target and prerequisites of that rule. make provides automatic variables for this purpose:

• $@ — the target of the rule (the @ sign looks a bit like a target)
• $< — the first prerequisite of the rule (the < points to the left towards the first prerequisite)
• $^ — all the prerequisites of the rule (the ^ points up to all the prerequisites listed above)

Thus, instead of writing

cow.o: cow.cpp cow.hpp
	$(CXX) $(CXXFLAGS) -c cow.cpp

we can write

cow.o: cow.cpp cow.hpp
	$(CXX) $(CXXFLAGS) -c $<

meaning “compile the first prerequisite to make the target”.

Similarly, for

main: main.o cow.o
	$(CXX) $(CXXFLAGS) -o main main.o cow.o

we can write

main: main.o cow.o
	$(CXX) $(CXXFLAGS) -o $@ $^

meaning “link all the prerequisites to make the target”.

Suffix Rules

In a suffix rule, we say “here's how to transform a .xxx file into a .yyy file”. For example, we can say “here's how to transform a .cpp file into a .o file”:

.cpp.o:
	$(CXX) $(CXXFLAGS) -c $<

This special kind of rule says “if no one gave you any build commands for building a particular .o file from a .cpp file, use these commands”.

So you can just define a single suffix rule for .cpp to .o and then you only need to write the dependencies for each .o file, as you can see in example3.mak.

clang++ -std=c++17 -MM *.cpp

Want More Information or Another Perspective?

The Working with Makefiles help page provides a summary of the key aspects of Makefiles, using a different program as the running example.

(When logged in, completion status appears here.)