An Introduction to fakemake

Andreas Dominik Cullmann

2017-11-19, 21:05:35

Why Mock the Unix Make Utility?

There are many build systems, and more uses for build systems (see Powers et al. 2002, sec. 11.10 and 11.11).

I have been using the unix make utility when developing R packages since 2012. But sometimes I get caught on a machine where make is not available and where I am not entitled to install it1.

This is why I wrote fakemake: to build an R package conditionally on the modification times of (file) dependencies without having to rely on external software. If you have any proper build system at hand: stick to it, do not use fakemake.

withr and knitr

Throughout this vignette I use R’s temporary directory, often by using withr::with_dir(tempdir(), ...). Because this is a vignette and the codes are examples. In real life, we would skip the temporary directory stuff.

Vignettes are built using knitr, which itself uses sink(). As sink() is central to fakemake for redirecting output to files in the make chain, I have to disable some of knitr’s output here and there. Don’t worry, it’s just because knitr and fakemake both want to use sink() exclusively and it only affects vignettes built with knitr.

Makelists

A makelist is the fakemake`s representation of a Makefile. It’s just a list of lists. Look at the minimal makelist provided by fakemake:

str(fakemake::provide_make_list("minimal", clean_sink = TRUE))
## List of 4
##  $ :List of 3
##   ..$ target       : chr "all.Rout"
##   ..$ prerequisites: chr [1:2] "a1.Rout" "a2.Rout"
##   ..$ code         : chr "print(\"all\")"
##  $ :List of 2
##   ..$ target: chr "a2.Rout"
##   ..$ code  : chr "print(\"a2\")"
##  $ :List of 3
##   ..$ target       : chr "a1.Rout"
##   ..$ prerequisites: chr "b1.Rout"
##   ..$ code         : chr "print(\"a1\")"
##  $ :List of 2
##   ..$ target: chr "b1.Rout"
##   ..$ code  : chr "print(\"b1\")"

Each sublist represents a Makefile’s target rule and has several items: at least a target and either code or prerequisites, possibly both. This makelist would still be a Makefile’s valid representation if target rule #3 with target “a1.Rout” had no (or an empty) code entry.

Other possible target rule entries are:

A Minimal Example

Suppose we would have a minimal makelist:

ml <- fakemake::provide_make_list("minimal", clean_sink = TRUE)

Building and Rebuilding

Now build the “all.Rout” target:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

We can see the files created:

show_file_mtime <- function(files = list.files(tempdir(), full.names = TRUE,
                                               pattern = "^.*\\.Rout")) {
    return(file.info(files)["mtime"])
}
show_file_mtime()
##                                        mtime
## /tmp/Rtmpo1WOuV/a1.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/a2.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/all.Rout 2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/b1.Rout  2017-11-24 22:30:56

If we wait for a second and rerun the build process, we get:

# ensure the modification time would change if the files were recreated
Sys.sleep(1)
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## NULL
show_file_mtime()
##                                        mtime
## /tmp/Rtmpo1WOuV/a1.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/a2.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/all.Rout 2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/b1.Rout  2017-11-24 22:30:56

Nothing changed. Good. Now, we change one file down the build chain:

fakemake::touch(file.path(tempdir(), "b1.Rout"))
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "a1.Rout"  "all.Rout"
show_file_mtime()
##                                        mtime
## /tmp/Rtmpo1WOuV/a1.Rout  2017-11-24 22:30:57
## /tmp/Rtmpo1WOuV/a2.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/all.Rout 2017-11-24 22:30:57
## /tmp/Rtmpo1WOuV/b1.Rout  2017-11-24 22:30:57

Since a1.Rout depends on b1.Rout and all.Rout depends on a1.Rout, these targets get rebuilt while a2.Rout stays untouched.

Had we touched a1.Rout, b1.Rout would not have been rebuilt:

fakemake::touch(file.path(tempdir(), "a1.Rout"))
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "all.Rout"
show_file_mtime()
##                                        mtime
## /tmp/Rtmpo1WOuV/a1.Rout  2017-11-24 22:30:58
## /tmp/Rtmpo1WOuV/a2.Rout  2017-11-24 22:30:56
## /tmp/Rtmpo1WOuV/all.Rout 2017-11-24 22:30:58
## /tmp/Rtmpo1WOuV/b1.Rout  2017-11-24 22:30:57

Forcing the Build

If you set the force option, you can force the target and all its prerequisites down the build chain to be built:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml, force = TRUE)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

If you want to force the target itself, but not all its prerequisites, set recursive = FALSE:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml, force = TRUE,
                                                recursive = FALSE)))
## [1] "all.Rout"

Using Aliases

If you find a target rule’s target too hard to type, you can use an alias:

i <- which(sapply(ml, "[[", "target") == "all.Rout")
ml[[i]]["alias"] <- "all"
withr::with_dir(tempdir(), print(fakemake::make("all", ml, force = TRUE)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

This is pointless here, but targets might be files down a directory tree like ‘log/testthat.Rout’ when building R packages.

Diverting Output / Programmatically Creating a Target Rule’s Target

Target rule b1 dumps its output to b1.Rout:

cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## [1] "b1"

Suppose it would programmatically create the target:

i <- which(sapply(ml, "[[", "target") == "b1.Rout")
ml[[i]]["code"]  <- paste(ml[[i]]["code"],
                      "cat('hello, world\n', file = \"b1.Rout\")",
                      "print(\"foobar\")",
                      sep = ";")
withr::with_dir(tempdir(), print(fakemake::make("b1.Rout", ml, force = TRUE)))
## [1] "b1.Rout"
cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## hello, wo[1] "foobar"

You end up with a broken target file, so you need to add a sink:

ml[[i]]["sink"] <- "b1.txt"
withr::with_dir(tempdir(), print(fakemake::make("b1.Rout", ml, force = TRUE)))
## [1] "b1.Rout"

Now you get what you wanted:

cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## hello, world
cat(readLines(file.path(tempdir(), "b1.txt")), sep = "\n")
## [1] "b1"
## [1] "foobar"

No Code Targets

Rule a1 has code

i <- which(sapply(ml, "[[", "target") == "a1.Rout")
ml[[i]]["code"]
## $code
## [1] "print(\"a1\")"

that prints “a1” into “a1.Rout”:

cat(readLines(file.path(tempdir(), "a1.Rout")), sep = "\n")
## [1] "a1"

If we remove that code and its output file and rerun

ml[[i]]["code"]  <- NULL
withr::with_dir(tempdir(), print(fakemake::make("a1.Rout", ml, force = TRUE)))
## [1] "b1.Rout" "a1.Rout"

the file is still created (note that target rule b1 down the make chain is run since we did not set recursive = FALSE ) but empty:

file.size(file.path(tempdir(), "a1.Rout"))
## [1] 0

Phony Targets

As you have seen, you can temporarily force a build. You may set a target to be .PHONY which forces it (but not its prerequisites) to be built:

ml[[i]][".PHONY"]  <- TRUE
withr::with_dir(tempdir(), print(fakemake::make("a1.Rout", ml)))
## [1] "a1.Rout"

Building R Packages Using Fakemake

We will now look a fakemake’s main purpose: building packages. First, we need to create a sample package.

Creating the Package

First, we create a package skeleton:

pkg_path <- file.path(tempdir(), "fakepack")
unlink(pkg_path, force = TRUE, recursive = TRUE)
devtools::create(pkg_path)

We add a minimal R code file and a log directory:

file.copy(system.file("templates", "throw.R", package = "fakemake"),
          file.path(pkg_path, "R"))
## [1] TRUE
dir.create(file.path(pkg_path, "log"))

This package does not make any sense. It is just a minimal working example (in the sense that it passes R CMD build and a simple R CMD check). It does not provide any functionality apart from a single internal function that is not exported via the package’s NAMESPACE. It is just there to exemplify the usage of fakemake.

Setting Up the Makelist

Then we get a package makelist from fakemake:

ml <- fakemake::provide_make_list("package")

This list is a bit more complex than the minimal example above, so we visualize it:

withr::with_dir(pkg_path, fakemake::visualize(ml))

Obviously the tarball depends on many files and the only target that’s no other target’s prerequisite is “log/check.Rout”. If you are more into hierarchical depictions, you can use the terminal target as root:

withr::with_dir(pkg_path, fakemake::visualize(ml, root = "log/check.Rout"))

But then you might be interested in this python program, it would leave with this graph: makefile2graph output

I regularly use it to visualize complex Makefiles.

More on Target Rules

Let’s take a look at the target rule that builds the tarball:

index <- which(sapply(ml, function(x) x["alias"] == "build"))
ml[[index]]
## $alias
## [1] "build"
## 
## $target
## [1] "get_pkg_archive_path(absolute = FALSE)"
## 
## $code
## [1] "devtools::build(pkg = \".\", path = \".\")"
## 
## $sink
## [1] "log/build.Rout"
## 
## $prerequisites
## [1] "list.files(\"R\", full.names = TRUE, recursive = TRUE)"  
## [2] "list.files(\"man\", full.names = TRUE, recursive = TRUE)"
## [3] "DESCRIPTION"                                             
## [4] "file.path(\"log\", \"lintr.Rout\")"                      
## [5] "file.path(\"log\", \"cleanr.Rout\")"                     
## [6] "file.path(\"log\", \"spell.Rout\")"                      
## [7] "file.path(\"log\", \"covr.Rout\")"                       
## [8] "file.path(\"log\", \"testthat.Rout\")"                   
## [9] "file.path(\"log\", \"roxygen2.Rout\")"

Note that some of its items are strings giving file names, some are strings that parse as R expressions, and prerequisites is a mix of both. Obiously, fakemake parses and evaluates these character strings dynamically.

Let us take a look at the prerequisites for testthat:

index <- which(sapply(ml, function(x) x["alias"] == "testthat"))
ml[[index]][["prerequisites"]]
## [1] "list.files(\"R\", full.names = TRUE, recursive = TRUE)"    
## [2] "list.files(\"tests\", full.names = TRUE, recursive = TRUE)"
## [3] "list.files(\"inst\", full.names = TRUE, recursive = TRUE)"

If you wonder why the only prerequisite for testthat shown in the above graphs is R/throw.R: there are no files to be found under tests/ or inst/ yet. We will come back to this later.

Building the Package

Now we build and check the package in one go:

withr::with_dir(pkg_path, print(fakemake::make("check", ml)))
## [1] "log/cleanr.Rout"            "log/covr.Rout"             
## [3] "log/lintr.Rout"             "log/roxygen2.Rout"         
## [5] "log/spell.Rout"             "log/testthat.Rout"         
## [7] "fakepack_0.0.0.9000.tar.gz" "log/check.Rout"

We see the files created in the log directory correspond to the names given by make:

list.files(file.path(pkg_path, "log"))
## [1] "build.Rout"    "check.Rout"    "cleanr.Rout"   "covr.Rout"    
## [5] "lintr.Rout"    "roxygen2.Rout" "spell.Rout"    "testthat.Rout"

and we can take a look at one:

cat(readLines(file.path(pkg_path, "log", "roxygen2.Rout")), sep = "\n")
## Updating roxygen version in /tmp/Rtmpo1WOuV/fakepack/DESCRIPTION
## Writing NAMESPACE
## Writing throw.Rd

Rebuilding the package does not do anything (NULL is returned instead of a the names of targets above), you save quite some CPU time compared to unconditionally reruning the codes in the makelist:

system.time(suppressMessages(withr::with_dir(pkg_path,
                                             print(fakemake::make("check",
                                                                  ml)))))
## NULL
##    user  system elapsed 
##   0.048   0.012   0.062

Changing Files and Rebulding the Package

Let us take a look at our testing coverage:

cat(readLines(file.path(pkg_path, "log", "covr.Rout")), sep = "\n")
##    filename functions line value
## 1 R/throw.R     throw   18     0
## 2 R/throw.R     throw   19     0
## 3 R/throw.R     throw   20     0
## 4 R/throw.R     throw   21     0
## fakepack Coverage: 0.00%
## R/throw.R: 0.00%

Well, poor. So we add a test file:

dir.create(file.path(pkg_path, "tests", "testthat"), recursive = TRUE)
file.copy(system.file("templates", "testthat.R", package = "fakemake"),
          file.path(pkg_path, "tests"))
## [1] TRUE
file.copy(system.file("templates", "test-throw.R", package = "fakemake"),
          file.path(pkg_path, "tests", "testthat"))
## [1] TRUE

Now we see - besides other things - that testthat has two new prerequisites: tests/testthat.R and tests/testthat/test-throw.R:

withr::with_dir(pkg_path, fakemake::visualize(ml))

Now we re-build the package’s tarball again (of course we could make("check", ml) again, but for the sake of (CRAN’s) CPU time, I skip the check):

withr::with_dir(pkg_path, print(fakemake::make("build", ml)))
## [1] "log/cleanr.Rout"            "log/covr.Rout"             
## [3] "log/lintr.Rout"             "log/testthat.Rout"         
## [5] "fakepack_0.0.0.9000.tar.gz"

We see that most of the build chain is rerun, except roxygenising, since the files under “tests/” are not prerequisites to log/royxgen2.Rout. Ah, and the test coverage is improved:

cat(readLines(file.path(pkg_path, "log", "covr.Rout")), sep = "\n")
## [1] filename  functions line      value    
## <0 rows> (or 0-length row.names)
## fakepack Coverage: 100.00%
## R/throw.R: 100.00%

References

Powers, Shelley, Jerry Peek, Tim O’Reilly, and Mike Loudikes. 2002. Unix Power Tools. O’Reilly & Associates.

  1. This is a nice example of what restrictive software policies are good for: you end up with a buggy imitation like fakemake instead of the well established original. You should not regulate software installations for programmers, unless you take away their interpreters/compilers.↩