RFC-4: Incremental Build

In this RFC we propose persisting build metadata on every fastn build. This will enable as to only rebuild only the minimum number of files needed in the output directory, and will significantly cut down build time.

📝

This is a RFC document

This document exists to describe a proposal for enhancing the fastn language. This is a Request For Comment. Please share your comments by posting them in the pull request for this RFC if this RFC is not merged yet. If the RFC is merged, you can post comment on our official Discord, or open a discussion on Github.

Learn about our RFC process. View all active RFCs. WIP RFCs, and RFCs awaiting initial comments can be found on Github, as Pull Requests, with label rfc.

Status

accepted

Motivation

Current we rebuild every document present in current package, and recreate entire .build folder. If we have cache metadata about the previous, we can do incremental build, and achieve much faster builds.

Detailed Design

We will create a new cache data, build-cache.json, which will contain the following information:

-- record build-cache:
string fastn-version:
document list documents:
file list assets:
file list fonts:

-- record file:
string path:
string checksum:

-- record document:
file file:
string html-checksum:
string list dependencies:

Lang:

ftd

Every time fastn build runs, it will load the existing build-cache.json, and scan the current folder, .packages and .build folders. From these we can compute what all files must exist in the final .build folder, and which ever is missing from .build folder, or have wrong checksum we will overwrite those files.

Configurable Build/Cache Folders

We will allow environment variables FASTN_BUILD_DIR and FASTN_CACHE_DIR to overwrite where we store the build and cache files. By default if FASTN_BUILD_DIR is missing we will continue to use .build folder and if FASTN_CACHE_DIR is missing we will use OS specific cache directory.

Alternatives

Rejected: `fastn build --ignore-cache`

We can also allow this command which will ignore cache and rebuild everything.

We rejected this because this is clearly a bug in fastn, and never a feature that end users would want. We can instead give fastn clean which will delete .build folder, the entire cache folder and so on.

Remote Caching

Since this feature requires us to preserve cache across fastn build, and on CI systems it will require CI provider specific steps, we can offer a free remote build cache service, simplifying this step.

This was rejected because we will have to cache both the build-cache.json and the content of the .build folder, later being much bigger.

Teaching Notes

The feature itself requires no training as this is an internal optimisation.

Confiruing CI systems to preserve build cache across builds is required. We will be updating our fastn-template Github Action to include build caching. We will also have to write blog post on how to enable build caching on Vercel, and other hosting providers who give caching.