RFC-2: Vendoring And fastn update
fastn update command, which helps you ugprade
your versioned dependencies.This document exists to describe a proposal for enhancing the fastn language.
This is a Request For Comment. Please share your comments by posting them in the
pull request for this RFC if this RFC is not merged yet. If the RFC is merged,
you can post comment on our official Discord, or
open a discussion on Github.
rfc.Status
Development Status
-
fastn buildto generatemanifest.json -
fastn update -
fastn update --statuscommand -
fastn check -
fastn {serve,build} --release
Motivation
fastn downloads and stores all the dependencies in a folder named .packages.
We also have what we call download on demand, so when a dependency is
encountered we do not download the entire dependency, as is the case with most
other package managers, but the individual documents in the dependency, based on
demand.
We currently do not have package versioning, so we have been asking people to
not checking .packages in the version control system along with their source
code. We do that so latest versions of the dependencies are downloaded and
used when the package is being built. This has allowed us to quickly do
dependency updates and have the ecosystem get the latest updates.
This is not a good long term solution though. We have been able to manage a level of quality control on our changes, not making any breaking changes, not making logical or surpisiing changes in packages, etc, mostly bug fixes and minor enhancements.
Backward incompatible or logical changes in package may happen when there are
more packages, e.g. from more authors and programmers who are learning fastn,
and to keep our package ecosystem healthy and reliable, we have to implement
version pinning of dependencies as well.
.packages contained all dependencies, and .packages was managed by fastn update, rest of fastn operations will not be making any HTTP requests, and will
be have consistent performance.Detailed Design
Package Manifest File
-- record package:
string name: ;; full package name
document list documents: ;; all the `fastn` files
file list assets: ;; all the images etc assets
file list fonts: ;; if the package is a font package
-- record file:
string full-name: ;; relative to FASTN.ftd
string checksum: ;; sha256 checksum of the document content
integer file-size: ;; in bytes
-- record document:
string list dependencies: ;; this is all the direct dependencies
file file:
fastn build
fastn build will create the /-/manifest.json file.Structure of .packages folder
fastn-community.github.io/doc-site will be stored in:
.packages/fastn-community.github.io/doc-site, and it's manifest file will be
in .packages/fastn-community.github.io/doc-site/-/manifest.json.fastn update --status
fastn update --status will download latest manifest.json for each package,
and compare the one .packages folder and list all the packages and files that
are out of date.fastn update <optional package-name>
fastn update, without package name will update all the out of date
files in .packages. If package-name is passed, only that package will be
updated. If during package update we encounter new dependencies we do not
already have we download them as well.Transitive Dependencies
fastn update will download all dependencies including transitive dependencies.How The Packages Will Be Developed Locally?
.packages folder.fastn serve --release flag
The download on demand feature would be only available in "debug mode", which is
the default. For deploying on production server, --release should be used,
which considers a missing document an error.
fastn check can be used to check if the package has all its
dependencies available in .packages folder. Check will in future perform more
checks.Alternatives
Lockfile Approach
Other package ecosystems do not typically vendor dependencies, instead create a lockfile with exact versions. Dependencies are downloaded on every developer and CI machine, prod machine during deployment.
We consider vendoring a superior approach as it reduces overall load on the central package repository, reduces total network activity in general. Vendoring also allows local changes to upstream packages.fastn update Only Downloads
One option we considered is to ensure only fastn udpate does any network
activity. If any document is missing we rest of fastn will report a hint asking
user to run fastn udpate to fix it. This simplifies our code, and gaurantees
no unneeded network call. Possibly.
This was rejected because a. we would have still wanted to give "automatically
run fastn update on need, at least in dev mode".
We rejected this option as for fastn update to detect all the dependencies we
have to parse all documents in the package, and this takes time, making fastn update a slow process. A slow process that has to be used is a lot is a bad
user experience.
If we make the process itself fast using incremental analysis approach (only analyse documents that have changed since last run) we can make this fast and use this.
Since this is a small decision, rest of the RFC is applicable in both cases, we have decided to start working on it for now till we implment the incremental anaysis approach.Teaching Notes
fastn update --status
and taking decisions about if a package is safe to update or not is actually
tricky, as how does one decide? One can give instructions to just try out and
see if nothing fails so it is safe to update. Can we let fastn update --status
report more information, like if the package will build if this particular
dependency was updated, and so it is safe to update?Unresolved Questions
Would This Lead To Conflicts?
Would having ability to modify dependency code cause ecosystem issues?
Like if I am vendoring code, it's trivial for me to modify them, and people will start modifying them, and so is that a good thing, or a bad, and we start building features to disallow that (e.g., fastn update complaining about checksum mismatch).
If it is a good thing we can makefastn update do a three way merge to keep
your local changes while updating dependencies.
