Monorepo 탐색 #5: 완벽한 도커

It's critical to understand that Docker caches each line in the Dockerfile, and that the output of one line is the input of the next. So if a line generates new output all subsequent caches are invalidated. With that in mind, here's a common Docker anti-pattern that causes issue 1:

COPY . .
RUN pnpm install

If anything changes in any file then pnpm install has to run from scratch, because the COPY . . would produce a different output. This should always be optimized so only the files necessary to install dependencies are copied in first, then dependencies are installed, and then the rest of the source-files are copied in. Something like this:

COPY package.json .
COPY pnpm-lock.yaml .
COPY pnpm-workspaces.yaml .
COPY apps/web/package.json ./apps/web/
COPY libs/types/package.json ./libs/types/
RUN pnpm install
COPY . .

Now all steps up to and including pnpm install remain cached so long as none of those meta-files change, and so Docker will skip all those steps. This is a massive speedup.

The downside is we're now manually specifying all those meta-files ☹️. And that leads to issue 2:

Using the COPY <meta-file> construct scales poorly because we have to author each Dockerfile with explicit and detailed information about which dependencies to copy in. And by using the COPY . . construct we copy all monorepo files, which needlessly bloats the image because for this example we only need the source-files from apps/web and libs/types (it's but web only depends on types ).

The key insight is that pnpm already understands how dependencies depend on each other, so we should be able to leverage that. We can't use pnpm directly from Dockerfile's COPY construct, but what if we use pnpm to generate a context that only contains the files needed for a specific package? Then the Dockerfile for that package could use COPY . . but it'd actually only copy in just the right files…

And, hang on, lets consider the meta-files too. The challenge is we can't isolate all the package.json files easily so we resort to path-specific COPY commands, but what if we get really clever and create our custom context such that all the meta-files are placed in a /meta folder inside the context for easy copying, and we put the rest of the source-files in another folder?

Let's see if that'll work!

We introduced the custom context technique where we simply piped tar into Docker:

$ cd apps/web
$ tar -cf - ../.. | docker build -f apps/web/Dockerfile -

Now it's time we discard the naive tar command and come up with something more bespoke.

I've made a script that takes a Dockerfile and finds just the right files needed for that package, and outputs it all as a tarball so it's a drop-in replacement for the tar command.

It's fine we preserve the pnpm install cache as much as we can, but when it does have to run it frustratingly re-downloads every single dependency from scratch. That's very wasteful in time and bandwidth! On our own machines pnpm downloads to a persisted store so it never has to re-download a package, but that store never gets persisted inside Docker because it evaporates as soon as a meta-file changes.

But Docker has a mechanism for exactly this: It allows a RUN command to mount a folder which is persisted on the host machine, so when the command runs it has access to files from previous runs. The code for this ends up a bit complex-looking, but it's worth the performance boost so let's try it out:

ARG PACKAGE_PATH
COPY ./meta .
RUN --mount=type=cache,id=pnpm-store,target=/root/.pnpm-store\
 # ↑ By caching the content-addressable store we stop
 # downloading the same dependencies again and again.
 # Unfortunately, doing this causes Docker to place 
 # the pnpm content-addressable store on a different
 # virtual drive, which prohibits pnpm from 
 # symlinking its content to its virtual store,
 # and that causes pnpm to fall back on copying the
 # files, and… that's totally fine! Except pnpm emits 
 # many warnings that its not using symlinks, so 
 # we also must use `grep` to filter out those warnings.
 pnpm install --filter "{${PACKAGE_PATH}}..." \
     --frozen-lockfile\
 | grep --invert-match "cross-device link not permitted\|Falling back to copying packages from store"
# ↑ Using `--invert-match` to discard annoying output

It would be nice if we could tell pnpm to be quiet when it can't symlink, but we can survive this complexity.

We've reached the last issue: We're bloating the final image with dev-dependencies because we don't clean up after building apps/web inside the image. It's a waste we shouldn't allow.

The solution is to reset back to having no dependencies installed, and then only installing the production dependencies. This is pretty straightforward to do by using Docker stages:

FROM node:16-alpine AS base
# Install pnpm

FROM base AS dev
# Install all dependencies and build the package

FROM base as prod
# Install just prod dependencies

With this approach the "prod" stage isn't affected by whatever happens in the "dev" stage. Nice! But because dev builds the package we do need some way to transfer files from dev to prod, because we need the final build code to be moved to prod stage. For that we can introduce an "assets" layer where we isolate just the files that should go into the prod stage. So we can do something like this:

FROM node:16-alpine AS base
RUN npm --global install pnpm
WORKDIR /root/monorepo

FROM base AS dev
# Install all dependencies and build the package

FROM dev AS assets
RUN rm -rf node_modules && pnpm recursive exec -- rm -rf ./node_modules ./src
# ↑ Reset back to no dependencies installed, and delete all
# src folders because we don't need source-files. 
# This way whatever files got built are left behind.

FROM base as prod
pnpm install --prod --filter "{${PACKAGE_PATH}}..."
# ↑ Install just prod dependencies
COPY --from=assets /root/monorepo .

So here the "assets" stage isolates whatever code was generated in the dev stage, which the prod stage then copies into itself. Does it work?

$ cd apps/web
$ pnpm build
$ docker run mono-web
[razzle] > Started on port 3000

🎉

It's one thing to get all this working locally, but we also need to update our GitHub Actions CI script.

The issues we set out to vanquish have been resolved 🎉. We now build images that play nice with Docker caching, the Dockerfiles are free from manually-specified dependency concerns, and the final images are very lean and optimal. Quite nice!

I feel the pnpm investment is really paying off, it was already a nice CLI to use but how amazing they also have a pretty straightforward API to use programmatically to do our dependency-graph logic!

This article's title promised "perfect", did we achieve that? Well, no, perfection is a high bar, but we've addressed all the practical concerns I've experienced so I'm happy to call it a day here. We wouldn't want to get too carried away after all 👀 (I think for some, this entire article-series is already deep into "carried away" territory).

I'd love to hear if you have any questions or comments, or if there are any directions you'd like to see explored in future articles. So please leave a comment.