Efficiently sharing a large node_modules directory between multiple TeamCity build jobs

The CI flow for our Node.js app looks roughly like this:

enter image description here

Currently, this all takes place in a single TeamCity ‘job’ with three ‘steps’ (the Test step runs 4 concurrent child processes).

Problems with the current approach:

  • The whole job takes too long – 15 minutes. (The Test subprocesses run in parallel, but this only shaves about 15% compared to running them serially.)
  • The Test step has jumbled log output from 4 child processes, and it’s painful figuring out what failed.

Desired approach

I want to split the above into six TeamCity jobs, using artifact and/or snapshot dependencies to compose them into the desired flow. This should make better use of our pool of four build agents (better parallelism), and it should make failures easier to pinpoint.

But I’m having trouble with sharing the node_modules from the first step so it can be reused by all four jobs in the Test phase. It takes about 3-5 minutes to run yarn (to set up node_modules), so I want to avoid repeating it on every Test job.

Also, most git pushes don’t actually change the npm dependencies, so the ‘Setup’ phase could often be bypassed for speed. CircleCI has a nice way to do this: it lets you cache your node_modules directory with a custom key such as node_modules.<HASH>, using a hash of your lockfile (yarn.lock or package-lock.json) – because the complete node_modules directory is more or less a function of the lockfile.

But my company won’t let me use CircleCI. We have to use TeamCity.

What I’ve tried:

  • Configuring the first TC job to export node_modules as an artifact, but this seems to take forever on TeamCity (>10 minutes for a large node_modules dir), compared to a few seconds on CircleCI. Also, TC doesn’t make it easy to have a dynamic cache key like Circle does.
  • I’ve tried a custom solution: I save a tarball of node_modules to S3 (with cache key based on lockfile), then each Test job streams it down and untars it into node_modules locally, but this ends up taking just as long as running yarn from scratch on each job, so there’s no point.

I’m stuck. Has anyone had any success setting up a CI flow like this on TeamCity?