The CI flow for our Node.js app looks roughly like this:
Currently, this all takes place in a single TeamCity ‘job’ with three ‘steps’ (the Test step runs 4 concurrent child processes).
Problems with the current approach:
- The whole job takes too long – 15 minutes. (The Test subprocesses run in parallel, but this only shaves about 15% compared to running them serially.)
- The Test step has jumbled log output from 4 child processes, and it’s painful figuring out what failed.
I want to split the above into six TeamCity jobs, using artifact and/or snapshot dependencies to compose them into the desired flow. This should make better use of our pool of four build agents (better parallelism), and it should make failures easier to pinpoint.
But I’m having trouble with sharing the
node_modules from the first step so it can be reused by all four jobs in the Test phase. It takes about 3-5 minutes to run
yarn (to set up
node_modules), so I want to avoid repeating it on every Test job.
Also, most git pushes don’t actually change the npm dependencies, so the ‘Setup’ phase could often be bypassed for speed. CircleCI has a nice way to do this: it lets you cache your
node_modules directory with a custom key such as
node_modules.<HASH>, using a hash of your lockfile (
package-lock.json) – because the complete
node_modules directory is more or less a function of the lockfile.
But my company won’t let me use CircleCI. We have to use TeamCity.
What I’ve tried:
- Configuring the first TC job to export
node_modulesas an artifact, but this seems to take forever on TeamCity (>10 minutes for a large
node_modulesdir), compared to a few seconds on CircleCI. Also, TC doesn’t make it easy to have a dynamic cache key like Circle does.
- I’ve tried a custom solution: I save a tarball of
node_modulesto S3 (with cache key based on lockfile), then each Test job streams it down and untars it into
node_moduleslocally, but this ends up taking just as long as running
yarnfrom scratch on each job, so there’s no point.
I’m stuck. Has anyone had any success setting up a CI flow like this on TeamCity?