Archiving Pipelines
an abridged update
Resources v2 is now an idea of a 'generalized resource interface', decoupled from artifact versioning.
Users will be able to use v1 and v2 resources side-by-side, and users won't even have to upgrade to v2 to use new Concourse features.
It's just a refinement of the same idea.
For these slides, remember that 'version' is now called 'config fragment'.
2017-2019
we hardly knew ye
Behind the name, there are a few great features:
Dynamically running jobs across 'spatial' change, i.e. branches and PRs.
"Fanning in" using passed constraints across spaces.
Automatically cleaning up spaces for closed PRs, etc.
These still make sense. So why is 'spaces' dead?
Our first approach was to extend resources and jobs to support finding versions and running builds across spaces.
This approach proved to be strategically expensive.
We thought of it as one big feature, and we baked it into the entire stack, from the web UI down to the resource protocol.
Having a single feature touch so many things led to lot of second-guessing and many different iterations.
Can we break this big feature up?
Recently we were able to decouple 'Resources v2' from 'Spaces' by taking a completely different approach, leveraging composition instead of directly coupling them.
Doing this was liberating: suddenly we could deliver two big features independently instead of all at once.
Can we do the same for 'spaces'?
Let's go on a journey.
I've got a few features to propose, which I hope are independently useful.
When they're used together, they become way more powerful.
Hang in there - the dots should connect as we go along!
Pipelines can be archived when they are no longer needed.
When archived, a pipeline becomes completely inactive, and its name can be used for later pipelines. This is basically a soft-delete.
$ fly archive-pipeline -p pipeline-name
This is a pretty simple feature on its own, but it's also a precursor to "instanced pipelines."
Someone actually started on a PR for this a while back. It doesn't seem too hard.
API: small - support archiving/unarchiving, support re-using names of archived pipelines.
UX: medium - 'dim' and/or stash away archived pipelines.
Core: x-small - don't schedule archived pipelines.
Pipeline templates can be "instantiated" with ((vars)) that also become part of the pipeline's identifier.
$ fly set-pipeline -p release \
-i version:5.3 \
-c ci/pipelines/release.yml
This is a small UX change that allows simple pipeline hierarchies to form - 2 levels deep instead of N.
API: medium? - allow multiple instances of same pipeline name with different vars.
Core: x-small - schedule all pipeline instances.
UX: medium - probably a lot of navigation UX to figure out, but hopefully simpler than 'spaces'.
set_pipeline step
set_pipeline step (cont'd)Pipelines can be configured by using a new set_pipeline step in a build plan:
plan: - get: ci - set_pipeline: concourse file: ci/pipelines/concourse.yml
The pipeline will be set within the team running the build, and will start un-paused.
set_pipeline + instances (cont'd)The set_pipeline step can be used to 'sync' all instances of a pipeline:
plan:
- get: ci
- set_pipeline: concourse
file: ci/pipelines/concourse.yml
- set_pipeline: release
file: ci/pipelines/release.yml
instance_vars: {version: 5.3}
- set_pipeline: release
file: ci/pipelines/release.yml
instance_vars: {version: 5.2}
set_pipeline + archiving (cont'd)A build which uses set_pipeline for setting instances determines the full set of active instances for the pipeline.
All other instances will be automatically archived when the build completes.
This way release pipelines will automatically become archived when they're no longer relevant.
This will be more important later!
API: x-small - support parsing it as part of the build plan.
Core: small - implement the step. medium once we do instance archiving.
UX: small - support rendering the step.
across step
across: build matrixes (cont'd)An across step runs a given step across all versions returned by a resource check:
across: supported-ruby-images as: ruby do: - task: unit image: ruby file: ci/tasks/unit.yml
The across step can be nested to do N-dimensional matrixes.
across: pipeline matrixes (cont'd)By composing these two steps and using instance_fragment , we now have pipeline matrixes:
across: concourse-branches as: branch do: - set_pipeline: branch instance_fragment: branch file: ci/pipelines/branch.yml
When a branch goes away, its instance will become archived.
across with trigger (cont'd)The across step will support trigger: true. This way the build will fire on any change to the set.
across: concourse-branches as: branch trigger: true do: - set_pipeline: branch instance_fragment: branch file: ci/pipelines/branch.yml
API: x-small - support it as part of the build plan.
Core: medium/large - implement a new type of scheduling and triggering, similar to get step handling but for sets of versions. Construct a build plan that repeats the step with a get for each version.
UX: medium - show all the steps, probably in some sort of tab UI. There's some prior art with attempts:.
A project is a namespace for pipelines, resources, and tasks.
A project is bootstrapped by a resource containing the project's configuration, along with its tasks, resources, and pipelines.
Many projects may exist within a team.
fly set-project (cont'd)Projects are configured with fly set-project:
$ fly -t ci set-project \
--project booklit \
--type git \
--source uri=https://github.com/vito/booklit \
--config-path ci
The --config-path flag identifies the sub-directory within the resource from which to load the project config.
Concourse will check for new versions of the project and load the project's config, resources, tasks, etc.:
ci/project.yml
ci/tasks/test.yml
ci/resources/booklit.yml
ci/pipelines/booklit.yml
Pipelines shrink down to just job definitions, removing hundreds of lines of YAML. Separate files are much easier to work with.
A minimal project configuration contains a name and a plan:
name: booklit
plan:
- task: test
The project's plan: will run every time a new version of the project is found, with the project resource itself available under the configured name.
Kinda like Travis/Circle CI.
Projects, when combined with the set_pipeline step, allow your entire project to be automated and reproducible.
It's a full blown build plan, you can even do pipeline templating if necessary:
name: ci plan: - task: generate-template - set_pipeline: fancy-templated-pipeline file: generated-pipeline/foo.yml
Pipelines within a project change in a few ways:
Every job will automatically have the project resource available to its build plan, just like the project's own plan.
Pipelines will no longer list their own resource definitions. Instead, there will be project-wide resource definitions.
Pipelines will be able to reference each other's jobs via passed constraints. This is the missing piece for 'spaces'.
Before:
in_parallel: - task: fly-linux file: ci/tasks/fly-linux.yml - task: fly-darwin file: ci/tasks/fly-darwin.yml - task: fly-windows file: ci/tasks/fly-windows.yml
After:
in_parallel: - task: fly-linux - task: fly-darwin - task: fly-windows
This reads a lot better, builds on existing usage patterns, and task names are actually useful now!
Projects define credential managers as var_sources:
name: ci var_sources: - type: vault config: # ... plan: # ...
The proximity to plan: makes it easy to visibly audit credential access within the project.
API: medium - there's a new object owner in town, so I expect some API noise. Maybe we should invest in non-hierarchical URLs at this point?
Core: medium/large - this is a bit of an epic, but it's at least grounded in well-understood ideas (i.e. build plans). I think there are interesting, relatively simple ways to design this.
UX: medium/large - we'll need project builds to show up in the UI somewhere. And we'll probably want to think about any navigation implications.
Circling back to 'spaces', let's see how these features add up:
Dynamically running pipelines across branches and PRs: projects + across step + set_pipeline step.
"Fanning in" across spaces is possible with projects.
Automatically cleaning up spaces which no longer exist: instanced pipelines become automatically archived.
name: booklit plan: - task: test
fly set-project -p booklit -t git \
-s uri=https://github.com/vito/booklit \
-c ci
This should feel intuitive for smaller projects which may not need a sophisticated pipeline system. Concourse has been described as 'overkill' for such use cases - hopefully this bridges the gap.
name: ci var_sources: - type: vault config: # ... plan: - set_pipeline: concourse - set_pipeline: prs
In this case it might make more sense for prs to be a separate project so it doesn't share the credential manager config.
name: ci
plan:
- set_pipeline: concourse
- across: release-branches
as: branch
do:
- set_pipeline: release
instance_fragment: branch
- across: feature-branches
as: branch
do:
- set_pipeline: branch
instance_fragment: branch
a brief side-tangent
...Ok, the Exodia metaphor is a little broken.
With Exodia, each card on its own is completely worthless until you have all five.
The entire point of this roadmap is that each feature is useful independently.
A better metaphor would be "Blue Eyes White Dragon":
= 
...but we have five features, not three.
And we all know how that turned out anyway.
What will we have achieved?
Projects accomplish a Travis/Circle CI -like workflow. For simpler projects this may be all you need.
At this point, the user will be introduced to resources, tasks, and build plans.
If and when they need to take the next step, then they can start using pipelines and adopt a "git ops" workflow.
By then, the only new concept is passed constraints.
Projects allow pipelines to reference each other in passed constraints.
This way users can configure independent pipelines for 'matrix' style workflows, i.e. testing across IaaSes.
The set_pipeline step allows for pipelines to be configured as part of a build plan.
Along the way, we can deprecate the concourse-pipeline resource, which has two smelly problems:
authenticating with Concourse
keeping fly in sync with your Concourse version
Pipeline instances allow for a common pipeline template to use some of its ((vars)) as part of the pipeline identifier.
This removes a lot of the need for 'hierarchical pipelines', which has been trotted around as an idea for a long time.
Pipeline instances should feel less complicated than arbitrary hierarchies (depth 2 is simpler than depth N).
The across step is the true 'root' of spatial automation. And because it just happens within a single build, it's a lot easier to reason about.
By composing the across step with the set_pipeline step, the mechanics of spatial pipelines should also be pretty easy to reason about.
For a long while now 'spaces' has been thought of as one big feature with a whole lot of implications.
By thinking about it as the intersection of five smaller features, our roadmap is heavily de-risked.
Resources v2: let's finally implement this
set_pipeline step: deprecate concourse-pipeline resource
Projects: smoothen learning curve
Archiving pipelines: nice-to-have
Instanced pipelines: automatic archiving, figure out nav
across step: open the spatial floodgates