Concourse v10

Resources v2

an abridged update

Resources v2 (cont'd)

Resources v2 is now an idea of a 'generalized resource interface', decoupled from artifact versioning.

Users will be able to use v1 and v2 resources side-by-side, and users won't even have to upgrade to v2 to use new Concourse features.

It's just a refinement of the same idea.

For these slides, remember that 'version' is now called 'config fragment'.

RIP Spaces


we hardly knew ye

What was spaces? (cont'd)

Behind the name, there are a few great features:

These still make sense. So why is 'spaces' dead?

Analysis paralysis (cont'd)

Our first approach was to extend resources and jobs to support finding versions and running builds across spaces.

This approach proved to be strategically expensive.

We thought of it as one big feature, and we baked it into the entire stack, from the web UI down to the resource protocol.

Having a single feature touch so many things led to lot of second-guessing and many different iterations.

Learning to walk again (cont'd)

Can we break this big feature up?

Recently we were able to decouple 'Resources v2' from 'Spaces' by taking a completely different approach, leveraging composition instead of directly coupling them.

Doing this was liberating: suddenly we could deliver two big features independently instead of all at once.

Can we do the same for 'spaces'?

Hold my hand... (cont'd)

Let's go on a journey.

I've got a few features to propose, which I hope are independently useful.

When they're used together, they become way more powerful.

Hang in there - the dots should connect as we go along!

Archiving Pipelines

Archiving Pipelines (cont'd)

Pipelines can be archived when they are no longer needed.

When archived, a pipeline becomes completely inactive, and its name can be used for later pipelines. This is basically a soft-delete.

$ fly archive-pipeline -p pipeline-name

This is a pretty simple feature on its own, but it's also a precursor to "instanced pipelines."

Complexity estimate (cont'd)

Someone actually started on a PR for this a while back. It doesn't seem too hard.

API: small - support archiving/unarchiving, support re-using names of archived pipelines.

UX: medium - 'dim' and/or stash away archived pipelines.

Core: x-small - don't schedule archived pipelines.

Instanced Pipelines

Instanced Pipelines (cont'd)

Pipeline templates can be "instantiated" with ((vars)) that also become part of the pipeline's identifier.

$ fly set-pipeline -p release \
    -i version:5.3 \
    -c ci/pipelines/release.yml

This is a small UX change that allows simple pipeline hierarchies to form - 2 levels deep instead of N.

Complexity estimate (cont'd)

API: medium? - allow multiple instances of same pipeline name with different vars.

Core: x-small - schedule all pipeline instances.

UX: medium - probably a lot of navigation UX to figure out, but hopefully simpler than 'spaces'.

set_pipeline step

set_pipeline step (cont'd)

Pipelines can be configured by using a new set_pipeline step in a build plan:

- get: ci
- set_pipeline: concourse
  file: ci/pipelines/concourse.yml

The pipeline will be set within the team running the build, and will start un-paused.

set_pipeline + instances (cont'd)

The set_pipeline step can be used to 'sync' all instances of a pipeline:

- get: ci
- set_pipeline: concourse
  file: ci/pipelines/concourse.yml
- set_pipeline: release
  file: ci/pipelines/release.yml
  instance_vars: {version: 5.3}
- set_pipeline: release
  file: ci/pipelines/release.yml
  instance_vars: {version: 5.2}

set_pipeline + archiving (cont'd)

A build which uses set_pipeline for setting instances determines the full set of active instances for the pipeline.

All other instances will be automatically archived when the build completes.

This way release pipelines will automatically become archived when they're no longer relevant.

This will be more important later!

Complexity estimate (cont'd)

API: x-small - support parsing it as part of the build plan.

Core: small - implement the step. medium once we do instance archiving.

UX: small - support rendering the step.

across step

across: build matrixes (cont'd)

An across step runs a given step across all versions returned by a resource check:

across: supported-ruby-images
as: ruby
- task: unit
  image: ruby
  file: ci/tasks/unit.yml

The across step can be nested to do N-dimensional matrixes.

across: pipeline matrixes (cont'd)

By composing these two steps and using instance_fragment , we now have pipeline matrixes:

across: concourse-branches
as: branch
- set_pipeline: branch
  instance_fragment: branch
  file: ci/pipelines/branch.yml

When a branch goes away, its instance will become archived.

across with trigger (cont'd)

The across step will support trigger: true. This way the build will fire on any change to the set.

across: concourse-branches
as: branch
trigger: true
- set_pipeline: branch
  instance_fragment: branch
  file: ci/pipelines/branch.yml

Complexity estimate (cont'd)

API: x-small - support it as part of the build plan.

Core: medium/large - implement a new type of scheduling and triggering, similar to get step handling but for sets of versions. Construct a build plan that repeats the step with a get for each version.

UX: medium - show all the steps, probably in some sort of tab UI. There's some prior art with attempts:.


What is a project? (cont'd)

A project is a namespace for pipelines, resources, and tasks.

A project is bootstrapped by a resource containing the project's configuration, along with its tasks, resources, and pipelines.

Many projects may exist within a team.

fly set-project (cont'd)

Projects are configured with fly set-project:

$ fly -t ci set-project \
    --project booklit \
    --type git \
    --source uri= \
    --config-path ci

The --config-path flag identifies the sub-directory within the resource from which to load the project config.

Project structure (cont'd)

Concourse will check for new versions of the project and load the project's config, resources, tasks, etc.:


Pipelines shrink down to just job definitions, removing hundreds of lines of YAML. Separate files are much easier to work with.

It's just a build plan! (cont'd)

A minimal project configuration contains a name and a plan:

name: booklit
- task: test

The project's plan: will run every time a new version of the project is found, with the project resource itself available under the configured name.

Kinda like Travis/Circle CI.

Git Ops (cont'd)

Projects, when combined with the set_pipeline step, allow your entire project to be automated and reproducible.

It's a full blown build plan, you can even do pipeline templating if necessary:

name: ci
- task: generate-template
- set_pipeline: fancy-templated-pipeline
  file: generated-pipeline/foo.yml

Pipelines within projects (cont'd)

Pipelines within a project change in a few ways:

Simpler build plans (cont'd)


- task: fly-linux
  file: ci/tasks/fly-linux.yml
- task: fly-darwin
  file: ci/tasks/fly-darwin.yml
- task: fly-windows
  file: ci/tasks/fly-windows.yml


- task: fly-linux
- task: fly-darwin
- task: fly-windows

This reads a lot better, builds on existing usage patterns, and task names are actually useful now!

Project-level creds (cont'd)

Projects define credential managers as var_sources:

name: ci

- type: vault
  config: # ...

plan: # ...

The proximity to plan: makes it easy to visibly audit credential access within the project.

Complexity estimate (cont'd)

API: medium - there's a new object owner in town, so I expect some API noise. Maybe we should invest in non-hierarchical URLs at this point?

Core: medium/large - this is a bit of an epic, but it's at least grounded in well-understood ideas (i.e. build plans). I think there are interesting, relatively simple ways to design this.

UX: medium/large - we'll need project builds to show up in the UI somewhere. And we'll probably want to think about any navigation implications.

Putting it all together...

Spaces? (cont'd)

Circling back to 'spaces', let's see how these features add up:

Demo 1: simple project (cont'd)

name: booklit
- task: test
fly set-project -p booklit -t git \
  -s uri= \
  -c ci

This should feel intuitive for smaller projects which may not need a sophisticated pipeline system. Concourse has been described as 'overkill' for such use cases - hopefully this bridges the gap.

Demo 2: git ops project (cont'd)

name: ci
- type: vault
  config: # ...
- set_pipeline: concourse
- set_pipeline: prs

In this case it might make more sense for prs to be a separate project so it doesn't share the credential manager config.

Demo 3: spatial pipelines (cont'd)

name: ci
- set_pipeline: concourse
- across: release-branches
  as: branch
  - set_pipeline: release
    instance_fragment: branch
- across: feature-branches
  as: branch
  - set_pipeline: branch
    instance_fragment: branch

your metaphor sucks

a brief side-tangent

Broken metaphor (cont'd)

...Ok, the Exodia metaphor is a little broken.

With Exodia, each card on its own is completely worthless until you have all five.

The entire point of this roadmap is that each feature is useful independently.

Better metaphor? (cont'd)

A better metaphor would be "Blue Eyes White Dragon":


...but we have five features, not three.

Better metaphor? (cont'd)

And we all know how that turned out anyway.


What will we have achieved?

Scales with more projects (cont'd)

Projects accomplish a Travis/Circle CI -like workflow. For simpler projects this may be all you need.

At this point, the user will be introduced to resources, tasks, and build plans.

If and when they need to take the next step, then they can start using pipelines and adopt a "git ops" workflow.

By then, the only new concept is passed constraints.

Inter-dependent pipelines (cont'd)

Projects allow pipelines to reference each other in passed constraints.

This way users can configure independent pipelines for 'matrix' style workflows, i.e. testing across IaaSes.

Pipeline automation (cont'd)

The set_pipeline step allows for pipelines to be configured as part of a build plan.

Along the way, we can deprecate the concourse-pipeline resource, which has two smelly problems:

Pipeline hierarchies (cont'd)

Pipeline instances allow for a common pipeline template to use some of its ((vars)) as part of the pipeline identifier.

This removes a lot of the need for 'hierarchical pipelines', which has been trotted around as an idea for a long time.

Pipeline instances should feel less complicated than arbitrary hierarchies (depth 2 is simpler than depth N).

Build/pipeline matrixes (cont'd)

The across step is the true 'root' of spatial automation. And because it just happens within a single build, it's a lot easier to reason about.

By composing the across step with the set_pipeline step, the mechanics of spatial pipelines should also be pretty easy to reason about.

De-risked roadmap (cont'd)

For a long while now 'spaces' has been thought of as one big feature with a whole lot of implications.

By thinking about it as the intersection of five smaller features, our roadmap is heavily de-risked.

So...what's the roadmap?