ReadyTalk Pipeline DSL

If you’ve used Jenkins for any amount of time you might like its configurability and extensiblity but also understand that this flexibility poses serious drawbacks to his continued use. I feel as though Jenkins suffers mostly from a lack of a few fundamental software tooling patterns on a job configuration basis:

I’ve always enjoyed the model of TravisCI‘s in-repository yaml file build job configuration, which provides all these aspects… but wasn’t available for Jenkins jobs. I also appreciated the actual implementation of a Jenkins deployment pipeline using Gradle that Benjamin Muschko and Peter Niederwieser from Gradleware presented last year, but thought it could be a lot simpler to configure. So I spent some time a few months ago combining some publicly available libraries and tools to get something that I think has the potential to be even better than the TravisCI yaml file construct with introduce easy to manage CD pipelines in Jenkins.

 The Current State

Today, we have the ability to create multi-job deployment pipelines with the elegance of a few lines of Groovy via our own ReadyTalk Pipeline DSL:

pipeline {
  name "integration1"
}

This defines:

All these jobs are wired up correctly via upstream/downstream triggers and also expose a few parameters on the jobs themselves (with sensible defaults). So one could run the deployment job and specify a specific git branch to build and deploy.

This pipeline concept in our own dsl is easily configurable too, people could add more jobs for mid level integration tests too:

pipeline {
  environmentName = "integration1"

  create(type: VoipTestJob)
}

Which would actually create a job called integration1-voip-test and wire it up in the pipeline right after the deployment (running in parallel to the acceptance smoke test, since this voip test concept completes in less than 5 minutes).

Isn’t that neat? This is surely what I’ve always wanted after reading the Continuous Delivery book by Jez Humble and David Farley.

 How Does It Work?

 The Tools

A while ago I came across the jenkins-job-dsl project (thanks Netflix and Justin Ryan!) which provides an elegant way to provide a similar way to configure Jenkins jobs.

I put together a plugin to manage Jenkins jobs from a Gradle build, but then ran across Gary Hale’s excellent work with the Gradle Jenkins plugin (he now works for Gradleware).

 Repository Layout

We have a repository called “jenkins-jobs” that’s laid out something like this:

├── README.md
├── build.gradle
├── buildSrc
│   ├── build.gradle
│   └── src
│       ├── main
│       │   └── groovy
│       │       └── com
│       │           └── readytalk
│       │               └── jenkins
│       │                   ├── dsl (contains ReadyTalk dsl additions)
│       │                   │   ├── ...
│       │                   │   └── ReadyTalkJenkinsDsl.groovy
│       │                   └── jobs (contains the following and many more
│       │                                     job types)
│       │                       ├── BasicJob.groovy
│       │                       ├── DeployJob.groovy
│       │                       ├── ...
│       │                       ├── Pipeline.groovy
│       │                       └── VoipTestJob.groovy
│       └── test
├── gradle (contains standard Gradle files)
├── gradle.properties
├── gradlew
├── gradlew.bat
├── settings.gradle
└── src
    └── main
        └── jenkins
            └── jobs (contains .groovy files where each team/developer can
                                create their own pipelines/jobs)
               ├── dev.groovy
               └── ...

Where the majority of ReadyTalk engineers configure their pipelines and jobs in src/main/jenkins/jobs/dev.groovy Here’s my own pipeline job definition in that file:

import com.readytalk.jenkins.dsl.*
import com.readytalk.jenkins.jobs.*

use(ReadyTalkJenkinsDsl) {
  extensions {
    defaults {
      # all pipelines will run in the office Jenkins slave cluster
      datacenter = Datacenter.OFFICE
    }
  }

  pipeline {
    # this will reset my personal test cluster every night at 10PM (plus some
    # change)
    environmentName = "sgoings"
  }
}

The magic (where we actually inject our own DSL) comes via the ReadyTalkJenkinsDsl.groovy file in the buildSrc structure:

package com.readytalk.jenkins.dsl

import com.readytalk.jenkins.jobs.*
import javaposse.jobdsl.dsl.JobParent

@Category(JobParent)
class ReadyTalkJenkinsDsl {

  static ExtensionHandler extensions = new ExtensionHandler([:])

  ExtensionHandler extensions(Closure closure) {
    extensions.with(closure)

    return extensions
  }

  JobHandler pipeline(Closure closure) {
    create([type: Pipeline], closure)
  }

  JobHandler create(Map<String, String> options, Closure closure) {
    def job = options['type'].newInstance(jm)

    job.apply(extensions)

    job.with(closure)

    referencedJobs.addAll(job.generate())

    return job
  }

  JobHandler build(Closure closure) {
    create([type: BuildJob], closure)
  }
}

The build.gradle file at the root of the project directory is in charge of using the Gradle Jenkins plugin to create Jenkins pipelines + jobs from the src/main/jenkins/jobs/*.groovy files. It looks something like:

buildscript {
  dependencies {
    classpath (
      'com.terrafolio:gradle-jenkins-plugin:1.2.2'
    )
  }
}

apply plugin: "com.terrafolio.jenkins"

jenkins {
  servers {
    production {
      url "<our internal jenkins instance url>"
      secure true
      username "<our internal jenkins user>"
      password System.env.JENKINS_PASSWORD
    }
  }

  defaultServer servers.production

  dsl fileTree('src/main/jenkins/jobs').include('*.groovy')
}

The buildSrc/build.gradle file, which is in charge of creating the pipeline abstractions and job types (remember pipeline and VoipTestJob?) looks something like:

apply plugin: 'groovy'

dependencies {
  compile (
    gradleApi(),
    localGroovy()
  )

  compile('org.jenkins-ci.plugins:job-dsl-core:1.24')
}

Most of our Jenkins jobs within the pipeline dsl extend from the BasicJob.groovy file (stored in buildSrc/src/main/jenkins/...) which gives an idea of how we’re abstracting our DSL away from the jenkins-job centric dsl:

package com.readytalk.jenkins.jobs

import javaposse.jobdsl.dsl.Job
import javaposse.jobdsl.dsl.JobManagement
import com.readytalk.jenkins.dsl.*

class BasicJob extends Job implements JobHandler {

  @Delegate Defaults defaults
  String environmentName
  String server
  String email

  BasicJob(JobManagement jm, Map<String, String> arguments = [:]) {
    super(jm, arguments)

    description("""This job is managed by the Jenkins Pipeline DSL project.
This means that changes to the job config (including enabling/disabling the job) may be overwritten.
""")

    wrappers {
      colorizeOutput('xterm')
    }
  }

  void apply(ExtensionHandler extensions) {
    defaults = new Defaults(extensions.defaults)
  }

  void setEnvironmentName(String envName) {
    environmentName = envName
  }

  Set<Job> generate() {
    return [this]
  }

  void addDatacenter() {
    switch(datacenter) {
      case Datacenter.OFFICE:
        label("wheezy && office")
    ...
    }
  }

  void addDownstream(Job job) {
    addDownstream(job.name)
  }

  void addDownstream(String jobName) {
    blockOnUpstreamProjects()
    blockOnDownstreamProjects()
    publishers {
      downstreamParameterized {
        trigger(jobName, 'SUCCESS', true) {
          currentBuild()
        }
      }
    }
  }

}

 Next Steps

I realize that browsing through code on a blog is the least interesting thing and working code in your own hands is far more effective, so I’m actively working on making the ReadyTalk Pipeline DSL publicly available. Obviously our pipeline constructs won’t be exactly what you need, but it’ll at least get you started in defining fantastically easy-to-use and maintain pipelines for you and your company. I’ll publicize its location once available.

 
33
Kudos
 
33
Kudos

Now read this

If Only “They” Knew

Do you ever wonder why so much software in the world seems to be overly complex, incomprehensible, or just plain messy? And then, say, you contact “the owner” to address the situation and they see nothing wrong? Doesn’t that infuriate... Continue →