Kubernetes Fundamentals

Deploying Jenkins with Kubernetes

Adding Persistent Volumes to Jenkins with Kubernetes

Previous Next

Exploring Kubernetes Init Container, ConfigMap & Secrets

by John Turner

Posted on February 22, 2017

Kubernetes, Docker, Jenkins

At the end of my last post I created a docker image for Jenkins that extended the official docker image by disabling the setup wizard, installing the default plugins and installing maven. I skipped over:

• setting up ssh for GitHub.
• automating the configuration of the Jenkins job(s).
• creating appropriate Jenkins slave images.

In this post I’ll describe how to set up SSH for GitHub.

Setting Up SSH for GitHub

To set up SSH for GitHub I created a ConfigMap containing the ssh config. I did this by creating the ConfigMap from a file.

Host github.com
    HostName ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/id_jenkins_rsa

kubectl create configmap jenkins-ssh-config --from-file=.ssh/config --from-file=.ssh/known_hosts

The SSH config specifies the identity to use as ~/.ssh/id_jenkins_rsa. As the private key is something I want to secure, I will generate the key and store it as a Kubernetes secret. First, follow the instructions on GitHub to generate a new ssh key and add it to your Github account. Then you can store the private and public key as a Kubernetes secret.

kubectl create secret generic jenkins-ssh-key --from-file=id_jenkins_rsa=.ssh/id_jenkins_rsa --from-file=id_jenkins_rsa.pub=.ssh/id_jenkins_rsa.pub

Now that we have made the SSH config and keys available to Kubernetes we need to configure the Jenkins container. I’ve done this using the Kubernetes Init Container feature which is in beta as of version 1.5.

Building a Custom Jenkins Docker Image - Part 2

by John Turner

Posted on February 15, 2017

Kubernetes, Docker, Jenkins

In Building a Custom Jenkins Docker Image - Part 1 we disabled the Jenkins install wizard and installed the default plugins. In part 2, I want to focus on ensuring the build tools are installed. The target end state of our build infrastructure must include the installation and configuration of:

• Java
• Jenkins
• Jenkins plugins
• Jenkins jobs (using JobDSL)
• Git
• Apache Maven

We can review the official Jenkins image source on the jenkinsci docker GitHub repository to understand which of these tools are already provided for. From the Dockerfile we can see the following:

FROM openjdk:8-jdk

RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/lists/*

This means that the official Jenkins image is based on the openjdk:8-jdk image (Java 8 JDK is installed) and that the Git client is installed. There is nothing to suggest that Maven is installed and we can verify this by creating a new bash session in the container and listing the installed packages.

$ kubectl exec jenkins-155111175-p72qf -it -- bash
jenkins@jenkins-155111175-p72qf:/$ apt list --installed
Listing... Done
acl/now 2.2.52-2 amd64 [installed,local]
...
lsb-base/now 4.1+Debian13+nmu1 all [installed,local]
mawk/now 1.3.3-17 amd64 [installed,local]
mercurial/now 3.1.2-2+deb8u3 amd64 [installed,local]
mercurial-common/now 3.1.2-2+deb8u3 all [installed,local]
mime-support/now 3.58 all [installed,local]
mount/now 2.25.2-6 amd64 [installed,local]
multiarch-support/now 2.19-18+deb8u7 amd64 [installed,local]
ncurses-base/now 5.9+20140913-1 all [installed,local]
...
zlib1g/now 1:1.2.8.dfsg-2+b1 amd64 [installed,local]
jenkins@jenkins-155111175-p72qf:/$

To install maven we use the Docker RUN statement in the Jenkins/Dockerfile to invoke apt-get install. This requires us first to become the root user and subsequently to resume as the jenkins user.

# install Maven
USER root
RUN apt-get update && apt-get install -y maven
USER jenkins

Building a Custom Jenkins Docker Image - Part 1

by John Turner

Posted on February 14, 2017

Kubernetes, Docker, Jenkins

Last week I spent some time learning how to utilize Kubernetes and Jenkins to form the foundation of a build infrastructure. I documented some of those learnings in the posts below:

• Kubernetes Fundamentals
• Deploying Jenkins with Kubernetes
• Adding Persistent Volumes to Jenkins with Kubernetes
• Auto-Scaling Jenkins with Kubernetes

This is a pretty good start but having built out VM based build infrastructure a number of times in the past I’m well aware that there is a long way to go before I have something I can use, manage and maintain. When working with VM’s I’ve perviously chosen to use Chef to manage and maintain the Jenkins master and slave hosts. Typically, Chef facilitated automation of the installation and configuration of:

• Java
• Jenkins
• Jenkins plugins
• Jenkins jobs (using JobDSL)
• Git
• Apache Maven

To achieve the same level of automation with Docker and Kubernetes, I will need to be able to perform all of the above and distribute as a set of Kubernetes resource definition files and Docker image(s). Before we start doing anything meaningful, I want to disable the setup wizard because, after all, we will be automating the Jenkins setup. To do this I modify the Kubernetes deployment resource file to specify a JAVA_OPTS environment variable.

Auto-Scaling Jenkins with Kubernetes

by John Turner

Posted on February 09, 2017

Kubernetes, Docker, Jenkins

Jenkins is a great piece of software (ok, it has problems but I couldn’t imagine software development without it). But one of the challenges with maintaining a Jenkins cluster is capacity management. It’s fairly typical to start out with a single master instance. Over time the number and size of Jenkins jobs increases placing more and more demand on the server. The first fix people apply when this happens is to vertically scale the Jenkins server (In fact I recently interviewed for a position and was told their Jenkins server hardware had 40 cores and 512GB of RAM). Some of the problems with scaling vertically include:

• Cost per unit of scale increases with the size of the hardware.
• Complex software configuration required to support a large variety of job types.
• Greater risk of ‘noisy neighbours’ impacting:
  • job performance.
  • server stability.

The alternatives to vertically scaling the Jenkins master are to:

• Deploy multiple Jenkins masters allocated:
  • per environment.
  • per organisational unit.
  • per product line.
• Deploy statically provisioned Jenkins slaves.
• Deploy dynamicaly provisioned Jenkins slaves.
• Deploy multiple Jenkins masters with statically or dynamically provisioned Jenkins slaves.

There are a few things to consider when choosing how to scale your Jenkins infrastructure. If you choose to deploy multiple master you should have an efficient and effective way to manage them. At a minimum, you should use configuration management or orchestration tooling to manage the lifecycle of the instances themselves. You should also consider similar for managing plugins, jobs etc. In the past I have had great success using tools like Chef and Job DSL to manage build infrastructure.

If you choose to use slaves, consider if you should provision bloated slaves capable of performing any build Job or if you should provision specialized slaves.

Adding Persistent Volumes to Jenkins with Kubernetes

by John Turner

Posted on February 08, 2017

Kubernetes, Docker, Jenkins

Having had a first attempt at Deploying Jenkins with Kubernetes I’ll next focus on incrementally making the Jenkins deployment more robust. The first thing that I’ll investigate is how I can preserve the contents of the JENKINS_HOME directory. Jenkins stores all of its important information within the JENKINS_HOME such as:

• build server configuration.
• build jobs.
• build artifacts.
• user accounts.
• user installed plugins.

By default disk files in a container are ephemeral. This means that when the Jenkins container fails and is recovered (or indeed when an upgrade occurs) that the data within the JENKINS_HOME directory is lost forever. It would be desirable to have the JENKINS_HOME directory preserved across failure, recovery and upgrade processes. This can be achieved using Kubernetes Volumes.

Container Filesystem

We cannot talk about Kubernetes volumes without understanding the Docker filesystem. The Docker filesystem is composed of a number of layers starting with one or more read only image layers. When a container is created, a read/write container layer is added to the top of the stack. When a file is read from the filesystem each layer is inspected from the top of the stack to the bottom until the file is found or the bottom of the stack is reached. When a file is modified it is first copied to the top of the stack and so the next time it is read the modified file is found first.