What is Docker BuildKit and What can I use it for?

BuildKit has been lingering in the background of Docker builds for some time now as an experimental feature. Since 19.03, with an environment variable BuildKit can be enabled and unleash some massive performance features.

The standard Docker build command performs builds on Dockerfiles serially, which means it reads and builds each line or layer of the Dockerfile one layer at a time. When BuildKit is enabled, it allows for parallel build processing resulting in better performance and faster build times.

BuildKit is a new project under the Moby umbrella for building and  packaging software using containers. It’s a new codebase meant to  replace the internals of the current build features in the Moby Engine.

Key BuildKit Features according to the Moby BuildKit Project

• Automatic garbage collection
• Extendable frontend formats
• Concurrent dependency resolution
• Efficient instruction caching
• Build cache import/export
• Nested build job invocations
• Distributed workers
• Multiple output formats
• Pluggable architecture
• Execution without root privilege

To summarize, BuildKit has better performance and uses the same docker build interface which we are already familiar with. Additionally, BuildKit enables the use of cache and storing cache in remote container repositories like DockerHub for better build performance as we don't have to rebuild every layer of an image.

BuildKit for Development

Since BuildKit is so easy to enable I recommend enabling everywhere to take advantage of the powerful features. Best place to start is with your Development environment to better understand the benefits.  BuildKit will be a pleasant upgrade unlocking more use cases for building images.

Start early and start in development. Enable BuildKit now to understand how best it will work with your projects. It is an easy way to start and then you can keep moving BuildKit forward in your Development enviornment to CI/CD as a next step.

Using BuildKit in your CI/CD Pipeline

Now that we understand the background about BuildKit let's now take a look at why we should enable BuildKit inside a CI/CD pipeline. Since most of the available CI/CD tools like CircleCI, GitLab, TravisCI, etc use or can use Docker to manage your jobs. This is the perfect use case to enable BuildKit.

Time is money and BuildKit will help reduce build time. Do I need to go further? Honestly, I enabled BuildKit on several projects and I keep rolling it out to more projects and have seen vast improvements in build times using GitLab CI. Timings will vary based on how your Dockerfile is written and configuration of your CI/CD servers.

Even if you are using the SaaS version of your CI/CD tools you can still enable BuildKit for your jobs. Cool, right?

Enable BuildKit

BuildKit can be enabled in two different ways. Enable BuildKit simply by setting an environment variable or configure Docker daemon to use BuildKit as default for all builds. BuildKit is only available for Linux and macOS.

The benefit of enabling BuildKit with a variable is maybe you only want to enable BuildKit for environment or a specific build. Set the environment variable by export DOCKER_BUILD=1.

To enable BuildKit when running a Build:

$ DOCKER_BUILDKIT=1 docker build .

Another option is setting BuidKit as a global setting for the entire Docker host by adding BuildKit to the Docker daemon. You can do this two different ways.

1. Edit the daemon file directly /etc/docker/daemon.json and add the below

{
"experimental" : false,
"debug" : true,
"features":{
"buildkit" : true}
}

2. If you are using Docker for Mac, for developing you can perform this directly via the Docker menu. Docker for Mac Menu -> Preferences ->Daemon -> Advanced

Copy the settings into the Docker Daemon configuration and click Apply & Restart.

Don't wait any longer. Enable BuildKit now!

What's Next? buildx

Now that we understand the benefits of BuildKit we can actually take our builds even further. What, further? Yes, you read that correctly.

BuildX sounds like something from an Elon Musk project. However, buildx is an expansion of BuildKit enabling even further features. buildx is still experimental but from my initial testing and now using in production works really well and I am still learning all the possible features. The greatest feature that fits my use cases is using the builders to isolate build enviornments and build mulit-architecutre images.

buildx Features

• Familiar UI from docker build
• Full BuildKit capabilities with container driver
• Multiple builder instance support
• Multi-node builds for cross-platform images
• Compose build support
• WIP: High-level build constructs (bake)
• TODO: In-container driver support

Jiang Huan wrote a great article how they are using BuildKit combined with buildx and leveraging build cache has reduce buildtimes significantly. I am still researching all the possabilites and will report the details of using both BuildKit and buildx and some more use cases. Stay tuned for more findings.

Read Jiang's findins below as he expalins how to setup each components and details out how to leverage build caching.

Docker build cache sharing on multi-hosts with BuildKit and buildx

To share how BuildKit is used along with buildx to speed up our image build jobs on multiple hosts such as shared Gitlab CI runners. Currently in Titansoft, we are on a journey of infrastructure…

Titansoft Engineering BlogJiang Huan

Follow me

If you liked this article be sure to Follow Me on Twitter to stay updated!