What is Docker BuildKit and What can I use it for?

BuildKit has been lingering in the background of Docker builds for some time now as an experimental feature. Since 19.03, with an environment variable BuildKit can be enabled and unleash some massive performance features.

The standard Docker build command performs builds on Dockerfiles serially, which means it reads and builds each line or layer of the Dockerfile one layer at a time. When BuildKit is enabled, parallel build processing is allowed, resulting in better performance and faster build times.

BuildKit is a new project under the Moby umbrella for building and packaging software using containers. It’s a new codebase meant to replace the internals of the current build features in the Moby Engine.

Key BuildKit Features according to the Moby BuildKit Project

• Automatic garbage collection
• Extendable frontend formats
• Concurrent dependency resolution
• Efficient instruction caching
• Build cache import/export
• Nested build job invocations
• Distributed workers
• Multiple output formats
• Pluggable architecture
• Execution without root privilege

To summarize, BuildKit has better performance and uses the same docker build interface, which we are already familiar with. Additionally, BuildKit enables the use of cache and storing cache in remote container repositories like DockerHub for better build performance, as we don't have to rebuild every layer of an image.

BuildKit for Development

Since BuildKit is so easy to enable, I recommend enabling it everywhere to take advantage of the powerful features. The best place to start is with your Development environment to better understand the benefits. BuildKit will be a pleasant upgrade, unlocking more use cases for building images.

Start early and start in development. Enable BuildKit now to understand how it will work best with your projects. It is an easy way to start, and then you can keep moving BuildKit forward in your Development environment to CI/CD as a next step.

Using BuildKit in your CI/CD Pipeline

Now that we understand the background of BuildKit, let's take a look at why we should enable It inside a CI/CD pipeline. Since most of the available CI/CD tools, such as CircleCI, GitLab, TravisCI, etc., use or can use Docker to manage jobs, this is the perfect use case for enabling BuildKit.

Time is money and BuildKit will help reduce build time. Do I need to go further? Honestly, I enabled BuildKit on several projects and I keep rolling it out to more projects and have seen vast improvements in build times using GitLab CI. Timings will vary based on how your Dockerfile is written and configuration of your CI/CD servers.

Even if you use the SaaS version of your CI/CD tools, you can still enable BuildKit for your jobs. Cool, right?

Enable BuildKit

BuildKit can be enabled in two different ways. Enable BuildKit simply by setting an environment variable or configure Docker daemon to use BuildKit as default for all builds. BuildKit is only available for Linux and macOS.

The benefit of enabling BuildKit with a variable is maybe you only want to enable BuildKit for environment or a specific build. Set the environment variable by export DOCKER_BUILD=1.

To enable BuildKit when running a Build:

$ DOCKER_BUILDKIT=1 docker build .

Another option is setting BuidKit as a global setting for the entire Docker host by adding BuildKit to the Docker daemon. You can do this two different ways.

1. Edit the daemon file directly /etc/docker/daemon.json and add the below

{
"experimental" : false,
"debug" : true,
"features":{
"buildkit" : true}
}

2. If you are using Docker for Mac for development, you can perform this directly via the Docker menu. Docker for Mac Menu -> Preferences ->Daemon -> Advanced

Copy the settings into the Docker Daemon configuration and click Apply & Restart.

Don't wait any longer. Enable BuildKit now!

What's Next? buildx

Now that we understand BuildKit's benefits, we can take our builds even further. What, further? Yes, you read that correctly.

BuildX sounds like something from an Elon Musk project. However, buildx is an expansion of BuildKit, enabling even further features. buildx is still experimental, but from my initial testing and now using it in production, it works really well, and I am still learning all the possible features.

The greatest feature that fits my use cases is using the builders to isolate build enviornments and build mulit-architecutre images.

buildx Features

• Familiar UI from docker build
• Full BuildKit capabilities with container driver
• Multiple builder instance support
• Multi-node builds for cross-platform images
• Compose build support
• WIP: High-level build constructs (bake)
• TODO: In-container driver support

Jiang Huan wrote a great article about how they are using BuildKit combined with builds and leveraging build cache to significantly reduce build times. I am still researching all the possibilities and will report the details of using both BuildKit and build and some more use cases. Stay tuned for more findings.

Read Jiang's findings below as he explains how to set up each component and details how to leverage build caching.

Docker build cache sharing on multi-hosts with BuildKit and buildx

To share how BuildKit is used along with buildx to speed up our image build jobs on multiple hosts such as shared Gitlab CI runners. Currently in Titansoft, we are on a journey of infrastructure…

Titansoft Engineering BlogJiang Huan

FAQ Section for "What is Docker BuildKit?"

What is Docker BuildKit?

Docker BuildKit is a feature available in Docker 19.03 and later, designed to enhance build performance and efficiency through parallel processing and advanced caching.

How does Docker BuildKit improve build performance?

BuildKit enables parallel processing of Dockerfile instructions, resulting in faster builds and improved performance compared to traditional serial processing.

How can I enable Docker BuildKit?

You can enable BuildKit by setting the environment variable DOCKER_BUILDKIT=1 before running your build command, or by configuring the Docker daemon.

What are the key features of Docker BuildKit?

Key features include automatic garbage collection, efficient instruction caching, concurrent dependency resolution, and support for multiple output formats.Is Docker BuildKit available for all operating systems?

BuildKit is available for Linux and macOS. It can be enabled either through environment variables or Docker daemon configuration.

Follow me

If you liked this article, Follow Me on Twitter to stay updated!