How to Monitor a Traefik Reverse Proxy with Prometheus
Recently, I started using the reverse proxy Traefik as a default for my projects. Out of the box, Traefik supports Docker Swarm, Kubernetes, and many others. What's unique about Traefik compared to NGINX or Apache is that it dynamically listens to your Orchestrator like Docker and knows each time a container is added, removed, killed or upgraded, and can generate its configuration automagically. Really? Tell me more.
I'm running Docker as an Orchestrator (obviously). Traefik listens to the Docker API for containers starting or stopping on Docker and it dynamically registers this information with Traefik. How cool is that? Traefik comes out of the box with nice dashboards for your services and the health of these services.
The Traefik dashboards are nice but I want this information in a central location with the rest of my monitoring information. After quite some digging through Traefik configurations, GitHub issues, and tons of side discussions I finally found that Traefik exports metrics in a Prometheus friendly format. Wow, this is great news.
The time has come to start playing with Docker (My favorite time). We will launch 2 separate Docker stacks. Stack 1 will contain our Prometheus stack for monitoring Traefik and Stack 2 will be the Traefik stack. This might be new for some as we will join the Traefik container to the Prometheus network. This allows us to have 2 different docker-compose files yet all containers can interact with each other via Docker networking. Cool, right?
First, we need to clone the Traefik repo
$ git clone https://github.com/vegasbrianc/docker-traefik-prometheus
Let's quickly take a look at the
docker-compose.yml file to understand how Traefik will communicate to Prometheus. The
docker-compose.yml file is very simple in terms of compose files go.
command key is where both the Docker endpoint and Prometheus metrics are enabled.
At the bottom of the docker-compose file is the
networks section. Here is where we connect to the network of the Prometheus stack. This connects the Traefik container to the Prometheus network or additional networks if desired.
version: '3' services: traefik: image: traefik command: --web --web.metrics.prometheus --web.metrics.prometheus.buckets="0.1,0.3,1.2,5.0" --docker --docker.domain=docker.localhost --logLevel=DEBUG networks: - prometheus ports: - "80:80" - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock networks: prometheus: external: name: dockerprometheus_back-tier
Time to start Traefik
From the Traefik directory let's start the Traefik stack.
$ docker-compose up -d
Once Traefik is up and running open a browser tab and open the Trafik Dashboard with
http://0.0.0.0:8080Since nothing is running the dashboard should be blank. Click on the
health tab at the top of the page and this will display some basic health statistics about Traefik.
Next, we need to deploy a Prometheus stack. Lucky for you I have a Docker stack containing Docker, Prometheus, and Grafana ready for such a task.
Clone the below repository to your Docker host. This repository contains instructions in case you have never set this up Prometheus before.
$ git clone https://github.com/vegasbrianc/prometheus
Before we can launch the stack we need to adjust the targets which Prometheus will scrape. A target is an endpoint which Prometheus listens to for metrics.
In the Prometheus project directory edit the following file
Navigate to the end of the file and replace localhost with Traefik's metric endpoint
static_configs: - targets ['traefik:8080','cadvisor:8080','node-exporter:9100']
Once we have made this change we can start the Prometheus stack. From the Prometheus project directory run:
$ docker-compose up -d
Now that both Traefik and Prometheus have been deployed, let's take a look at what Traefik is doing. Open up the Traefik dashboard again and have a look
http://0.0.0.0:8080. We should now see that Traefik has automatically populated the Providers tab of the dashboard with all the running containers. Traefik even assigned a local domain name to all of our services (Mind blown).
We can now choose a hostname from one of our services and access it directly via the hostname. For demo purposes, let's use Grafana as this is a service we know that is exposed. In your browser, open
http://grafana.dockerprometheus.docker.localhost. This should now open the Grafana login page.
If you have never setup Grafana be sure to check the setup instructions in the Prometheus Repo.
Setup the Traefik Grafana Dashboard
We covered a lot of ground. The next steps will start to show how this is all comes together inside Grafana. In the previous section, you configured Grafana and Prometheus as a Data Source.
Several starter dashboards are available via Grafana. I took the liberty of combining a couple dashboards into a new one to offer a little more functionality. Open the Grafana menu -> Dashboards -> Import. You will now be prompted with an import screen. Enter
2870 for the Dashboard ID to import. This will import the Dashboard directly from Grafana.
Next, select the Data Source for Prometheus which we created earlier and click Import
Great, now you should see the Traefik dashboard inside Grafana. Have a look around and don't forget to write a comment if you have a question.
If you liked this article be sure to Follow Me on Twitter to stay updated!