Learning Prometheus, Grafana, and Loki with Docker

If you’ve read this blog for any length of time, you’ll know that I’m a big fan of Splunk, and using Splunk to solve everyday problems. But while Splunk excels at being able to eat nearly any kind of event or data, sometimes you need a more specific tool when working with a specific type of data.

That’s where Prometheus, Loki, and Grafana all come in. Prometheus is a time series database built for storing metrics. Loki is a log collection system which scales horizontally and is useful for collecting application logs, and Grafana is the dashboard app which is used to view metrics from either platform!

I wanted to learn more about each of these apps, and I figured the best way to do so was to build out something in Docker that let me ingest data immediately, and then to build some sample dashboards on top of that. I then open sourced it, and the entire project can be found at https://github.com/dmuth/grafana-playground

Getting Started

First, clone the repo and start up all of the Docker containers:

git clone https://github.com/dmuth/grafana-playground.git
cd grafana-playground
docker-compose up -d

This will start up several containers, some of which will ingest data, some of which will store data.

Next, go to http://localhost:3000/ and log in with the username/password combination of admin/admin. You can change the password if you want. From there, you'll want to go into the API configuration section and create a new API key with admin access. Be sure to note what the API key is, as you're gonna need it shortly.

Finally, you’ll need to import the pre-built dashboards and data sources into Grafana, and this can be done with a special Docker container called tools, which can be used to access the rest of the environment. Here's how to get into that container and import those things:

docker-compose exec tools bash # Start a bash shell in the tools container
cat /mnt/config/dashboards.json | /mnt/bin/manage-dashboards.py --import --api-key API_KEY
/mnt/bin/manage-data-sources.py --api-key API_KEY
exit # Exit the tools container

Obviously, replace API_KEY with the API key you got above.

At this point, data sources for Loki and Prometheus have been loaded, along with all pre-built dashboards. You can view them at http://localhost:3000/dashboards immediately.

Viewing Dashboards

Now that you’ve imported all of the pre-built dashboards, here’s a list of available dashboards:

• Ping Results — Shows ping time and packet loss for specified hosts. The hosts can be changed.
• Ping Results, but from Prometheus — Similar to the original ping dashboard, this pulls metrics from Prometheus, which are aggregated, and the results will be in lower resolution.
• Syslog Volume — Covers syslog, synthetic logs, and ping events.
• Docker Logs — This playground ingests logs from its own Docker containers, which can be viewed here.
• Loki Stats — Statistics on the Loki Database
• Promtail Stats — Statistics on the Promtail instance
• Docker Host Stats — System Metrics from Prometheus (fed in by Telegraf)
• SEPTA Regional Rail Stats — Stats on SEPTA Regional Rail

Exporting Dashboards

If you want to export any dashboards that you changed, here’s how to do that:

docker-compose exec tools bash # Spawn a bash shell in tools container
/mnt/bin/manage-dashboards.py --export --api-key API_KEY > /mnt/dashboards.json
exit # Leave tools container

Running Ad-hoc Queries in Grafana

To run a specific query, click the Compass on the left side of any Grafana screen which puts you into Explorer Mode. Then paste in this query: { filename=~"/logs/synthetic/.*" }. That should immediately show you the most recent logs that have been written. If this shows nothing, then data is not making it into Loki.

Manually Injecting Logs

If you want to manually inject an arbitrary number of logs, that can be done with this command:

docker-compose run logs n

Replace n with the number of logs you want to write. They will go into the file /logs/synthetic/manual.log in the logs volume, which will then be picked up by the promtail container. They can be viewed in Grafana with this query:

{filename=~"/logs/synthetic/manual.log"}

Changing Which Hosts are Pinged

• Edit docker-compose.yml
• Change the HOSTS variable for the ping container.
• Restart the ping container with docker-compose kill ping; docker-compose up -d ping
• Current hosts being pined can be inspected with docker inspect grafana-playground_ping_1 | jq .[].Config.Env (adjust the container name accordingly).

Command Line Utilities

If you want to query Loki directly, I write a command-line script for that:

./bin/query.sh - Query the Dockerized instance of Loki on the command line.

Examples:

• ./bin/query.sh '{job="logs-ping"}'
• ./bin/query.sh '{job="logs-ping"}' 5
• ./bin/query.sh '{job="logs-ping",host="docker"}'
• ./bin/query.sh '{job="logs-ping",filename="/logs/ping/google.com.log"}'
• ./bin/query.sh '{job="logs-ping",filename=~"/logs/ping.*"}'
• ./bin/query.sh '{job="logs-ping",filename=~"/logs/ping.*"}' 10

Closing Thoughts

There’s more I can go into, but it’s mostly content that is highly technical. So instead I’d like to refer you to the GitHub project at https://github.com/dmuth/grafana-playground, which contains a breakdown of all Docker containers and their functions, a FAQ, and more!

Feel free to let me know what you think in the comments below, or file an issue on the GitHub project!

(Original Blog post)