Learning Prometheus, Grafana, and Loki with Docker

If you’ve read this blog for any length of time, you’ll know that I’m a big fan of Splunk, and using Splunk to solve everyday problems. But while Splunk excels at being able to eat nearly any kind of event or data, sometimes you need a more specific tool when working with a specific type of data.

That’s where Prometheus, Loki, and Grafana all come in. Prometheus is a time series database built for storing metrics. Loki is a log collection system which scales horizontally and is useful for collecting application logs, and Grafana is the dashboard app which is used to view metrics from either platform!

I wanted to learn more about each of these apps, and I figured the best way to do so was to build out something in Docker that let me ingest data immediately, and then to build some sample dashboards on top of that. I then open sourced it, and the entire project can be found at https://github.com/dmuth/grafana-playground

Getting Started

First, clone the repo and start up all of the Docker containers:

git clone https://github.com/dmuth/grafana-playground.git
cd grafana-playground
docker-compose up -d

This will start up several containers, some of which will ingest data, some of which will store data.

Next, go to http://localhost:3000/ and log in with the username/password combination of . You can change the password if you want. From there, you'll want to go into the API configuration section and create a new API key with admin access. Be sure to note what the API key is, as you're gonna need it shortly.

Finally, you’ll need to import the pre-built dashboards and data sources into Grafana, and this can be done with a special Docker container called , which can be used to access the rest of the environment. Here's how to get into that container and import those things:

docker-compose exec tools bash # Start a bash shell in the tools container
cat /mnt/config/dashboards.json | /mnt/bin/manage-dashboards.py --import --api-key API_KEY
/mnt/bin/manage-data-sources.py --api-key API_KEY
exit # Exit the tools container

Obviously, replace with the API key you got above.

At this point, data sources for Loki and Prometheus have been loaded, along with all pre-built dashboards. You can view them at http://localhost:3000/dashboards immediately.

Viewing Dashboards

Now that you’ve imported all of the pre-built dashboards, here’s a list of available dashboards:

Exporting Dashboards

If you want to export any dashboards that you changed, here’s how to do that:

docker-compose exec tools bash # Spawn a bash shell in tools container
/mnt/bin/manage-dashboards.py --export --api-key API_KEY > /mnt/dashboards.json
exit # Leave tools container

Running Ad-hoc Queries in Grafana

To run a specific query, click the Compass on the left side of any Grafana screen which puts you into Explorer Mode. Then paste in this query: . That should immediately show you the most recent logs that have been written. If this shows nothing, then data is not making it into Loki.

Manually Injecting Logs

If you want to manually inject an arbitrary number of logs, that can be done with this command:

docker-compose run logs n

Replace with the number of logs you want to write. They will go into the file in the logs volume, which will then be picked up by the promtail container. They can be viewed in Grafana with this query:

{filename=~"/logs/synthetic/manual.log"}

Changing Which Hosts are Pinged

  • Edit
  • Change the variable for the container.
  • Restart the container with
  • Current hosts being pined can be inspected with (adjust the container name accordingly).

Command Line Utilities

If you want to query Loki directly, I write a command-line script for that:

- Query the Dockerized instance of Loki on the command line.

Examples:

Closing Thoughts

There’s more I can go into, but it’s mostly content that is highly technical. So instead I’d like to refer you to the GitHub project at https://github.com/dmuth/grafana-playground, which contains a breakdown of all Docker containers and their functions, a FAQ, and more!

Feel free to let me know what you think in the comments below, or file an issue on the GitHub project!

(Original Blog post)