Learning Prometheus, Grafana, and Loki with Docker

Getting Started

git clone https://github.com/dmuth/grafana-playground.git
cd grafana-playground
docker-compose up -d
docker-compose exec tools bash # Start a bash shell in the tools container
cat /mnt/config/dashboards.json | /mnt/bin/manage-dashboards.py --import --api-key API_KEY
/mnt/bin/manage-data-sources.py --api-key API_KEY
exit # Exit the tools container

Viewing Dashboards

Exporting Dashboards

docker-compose exec tools bash # Spawn a bash shell in tools container
/mnt/bin/manage-dashboards.py --export --api-key API_KEY > /mnt/dashboards.json
exit # Leave tools container

Running Ad-hoc Queries in Grafana

Manually Injecting Logs

docker-compose run logs n
{filename=~"/logs/synthetic/manual.log"}

Changing Which Hosts are Pinged

  • Edit docker-compose.yml
  • Change the HOSTS variable for the ping container.
  • Restart the ping container with docker-compose kill ping; docker-compose up -d ping
  • Current hosts being pined can be inspected with docker inspect grafana-playground_ping_1 | jq .[].Config.Env (adjust the container name accordingly).

Command Line Utilities

  • ./bin/query.sh '{job="logs-ping"}'
  • ./bin/query.sh '{job="logs-ping"}' 5
  • ./bin/query.sh '{job="logs-ping",host="docker"}'
  • ./bin/query.sh '{job="logs-ping",filename="/logs/ping/google.com.log"}'
  • ./bin/query.sh '{job="logs-ping",filename=~"/logs/ping.*"}'
  • ./bin/query.sh '{job="logs-ping",filename=~"/logs/ping.*"}' 10

Closing Thoughts

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Douglas Muth

Douglas Muth

77 Followers

Engineer. AWS, CyberSec, DMARC, Docker, Splunk, White Mage. Staffs way too many cons. he/him. 28% Cheetah.