How to create metrics from logs with Loki and Grafana

August 23, 2023

I have an application on a raspberrypi, whose only job is to check the temperature in the bedrooms and adjust the heating accordingly. It also produces logs like these:

{"level":"debug","type":"room_status","room_name":"Bedroom 1","measured_temp":18.4}
{"level":"debug","type":"room_status","room_name":"Bedroom 2","measured_temp":17.4}
{"level":"info","type":"room_request","message":"the room temperature is lower than expected, requesting heating","room_name":"Bedroom 2"}

For my monitoring setup, I've done the following steps:

  • install Grafana
  • install Loki
  • register Loki as a Data Source in Grafana
  • push logs to Loki using Promtail

Logs are then accessible from Grafana:

And now the best part: With Loki & LogQL, I can write a simple query to extract metrics from the logs:

avg by (room_name) (avg_over_time({service="docker/demo"} |~ "room_status" | json | unwrap measured_temp [15m]))

And the result, when using the query with more data in a Grafana panel:

note: I noticed that one of the bedrooms rise to 27.5°C around 8PM each day. It's not a bug, it's only because it's the exact time with the perfect angle where the sun reach and overheat the sensor.

That's all, I was expecting it to be more complicated but it was quit straightforward to setup. From there, I am now able to easily create graphs and alerts for all my docker applications.
If you want to know more about the configuration and installation steps, I will go into more details in the next post.