Michal Vaško is a DevOps engineer at cloudWerkstatt, with a passion for open source technology and a deep love for observability.
While operations or platform teams have long relied on visibility into metrics to react swiftly, the idea of doing the same thing with logs was once just a dream. Thankfully, Grafana Loki has revolutionized the logging stack, giving you the same level of visibility with logs that you get with metrics.
If you haven’t embarked on the Loki journey yet, now is the time with the release of Loki 3.0!
There are several deployment methods available for various deployment modes: Helm, Tanka, Docker or Docker Compose, local, and from source. For more details on all these methods, you can refer to the Grafana Loki installation guide.
All these installation methods are great, serving their purpose and providing flexibility to users so you can deploy Loki in a manner that fits your specific scenario. But let me introduce you to another approach — the Ansible Loki Role. This role supports Debian, Ubuntu, and Red Hat. Originally migrated from Voidquark’s Loki repository to the Grafana Ansible collection, I’m excited to be part of the ongoing maintenance and improvements to the role.
Deployment of Loki is primarily focused on Kubernetes, for scalability reasons. But what if you’re not at that phase yet? Or you simply want to experiment? Or you have a small organization where deploying Loki is desired and you are already familiar with Ansible? Well, you’re in the right place. Let’s dive into setting up this Ansible role.
What you need to get started
- Ensure you have an Ansible controller deployed.
- Install the Ansible Grafana collection on your Ansible controller.
- Familiarize yourself with the README of the Ansible Loki Role.
- Create a simple default playbook or customize it to fit your requirements.
I’m going to assume you already have Ansible set up and have read the Loki role README to become more familiar with it.
Now, let’s proceed with installing the Ansible Grafana collection:
ansible-galaxy collection install grafana.grafanaHow to use the default deployment
Let’s demonstrate an out-of-the-box deployment with the default configuration.
First, create a playbook named function_loki-default.play. This playbook is straightforward and deploys Loki with default settings.
- name: Deploy Loki using the default configuration
hosts: loki
become: true
roles:
- role: grafana.grafana.lokiNext, execute this playbook to deploy Loki:
ansible-playbook function_loki-default.playThe default deployment for Loki is now in place, but most people will want to tweak it for their own needs. Let’s move on to customizing it.
How to customize your deployment
What if you want to tinker a bit more? Well, you’re not limited! You can configure each component exactly as you wish. Let’s delve into an example where we’ll use:
- Filesystem storage
- A ruler with Alertmanager
- A simple SSH alerting rule
- A compactor with a four-week retention period
- Other custom parameters
Let’s create a playbook named function_loki-customized.play for this customized deployment:
- name: Deploy Loki using the local filesystem
hosts: loki
become: true
roles:
- role: grafana.grafana.loki
vars:
loki_querier:
max_concurrent: 16
engine:
max_look_back_period: 672h
loki_storage_config:
tsdb_shipper:
active_index_directory: "{{ loki_working_path }}/tsdb-index"
cache_location: "{{ loki_working_path }}/tsdb-cache"
filesystem:
directory: "{{ loki_working_path }}/chunks"
loki_ingester:
wal:
enabled: true
dir: "{{ loki_working_path }}/wal"
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h
max_chunk_age: 2h
chunk_target_size: 1048576
query_store_max_look_back_period: 672h
loki_limits_config:
split_queries_by_interval: 0
reject_old_samples: true
reject_old_samples_max_age: 168h
max_query_length: 0
max_query_series: 50000
retention_period: 672h
allow_structured_metadata: false
max_query_lookback: 672h
loki_compactor:
working_directory: "{{ loki_working_path }}/compactor"
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
loki_common:
path_prefix: "{{ loki_working_path }}"
storage:
filesystem:
rules_directory: "{{ loki_working_path }}/rules"
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
loki_ruler:
rule_path: "{{ loki_working_path }}/rules_tmp"
ring:
kvstore:
store: inmemory
enable_api: true
enable_alertmanager_v2: true
alertmanager_url: http://localhost:9093
loki_ruler_alerts:
- name: Logs.sshd
rules:
- alert: SshLoginFailed
expr: |
count_over_time({job=~"secure"} |="sshd[" |~": Failed|: Invalid|: Connection closed by authenticating user" | __error__="" [15m]) > 6
for: 0m
labels:
severity: critical
annotations:
summary: "{% raw %}SSH authentication failure (instance {{ $labels.instance }}).{% endraw %}"
description: "{% raw %}Increase of SSH authentication failures in last 15 minutes\n VALUE = {{ $value }}{% endraw %}"
Now that your customized Loki instance is deployed and ready to use, the next step is to send logs to Loki. There are various ways to send logs to Loki, such as using Grafana Alloy or Promtail. All methods are described in this section of the documentation.
Once your logs are in Loki, you can query them using the LogCLI utility or Grafana. However, before you can explore logs from Grafana, you must connect the “Loki” data source in your Grafana instance. If you’re deploying Grafana via Ansible, note that the Grafana Ansible role is included in the same collection. You can configure this data source in your inventory and re-run the Grafana Ansible role.
If you’re not using the Ansible role, you can simply add the data source as described in the documentation on managing data sources.
Once the data source is set up, simply navigate to your Grafana URL, followed by “/explore,” and select the “Loki” data source to start exploring your logs. For more details on how to explore logs in Grafana, check out this section of the documentation.
One more thing: The role is idempotent, designed to deploy Loki with configurations aligned to your inventory. If you want to clean up and remove the entire deployment, simply use the loki_uninstall tag. If no tag is specified, the role operates in deployment mode.
ansible-playbook function_loki-customized.play -t loki_uninstallWhat’s next?
Your Loki instance is now up and running — congrats! — and I assume your logs are being delivered to Loki.
To continue learning about Loki, try creating amazing dashboards tailored to your needs. If you need inspiration, check out the NGINX logs dashboard or my dashboards like SSH Logs, Promtail Monitoring, SUDO Logs or Nextcloud Logs, which utilize Loki as a data source. You can also explore the Loki configuration and Loki role documentation.
I wish you a smooth deployment and lots of enjoyment on your Loki journey! And if managing the Loki stack seems overwhelming, consider Grafana Cloud. With Grafana Cloud, you can enjoy hassle-free log management without the need to manage Loki.