[Guide: Fetching Large Logs From Loki in Kubernetes]

In our most recent project, we faced a significant challenge: extracting application logs per day from Loki in a Kubernetes environment. When cold, the application writes about 60 lines per minute, and when someone interacts with the application, it can write 2000-5000 lines of logs per minute – it turned out that we needed to get more than 300,000 lines of logs in total. Our project setup did not include a configured log export, and our primary log viewing tool was Grafana, which imposes a 5000 line limit on log retrieval. Increasing this limit was not feasible as it would have been a significant drain on our resources and was unnecessary for this one-time task. Additionally, accessing logs directly from the Kubernetes pod was not an option due to storage limitations within the pod itself.

So, we need to download logs directly from Loki without changing any configurations.

Preparation

Copy link

Used tools

Copy link

• LogCli
• Kubectl

Additional steps

Copy link

To ensure that the query we will use to search for logs is correct, follow these steps:

1. Navigate to Grafana Explore:

Go to Grafana > Explore

2. Set the Required Label:

Apply the necessary label to filter logs by the service.

3. Filter by Date:

Use the operation filter to display lines containing the desired date.

Example query:

{instance="our-service"} |= `2024-07-12`

Execution

Copy link

1. Install LogCli:

Download the LogCli binary from the Loki releases page

2. Set Loki Address:

Configure the Loki address for LogCli using an environment variable:

export LOKI_ADDR=http://localhost:8000

3. Port Forwarding:

Forward local ports to the Loki pod to allow local access:

kubectl --namespace loki port-forward svc/loki-stack 8000:3100

4. Extract Logs:

Use LogCli to query and save the logs to a file:

logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log

In this command:

• --limit is set with a high value to ensure all logs are captured.
• --since is set to 72 hours to cover a sufficient time range.

Conclusion

Copy link

The whole process took approximately 10 minutes and resulted in a file containing the complete application logs for the specified date. If required, this process can be further optimised or automated.