Build a Log Collection Pipeline
What you'll build
A log collection pipeline using Fluent Bit to collect container logs from Kubernetes and send them to Loki for storage and querying.
Goals
- Understand how log collection works in Kubernetes
- Deploy Fluent Bit as a DaemonSet to collect logs
- Deploy Loki for log storage
- Query logs using Grafana
Prerequisites
- A Kubernetes cluster with Helm
- Grafana already deployed (or deploy it as part of this project)
- Basic understanding of logging concepts
Steps
- 1
Add the Grafana Loki Helm repo
Add the Grafana Helm repository which contains charts for Loki and Fluent Bit.
Codehelm repo add grafana https://grafana.github.io/helm-charts helm repo updateExpected result
The Grafana Helm repository is added.
- 2
Install Loki
Deploy Loki in the monitoring namespace for log storage.
Codehelm install loki grafana/loki-stack -n monitoring --set fluent-bit.enabled=true,promtail.enabled=falseExpected result
Loki and Fluent Bit are deployed. Fluent Bit starts collecting container logs.
- 3
Add Loki as a Grafana data source
In Grafana, go to Configuration > Data Sources > Add data source > Loki. Set the URL to http://loki:3100 and save.
Expected result
Loki is configured as a data source in Grafana.
- 4
Query logs in Grafana
Go to Explore in Grafana, select the Loki data source, and run a log query like {namespace="default"} to see logs from the default namespace.
Code{namespace="default"}Expected result
You see container logs from the default namespace in real-time.
- 5
Add a health check endpoint
Add a /health endpoint returning 200 OK for orchestration systems (Kubernetes, load balancers) to probe liveness. Health checks are foundational for production deployment.
Code# app.py (Flask example) from flask import Flask, jsonify app = Flask(__name__) @app.route("/health") def health(): return jsonify({"status": "ok"}), 200 @app.route("/ready") def ready(): # Check DB connection etc. return jsonify({"ready": True}), 200Expected result
curl localhost:5000/health returns {"status":"ok"} with a 200 status code.
Try it yourself
What you learned
You built a log collection pipeline that gathers all container logs and makes them searchable in Grafana. Centralized logging is critical for debugging in a containerized environment.
Related Resources
Related Glossary
A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.