Skip to content

Build a Log Collection Pipeline

Easy ~35 min

What you'll build

A log collection pipeline using Fluent Bit to collect container logs from Kubernetes and send them to Loki for storage and querying.

Goals

  • Understand how log collection works in Kubernetes
  • Deploy Fluent Bit as a DaemonSet to collect logs
  • Deploy Loki for log storage
  • Query logs using Grafana

Prerequisites

  • A Kubernetes cluster with Helm
  • Grafana already deployed (or deploy it as part of this project)
  • Basic understanding of logging concepts

Steps

  1. 1

    Add the Grafana Loki Helm repo

    Add the Grafana Helm repository which contains charts for Loki and Fluent Bit.

    Code
    helm repo add grafana https://grafana.github.io/helm-charts
    helm repo update

    Expected result

    The Grafana Helm repository is added.

  2. 2

    Install Loki

    Deploy Loki in the monitoring namespace for log storage.

    Code
    helm install loki grafana/loki-stack -n monitoring --set fluent-bit.enabled=true,promtail.enabled=false

    Expected result

    Loki and Fluent Bit are deployed. Fluent Bit starts collecting container logs.

  3. 3

    Add Loki as a Grafana data source

    In Grafana, go to Configuration > Data Sources > Add data source > Loki. Set the URL to http://loki:3100 and save.

    Expected result

    Loki is configured as a data source in Grafana.

  4. 4

    Query logs in Grafana

    Go to Explore in Grafana, select the Loki data source, and run a log query like {namespace="default"} to see logs from the default namespace.

    Code
    {namespace="default"}

    Expected result

    You see container logs from the default namespace in real-time.

  5. 5

    Add a health check endpoint

    Add a /health endpoint returning 200 OK for orchestration systems (Kubernetes, load balancers) to probe liveness. Health checks are foundational for production deployment.

    Code
    # app.py (Flask example)
    from flask import Flask, jsonify
    app = Flask(__name__)
    
    @app.route("/health")
    def health():
        return jsonify({"status": "ok"}), 200
    
    @app.route("/ready")
    def ready():
        # Check DB connection etc.
        return jsonify({"ready": True}), 200

    Expected result

    curl localhost:5000/health returns {"status":"ok"} with a 200 status code.

Try it yourself

What you learned

You built a log collection pipeline that gathers all container logs and makes them searchable in Grafana. Centralized logging is critical for debugging in a containerized environment.

Related Resources

Related Glossary

Related Glossary

A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.

Decode error messages

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

View errors

← Back to topic