Skip to content

Monitoring Data Lost

Error message

                Error: Prometheus missing metrics for the last 30 minutes
              

Quick summary

Prometheus or another monitoring system has gaps in collected metrics, usually due to scrape failures or storage issues.

Why this happens

  • Prometheus cannot scrape targets (application down, network issues)

  • The Prometheus storage is full or has retention issues

  • The application's /metrics endpoint is not working or is slow

  • Prometheus itself is overloaded and dropping samples

Minimal example

✗ Broken code
# Prometheus cannot scrape my-app
# Target is down at http://my-app:8080/metrics
✓ Fixed code
# Check if my-app is running
kubectl get pods -l app=my-app
# Restart if needed
kubectl rollout restart deployment/my-app
# Verify metrics endpoint
kubectl exec -it <prometheus-pod> -- curl http://my-app:8080/metrics

Prometheus cannot scrape the application because it is down. The fix restarts the application and verifies the metrics endpoint.

How to diagnose

  • Check Prometheus targets: visit http://prometheus:9090/targets

  • Check Prometheus logs for errors: kubectl logs <prometheus-pod>

  • Verify the application's /metrics endpoint is responding

  • Check Prometheus storage usage and retention settings

How to fix

  • Fix the scrape target (restart the application, fix network)

  • Increase Prometheus storage or adjust retention period

  • Fix the application's metrics endpoint

  • Scale Prometheus or use remote write to offload storage

How to prevent

  • Monitor Prometheus health and alert on scrape failures

  • Set up long-term storage (Thanos, Cortex) for metrics retention

  • Use ServiceMonitors to automatically manage scrape targets

Related Resources

Related Glossary

← Back to language