feat: add observability stack and background task infrastructure
Add OpenTelemetry instrumentation with distributed tracing and metrics: - Structured JSON logging with trace context correlation - Auto-instrumentation for FastAPI, asyncpg, httpx, redis - OTLP exporter for traces and Prometheus metrics endpoint Implement Celery worker and notification task system: - Celery app with Redis/SQS broker support and configurable queues - Notification tasks for incident fan-out, webhooks, and escalations - Pluggable TaskQueue abstraction with in-memory driver for testing Add Grafana observability stack (Loki, Tempo, Prometheus, Grafana): - OpenTelemetry Collector for receiving OTLP traces and logs - Tempo for distributed tracing backend - Loki for log aggregation with Promtail DaemonSet - Prometheus for metrics scraping with RBAC configuration - Grafana with pre-provisioned datasources and API overview dashboard - Helm templates for all observability components Enhance application infrastructure: - Global exception handlers with structured ErrorResponse schema - Request logging middleware with timing metrics - Health check updated to verify task queue connectivity - Non-root user in Dockerfile for security - Init containers in Helm deployments for dependency ordering - Production Helm values with autoscaling and retention policies
This commit is contained in:
@@ -8,7 +8,7 @@ from app.schemas.auth import (
|
||||
SwitchOrgRequest,
|
||||
TokenResponse,
|
||||
)
|
||||
from app.schemas.common import CursorParams, PaginatedResponse
|
||||
from app.schemas.common import CursorParams, ErrorDetail, ErrorResponse, PaginatedResponse
|
||||
from app.schemas.incident import (
|
||||
CommentRequest,
|
||||
IncidentCreate,
|
||||
@@ -35,6 +35,8 @@ __all__ = [
|
||||
"TokenResponse",
|
||||
# Common
|
||||
"CursorParams",
|
||||
"ErrorDetail",
|
||||
"ErrorResponse",
|
||||
"PaginatedResponse",
|
||||
# Incident
|
||||
"CommentRequest",
|
||||
|
||||
Reference in New Issue
Block a user