feat: add observability stack and background task infrastructure
Add OpenTelemetry instrumentation with distributed tracing and metrics: - Structured JSON logging with trace context correlation - Auto-instrumentation for FastAPI, asyncpg, httpx, redis - OTLP exporter for traces and Prometheus metrics endpoint Implement Celery worker and notification task system: - Celery app with Redis/SQS broker support and configurable queues - Notification tasks for incident fan-out, webhooks, and escalations - Pluggable TaskQueue abstraction with in-memory driver for testing Add Grafana observability stack (Loki, Tempo, Prometheus, Grafana): - OpenTelemetry Collector for receiving OTLP traces and logs - Tempo for distributed tracing backend - Loki for log aggregation with Promtail DaemonSet - Prometheus for metrics scraping with RBAC configuration - Grafana with pre-provisioned datasources and API overview dashboard - Helm templates for all observability components Enhance application infrastructure: - Global exception handlers with structured ErrorResponse schema - Request logging middleware with timing metrics - Health check updated to verify task queue connectivity - Non-root user in Dockerfile for security - Init containers in Helm deployments for dependency ordering - Production Helm values with autoscaling and retention policies
This commit is contained in:
29
app/db.py
29
app/db.py
@@ -6,7 +6,6 @@ from contextvars import ContextVar
|
||||
|
||||
import asyncpg
|
||||
from asyncpg.pool import PoolConnectionProxy
|
||||
import redis.asyncio as redis
|
||||
|
||||
|
||||
class Database:
|
||||
@@ -46,34 +45,8 @@ class Database:
|
||||
yield conn
|
||||
|
||||
|
||||
class RedisClient:
|
||||
"""Manages Redis connection."""
|
||||
|
||||
client: redis.Redis | None = None
|
||||
|
||||
async def connect(self, url: str) -> None:
|
||||
"""Create Redis connection."""
|
||||
self.client = redis.from_url(url, decode_responses=True)
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
"""Close Redis connection."""
|
||||
if self.client:
|
||||
await self.client.aclose()
|
||||
|
||||
async def ping(self) -> bool:
|
||||
"""Check if Redis is reachable."""
|
||||
if not self.client:
|
||||
return False
|
||||
try:
|
||||
await self.client.ping()
|
||||
return True
|
||||
except redis.RedisError:
|
||||
return False
|
||||
|
||||
|
||||
# Global instances
|
||||
# Global instance
|
||||
db = Database()
|
||||
redis_client = RedisClient()
|
||||
|
||||
|
||||
_connection_ctx: ContextVar[asyncpg.Connection | PoolConnectionProxy | None] = ContextVar(
|
||||
|
||||
Reference in New Issue
Block a user