feat: add observability stack and background task infrastructure

Add OpenTelemetry instrumentation with distributed tracing and metrics:
- Structured JSON logging with trace context correlation
- Auto-instrumentation for FastAPI, asyncpg, httpx, redis
- OTLP exporter for traces and Prometheus metrics endpoint

Implement Celery worker and notification task system:
- Celery app with Redis/SQS broker support and configurable queues
- Notification tasks for incident fan-out, webhooks, and escalations
- Pluggable TaskQueue abstraction with in-memory driver for testing

Add Grafana observability stack (Loki, Tempo, Prometheus, Grafana):
- OpenTelemetry Collector for receiving OTLP traces and logs
- Tempo for distributed tracing backend
- Loki for log aggregation with Promtail DaemonSet
- Prometheus for metrics scraping with RBAC configuration
- Grafana with pre-provisioned datasources and API overview dashboard
- Helm templates for all observability components

Enhance application infrastructure:
- Global exception handlers with structured ErrorResponse schema
- Request logging middleware with timing metrics
- Health check updated to verify task queue connectivity
- Non-root user in Dockerfile for security
- Init containers in Helm deployments for dependency ordering
- Production Helm values with autoscaling and retention policies

app/api/v1/health.py

@@ -2,7 +2,8 @@
 
 from fastapi import APIRouter, Response, status
 
-from app.db import db, redis_client
+from app.db import db
+from app.taskqueue import task_queue
 
 router = APIRouter()
 
@@ -16,14 +17,14 @@ async def healthz() -> dict[str, str]:
 @router.get("/readyz")
 async def readyz(response: Response) -> dict[str, str | dict[str, bool]]:
     """
-    Readiness probe - checks database and Redis connectivity.
+    Readiness probe - checks database and task queue connectivity.
     - Check Postgres status
-    - Check Redis status
+    - Check configured task queue backend
     - Return overall healthiness
     """
     checks = {
         "postgres": False,
-        "redis": False,
+        "task_queue": False,
     }
 
     try:
@@ -34,7 +35,7 @@ async def readyz(response: Response) -> dict[str, str | dict[str, bool]]:
     except Exception:
         pass
 
-    checks["redis"] = await redis_client.ping()
+    checks["task_queue"] = await task_queue.ping()
 
     all_healthy = all(checks.values())
     if not all_healthy: