Observability
Aktilot ships with a full observability stack out of the box. Every component of the RAG pipeline — from document ingestion to LLM calls to ChromaDB queries — is instrumented with OpenTelemetry and visualised in Grafana. The stack starts automatically with docker compose up; no additional setup is required.
│ OTLP/gRPC
▼
OpenTelemetry Collector
├──► Prometheus (metrics)
└──► Grafana Tempo (traces) ──► Grafana (dashboards)
Services
| Service | Local URL | Purpose |
|---|---|---|
| Grafana | http://localhost:3002 | Dashboards (metrics + traces) |
| Prometheus | http://localhost:9090 | Metrics store + query engine |
| Grafana Tempo | http://localhost:3200 | Distributed trace storage |
| OTEL Collector | http://localhost:8889/metrics | Raw metrics scrape endpoint |
| Temporal UI | http://localhost:8233 | Workflow execution history |
Default Grafana login: admin / admin
7 pre-built dashboards
1. RAG Pipeline Overview
/d/rag-overview
Your first stop. Answers “is the system healthy right now?” — LLM response latency, chat volume, average retrieval top-1 score, and an activity latency breakdown showing which pipeline step is the bottleneck.
2. Retrieval Quality
/d/retrieval-quality
Open this when answers feel irrelevant. Reranker top-1 score over time, docs discarded ratio, retrieval score by type (hybrid vs. vector vs. BM25), and top-K requested vs. returned.
3. Prompt Intelligence
/d/prompt-intelligence
Open this when token costs spike. Stacked prompt token composition (context / system / question), context tokens by project, and chunks included vs. top-K requested.
4. LLM Performance
/d/llm-performance
Open this when LLM calls feel slow. Response latency by purpose, finish-reason distribution, and requests by purpose to catch silently failing steps.
5. Vector DB Health
/d/vectordb-health
Open this after uploading documents. Collection size over time, queries by collection, and search/insert latency.
6. Token & Cost Intelligence
/d/token-cost
Open this at end of month. Cumulative input/output tokens by model and purpose, embedding tokens by call site, and the input/output token ratio.
7. Temporal Workflows
/d/temporal-workflows
Open this when workflows are slow or failing. Activities executed, retries, terminal failures, activity latency by type, worker task-slot saturation, and queue delay.
Traces
Every API request and Temporal workflow activity is traced end to end and stored in Grafana Tempo. Traces link API requests → workflow executions → individual activity spans, so you can follow a single user query from the HTTP endpoint through every pipeline step. Open Grafana → Explore → select the Tempo datasource, then search by service name: aktilot-api or aktilot-worker.
Metrics reference
All custom metrics use the rag.* prefix and are exported via OTLP to the collector. Temporal SDK built-in metrics use the temporal_* prefix.
| Prefix | Source | Examples |
|---|---|---|
| rag_llm_* | Chat activities | rag_llm_request_latency_ms, rag_llm_requests_total |
| rag_embedding_* | Embedding calls | rag_embedding_latency_ms, rag_embedding_tokens_total |
| rag_retrieval_* | Vector + BM25 search | rag_retrieval_top_k_returned, rag_retrieval_score_avg |
| rag_reranker_* | Hybrid reranking | rag_reranker_score_top1, rag_reranker_latency_ms |
| rag_prompt_* | Prompt assembly | rag_prompt_tokens_context, rag_prompt_chunks_included |
| rag_vectordb_* | ChromaDB operations | rag_vectordb_collection_size, rag_vectordb_search_latency_ms |
| rag_tokens_* | Cost tracking | rag_tokens_input_total, rag_tokens_embedding_total |
| rag_workflow_* | Temporal activities | rag_workflow_activity_duration_ms, rag_workflow_retries_total |
| temporal_* | Temporal SDK | temporal_activity_execution_latency, temporal_worker_task_slots_available |