Metrics Catalog

Scrape

Metrics are exposed by a separate metrics listener. See Metrics Monitoring for configuration.

curl -fsS \
  -H "Authorization: Bearer $METRICS_TOKEN" \
  http://localhost:9090/metrics

SyncTV registers every metric definition when the metrics listener starts. Invalid or duplicate definitions stop startup instead of producing a partial registry.
Labels use bounded values. HTTP path is an Axum route template such as /api/rooms/{room_id}; resource IDs, query strings, and raw error messages are excluded.
Existing metric names and label sets are compatibility-sensitive. Review dashboards and alerts before changing them.
A vector metric may remain absent until its feature records the first labeled sample.

Metric	Type	Labels	Meaning
`http_requests_total`	counter	`method`, `path`, `status`	HTTP requests
`http_request_duration_seconds`	histogram	`method`, `path`	HTTP request duration
`http_requests_in_flight`	gauge	none	In-flight HTTP requests
`websocket_connections_active`	gauge	none	Active WebSocket connections
`websocket_connections_total`	counter	`status`	WebSocket connection attempts
`websocket_errors_total`	counter	`error_type`	WebSocket errors
`users_online`	gauge	none	Online users
`rooms_active`	gauge	none	Active rooms

Metric	Type	Labels	Meaning
`db_connections_active`	gauge	none	Active DB connections
`db_connections_idle`	gauge	none	Idle DB connections
`db_pool_size_max`	gauge	none	Configured maximum DB pool size across pools
`db_pool_utilization_ratio`	gauge	`pool`	Pool utilization, from 0 to 1
`cache_hits_total`	counter	`cache_type`, `level`	Cache hits
`cache_misses_total`	counter	`cache_type`, `level`	Cache misses
`cache_evictions_total`	counter	`cache_type`	Cache evictions
`cache_errors_total`	counter	`cache_type`, `operation`	Cache operation errors
`cache_invalidations_total`	counter	`cache_type`	Cache invalidations
`cache_operation_duration_seconds`	histogram	`operation`	Cache operation duration
`cache_lag_flush_total`	counter	`component`	Full L1 flushes after invalidation-channel lag
`cache_fence_operations_total`	counter	`domain`, `operation`, `result`	Version-fence operations
`cache_db_fallback_total`	counter	`domain`, `reason`	Strong reads that fell back to PostgreSQL
`cache_stale_write_reject_total`	counter	`cache_type`, `level`	Rejected stale cache writes
`cache_fence_pending`	gauge	`domain`	Domains with a pending version fence
`cache_fence_repair_total`	counter	`domain`, `result`	Read-time fence repair outcomes
`cache_fence_db_compare`	gauge	`domain`, `relation`	Latest DB-to-fence patrol comparison

Metric	Type	Labels	Meaning
`grpc_requests_total`	counter	`service`, `method`, `status`	Completed gRPC requests
`grpc_request_duration_seconds`	histogram	`service`, `method`, `status`	gRPC request duration

Metric	Type	Labels	Meaning
`chat_messages_total`	counter	none	Persisted chat messages
`synctv_file_object_delete_attempts_total`	counter	`origin`, `backend`	File object delete attempts
`synctv_file_object_delete_failures_total`	counter	`origin`, `backend`	File object delete failures
`synctv_file_cleanup_jobs_due`	gauge	none	Due file cleanup jobs waiting for retry
`synctv_file_cleanup_jobs_total`	counter	`action`, `origin`, `backend`	File cleanup retry job actions
`webrtc_peers_active`	gauge	none	Active WebRTC peers
`active_connections`	gauge	none	Active connections
`spawned_task_panics_total`	counter	`task_name`	Background task panics caught by `spawn_monitored`
`logging_dropped_lines_total`	counter	`component`	Log lines dropped by full non-blocking queues
`email_delivery_queue_depth`	gauge	none	Email jobs awaiting delivery in the PostgreSQL outbox
`email_delivery_in_flight`	gauge	none	Email jobs being delivered by this instance
`email_delivery_jobs_total`	counter	`kind`, `status`	Email job outcomes: `sent`, `retry`, `dead`, `superseded`, `fenced`, `persist_failed`, or `ack_failed`
`email_delivery_duration_seconds`	histogram	`kind`, `status`	Delivery duration by job kind and outcome
`rate_limit_redis_fallbacks_total`	counter	`category`	In-memory fallback after Redis rate-limit errors

Metric	Type	Labels	Meaning
`synctv_cluster_connections_total`	gauge	none	Cluster connections on this node
`synctv_realtime_events_published_total`	counter	`event_type`	Published realtime events
`synctv_realtime_events_received_total`	counter	`event_type`	Received realtime events
`synctv_realtime_events_dropped_total`	counter	`reason`	Dropped realtime events
`synctv_cluster_heartbeat_failures`	gauge	none	Consecutive heartbeat failures
`synctv_node_active_rooms`	gauge	none	Active rooms on this node
`synctv_cluster_leader_election_state`	gauge	none	Leader election state; 1 means leader
`synctv_cluster_leader_election_epoch`	gauge	none	Current leader epoch
`synctv_cluster_leader_election_consecutive_failures`	gauge	none	Consecutive leader election failures
`synctv_cluster_epoch_mismatch_quarantine`	gauge	none	Epoch mismatch quarantine state
`synctv_cluster_leader_election_mode`	gauge	none	Election mode: 0 standalone, 1 Redis, 2 Kubernetes Lease
`synctv_cluster_distributed_counter_ttl_refreshes_total`	counter	`result`	Distributed counter TTL refresh outcomes
`synctv_cluster_distributed_counter_ttl_keys_refreshed`	gauge	none	Keys refreshed in the latest TTL cycle
`synctv_cluster_distributed_counter_ttl_consecutive_failures`	gauge	none	Consecutive TTL refresh failures

Metric	Type	Labels	Meaning
`stream_relay_duration_seconds`	histogram	`stream_type`	Stream relay operation duration
`active_relay_streams`	gauge	none	Active relay streams
`stream_errors_total`	counter	`stream_type`, `error_type`	Stream errors
`streamhub_restarts_total`	counter	`reason`	StreamHub event loop restarts
`streams_active`	gauge	none	Active tracked streams
`synctv_publisher_heartbeat_failures_total`	counter	none	Publisher cleanups after heartbeat failure
`livestream_active_publishers`	gauge	none	Active livestream publishers
`livestream_active_viewers`	gauge	none	Active livestream viewers
`livestream_relay_frame_drops_total`	counter	none	Relay frame drops caused by backpressure
`livestream_flv_slow_client_terminations_total`	counter	none	FLV slow-client terminations