Observability

Overview

ScryData's observability system captures rich metadata for every query flowing through the proxy:

Query Execution
      ↓
Extract Metadata (query, timing, success/failure)
      ↓
Create QueryEvent
      ↓
Send to EventBatcher (async, non-blocking)
      ↓
Batch accumulates events
      ↓
Flush on size (100 events) OR time (1000ms)
      ↓
EventPublisher publishes batch
                            

Query Events

Every query generates an event containing:

Example Event (Anonymized)

{
  "event_id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": "2025-12-06T10:00:00.123Z",
  "query": "SELECT * FROM users WHERE id = ?",
  "normalized_query": "SELECT * FROM users WHERE id = ?",
  "value_fingerprints": ["blake3:abc123def456..."],
  "duration_ms": 1.23,
  "rows": 1,
  "success": true,
  "database": "production",
  "connection_id": "conn-123"
}

Event Batching

Events are batched to reduce network overhead:

Memory Safety: The channel is bounded with ring buffer semantics. New events replace oldest events if full, preventing unbounded memory growth. Queries never block on event publishing.

Memory Usage: ~100 bytes per event → 10,000 events ≈ 1MB

Publishers

DebugLoggerPublisher

Logs events as JSON to console. Use for development and debugging.

[publisher]
publisher_type = "debug"

HttpPublisher

Publishes batches to remote HTTP endpoint using FlexBuffers serialization (50% smaller than JSON).

[publisher]
publisher_type = "http"
http_endpoint = "https://analytics.example.com/events"
http_timeout_ms = 500
http_max_retries = 2
http_api_key = "sk-..."
http_compression = true

Features:

• Retries with exponential backoff on failure
• Optional gzip compression (default: enabled)
• API key authentication
• Monotonic batch sequence number for ordering

FlexBuffers Serialization

ScryData uses FlexBuffers for efficient binary serialization:

Benefits:

• Schema-less: No code generation required
• Fast: Zero-copy deserialization
• Compact: 50% smaller than JSON
• Cross-platform: FlatBuffers ecosystem support

Hot Data Detection

ScryData tracks frequently accessed value fingerprints using Count-Min Sketch + Top-K heap:

curl http://localhost:9090/debug/hot_data

{
  "top_k": [
    {
      "fingerprint": "blake3:a1b2c3d4e5f6...",
      "access_count": 15234
    },
    {
      "fingerprint": "blake3:f6e5d4c3b2a1...",
      "access_count": 8901
    }
  ]
}

Use Cases:

• Cache Optimization: Cache hot fingerprint results
• Anomaly Detection: Sudden spike in specific fingerprint access
• Performance Tuning: Optimize queries accessing hot data
• Security: Detect unusual access patterns (credential stuffing)

Prometheus Metrics

# Total queries processed
scry_queries_total 10000

# Query latency percentiles
scry_query_latency_seconds{quantile="0.5"} 0.000123
scry_query_latency_seconds{quantile="0.99"} 0.000456

# Pool connections
scry_pool_connections_total 15
scry_pool_connections_available 8
scry_pool_utilization 0.46

# Circuit breaker state
scry_circuit_breaker_state 0

# Error rate
scry_query_error_rate 0.001