scry-backfill

Replicate PostgreSQL data to scry-platform for shadow databases, migration testing, and schema validation.

What is scry-backfill?

scry-backfill replicates your PostgreSQL database to scry-platform, creating a shadow database that enables safe migration testing and schema validation without affecting production.

scry-backfill handles the complete replication lifecycle:

  • Schema extraction: Captures your database schema including tables, indexes, constraints, and sequences
  • Data streaming: Efficiently transfers existing data and ongoing changes to scry-platform
  • Automatic mode selection: Intelligently chooses between snapshot and CDC modes based on your database state

When You Need It

scry-backfill is essential when you need a copy of your production data for testing:

  • Shadow database for migration testing: Create a replica to test migrations against real data before applying to production
  • Replay production queries against data copy: Use captured queries from scry-proxy against your shadow database to validate behavior
  • Validate schema changes before production: Test ALTER TABLE, index additions, and constraint changes safely
Note: If you only need query capture and analysis without data replication, scry-proxy alone is sufficient. scry-backfill is specifically for creating shadow databases with your production data.

How It Works

scry-backfill connects to your PostgreSQL database using logical replication to stream changes in real-time:

┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
│                 │         │                 │         │                 │
│   PostgreSQL    │────────▶│  scry-backfill  │────────▶│  scry-platform  │
│                 │         │                 │         │                 │
│  (Production)   │  Logical│  Schema + Data  │  HTTP/2 │ (Shadow Database│
│                 │  Repli- │   Extraction    │  Stream │   + Analytics)  │
│                 │  cation │                 │         │                 │
└─────────────────┘         └─────────────────┘         └─────────────────┘

Logical Replication

scry-backfill uses PostgreSQL's native logical replication protocol. This provides:

  • Row-level change capture without triggers or application changes
  • Minimal impact on production database performance
  • Support for selective table replication

CDC Streaming

After the initial snapshot, scry-backfill switches to Change Data Capture (CDC) mode, streaming INSERT, UPDATE, and DELETE operations as they occur. This keeps your shadow database continuously synchronized with production.

Why Not Debezium?

Debezium is a popular open-source CDC platform, but scry-backfill is purpose-built for database migration validation. Here's how they compare:

Aspect Debezium scry-backfill
Infrastructure Requires Kafka + Kafka Connect cluster Direct HTTP/2 streaming, no middleware
Schema Changes Often requires connector restart Handled gracefully with automatic adaptation
Mode Selection Manual configuration of snapshot vs streaming Automatic mode selection based on database state
Rate Limiting External configuration required Built-in rate limiting with WAL monitoring
Platform Integration Separate pipeline to target system Native integration with scry-platform

scry-backfill eliminates the operational complexity of managing a Kafka cluster while providing tight integration with scry-platform's migration validation features.

Ready to Get Started?

Set up scry-backfill and create your first shadow database for migration testing.

Request Early Access