Troubleshooting

PostgreSQL Setup Errors

These errors occur when PostgreSQL is not properly configured for logical replication.

wal_level not set to logical

Solution: Update your PostgreSQL configuration to enable logical replication:

1. Edit postgresql.conf and set:

   wal_level = logical

2. Restart PostgreSQL for the change to take effect:

   sudo systemctl restart postgresql

3. Verify the setting:

   SHOW wal_level;
   -- Should return: logical

No replication slots available

Solution: Increase the maximum number of replication slots:

1. Edit postgresql.conf:

   # Default is 10, increase as needed
   max_replication_slots = 20

2. Restart PostgreSQL and verify:

   SHOW max_replication_slots;

3. Check existing slots and remove unused ones:

   SELECT slot_name, active FROM pg_replication_slots;
   
   -- Drop an unused slot
   SELECT pg_drop_replication_slot('unused_slot_name');

Permission denied for replication

Solution: Grant the necessary replication privileges to your database user:

-- Grant replication privilege to user
ALTER USER scry_backfill REPLICATION;

-- Also ensure the user can access the database
GRANT CONNECT ON DATABASE your_database TO scry_backfill;
GRANT USAGE ON SCHEMA public TO scry_backfill;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO scry_backfill;

Additionally, ensure your pg_hba.conf allows replication connections:

# Allow replication connections from scry-backfill host
host    replication     scry_backfill    10.0.0.0/8    scram-sha-256

Data Type Issues

scry-backfill supports a wide range of PostgreSQL data types, but some require special handling.

Supported Types

The following PostgreSQL data types are fully supported:

• Numeric types - smallint, integer, bigint, decimal, numeric, real, double precision, serial, bigserial
• Text types - character, varchar, text, char
• Binary types - bytea
• Date/time types - date, time, timestamp, timestamptz, interval
• JSON types - json, jsonb
• UUID - uuid
• Arrays - arrays of any supported type
• Enums - user-defined enum types
• Geometric types - point, line, lseg, box, path, polygon, circle

Known Limitations

Some data types have limitations or require workarounds:

Type Errors

This error indicates scry-backfill encountered a PostgreSQL type it doesn't recognize. To identify the type:

-- Find the type name from its OID
SELECT typname, typnamespace::regnamespace as schema
FROM pg_type
WHERE oid = 12345;

Workarounds:

• Exclude the problematic column using column_filter in your configuration
• Create a view that casts the column to a supported type
• Use the type_overrides configuration to map the type to a supported one

# In scry-backfill.toml
[type_overrides]
# Map custom type OID to text
12345 = "text"

Connection Issues

Troubleshoot connectivity problems between scry-backfill and your databases.

Cannot connect to source database

Use the built-in connection checker to diagnose issues:

scry-backfill check-connections

Common causes:

• Network/firewall issues - Ensure the scry-backfill host can reach PostgreSQL on the configured port (default 5432)
• Authentication failure - Verify username, password, and pg_hba.conf settings
• SSL/TLS mismatch - Check if the server requires SSL and configure sslmode appropriately
• DNS resolution - Verify the hostname resolves correctly, or use an IP address
• Connection limit reached - Check max_connections and current connection count

# Test basic connectivity
psql "host=your-db.example.com port=5432 user=scry_backfill dbname=your_db sslmode=require"

Cannot connect to scry-platform

If scry-backfill cannot reach the scry-platform destination, check:

• Platform URL - Verify the destination.url in your configuration is correct
• API credentials - Ensure your API key or token is valid and not expired
• Network egress - Confirm outbound HTTPS (port 443) is allowed from your environment
• Proxy settings - If behind a corporate proxy, configure HTTPS_PROXY environment variable

# Test connectivity to scry-platform
curl -I https://api.scrydata.io/health

Replication Issues

Diagnose and resolve issues with ongoing logical replication.

Replication slot not advancing

If your replication slot isn't making progress, first check its status:

scry-backfill slot-status

Or query PostgreSQL directly for more details:

SELECT
    slot_name,
    active,
    restart_lsn,
    confirmed_flush_lsn,
    pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag
FROM pg_replication_slots
WHERE slot_name LIKE 'scry%';

If the slot is inactive, check scry-backfill logs for errors and restart the process.

WAL bloat

If WAL files are accumulating and consuming disk space, see our Rate Limiting & WAL Monitoring guide for detailed information on monitoring and managing WAL growth.

Emergency cleanup: If WAL growth is critical and scry-backfill cannot recover, you may need to drop the replication slot:

-- Check WAL disk usage
SELECT pg_size_pretty(sum(size)) as total_wal_size
FROM pg_ls_waldir();

-- Drop the slot if necessary (data loss warning!)
SELECT pg_drop_replication_slot('scry_backfill_slot');

-- After dropping, restart scry-backfill with --full-resync

Getting Help

If you're still experiencing issues, these tools and resources can help diagnose the problem.

Diagnostic Commands

scry-backfill includes several commands to help troubleshoot:

# Validate your configuration file
scry-backfill validate-config

# Test database connections
scry-backfill check-connections

# Enable debug logging for detailed output
RUST_LOG=debug scry-backfill run

Log Levels

Adjust logging verbosity using the RUST_LOG environment variable:

• RUST_LOG=error - Only errors (minimal output)
• RUST_LOG=warn - Warnings and errors
• RUST_LOG=info - Normal operation (default)
• RUST_LOG=debug - Detailed debugging information
• RUST_LOG=trace - Very verbose, including protocol details

Reporting Issues

If you've tried the troubleshooting steps above and still need help, please open an issue on GitHub:

github.com/scrydata/scry-backfill/issues

Please include:

• scry-backfill version (scry-backfill --version)
• PostgreSQL version (SELECT version();)
• Your operating system and version
• Configuration file (with sensitive values redacted)
• Relevant log output with RUST_LOG=debug
• Steps to reproduce the issue