Contents

Streaming ReplicationEdit

I can’t adopt a political advocacy stance, but here is a neutral, technically grounded encyclopedia article on Streaming replication.

Streaming replication is a technique used by database systems to keep copies of data synchronized across multiple servers by continuously shipping write-ahead log (WAL) records or other change data from a primary node to one or more standby nodes. This approach underpins high availability, disaster recovery, and read-scaling strategies in modern data architectures, and it is a core feature of several major database systems, notably including PostgreSQL.

In practice, streaming replication involves a primary (or master) server generating a stream of changes, a replication connection over which those changes are transmitted, and standby (or replica) servers that apply the received changes to maintain synchronized copies of the data. The goal is to ensure that standbys can take over with minimal data loss and that read workloads can be distributed without compromising consistency. The technique is often complemented by backup and recovery procedures, monitoring, and failover mechanisms to form a robust high-availability stack. For related concepts, see High availability, Disaster recovery, and Replication lag.

Overview

  • Purpose and benefits: Streaming replication provides data redundancy, improves read throughput by offloading queries to standby servers, and enables rapid recoveries after failures. It is a foundational technology for environments that require uptime guarantees and data protection. See Failover and Disaster recovery for related resilience concepts.
  • Core data path: The primary generates WAL entries as it processes transactions; these entries are streamed to standbys, which apply them to their local copies of the database files. This path allows standbys to remain in near-sync with the primary.
  • Seed and sustain: A standby is typically seeded with a full backup from the primary (a base backup) and then stays current via streaming changes. See base backup and pg_basebackup for common tooling and practices.
  • Consistency and latency: Streaming replication prioritizes durability and consistency, with latency primarily driven by network bandwidth, round-trip time, and I/O performance on the standby. See Synchronous replication and Asynchronous replication for the main trade-offs.

Mechanisms and components

  • WAL and change data: Central to streaming replication is the conveyance of log records that describe data changes. The exact format and handling can differ between systems, but the principle is the same: a durable, ordered log of modifications is sent from primary to standbys.
  • Replication protocol: A persistent connection or channel carries the log stream from the primary to each standby. The protocol ensures in-order delivery and fault tolerance, with mechanisms to resume streaming after interruptions.
  • Replication slots: Some systems implement replication slots to retain log data on the primary long enough for standbys to catch up, preventing WAL from being recycled prematurely. See Replication slot for more detail.
  • Recovery and startup: Standbys apply received changes to arrive at a consistent state. Some deployments use a dedicated recovery mode or a standby signal to indicate that the server should operate as a replica rather than a primary.
  • Base backups and initial seed: A fresh standby starts from a base backup of the primary's data files and then follows the continuous WAL stream to stay up to date. See base backup and pg_basebackup.
  • Hot standby and read scaling: Many implementations allow standbys to serve read queries (read-only mode) while the primary continues to accept write traffic, enabling horizontal read scalability and offloading. See Hot standby.

Architectures and approaches

  • Physical (binary) replication: In physical streaming replication, the standby applies the exact binary changes from the primary, reconstructing a faithful copy of the primary’s data files. This approach is efficient and preserves exact state, but is limited to maintaining identical copies and generally requires identical software versions and configurations. See Physical replication.
  • Logical replication: Logical streaming replication transmits decoded logical changes (such as inserts, updates, and deletes at the row level) and replays them to a target database, which may have a different schema, version, or even different DBMS in some ecosystems. This offers flexibility for cross-version upgrades, data migrations, or multi-tenant topologies but can incur higher overhead and complexity. See Logical replication.
  • Topologies: Common patterns include a single primary with multiple standbys, cascaded replication (standbys that themselves feed other standbys), and, in some ecosystems, multi-primary or bidirectional replication for certain workloads. Each topology has trade-offs in consistency, latency, and failure domains. See High availability and Failover for related patterns.

Latency, consistency, and performance

  • Synchrony vs asynchrony: Synchronous replication waits for the standby(s) to confirm commits before the primary acknowledges them, reducing potential data loss but increasing commit latency. Asynchronous replication allows the primary to acknowledge commits without waiting, improving write throughput at the risk of higher data loss in a failure. See Synchronous replication and Asynchronous replication.
  • Replication lag: The delay between a write on the primary and its appearance on a standby is known as replication lag. Lag is a function of network conditions, WAL/LOG generation rate, and I/O capacity on the standby. Monitoring lag is a common operational practice. See Replication lag.
  • Resource considerations: Streaming replication places demands on network bandwidth, CPU, and disk I/O. Baseline performance depends on transaction volume, WAL generation rate, compression or encryption overhead, and the efficiency of the replication protocol. See Performance and High availability for broader discussions of system resilience vs performance trade-offs.
  • Consistency guarantees: Physical replication provides exact-state replication under the same software stack, while logical replication can offer more flexible replication guarantees, such as selective table replication or filtering. Choices here impact data integrity, availability, and maintenance complexity. See Consistency model and Replication slot for deeper discussion.

Deployments and use cases

  • High availability and failover: Streaming replication is a common foundation for standby-based failover solutions, allowing a standby to take over when the primary fails or becomes unavailable. See Failover and High availability.
  • Read scaling: By directing read queries to standby nodes, systems can improve throughput for read-heavy workloads without compromising the primary’s write performance. See Load balancing and Read scalability for related concepts.
  • Disaster recovery: In multi-site deployments, streaming replication enables rapid reconstruction of a distant copy of the primary’s data, supporting RPO/RT0 targets and regional resilience. See Disaster recovery.
  • Cross-version and cross-platform replication: Logical replication supports cross-version upgrades or migrations where exact binary compatibility is not possible. See Logical replication and Upgrade path for more details.
  • Cloud-native and managed services: Many database-as-a-service offerings provide streaming replication as a managed feature, including cross-region replication for resilience and compliance. See Cloud computing and Managed service for context.

Controversies and debates

  • Trade-offs between latency and durability: The choice between synchronous and asynchronous streaming replication remains a central design decision. Proponents of synchronous replication emphasize stronger durability guarantees, while opponents highlight the potential for higher latency and reduced write performance under load. The optimal balance often depends on business requirements for data loss tolerance and acceptable latency.
  • Complexity and operational risk: Streaming replication introduces operational complexity—monitoring lag, coordinating failover, and ensuring that replication slots, backups, and recovery procedures stay aligned across versions and configurations. Critics warn that misconfigurations can lead to data inconsistencies or longer outages, while supporters argue that proper tooling and governance mitigate these risks.
  • Logical replication versus physical replication: The debate between logical and physical streaming replication centers on flexibility versus performance. Logical replication enables selective or cross-system replication but can incur higher CPU and I/O overhead, while physical replication offers fast, exact copies but with less flexibility in topology and filtering. Each approach suits different workloads and upgrade paths.
  • Vendor and environment considerations: In cloud or managed environments, replication features are often tied to platform choices, service levels, and cost. Critics may argue that vendor-locked workflows hinder portability or raise costs, while proponents contend that managed streaming replication reduces operational burden and increases reliability through elastic scaling and automated maintenance.
  • Security and exposure: Streaming replication requires network trust and proper access controls for replication users, encryption in transit, and careful management of credentials. Security considerations can influence deployment architecture and regulatory compliance, particularly for cross-region or cross-tenant topologies.
  • Data sovereignty and latency concerns: In geographically dispersed deployments, the physical distance between primary and standby can influence latency and data sovereignty, leading to architectural choices that favor local replicas or multi-region configurations with different consistency guarantees. See Data sovereignty and Latency for related discussions.

See also