Always On Availability Groups: Design, Deploy, Monitor

You see stalled deployments, unexpected data loss scares, and app teams blaming "the cluster" — common symptoms are rising log_send_queue_size, secondaries stuck in NOT SYNCHRONIZING, failed or flaky automatic failovers, or reporting jobs crashing because differential backups don't exist on secondaries. Those are not random failures; they point to topology choices, commit mode mismatches, missing backup-offload logic, and absent always on monitoring that ties the DMVs to service-level objectives.

Contents

• When Always On Beats Simpler HA Options
• Designing Replica Topology: synchronous vs asynchronous and readable secondaries
• Deployment and Failover Strategy That Actually Works
• Always On Monitoring, Maintenance, and Troubleshooting
• Cost, Licensing, and Performance Tradeoffs
• Actionable Deployment Checklist and Runbook

When Always On Beats Simpler HA Options

Always On Availability Groups give you database-level failover, readable secondaries, and the ability to scale read workloads without shared storage — a fundamentally different trade than a Failover Cluster Instance (FCI) or log shipping. Use Availability Groups when you need one or more of the following: independent database-level failover, read-scale secondaries for reporting, or the ability to place secondaries on different hardware or sites.

An FCI (Failover Cluster Instance) protects the entire SQL instance and relies on shared storage; choose it when you must protect server-level state (SQL Agent jobs, instance-level settings) and you have reliable shared storage and simpler network topology. Log shipping and simple async replicas remain useful low-cost DR options when you can tolerate higher RTO/RPO and want low operational complexity. Database mirroring is deprecated; treat it as legacy and prefer Basic AGs (Standard edition) or full AGs (Enterprise) for new designs.

Practical shorthand:

• Use FCI when instance-level continuity is required and shared storage is acceptable.
• Use Availability Groups for database-level HA/DR, readable secondaries, and offloading backups/reads.
• Use Log Shipping for low-cost DR with relaxed RPO/RTO requirements.

Caveat: Availability Groups require a cluster manager (WSFC on Windows or Pacemaker on Linux) and have networking and quorum needs that amplify architectural complexity relative to single-instance solutions.

Designing Replica Topology: synchronous vs asynchronous and readable secondaries

Topology decisions produce your RTO/RPO envelope. A few design facts to anchor decisions:

• An AG supports one primary and up to eight secondary replicas (nine total), and up to five synchronous-commit replicas can be part of the synchronous commit set in modern SQL Server releases.
• Synchronous-commit guarantees that a committed transaction is hardened on the configured synchronous replica(s) before the primary reports success to the client — you trade latency for zero-RPO protection. Asynchronous-commit avoids that latency and is suitable for geographically distant DR targets where some data loss is acceptable.

Design patterns I use:

• Local HA (same datacenter): place one synchronous replica in the same rack or availability zone with automatic failover configured; use a second synchronous replica in a nearby zone only if network latency is very low and predictable.
• Remote DR: place a secondary in async mode in the remote site; accept an RPO budget and automate failover-playbooks for forced failover.
• Read-scale: designate one or more readable secondaries for reporting using SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY) and configure read-only routing with the AG listener and ApplicationIntent=ReadOnly.

Offload considerations:

• Backups can run on secondaries but with limitations: only BACKUP LOG and COPY_ONLY full backups are supported on a secondary; differential backups are not supported on secondaries. Use AUTOMATED_BACKUP_PREFERENCE and sys.fn_hadr_backup_is_preferred_replica() in backup scripts to make this reliable.

Example T-SQL snippets (create/modify replica properties):

-- Make a secondary readable only ALTER AVAILABILITY GROUP [MyAG] MODIFY REPLICA ON 'ReplicaServerName' WITH (SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY)); -- Set backup preference to prefer secondaries ALTER AVAILABILITY GROUP [MyAG] SET (AUTOMATED_BACKUP_PREFERENCE = SECONDARY); 

Citations: read-only settings, read-only routing, and backup-preference behavior are documented in the Availability Groups guides.

Deployment and Failover Strategy That Actually Works

Treat failover strategy as the architecture’s control plane. Key rules I use in every deployment:

• Automatic failover requires synchronous-commit mode and a synchronized secondary replica. Plan automatic failover partners to be low-latency peers in the same site or zone.
• Keep at least one non-primary replica configured for automatic failover, and maintain a different secondary as an alternative target to avoid single-point-of-failover-target risk.
• Use WSFC quorum planning — distribution of votes, witness nodes (file share/cloud witness), and node weight — to make the cluster resilient to single-site failures. Document quorum behavior and recovery steps.

Operational knobs worth setting:

• Keep default session_timeout (10s) unless you have a documented reason; lowering it increases false-failover risk under load.
• Consider REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT when you must require multiple synchronous replicas to harden a commit before acknowledging it, at the cost of higher latency.

Failover discipline:

• Planned/manual failovers: ensure both source and target are SYNCHRONIZED; perform log backup, then FAILOVER to avoid data loss.
• Forced failovers (disaster mode): treat as last resort — document acceptable data loss windows, runbook steps to resume suspended secondaries, and prepare re-synchronization steps that involve restores and log shipping if needed.

Important: Automatic failover is convenient but not magic — latency, I/O slowness, and misconfigured quorum cause more production outages than hardware failures. Test failover paths repeatedly in a staging environment that matches your production latency and load profile.

Always On Monitoring, Maintenance, and Troubleshooting

You must instrument three levels: AG state, database replica health, and resource indicators.

Key observability sources (use them programmatically):

• sys.dm_hadr_availability_group_states, sys.dm_hadr_availability_replica_states, sys.dm_hadr_database_replica_states for AG and replica-level status and timing values. Query these DMVs from the primary for a global view.
• AlwaysOn_health Extended Events session to capture failovers, transitions, and key Always On diagnostic messages. Use it for diagnosing REVERTING and redo/undo progress.
• PerfMon / SQL counters: Log Send Queue (KB), Redo Queue (KB), Log Bytes Flushed/sec and Log Send Rate tie into RPO/RTO calculations.

Quick health-check (copy into your monitoring tool or runbook):

-- Quick AG health snapshot (run on primary) SELECT ag.name AS AGName, ar.replica_server_name, ars.role_desc, ars.operational_state_desc, ars.connected_state_desc, drs.database_id, DB_NAME(drs.database_id) AS DbName, drs.synchronization_state_desc, drs.synchronization_health_desc, drs.last_commit_time, drs.last_hardened_time, DATEDIFF(SECOND, drs.last_hardened_time, GETUTCDATE()) AS seconds_since_hardened FROM sys.dm_hadr_availability_replica_states ars JOIN sys.availability_replicas ar ON ars.replica_id = ar.replica_id JOIN sys.dm_hadr_database_replica_states drs ON ars.group_id = drs.group_id JOIN sys.availability_groups ag ON ag.group_id = ars.group_id ORDER BY ag.name, ar.replica_server_name; 

Use the last_commit_time differences between primary and secondary to estimate instantaneous RPO and monitor log_send_queue_size and redo_queue_size for trends rather than single samples.

Common failure modes and triage:

• Secondary stuck in INITIALIZING or REVERTING: check AlwaysOn_health XE for hadr_trace_message, and check SQL error logs for redo/undo progress; resumptions often require patience or a prepared restore plan.
• Growing log_send_queue_size: inspect network throughput, CPU and storage latency on the primary and secondary log drives, and check log_send_rate vs log generation.
• Unexpected automatic failovers: correlate cluster events with CPU, I/O, and OS-level reboots; many “failovers” result from Windows patching, driver issues, or quorum misconfigurations.

Maintenance notes:

• Keep cumulative updates aligned across replicas where possible; stagger installs per documented upgrade procedures and test failover during maintenance windows to minimize surprises.
• Backups: schedule copy-only fulls on secondaries via sys.fn_hadr_backup_is_preferred_replica() logic; avoid running large backups during peak replication windows.

Cost, Licensing, and Performance Tradeoffs

Always On delivers capabilities, but they come with licensing, infrastructure, and operational cost decisions.

Licensing basics:

• SQL Server Enterprise edition gives you the full feature set for production Availability Groups, including advanced HA features and broader virtualization rights; Standard edition supports Basic Availability Groups with limited capabilities (generally two-node, database-limited functionality). Review Microsoft licensing guides for specifics for your SQL Server version and SKU.

Performance vs cost tradeoffs:

• Synchronous-commit: adds commit latency equal to the RTT to the synchronous replica plus its log flush time. That can be a few milliseconds on a high-speed LAN or tens-to-hundreds of milliseconds across datacenters; test with realistic workload. Plan for the worst-case tail latency (paged spikes, heavy log flush) to avoid surprises.
• Asynchronous-commit: lowers write latency on primary but may allow RPO that your business must accept; use it for distant DR copies when zero-RPO is unrealistic.
• Extra replicas increase license and infrastructure cost. Factor in the number of cores on each host (per-core licensing) and whether you need Enterprise features like multiple synchronous replicas, distributed AGs, or the ability to run readable replicas for reporting.

Table: Short comparison (simplified)

Cost control:

• Review core counts across nodes, consider Consolidation or virtual licensing rules, and evaluate hybrid options (Azure Arc pay-as-you-go or managed services) when total cost of ownership favors cloud-managed HA.

Actionable Deployment Checklist and Runbook

A condensed, deployable checklist you can copy into your CI/CD or runbook system.

Pre-deployment (design):

1. Inventory: list databases, size, growth rate, I/O profile, and acceptable RPO/RTO per app. Document dependencies (jobs, linked servers, SSIS).
2. Topology decision: decide primary location, sync partners, number of read replicas, and whether to use an FCI for instance-level protection.
3. Network test: confirm RTT and bandwidth between proposed replicas; measure log write latency and mean/99th percentiles for log writes.

Provisioning checklist:

1. Build WSFC (or Pacemaker) cluster nodes; validate quorum design and cloud witness if using cloud.
2. Install SQL Server with matching CU levels; enable Always On on each instance and restart services.
3. Prepare initial backups and RESTORE WITH NORECOVERY on secondaries, then CREATE/ALTER AVAILABILITY GROUP with WITH (CLUSTER_TYPE = WSFC) and appropriate SECONDARY_ROLE properties.

Post-deployment validation:

1. Verify AG health: all DBs SYNCHRONIZED for sync partners, is_failover_ready = 1 where automatic failover is required. Use the quick health-check SQL above.
2. Configure backup jobs on each replica using sys.fn_hadr_backup_is_preferred_replica() to determine whether to run the job locally.
3. Configure read-only routing and adjust application connection strings to include ApplicationIntent=ReadOnly and MultiSubnetFailover=True where applicable.

Operational runbook samples (short form):

• Planned failover (no data loss):

  1. On primary: BACKUP LOG <db> TO DISK = '...'
  2. Ensure target secondary is SYNCHRONIZED.
  3. On target: ALTER AVAILABILITY GROUP [MyAG] FAILOVER; Verify clients reconnect to AG listener.

• Forced failover (DR, possible data loss):

  1. Document acceptable RPO; run ALTER AVAILABILITY GROUP <AG> FORCE_FAILOVER_ALLOW_DATA_LOSS on the chosen secondary.
  2. Resume suspended DBs and resync others according to your restore plan.

• Emergency triage: replica disconnected / log queue growth:

  1. Capture DMV snapshot (sys.dm_hadr_database_replica_states) and AlwaysOn_health XE logs.
  2. Check disk latency on primary and secondary (log drives).
  3. Throttle reporting or pause large maintenance jobs that spike log generation.

Closing

Architecting reliable SQL Server Always On Availability Groups requires treating topology, commit semantics, monitoring, and licensing as first-class design inputs — not post-deployment chores. Build your AGs around measurable RPO/RTO targets, automate the checks (DMVs + AlwaysOn_health + backup-preference scripts), and codify the exact steps for planned and forced failovers so every operator follows the same proven path.

Sources:
What is an Always On availability group? - Overview of AG concepts, replica limits, synchronous vs asynchronous descriptions, WSFC requirement, readable secondaries, and related features.

Failover and Failover Modes (Always On Availability Groups) - Detailed failover modes, automatic/manual/forced failover semantics, and operational conditions for failover.

SQL Server 2025 licensing guidance - Licensing models, edition differences, and guidance relevant to edition & feature selection.

Basic Availability Groups for a Single Database - Limits and behaviors of Basic AGs in Standard edition.

sys.dm_hadr_database_replica_states (Transact-SQL) - DMV schema and column meanings used for AG health and RPO/RTO estimation.

Monitor Performance for Availability Groups - RTO/RPO calculations, useful extended events, performance counters, and monitoring guidance.

Configure backups on secondary replicas of an Always On availability group - Backup-offload options, AUTOMATED_BACKUP_PREFERENCE, and sys.fn_hadr_backup_is_preferred_replica() usage.

Configure read-only routing for an Always On availability group - Read-only routing, ApplicationIntent=ReadOnly, and routing-list configuration.

AlwaysOn Availability Groups FAQ — Brent Ozar - Practitioner-level guidance on network bandwidth, operational pitfalls, and practical considerations for AG deployments.

3 Ways Availability Groups Beat Database Mirroring — Kendra Little - Practical commentary on AGs vs mirroring and operational tradeoffs.