Wait Type HADR_FABRIC_CALLBACK

Wait statistics, in the context of SQL Server, refer to the amount of time that a query spends waiting to access data in the database. When a client application requests data from the database, the request is placed in a queue and the client application must wait for its turn to access the data. The time that the query spends waiting is called a "wait" and is tracked by SQL Server. This information can be used to identify potential performance bottlenecks and optimize the performance of the database. Wait statistics are commonly used by database administrators to diagnose and troubleshoot performance issues in SQL Server.

SQL Server relies on wait types to provide insights into delays and bottlenecks in its processes. One specific wait type, HADR_FABRIC_CALLBACK, is part of the Always On Availability Groups feature, which ensures high availability and disaster recovery. This wait type indicates that SQL Server is waiting for a response from the High Availability and Disaster Recovery (HADR) fabric layer, which manages communication and coordination between replicas. In this blog post, we’ll take an in-depth look at the HADR_FABRIC_CALLBACK wait type, its causes, and how to troubleshoot and minimize its impact on your SQL Server environment.

What Is the HADR_FABRIC_CALLBACK Wait Type?

The HADR_FABRIC_CALLBACK wait type occurs when SQL Server is waiting for a response or acknowledgment from the HADR fabric layer. This fabric layer handles the critical communication between replicas in an Always On Availability Group, ensuring that the primary and secondary replicas remain synchronized and operational. The wait type represents the time a process must pause while waiting for this communication to complete.

For example, during log synchronization, the primary replica sends transaction log records to secondary replicas. The HADR fabric layer facilitates this communication, and the HADR_FABRIC_CALLBACK wait appears when the primary is waiting for a response from the secondary, such as confirmation that a log block has been received and processed.

When Does HADR_FABRIC_CALLBACK Appear?

This wait type is most commonly encountered in the following scenarios:

• Log Synchronization – When the primary replica sends log records to the secondary replicas, it waits for acknowledgment that the data has been received and processed.
• Failover Operations – During planned or automatic failovers, replicas use the HADR fabric to coordinate changes in roles and states.
• High Workloads – When a replica is under heavy load, it may delay its responses to the HADR fabric layer, causing waits.
• Network Issues – Poor network performance, high latency, or limited bandwidth between replicas can exacerbate this wait type.
• Replica Health Checks – Periodic checks between replicas also rely on the HADR fabric and may lead to this wait type under certain conditions.

Why HADR_FABRIC_CALLBACK Waits Matter

While some HADR_FABRIC_CALLBACK waits are normal, prolonged or frequent waits can indicate underlying issues that may affect the performance and reliability of your Availability Group. These waits are critical to monitor because they can signal:

• Synchronization Delays – Prolonged waits may indicate lag between the primary and secondary replicas, reducing failover readiness and impacting read-only queries on secondary replicas.
• Network or Resource Bottlenecks – Delays in communication may point to insufficient network capacity or overloaded replica resources.
• Replica Configuration Problems – Misconfigured replicas may not communicate efficiently with the HADR fabric.
• Large Transactions – Transactions generating substantial log records can cause delays in acknowledgment from secondary replicas.

How to Identify and Address HADR_FABRIC_CALLBACK Waits

1. Monitor Network Performance

Ensure that the network connecting your primary and secondary replicas is fast and reliable. High latency, packet loss, or bandwidth limitations can exacerbate HADR_FABRIC_CALLBACK waits. Consider using a dedicated network for Always On traffic to minimize congestion and improve performance.

2. Optimize Workloads

Analyze workloads running on the primary and secondary replicas. High transaction volumes or resource-intensive queries can lead to delays in HADR communication. Distribute workloads appropriately to avoid overwhelming any single replica.

3. Upgrade Hardware and Storage

Slow disk I/O or limited CPU and memory resources on either the primary or secondary replicas can delay log processing and synchronization. Use high-performance storage (e.g., SSDs) and ensure replicas have adequate hardware to handle the workload.

4. Adjust Synchronization Mode

If the secondary replicas are not critical, consider using asynchronous commit mode to reduce synchronization overhead. While this may introduce a slight lag, it can significantly improve performance in high-latency environments.

5. Monitor and Optimize Queries

Review and optimize queries generating large transactions on the primary replica. Reducing the size of log records sent to the secondary replicas can help alleviate HADR_FABRIC_CALLBACK waits.

Monitoring HADR_FABRIC_CALLBACK Waits

To monitor HADR_FABRIC_CALLBACK waits, use the following SQL Server tools:

• Dynamic Management Views (DMVs) – Query sys.dm_os_wait_stats to track wait statistics and identify trends.
• Always On Dashboard – Use the dashboard to monitor replica health and synchronization states.
• Replica-Specific DMVs – Analyze views like sys.dm_hadr_availability_replica_states and sys.dm_hadr_database_replica_states to get detailed insights into replica performance and synchronization status.

Conclusion

The HADR_FABRIC_CALLBACK wait type in SQL Server is a key indicator of communication delays between replicas in an Always On Availability Group. While these waits are normal during log synchronization and failover operations, excessive waits may signal network issues, resource bottlenecks, or inefficient workloads. By optimizing your network, improving hardware performance, and monitoring replica health, you can minimize HADR_FABRIC_CALLBACK waits and ensure smooth operation of your Always On environment.

If you need expert assistance with Always On Availability Groups, SQL Server performance tuning, or resolving wait types, Stedman Solutions offers managed services to help you maintain a high-performing and reliable SQL Server environment.

Watch on YouTube

Find out more about our SQL Server Managed Services

Applies to

Related Waits

See Also

• More About HADR_FABRIC_CALLBACK