Worker threads are running out

What this check looks for

Two signals:

  • THREADPOOL in sys.dm_os_wait_stats with waiting_tasks_count greater than zero, which means at least one request has had to queue for a worker since the last restart.
  • Any visible online scheduler with a work_queue_count above zero right now, which means work is queued at this moment.

The configured max worker threads value is read from sys.configurations and included, because zero there means SQL Server chooses the number, and that is what most instances have.

Why it matters

Unlike almost every other wait type, this one is not a matter of degree. Small amounts of most waits are normal and uninteresting. Any THREADPOOL wait at all is worth knowing about, because of what happens when the pool is fully consumed.

Once every worker thread is in use, SQL Server cannot accept a new connection, because accepting one requires a worker. That includes:

  • Every application connection, which fails or hangs.
  • The connection your monitoring tool uses.
  • The connection you would open to find out what is wrong.

Meanwhile the CPU sits idle, because the threads are all waiting rather than working. A server that is completely unreachable with no CPU load is one of the few symptoms that reliably gets reported as a network or hardware failure, and the investigation starts on the wrong team.

The underlying cause is almost always a blocking chain. One blocked session is one worker. A hundred sessions blocked behind it are a hundred workers, and on a machine with a few cores the default worker limit is only a few hundred. A single long blocking chain can consume the pool in minutes. The fix is upstream, at the head of the blocking chain.

Raising max worker threads usually makes it worse. More workers means more memory for thread stacks and more context switching, and the blocking chain still grows to fill whatever number you set. It converts a fast failure into a slow one.

How to confirm it yourself

SELECT [wait_type], [waiting_tasks_count],
       [wait_time_ms] / 1000 AS [wait_time_seconds],
       [max_wait_time_ms]
  FROM sys.dm_os_wait_stats WITH (NOLOCK)
 WHERE [wait_type] = 'THREADPOOL';

What the pool looks like right now:

SELECT SUM([current_workers_count])   AS [workers_in_use],
       SUM([active_workers_count])    AS [active_workers],
       SUM([work_queue_count])        AS [queued_work],
       SUM([runnable_tasks_count])    AS [runnable_tasks],
       MAX([max_workers_count])       AS [max_workers]
  FROM sys.dm_os_schedulers WITH (NOLOCK)
 WHERE [status] = 'VISIBLE ONLINE';

And, since the cause is normally blocking:

SELECT r.[session_id], r.[blocking_session_id], r.[wait_type], r.[wait_time],
       r.[status], DB_NAME(r.[database_id]) AS [database_name], t.
  FROM sys.dm_exec_requests AS r WITH (NOLOCK)
 OUTER APPLY sys.dm_exec_sql_text(r.[sql_handle]) AS t
 WHERE r.[blocking_session_id] <> 0
 ORDER BY r.[wait_time] DESC;

How to fix it

During the incident, when you cannot connect at all:

Use the dedicated admin connection. It has its own scheduler and its own worker, which is exactly what it exists for. Connect with sqlcmd -A -S YourServer or by prefixing the server name with ADMIN: in SQL Server Management Studio. Enable remote DAC in advance, because you cannot enable it during the outage.

From there, find the head of the blocking chain, the session with a null or zero blocking_session_id that everything else is waiting on, and kill it.

Afterwards, fix the thing that caused the chain:

  1. Find the blocking root. It is usually a long transaction, a missing index turning a seek into a scan under a lock, or a lock escalation.
  2. Look at the application’s connection pool. A pool that opens hundreds of connections and retries aggressively turns a small delay into thread exhaustion.
  3. Check for parallel query storms. A parallel plan consumes one worker per thread per branch, so a high MAXDOP plus high concurrency multiplies worker usage.
  4. Enable remote DAC now, before you need it:
EXEC sp_configure 'remote admin connections', 1;
RECONFIGURE;

Leave max worker threads at 0 unless you have a specific, measured reason not to.

How long it takes

About an hour and a half, mostly spent on whatever caused the blocking. Clearing the immediate chain takes a minute once you can connect.

Report Why you would go there
Blocking Tree The chain consuming the workers, drawn as a tree.
Blocking Queries The head of the chain and what it is running.
Waits THREADPOOL against every other wait on the instance.
Sessions How many connections exist, and which application opened them.
SQL CPU Schedulers Scheduler by scheduler worker and queue counts.
Configuration Values The configured max worker threads and remote DAC setting.
Check
A transaction has been open far too long The usual head of the blocking chain.
Remote DAC Whether the connection you will need in the incident is enabled.
Max degree of parallelism Parallel plans consume workers several at a time.
Memory grants are queuing The other resource shortage that looks like an idle hang.

Frequently asked questions

Only a few THREADPOOL waits since the last restart. Is that a problem? It means the instance came close at least once. It is not an emergency, but it is the warning that comes before the outage, and it is worth understanding what happened.

Should I raise max worker threads? Almost certainly not. It is a limit that exists so failure is fast and diagnosable rather than a slow collapse. Raising it treats the number as the problem when the blocking chain is the problem.

The server has plenty of CPU headroom. That is the characteristic symptom. The workers are waiting, not working, so the CPU is idle while the server is unusable.

Why does the check mention the dedicated admin connection? Because the situation it describes is one where no ordinary connection will succeed. If remote DAC is disabled, the check on that setting is the one to act on first.