Add metrics max queue latency

[nicktindall]

We already publish the 50th, 90th and 99th percentile queue latencies for time tracking thread pools, this PR adds a metric that publishes the most recent max queue latency observed by org.elasticsearch.action.admin.cluster.node.usage.TransportNodeUsageStatsForThreadPoolsAction

I added a new metrics class to make it available to the action, I'm open to the idea of putting these metrics somewhere else, it didn't seem like we had an obvious home for them currently. There is DesiredBalanceMetrics but that strikes me as being more focused on the balancer rather than the deciders.

Relates: ES-12631

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[mhl-b]

Why do you need a barrier? I think countdown is enough to block thread-pool and let one task sit in queue.

        var latch = new CountDownLatch(1);
        range(0, threads + 1).forEach(i -> pool.execute(() -> safeAwait(latch)));
        var queueTimeMs = between(100,200);
        safeSleep(queueTimeMs);
        latch.countDown();

[mhl-b]

There is DesiredBalanceMetrics but that strikes me as being more focused on the balancer rather than the deciders.

I don't mind having queue metrics in DesiredBalanceMetrics, at least, everything related to allocation in one place. Also allocators and balancer tend to use same metrics.

[Copilot]

Pull Request Overview

This PR adds a new metric that publishes the maximum queue latency for the write thread pool to support write load constraint decisions. The metric is collected by TransportNodeUsageStatsForThreadPoolsAction and exposed through the existing DesiredBalanceMetrics system.

• Refactors DesiredBalanceMetrics to be injected as a dependency instead of created internally
• Adds max queue latency metric registration and collection functionality
• Updates all test constructors to use the new DesiredBalanceMetrics.NOOP instance

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
DesiredBalanceMetrics.java	Adds NOOP instance and method to register write load decider max latency gauge
DesiredBalanceShardsAllocator.java	Changes constructor to accept DesiredBalanceMetrics instead of creating internally
ClusterModule.java	Creates and injects DesiredBalanceMetrics instance
TransportNodeUsageStatsForThreadPoolsAction.java	Adds max queue latency metric collection and registration
WriteLoadConstraintDeciderIT.java	Adds integration test for the new metric
Test files	Updates all test constructors to use DesiredBalanceMetrics.NOOP

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[mhl-b]

LGTM