Convert heavy queries from 5xx to 4xx

[eeldaly]

What this PR does:
This PR introduces a timeout in querier (default 59s) to timeout before we hit timeout. Once this is hit, we convert queries that took longer than X (default 40s) PromQL evaluation time from 5XX to 4XX. This conversion and 1s earlier timeout is disabled by default.

Default new configs:
querier.timeout-classification-enabled: false
querier.timeout-classification-deadline: 59s
querier.timeout-classification-eval-threshold: 40s

Response Outputs:

Current output on timeout:

'Response code: 504\n'
{'Date': 'Thu, 26 Mar 2026 20:59:06 GMT', 'Content-Type': 'text/plain', 'Content-Length': '24', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'ee98f602-651c-4783-946e-f547a8d88e32', 'server': 'amazon'}
''
upstream request timeout

New output on timeout (Less than 40s PromQL evaluation time):

'Response code: 504\n'
{'Date': 'Thu, 26 Mar 2026 22:21:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '75', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'ec88ba9b-7d21-45f4-8996-418b2e7b812b', 'server': 'amazon', 'vary': 'Origin'}
''
{"status":"error","errorType":"timeout","error":"upstream request timeout"}

New output on timeout (More than 40s PromQL evaluation time):

'Response code: 422\n'
{'Date': 'Thu, 26 Mar 2026 21:05:46 GMT', 'Content-Type': 'application/json', 'Content-Length': '138', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'bf5b69b7-8d11-4e9e-95be-09fbfa017bbd', 'server': 'amazon', 'vary': 'Origin'}
''
{"status":"error","errorType":"execution","error":"query timed out: query spent too long in evaluation - consider simplifying your query"}

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Do you think we can align some of the defaults with the default engine timeout in Cortex?

[yeya24]

One last ask from me is to add a doc talking about the usecase and how users can use this feature