Add CrashOnPanic middleware + wrap distributor push routes

[justinjung04]

What this PR does:

Adds a CrashOnPanic HTTP middleware that terminates the distributor process when a panic occurs during request handling, instead of letting Go's default net/http recovery keep the process running in a corrupted state.

This is motivated by an incident where a distributor panicked with growslice: len out of range during protobuf unmarshal of a corrupt remote_write payload. The net/http server caught the panic and kept the process alive, but the distributor entered a bad state — returning PROTOCOL_ERROR on all gRPC health checks to ingesters. Manually deleting the pod was required to recover.

2025-03-10T10:07:08.767Z  2025/03/10 10:07:08 http: panic serving 10.1.38.172:52484: runtime error: growslice: len out of range
goroutine 11412 [running]:
net/http.(*conn).serve.func1()
  net/http/server.go:1947 +0xbe
panic({0x2c21c00?, 0x38309d0?})
  runtime/panic.go:785 +0x132
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
  github.com/opentracing-contrib/go-stdlib/nethttp/server.go:159 +0x175
panic({0x2c21c00?, 0x38309d0?})
  runtime/panic.go:785 +0x132
github.com/cortexproject/cortex/pkg/cortexpb.(*TimeSeries).Unmarshal(0xc03c58b860, {0xc0537bec0e, 0x177, 0x29ee84})
  github.com/cortexproject/cortex/pkg/cortexpb/cortex.pb.go:3035 +0xaf5
github.com/cortexproject/cortex/pkg/cortexpb.(*PreallocTimeseries).Unmarshal(...)
  github.com/cortexproject/cortex/pkg/cortexpb/timeseries.go:74 +0x77
github.com/cortexproject/cortex/pkg/cortexpb.(*WriteRequest).Unmarshal(...)
  github.com/cortexproject/cortex/pkg/cortexpb/cortex.pb.go:2789 +0x918
github.com/cortexproject/cortex/pkg/cortexpb.(*PreallocWriteRequest).Unmarshal(...)
  github.com/cortexproject/cortex/pkg/cortexpb/timeseries.go:63 +0x86
github.com/cortexproject/cortex/pkg/util.ParseProtoReader(...)
  github.com/cortexproject/cortex/pkg/util/http.go:169 +0x3be
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1(...)
  github.com/cortexproject/cortex/pkg/util/push/push.go:32 +0x1ca

Since distributors are stateless, the correct behavior is to crash immediately and let Kubernetes restart the pod with clean state.

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• [N/A] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• [N/A] docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags