Skip to content

@package component templates missing _tier_preference after logs@settings v23 upgrade #7675

Description

@Supplementing

Problem Statement

Fleet-server's @package component templates (e.g., logs-fleet_server.output_health@package) do not set _tier_preference. Prior to a recent ES upgrade, the logs@settings built-in component template (now v23) explicitly set data_hot,data_content tier preference, ensuring new backing indices landed on hot-tier nodes with promotable shards. That setting was removed from logs@settings as part of Elastic's shift to Data Stream Lifecycle. Fleet-server's templates were not updated to compensate, leaving tier allocation to ES defaults. On affected deployments, new backing indices after a rollover land on nodes with unpromotable shards, which — combined with the missing maintenance privilege — causes 403s on output health writes.

This affects any customer on the ES version shipping logs@settings v23 whose fleet-server data streams have rolled over.

Goals

  • New fleet-server data stream backing indices are consistently allocated to the hot tier on standard deployments
  • The fix survives future rollovers without manual intervention

Non-Goals

  • Fixing the missing maintenance privilege in the service account role (tracked here)
  • Modifying tier routing for non-fleet data streams
  • Changing DSL lifecycle configuration

Acceptance Criteria

  • logs-fleet_server.* and metrics-fleet_server.* @package component templates include _tier_preference: data_hot,data_content
  • After a rollover, new backing indices are allocated to the hot tier on a standard ESS deployment
  • The fix does not interfere with DSL-managed lifecycle (data_retention: 7d) on the output health data stream
  • Serverless deployments are unaffected (tier preference is a no-op in serverless)

Spike / Investigation Notes

Confirmed via GET /_component_template/logs-fleet_server.output_health@package that no _tier_preference is set. The logs@settings built-in template (modified_date_millis diverges from created_date_millis, consistent with an ES upgrade) no longer carries this setting. The immediate workaround applied to the affected customer was adding _tier_preference to the logs-fleet_server.output_health@custom component template (the @custom slot is listed in ignore_missing_component_templates and safe to populate). The upstream fix belongs in the @package templates so it ships with the integration rather than requiring per-deployment remediation.

Open Questions

  • Do metrics-fleet_server.* @package templates have the same gap? Spot-checked logs-fleet_server.output_health only during the incident.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions