Problem Statement
Fleet-server's @package component templates (e.g., logs-fleet_server.output_health@package) do not set _tier_preference. Prior to a recent ES upgrade, the logs@settings built-in component template (now v23) explicitly set data_hot,data_content tier preference, ensuring new backing indices landed on hot-tier nodes with promotable shards. That setting was removed from logs@settings as part of Elastic's shift to Data Stream Lifecycle. Fleet-server's templates were not updated to compensate, leaving tier allocation to ES defaults. On affected deployments, new backing indices after a rollover land on nodes with unpromotable shards, which — combined with the missing maintenance privilege — causes 403s on output health writes.
This affects any customer on the ES version shipping logs@settings v23 whose fleet-server data streams have rolled over.
Goals
- New fleet-server data stream backing indices are consistently allocated to the hot tier on standard deployments
- The fix survives future rollovers without manual intervention
Non-Goals
- Fixing the missing
maintenance privilege in the service account role (tracked here)
- Modifying tier routing for non-fleet data streams
- Changing DSL lifecycle configuration
Acceptance Criteria
Spike / Investigation Notes
Confirmed via GET /_component_template/logs-fleet_server.output_health@package that no _tier_preference is set. The logs@settings built-in template (modified_date_millis diverges from created_date_millis, consistent with an ES upgrade) no longer carries this setting. The immediate workaround applied to the affected customer was adding _tier_preference to the logs-fleet_server.output_health@custom component template (the @custom slot is listed in ignore_missing_component_templates and safe to populate). The upstream fix belongs in the @package templates so it ships with the integration rather than requiring per-deployment remediation.
Open Questions
- Do
metrics-fleet_server.* @package templates have the same gap? Spot-checked logs-fleet_server.output_health only during the incident.
References
Problem Statement
Fleet-server's
@packagecomponent templates (e.g.,logs-fleet_server.output_health@package) do not set_tier_preference. Prior to a recent ES upgrade, thelogs@settingsbuilt-in component template (now v23) explicitly setdata_hot,data_contenttier preference, ensuring new backing indices landed on hot-tier nodes with promotable shards. That setting was removed fromlogs@settingsas part of Elastic's shift to Data Stream Lifecycle. Fleet-server's templates were not updated to compensate, leaving tier allocation to ES defaults. On affected deployments, new backing indices after a rollover land on nodes with unpromotable shards, which — combined with the missingmaintenanceprivilege — causes 403s on output health writes.This affects any customer on the ES version shipping
logs@settingsv23 whose fleet-server data streams have rolled over.Goals
Non-Goals
maintenanceprivilege in the service account role (tracked here)Acceptance Criteria
logs-fleet_server.*andmetrics-fleet_server.*@packagecomponent templates include_tier_preference: data_hot,data_contentdata_retention: 7d) on the output health data streamSpike / Investigation Notes
Confirmed via
GET /_component_template/logs-fleet_server.output_health@packagethat no_tier_preferenceis set. Thelogs@settingsbuilt-in template (modified_date_millisdiverges fromcreated_date_millis, consistent with an ES upgrade) no longer carries this setting. The immediate workaround applied to the affected customer was adding_tier_preferenceto thelogs-fleet_server.output_health@customcomponent template (the@customslot is listed inignore_missing_component_templatesand safe to populate). The upstream fix belongs in the@packagetemplates so it ships with the integration rather than requiring per-deployment remediation.Open Questions
metrics-fleet_server.*@packagetemplates have the same gap? Spot-checkedlogs-fleet_server.output_healthonly during the incident.References