lib-observability

lib-observability is Lerian's shared Go library for observability and telemetry. It provides a unified, OpenTelemetry-native instrumentation layer for tracing, metrics, structured logging, panic recovery with telemetry, and production-safe assertions — extracted from lib-commons to give observability its own versioning, dependency footprint, and release cadence.

What's in this library

Telemetry bootstrap and tracing (tracing)

Full OpenTelemetry SDK lifecycle management: OTLP/gRPC exporter setup for traces, metrics, and logs; TracerProvider, MeterProvider, and LoggerProvider construction via a single NewTelemetry(cfg) call; global provider opt-in with ApplyGlobals(); head sampling with SampleRatio (0 keeps the SDK default of recording every trace; a value in (0, 1] installs ParentBased(TraceIDRatioBased(ratio)), and anything else fails NewTelemetry with ErrInvalidSampleRatio); ForceFlush(ctx) to push buffered traces, metrics, and logs without shutting the providers down; and graceful shutdown with ShutdownTelemetry(). The exporters pass transport security explicitly — TLS credentials (TLS 1.2 floor) when InsecureExporter is false, plaintext when it is true — so an OTEL_EXPORTER_OTLP_* endpoint without a scheme no longer downgrades the connection; those environment variables are normalized in place with the matching scheme. Includes trace context propagation for HTTP, gRPC, and message queues (Kafka/Redpanda/RabbitMQ), span error/event recording helpers, struct-to-attribute conversion with automatic sensitive field redaction, and custom SpanProcessor implementations for context-carried attribute injection.

Metrics (metrics)

Thread-safe MetricsFactory with lazy instrument caching and a fluent builder API for Counters, Gauges, and Histograms, each in an int64 (Counter, Gauge, Histogram) and a float64 (Float64Counter, Float64Gauge, Float64Histogram) form — use float64 for fractional values such as cost and for durations in seconds, which is the unit the default histogram buckets assume. Provides .WithLabels() / .WithAttributes() chaining followed by .Add(), .Set(), or .Record() — all with explicit error returns. Includes pre-configured domain metric recorders (accounts, transactions, routes, operations) and system infrastructure gauges (CPU, memory). Ships a NewNopFactory() for tests and disabled-metrics environments.

HTTP client and server instrumentation (httpobs)

A thin, nil-safe wrapper over otelhttp that turns an outbound HTTP transport into an instrumented one: every outbound request is classified as a call to an external dependency (span kind CLIENT) and emits http.client.request.duration (seconds). NewTransport(base, opts...) wraps the transport the app already built (preserving its TLS/timeout/proxy config); NewClient(base, opts...) is a convenience returning a ready *http.Client. Bounded span name by default (HTTP <METHOD>), no-op when telemetry is off. See "Outbound call instrumentation" below.

NewHandler(next, opts...) is the inbound counterpart for a stdlib net/http server (a Fiber v3 app uses middleware.WithTelemetry instead — never both on the same server). It produces a SERVER span and emits http.server.request.duration (seconds). The default span name is the method plus the registered route pattern (r.Pattern, set by the Go 1.22+ ServeMux) — GET /users/{id}, or the method alone when nothing matched — never the concrete URL path. A method-qualified registration ("GET /users/{id}") names the span exactly like a bare one: the pattern's own method prefix is dropped rather than repeated. Bodies and the Authorization header are never recorded, and url.path is the route template rather than the concrete path (see "HTTP server telemetry safety" below). Inbound traceparent is ignored by default and every request starts a new root trace; pass httpobs.WithPropagators(otel.GetTextMapPropagator()) to continue a trusted caller's trace — the same trust decision TrustInboundTraceContext expresses for the Fiber and gRPC paths. A request that was already partly handled before it reached NewHandler (answered by an outer fast path, then replayed) can be served under httpobs.ContextWithStartTime(ctx, start), so its span starts at start and its duration sample is measured from it.

Manual client-span helper (tracing.StartClientSpan)

For outbound calls that have no dedicated wrapper (e.g. the document database, custom RPC/SDK), tracing.StartClientSpan(ctx, tracer, name, opts...) starts a span already classified as CLIENT, so hand-rolled network hops stop defaulting to INTERNAL. The CLIENT kind is an overridable default. See "Outbound call instrumentation" below.

Structured logging (log)

A minimal, implementation-agnostic Logger interface with five methods (Log, With, WithGroup, Enabled, Sync), four severity levels, and typed Field constructors (String, Int, Bool, Err, Any). Includes a stdlib-based GoLogger with CWE-117 log-injection prevention, a NopLogger for tests, production-aware error sanitization (SafeError, SanitizeExternalResponse), and a generated mock for unit testing.

Zap adapter with OTEL bridge (zap)

A zap adapter implementing the Logger interface, with automatic trace_id and span_id injection into every log entry. Bridges zap output to the OpenTelemetry Logs SDK via otelzap, enabling unified log collection through the OTLP pipeline. Supports environment-aware configuration (production, staging, development, local) and runtime log level adjustment. The configured level (Config.Level, else LOG_LEVEL, else the environment default) governs the OTLP bridge as well as the local sink: an entry below it is exported by neither, and Logger.Level().SetLevel moves both. Set Config.Output to send every encoded entry to your own io.Writer — a rotated file for a daemon, or anywhere off stdout for a full-screen terminal client — keeping the same encoder, level, sampling and OTLP bridge. Leaving it nil keeps the default sink, zap's own stderr; the caller owns the writer's lifecycle (rotation, redaction, closing). Set Config.Encoding ("json" or "console") to choose the encoder directly instead of deriving it from the environment — a process whose own config asks for JSON while its deployment environment is development no longer has to misdeclare the environment; empty keeps today's default, and an unrecognised value is an error from New rather than a silent fallback. Set Config.DisableSampling to drop the sampler: the production profile samples at 100:100, so past the 100th copy of a message inside a second only every 100th is written — right for a service under load, wrong for a diagnostic log a human reads afterwards (an agent harness, a CLI, a job whose output is the deliverable).

Panic recovery with telemetry (runtime)

Policy-driven panic recovery (KeepRunning / CrashProcess) with full observability integration: span event recording (panic.recovered), panic counter metrics (panic_recovered_total), structured logging, and optional external error reporter forwarding. Provides safe goroutine launchers (SafeGo, SafeGoWithContext) and HandlePanicValue for integration with HTTP/gRPC framework recovery middleware. Supports production mode for stack trace redaction.

Production assertions with instrumentation (assert)

A context-scoped Asserter that validates domain invariants at runtime without panicking — every assertion failure returns an error, records a span event (assertion.failed), and increments the assertion_failed_total metric counter. Includes a predicates library for financial domain validation (decimal precision, balance sufficiency, transaction state transitions, debit/credit equality) alongside general-purpose checks (NotNil, NotEmpty, NoError, ValidUUID).

Observability constants and context carriers

Shared OTEL attribute prefixes, metric names (including the gen_ai.* generative-AI attributes and client metrics), event names, database system identifiers, header constants (traceparent, Traceparent, Tracestate), label sanitization (SanitizeMetricLabel), and sensitive field detection for cross-cutting redaction. Context carrier helpers (ContextWithTracer, ContextWithMetricFactory, ContextWithLogger, ContextWithSpanAttributes) for propagating observability primitives through context.Context.

Redaction engine

A configurable Redactor with rule-based field processing supporting mask, hash (SHA-256), and drop actions. Applies automatically to span attributes via the RedactingAttrBagSpanProcessor and to struct-to-attribute conversion. Includes ObfuscateStruct for generic struct field obfuscation and integration with the sensitive field detection layer.

Design principles

• Explicit initialization — no implicit global state; NewTelemetry + ApplyGlobals is opt-in
• Nil-safe and no-op by default — every factory and logger has a null-object variant for safe degradation
• Errors over panics — metric/builder operations return errors; assertions return errors instead of panicking
• Redaction-first — sensitive fields are masked in spans, logs, and attributes by default
• Interface-driven — Logger, MetricsFactory, ErrorReporter, and DLQMetrics are all interface-bound for testability

Instrumentation scope and build identity

Two scopes, two owners.

Signals this library emits — the HTTP and gRPC middleware spans, messaging spans, the transport duration instruments (HTTP, gRPC, messaging) — are stamped with the library's own identity, resolved from the running binary:

Field	Value
instrumentation_scope.name	github.com/LerianStudio/lib-observability/v4
instrumentation_scope.version	the module version linked into the binary; (devel) for a source build or a local replace

Nothing configures this. The scope names the code that produced the signal, so a dashboard can tell "the library's HTTP middleware" from "a span the service opened by hand".

Operators: otel_scope_version on these series changes with every lib-observability release, so each upgrade replaces the transport metric series. Aggregate transport metrics without scope labels — sum by (http_request_method, http_response_status_code) (rate(http_server_request_duration_seconds_count[5m])), never by (otel_scope_version) — grep recording rules and alerts for otel_scope_version before upgrading, and note that the collector may drop the label altogether. The upgrade that introduced this scope is described in MIGRATION-v4.md.

Signals your service emits are attributed to your service, never to this library. That covers Telemetry.Tracer(name) and Telemetry.Meter(name), where the name you pass is your scope, and it also covers the carriers the middleware puts on the request context — the tracer and the metrics factory you pull out of observability.NewTrackingFromContext(ctx), and Telemetry.MetricsFactory. Those are scoped to TelemetryConfig.LibraryName exactly as configured, with no fallback: an empty LibraryName means an empty scope, as it always has. This library never rewrites that scope, so a business span or counter keeps its series identity across library upgrades.

Build identity on the resource

ServiceVersion and ServiceRevision describe the binary and travel on the OTel resource of traces, metrics, and logs:

Config field	Resource attribute	Source in the service
ServiceVersion	service.version	main.version, set at link time
ServiceRevision	vcs.ref.head.revision	main.revision, set at link time — the full git SHA

ServiceRevision is optional: blank omits the attribute and leaves the resource exactly as before. No format validation happens here. The values come from the binary, never from the environment: the service declares two variables in main, the build fills them with -ldflags "-X main.version=1.4.2 -X main.revision=$(git rev-parse HEAD)", and the service passes both in:

package main

import "github.com/LerianStudio/lib-observability/v4/tracing"

// Set at link time by -ldflags "-X main.version=... -X main.revision=...".
var version, revision string

func main() {
    telemetry, err := tracing.NewTelemetry(tracing.TelemetryConfig{
        ServiceName:     "midaz-ledger",
        ServiceVersion:  version,
        ServiceRevision: revision,
        DeploymentEnv:   "production",
        EnableTelemetry: true,
    })
    // ...
}

lib-commons is publishing a commons/buildinfo package in this same rollout that wraps these values (buildinfo.Get()); use it once it is available.

Outbound call instrumentation — span kind precedence

Outbound calls (crossing a process/network boundary: DB, cache, HTTP, another service) must be classified as CLIENT in telemetry. Inbound requests are SERVER; purely in-process work is INTERNAL. A raw tracer.Start(ctx, name) defaults to INTERNAL, so an outbound call instrumented by hand is mis-classified — which hides the latency/error of your external dependencies and inflates the INTERNAL series.

Rule of precedence — for humans and AI assistants:

1. Always prefer the dedicated wrapper. It sets the correct span kind (and metric) automatically, so you never have to remember WithSpanKind.

   Outbound call type	Use this	Span kind
   SQL database	sqlobs	CLIENT
   Redis / Valkey	redisobs	CLIENT
   HTTP client	httpobs	CLIENT
   Messaging (produce/consume)	messagingobs	PRODUCER / CONSUMER
   Inbound HTTP / gRPC (server)	middleware / grpcmiddleware	SERVER

2. Only when no wrapper exists, use tracing.StartClientSpan by hand — e.g. the document database (no stable driver instrumentation today) or a custom RPC/SDK call.

3. Never double-instrument. Do not wrap a call with a wrapper and also open a manual StartClientSpan around the same call — that produces two spans for one operation. Use the wrapper for wrapped call types; use StartClientSpan only for the rest.

Example — HTTP client (use the wrapper)

Wrap the transport your app already built so its custom TLS/timeout/proxy config is preserved. WithTracerProvider is required for the CLIENT span (otherwise only the metric is emitted).

import (
    "net/http"

    "github.com/LerianStudio/lib-observability/v4/httpobs"
)

func newInstrumentedClient(baseTransport http.RoundTripper) *http.Client {
    return httpobs.NewClient(baseTransport,
        httpobs.WithMeterProvider(meterProvider),
        httpobs.WithTracerProvider(tracerProvider), // required for the CLIENT span
    )
}

// When the app builds its own *http.Client, wrap only the transport:
//   client.Transport = httpobs.NewTransport(baseTransport,
//       httpobs.WithMeterProvider(meterProvider),
//       httpobs.WithTracerProvider(tracerProvider))

Migration note: once the transport is wrapped, remove any manual tracer.Start(...) you previously opened around the HTTP call (no double-instrumentation). The caller must fully read and close the response body — the span ends on body close.

Example — no wrapper (use StartClientSpan)

For an outbound call with no wrapper — e.g. the document database — replace the hand-rolled tracer.Start(...) so the span is CLIENT:

import "github.com/LerianStudio/lib-observability/v4/tracing"

// Before: outbound Mongo call rendered as INTERNAL
//   ctx, span := tracer.Start(ctx, "mongodb.find_holder")
// After: classified as an external-dependency call (CLIENT)
ctx, span := tracing.StartClientSpan(ctx, tracer, "mongodb.find_holder")
defer span.End()
// ... perform the document-database call ...

Credentials in outbound URLs (guaranteed, no opt-out): the URL recorded on a span never carries the query string, the fragment, userinfo or an opaque request target — url.full keeps scheme://host/path of a hierarchical URL and only scheme://host of an opaque one. An API key in the query (Gemini's ?key=, a pre-signed S3/GCS signature, any ?token=) is therefore never exported to the collector. The request on the wire is unchanged: the full URL is restored below the instrumentation, so the call, its propagation headers and its byte accounting are unaffected. Inbound (NewHandler) needs nothing for the query — the SERVER span records url.path and never the full URL, so an OAuth callback's ?code= is already safe; its url.path is covered in "HTTP server telemetry safety" below.

The path is kept, since it is what makes a span readable. If your outbound paths carry identifiers/PII, redact url.full in the OTel Collector (transform processor) — that is where path-shaped PII/cardinality redaction belongs.

HTTP server telemetry safety

WithHTTPLogging and WithTelemetry resolve the matched Fiber route only after the downstream handler returns. Access logs, server span names, and url.path therefore use the route template (for example, /v1/contracts/:contract_id) and omit the query string entirely. Unmatched traffic uses the stable /{unmatched} fallback; http.route remains absent, as required by OpenTelemetry.

httpobs.NewHandler gives the stdlib net/http path the same guarantee, with no opt-out. It resolves the route the same way — after the downstream handler returns, from r.Pattern — and rewrites url.path to the route template: a request to /users/42 is exported as url.path = http.route = /users/{id}, and traffic matching no route as url.path = /{unmatched} with http.route absent. An id, a CPF, or an account number in an inbound path therefore never leaves the process, and a scanner cannot inflate the attribute's cardinality.

WithoutCallerAttributes() additionally keeps the caller off the span: no user_agent.original, no client.address, no network.peer.address / network.peer.port. It is off by default; turn it on for a public listener, where those four are sourced from values an unauthenticated caller chooses (User-Agent, X-Forwarded-For, the peer address) onto a span that also carries this service's authenticated user and tenant ids. The handler below still receives the real User-Agent, X-Forwarded-For, and RemoteAddr, so access logs, rate limiters, and IP allowlists see exactly what arrived; server.address (this service's own host) is kept either way.

The HTTP middleware never derives tenant or customer identity from X-Tenant-Id. That client-controlled value is not added to access logs, server spans, or the built-in HTTP metrics. http.server.request.duration is limited to method, route template, response status, and error class.

Applications that need tenant-level HTTP telemetry can opt into four separate lerian.*.by_tenant instruments. The authentication layer must first attest a UUID and optional display name resolved from a validated credential; the metrics never fall back to a header, baggage, metadata, or generic span attribute.

Register exactly one HTTP telemetry middleware. The three variants are mutually exclusive: WithTelemetry (spans plus the standard HTTP server metrics), WithAuthenticatedTenantHTTPMetrics (the same, plus the per-tenant instruments), and WithTracingOnly (spans and no metric at all). Registering any two of them starts the server span twice. Registering both metric-emitting variants also records http.server.request.duration twice, corrupting RPS and error-rate queries; WithTracingOnly beside either of them doubles only the span, since it emits no metric.

mid := middleware.NewTelemetryMiddleware(telemetry)
app.Use(mid.WithAuthenticatedTenantHTTPMetrics(telemetry))
app.Use(func(c fiber.Ctx) error {
    claims := claimsFromValidatedCredential(c)
    tenantID, err := uuid.Parse(claims.TenantID) // JWT tenantId
    if err != nil {
        return fiber.ErrUnauthorized
    }
    tenantName := claims.TenantSlug // JWT tenantSlug; optional display label
    c.SetContext(observability.ContextWithAuthenticatedTenant(
        c.Context(), tenantID, tenantName,
    ))
    return c.Next()
})

tenant.id is the stable aggregation key. tenant.slug is a mutable display label only. Keep both in Grafana queries so a reused slug can never merge two different tenants:

sum by (tenant_id, tenant_slug) (
  rate(lerian_http_server_requests_by_tenant_total[5m])
)

Use {{tenant_slug}} as the legend, but never aggregate only by tenant_slug. A rename creates another attribute set for the same tenant.id. With the default cumulative SDK temporality, the old set can remain in the process aggregation state until the MeterProvider or process restarts; the backend series then remains for its retention period. Queries that need continuity across a rename must aggregate by tenant_id.

The instruments divide responsibility deliberately:

• lerian.http.server.requests.by_tenant counts volume by authenticated tenant and normalized route.
• lerian.http.server.responses_4xx.by_tenant counts HTTP 4xx responses by authenticated tenant and normalized route. Exact status codes stay out of the metric to prevent another cardinality multiplier.
• lerian.http.server.responses_5xx.by_tenant counts HTTP 5xx responses by authenticated tenant and normalized route. Dividing it by the request counter gives the route-level server-error rate.
• lerian.http.server.latency.by_tenant provides p50/p95/p99 by authenticated tenant and bounded response-status class, without route or method.
• Per-route tenant latency and exact status diagnosis belong in traces, which can carry tenant.id without metric-series multiplication.
• Global RED remains in http.server.request.duration; deployment collectors may add resource dimensions such as client_id without changing this contract.

This split is a correctness boundary, not only a cost optimization. With the current 14 explicit boundaries, a Prometheus histogram costs 17 series per attribute set; a counter costs one. tenant.slug is functionally 1:1 with tenant.id, so it does not multiply the steady-state set count. The validated steady-state 50-tenant × 30-route scenario produces 1,500 request sets, 500 4xx-response sets, 500 5xx-response sets, and 150 latency sets, with no overflow before any retained rename overlap. Putting route and status on one counter would instead create 9,000 sets and collapse 7,001 into otel.metric.overflow, silently losing tenant identity. This is an operational budget, not a universal guarantee: each adopter must recalculate authenticated tenants, normalized routes, and retained tenant-slug versions, then keep each instrument below the effective limit. The default is 2,000; applications can override it with NewTelemetryWithOptions(cfg, WithMetricCardinalityLimit(limit)). The SDK reserves one attribute set for overflow. For a counter, budget normalized routes × (tenants + retained rename versions) <= limit - 1. At the default limit, 30 routes and 50 tenants leave room for at most 16 retained full-route rename versions; the 17th can overflow. Count the stable unmatched-route fallback as a normalized route. Size the configured limit against projected tenant growth and rename history for the process lifetime, not only the current tenant count.

Telemetry may run before or after authentication and still observe the attested context. Register it first when the duration should include authentication latency. Requests without an explicitly authenticated tenant remain in the standard HTTP metric and are omitted from all tenant metrics. A later ContextWithAuthenticatedTenant or ContextWithAuthenticatedTenantID call replaces the earlier value; uuid.Nil clears it. The ID-only helper remains supported and emits the metrics without tenant.slug.

WithTracingOnly is for a service that already emits its own HTTP RED metrics and needs them emitted once, under its own instrument names, route template, and labels. It produces the same server span, request-id header, and context wiring as WithTelemetry, and records nothing itself — no http.server.request.duration, no http.server.active_requests, no per-tenant instrument, and no background host-metrics collector — even when the Telemetry carries a MeterProvider and a MetricsFactory. On the tracer path the metrics factory stays on the request context, so the application's own metrics are unaffected; with no TracerProvider configured this handler returns before that wiring, exactly as WithTelemetry does.

mid := middleware.NewTelemetryMiddleware(telemetry)
app.Use(mid.WithTracingOnly(telemetry))

Tenant ID propagation

The gRPC middleware can still read a tenant identifier from request metadata and propagate it through telemetry as the tenant.id attribute / log field. HTTP applications must attach authenticated identity explicitly if their application telemetry requires it.

Wire protocol

• gRPC: canonical metadata key tenant-id. No aliases.

Values are normalized (trimmed, control chars stripped) and dropped silently when empty or longer than 128 bytes to bound telemetry cardinality.

Where it shows up

Signal	How it gets there	Action required by caller
HTTP access logs, server spans, and built-in metrics	Tenant/customer identity is never inferred from X-Tenant-Id.	None; the header is deliberately ignored by HTTP telemetry.
gRPC logs and traces	WithGrpcLogging and the span processor propagate tenant.id from canonical metadata.	None.
Custom application metrics	Not automatic. Metric labels are a high-impact cardinality decision left to the caller.	Attach authenticated identity explicitly; middleware.RequestAttributes(ctx) can copy an application-populated request bag.

Example for custom metrics:

import "github.com/LerianStudio/lib-observability/v4/middleware"

counter, _ := factory.Counter("orders.created")
_ = counter.
    WithAttributes(middleware.RequestAttributes(ctx)...).
    Add(ctx, 1)

Trust boundary

ResolveTenantIDFromHTTP remains available for source compatibility, but the shared HTTP middleware no longer invokes it automatically. X-Tenant-Id is client-controlled and must not become infrastructure telemetry identity.

If an auth layer resolves the real tenant from a signed credential, it can call observability.ContextWithSpanAttributes(ctx, attribute.String("tenant.id", real)) for explicit application spans or business metrics. The standard HTTP duration histogram still excludes identity. For the opt-in tenant-attributed HTTP metrics, the auth layer must instead call observability.ContextWithAuthenticatedTenantID; generic span attributes are deliberately insufficient.

Relationship to lib-commons

This library was extracted from lib-commons to decouple observability infrastructure from service primitives and data connectors. Services that previously imported lib-commons for telemetry can migrate to lib-observability for a lighter dependency graph. lib-commons will depend on lib-observability for its own instrumentation needs (database spans, streaming metrics, middleware telemetry).