Emit account_id and job metadata as Kubernetes pod labels

[wesleyjellis]

Summary

Emit account/job metadata as Kubernetes pod labels rather than weaving it
into the pod name (which risks the 63-char limit and OpsLevel-side naming
conventions). This enables native DataDog filtering via podLabelsAsTags with
no pod-name parsing. The pod/ConfigMap/PDB name is unchanged.

Labels stamped on each job pod:

label	value	cardinality
app.kubernetes.io/name	opslevel-job	1
opslevel.com/mode	faktory | api	2
opslevel.com/account-id	account id (when set)	bounded by # accounts
opslevel.com/job-id	job id	unbounded — see note
opslevel.com/job-type	job kind / template slug (when set)	~13

Labels are added after building the PDB label selector so they stay out of the
selector, which the app.kubernetes.io/instance label already makes unique.

Also adds a getRunnerJobVariable helper (named explicitly since a similar
helper is anticipated for egress proxies).

Fixes

• ACCOUNT_ID lookup: the OpsLevel API uppercases/sanitizes variable keys
  before sending them (RunnerJobType#format_key_as_valid_env_var_name), so the
  account id arrives as ACCOUNT_ID, not account_id. The original lookup used
  the lowercase key and never matched, leaving the label always empty. Now
  matches the uppercase key.

Requires a companion OpsLevel change

opslevel.com/job-type is a no-op until the OpsLevel API emits a JOB_TYPE
variable. The job's kind is its template (Runners::JobRun#template_name,
~13 values) and is currently only carried by the DB template_id column — it is
never sent to the runner. To light up the label, add a derived variable in
app/models/runners/job_run.rb#derived_variables (alongside JOB_ID /
ACCOUNT_ID):

{ "key" => "JOB_TYPE", "value" => template_name.demodulize.underscore, "sensitive" => false },
# e.g. "Runners::RepoGrepV1" -> "repo_grep_v1"

Sending a clean slug from the server keeps the label value valid (raw
template_name contains ::, which is invalid in a k8s label value) and
low-cardinality.

Cardinality note: opslevel.com/job-id is unbounded (one value per job).
It's fine as a k8s label (kubectl filtering) and as a DataDog log
attribute, but should not be mapped into podLabelsAsTags as a metric
tag. job-type is the opposite — low cardinality and ideal for tagging.

Test plan

• go build ./...
• go test ./pkg passes
• TestGetRunnerJobVariable_* cover the helper, including a case-sensitivity
  guard for the ACCOUNT_ID bug

🤖 Generated with Claude Code

[archf]

This is super cool, thanks a heap!

Let's also add a changie record and we're good to go!

[archf]

can we 'dup' and add short form keys as well?

e.g.:

labels["accountiD"] = accountID ...
[...]

That would be cool for a CLI speedrun instead of typing out the long form.

e.g. kubectl get pods -l accountID=XXX

[wesleyjellis]

Do you want to dupe all or just accountID?

[archf]

I would do all (most?) in the shortest typeable form. It's not the most expensive thing and but it allows you to find/select pod(s) without any external grepping tool.

Omit whatever doesn't make sense. Worst case, we add them later if we wish we had them.

On the labels now that I look at them

I'm not certain what name means // how useful it is since they are all in the same job namespace already.

labels["app.kubernetes.io/name"] = "opslevel-job"

What is job-type? That seems very useful.

Maybe we don't need mode since it's inferred by the namespace?

We want labels that allow to select useful subsets of pods amongst all of them to tail logs or whatever and add useful tags in Datadog for metrics, monitors and dashboards.

[archf]

Ah, I saw in the PR description what these mean. Very cool!

For some it could be worthy of a code comment for the soul that looks at this later.

[wesleyjellis]

mode separates faktory from legacy jobs, maybe not the most useful but I'd keep it