atunnel: broker actor certificates through atelet by EItanya · Pull Request #708 · agent-substrate/substrate

Eitan Yarmush (EItanya) · 2026-08-03T04:24:11Z

Closes #706

Summary

broker short-lived actor certificates from atelet over a same-node mTLS Unix socket
keep the actor private key in atunnel and renew the certificate before expiry
authenticate egress CONNECT using the actor certificate instead of bearer tokens

Testing

make verify
go test -race ./internal/atunnel

Lior Lieberman (LiorLieberman) · 2026-08-03T16:09:39Z

Thanks Eitan Yarmush (@EItanya) !

Quick question before I review more deeply today.

With the goal of "having a JWT available to atunnel to append on a CONNECT request to the egress" -- What parts of the this PR addresses?

Here are the parts we needed (before this pr -- havent reviewed it yet);

Improve the JWT issuance flow as part of MintJWT (properly authorize atelets, having the JWT formats solid, etc)
Implement an in-memory cache with go background routine that refreshes certs for every active actor.
Implement the UDS socket between atunnel and atelet to pass the certs+jwts.

From a very brief glance - I think your PR is attempting to do all three. Is that correct?

Also, For (1), we likely need to sort out two things to make the JWTs useful:

Get the issuer a publicly accessible DNS name that is also oidc-compliant (cc: Taahir Ahmed (@ahmedtd))
Agree on the format for the Subject field on JWTs

See for ref -

substrate/cmd/ateapi/internal/actoridentity/actoridentity.go

Lines 135 to 138 in 3ed6aa0

    
           // TODO: This is currently API but it has to be a globally unique, oidc-compliant and accsible DNS name 
        
           Issuer: "https://api.ate-system.svc", 
        
           // TODO: this format is very likely going to change. 
        
           Subject:    fmt.Sprintf("atespaces:%s:actors:%s", req.GetAtespace(), req.GetActorName()),

Eitan Yarmush (EItanya) · 2026-08-03T17:13:59Z

Yes, with two clarifications: this PR renews JWTs, not actor certificates, and each worker has only one active actor.

JWT issuance/authorization: implemented. Atunnel authenticates to atelet over mTLS using the worker Pod identity. Atelet derives the worker UID from that certificate and resolves its current actor assignment; the request does not supply actor identity. Ateapi then independently revalidates the atelet’s node, worker assignment, actor UID, and permitted audience before minting. The JWT includes the actor UID/resource version and worker Pod UID.
In-memory renewal: implemented for the worker’s single active actor. Atunnel mints before activation, stores the JWT in memory, renews with roughly 10% lifetime remaining, retries until expiry, and fails closed for new tunnels after expiry. Deactivation clears the JWT and stops renewal.
UDS broker: implemented. The UDS carries only JWT requests/responses—not certificates. The Pod certificate is used to mutually authenticate the atunnel↔atelet connection, including verifying both are on the same node.

I agree the issuer and subject contracts remain unresolved. This PR intentionally retains the existing issuer/subject behavior and does not implement PEP verification yet. Before verification lands, we need a stable OIDC issuer with discovery/JWKS reachable by the PEP.

Lior Lieberman (LiorLieberman) · 2026-08-03T22:55:07Z

+			ActorResourceVersion: actor.GetMetadata().GetVersion(),
+			WorkerPodUid:         req.GetWorkerPodUid(),


why do we need these?

Lior Lieberman (LiorLieberman) · 2026-08-04T17:28:31Z

+	clientJWTIssuer      = pflag.String("client-jwt-issuer", "", "The expected issuer URL for client JWTs.")
+	clientJWTAudience    = pflag.String("client-jwt-audience", "", "The expected audience for client JWTs.")
+	actorIDJWTPoolFile   = pflag.String("actor-id-jwt-pool", "", "The file that contains the serialized JWT authority pool for signing actor JWTs")
+	egressGatewayAddress = pflag.String("egress-gateway-address", "", "Address of the egress PEP. Empty disables tunneled egress.")


why do need egressgateway on ateapi?

Lior Lieberman (LiorLieberman) · 2026-08-04T17:36:32Z

+	// control resolves the authenticated worker Pod to its current assignment.
+	control ateapipb.ControlClient
+	// identity revalidates that assignment and signs the actor certificate.
+	identity ateapipb.ActorIdentityClient


identity as "identityService"? feels weird to call a field identity that does not represent an identity

Lior Lieberman (LiorLieberman) · 2026-08-04T17:38:00Z

+	// TODO: Before release, request an atunnel-specific MintCert purpose and
+	// require the egress PEP to reject generic actor certificates.
+	// The request deliberately carries no actor identity. The authenticated Pod
+	// UID is the only input used to select a worker and its current assignment.
+	workerUID, err := authenticatedWorkerUID(ctx)


not sure I am following the TODO comment. I thought we were saying that we need it to be an actor cert thats specifically for atunnel use.

I think it would be fine to merge without that, and then add it in a follow up? Either way works.

Lior Lieberman (LiorLieberman) · 2026-08-04T17:42:43Z

+	return identity.PodUID, nil
+}
+
+func restrictClientToNode(node *substratex509.PodIdentity) func(tls.ConnectionState) error {


can we get a more descriptive name and/or a comment that tells what you are doing in this func?

Lior Lieberman (LiorLieberman) · 2026-08-04T21:23:14Z

 		containers:   req.GetSpec().GetContainers(),
 		assetPaths:   req.GetRuntimeAssetPaths(),

-		actorVersion:         req.GetActorVersion(),


whats the rationale for removing the actorVersion

Lior Lieberman (LiorLieberman) · 2026-08-04T21:24:00Z

 	// and in every ateom pod (which mounts them as overlay lowerdirs).
-	ImageCacheDir = filepath.Join(BasePath, "image-cache")
+	ImageCacheDir          = filepath.Join(BasePath, "image-cache")
+	CredentialBrokerSocket = filepath.Join(BasePath, "credential-broker.sock")


can we add a comment on top of CredentialBrokerSocket?

Lior Lieberman (LiorLieberman) · 2026-08-04T21:26:48Z

+	"google.golang.org/grpc/credentials"
+)
+
+// BrokerCertificateSource owns atunnel's actor private key and obtains the


I could be wrong but dont we want atelet to be the one that generate the private keys?

Lior Lieberman (LiorLieberman) · 2026-08-04T21:30:54Z

-// Activate allows egress for one actor. There can be only one active actor per
-// worker. bearerToken may be empty until actor JWT issuance is available.
-func (e *Egress) Activate(dialer EgressDialer, atespace, actorName string, actorVersion int64, bearerToken string) error {
+// Activate allows egress with a previously obtained actor certificate and


with a previously obtained actor certificate

thinking out loud - do we care about actorVersion(s)?

Lior Lieberman (LiorLieberman) · 2026-08-04T21:44:12Z

+
+message MintActorCertificateRequest {
+  // DER-encoded PKCS #10 certificate signing request. Atunnel retains the
+  // corresponding private key.


My initial thinking was that atelet is the thing we trust, and hence it generates the private keys and hands them to atunnel. Like kubelet. But your approach may be bettter, though keys would have to be regenerated across suspend/resume?

/cc Taahir Ahmed (@ahmedtd) for thoughts.

Taahir Ahmed (ahmedtd) · 2026-08-05T00:48:23Z

+
+  // Exact worker Pod requesting the certificate. Ateapi verifies that this
+  // worker is still assigned to the actor before signing.
+  string worker_pod_uid = 5;


Can you add an additional note here to the effect that it's safe to trust the worker_pod_uid specified in the request body because atelet is the client, and atelet is trusted to tell us which worker pod is requesting the certificate (as long as the worker pod is associated with that particular atelet).

Taahir Ahmed (ahmedtd) · 2026-08-05T00:59:25Z

 		},
 	}

 	if err := substratex509.AddActorIdentityToCertificate(&substratex509.ActorIdentity{


We need an additional indication here that this is an ateom or atunnel, not the actor using its own identity.

Maybe fields in the ActorIdentity extension for:

SystemComponentName

SystemComponentPodName

SystemComponentPodUID

Taahir Ahmed (ahmedtd) · 2026-08-05T01:01:11Z

+	// TODO: Before release, request an atunnel-specific MintCert purpose and
+	// require the egress PEP to reject generic actor certificates.
+	// The request deliberately carries no actor identity. The authenticated Pod
+	// UID is the only input used to select a worker and its current assignment.
+	workerUID, err := authenticatedWorkerUID(ctx)


I think it would be fine to merge without that, and then add it in a follow up? Either way works.

Taahir Ahmed (ahmedtd) · 2026-08-05T01:02:49Z

+		return nil, err
+	}
+	var assigned *ateapipb.Worker
+	for pageToken := ""; ; {


Looping over all workers isn't going to work even in the medium term. Do we have any way to fetch the correct worker in a single request?

Taahir Ahmed (ahmedtd) · 2026-08-05T01:04:20Z

+	ateapiAddress        = pflag.String("ateapi-address", "dns:///api.ate-system.svc:443", "ateapi gRPC target used by the credential broker.")
+	ateapiCAFile         = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ateapi.")
+	ateapiServerName     = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ateapi certificate.")


Normally ate-api-server dials out to atelet. In this case, we need to dial back.

Taahir Ahmed (ahmedtd) · 2026-08-05T01:09:56Z

+		serverboot.Fatal(ctx, "Failed to load atelet Pod identity", fmt.Errorf("credential bundle has no Pod identity"))
+	}
+	brokerTLS := tlsCfg.Clone()
+	brokerTLS.VerifyConnection = restrictClientToNode(ateletIdentity)


I think over a UDS we should be able to get this level of control just from filesystem permissions? Ie, we create a UDS socket file for each individual ateom, and make sure only that ateom can access it.

TLS over a UDS is a bit strange.

Eitan Yarmush (EItanya) marked this pull request as ready for review August 3, 2026 13:39

Eitan Yarmush (EItanya) requested review from Lior Lieberman (LiorLieberman) and Dmitry Berkovich (dberkov) August 3, 2026 13:39

Eitan Yarmush (EItanya) added 7 commits August 4, 2026 01:01

atunnel: broker actor JWTs through atelet

42f3216

atunnel: clarify credential lifecycle

b7a22b6

atunnel: clarify credential trust boundaries

36e58c5

atunnel: keep egress dialer private

cca7142

atunnel: clarify JWT expiry check

25706f7

atunnel: group egress gateway configuration

7137416

atunnel: authenticate egress with actor certificates

facebf3

Eitan Yarmush (EItanya) force-pushed the issue-706-atunnel-identity branch from 3306005 to facebf3 Compare August 4, 2026 01:03

Eitan Yarmush (EItanya) changed the title ~~atunnel: broker actor JWTs through atelet~~ atunnel: broker actor certificates through atelet Aug 4, 2026

Lior Lieberman (LiorLieberman) reviewed Aug 4, 2026

View reviewed changes

Eitan Yarmush (EItanya) mentioned this pull request Aug 4, 2026

Actor Identity Broker can issue an mTLS certificate that identifies a particular atunnel acting on behalf of a particular caller. #734

Open

Taahir Ahmed (ahmedtd) reviewed Aug 5, 2026

View reviewed changes

		ActorResourceVersion: actor.GetMetadata().GetVersion(),
		WorkerPodUid: req.GetWorkerPodUid(),

Uh oh!

Conversation

Eitan Yarmush (EItanya) commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 3, 2026

Uh oh!

Eitan Yarmush (EItanya) commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Eitan Yarmush (EItanya) commented Aug 3, 2026 •

edited

Loading