Managed identity cannot create Cosmos containers: data-plane RBAC has no create action, azd deploy never runs Bicep, and deployer container lists have drifted

[paullizer]

Issue

Deployments configured with AZURE_COSMOS_AUTHENTICATION_TYPE=managed_identity cannot create Cosmos DB containers. Whenever a container the application expects does not already exist, startup fails with a 403 at module import time — before Flask binds a port — and the App Service container never starts.

Previously reported in #693 (seen on both initial deployment and updates; worked around by switching to key auth). Related: #484. Not the same as #536, which fails earlier during CosmosClient construction and looks like endpoint resolution rather than RBAC.

Root cause

Three compounding causes.

1. Container creation cannot be expressed in Cosmos data-plane RBAC

application/single_app/config.py builds the schema at module import time against the data plane:

cosmos_database = cosmos_client.create_database_if_not_exists(cosmos_database_name)
# ... followed by 67 create_container_if_not_exists calls

These requests go to https://<account>.documents.azure.com, which evaluates only the account's native sqlRoleAssignments. Per the data plane security reference, the complete dataAction set is:

• Microsoft.DocumentDB/databaseAccounts/readMetadata
• .../sqlDatabases/containers/executeQuery, readChangeFeed, executeStoredProcedure, manageConflicts
• .../sqlDatabases/containers/items/*

There is no create action for databases or containers. Even the containers/* wildcard is documented as covering only "executing queries, reading the change feed, managing conflicts, and executing stored procedures". And native RBAC does not support notDataActions — "any action that isn't specified as an allowed dataAction is excluded automatically" — so a custom az cosmosdb sql role definition cannot express it either.

Creation is an Azure RBAC Action, evaluated by ARM (permissions reference):

Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write             Create a SQL database.
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write  Create or update a SQL container.

Consequences:

• No Cosmos data-plane role, built-in or custom, can authorize the current code path.
• Granting the managed identity Cosmos DB Operator or Contributor also does nothing, because the SDK request still goes to the data-plane endpoint. This is an easy dead end to spend a day in.
• The guidance recorded on Managed Identity Auth for Cosmos DB for Deployment and Updates to the Application #693 — run cosmosDb-postDeployPerms.sh — cannot fix it for two independent reasons: the script grants the signed-in user, not the App Service identity, and it grants data-plane Data Contributor, which lacks the action regardless.

The failure is intermittent-looking because create_container_if_not_exists reads first and only creates on 404. With readMetadata, containers that already exist read back fine — so managed identity appears to work whenever the schema happens to be complete.

2. azd deploy never provisions containers

Containers are created in exactly two places:

Path	Mechanism	Plane	Runs on
azd provision / azd up	cosmosDb.bicep cosmosContainers loop	Control (ARM, deployer identity)	provision only
App startup	create_container_if_not_exists	Data (documents.azure.com)	key auth only

azd deploy runs neither. Its only hook is predeploy, which builds and pushes the container image. postconfig.py performs get_container_client / read_item / upsert_item against settings only, and creates no schema.

So on the upgrade path, the application's own creation call is the only mechanism that can add a new container — and that call is authorized solely by the account key.

Every release that adds a container is therefore a breaking change for every managed-identity deployment. Key-based deployments self-heal silently on first boot, which is why this has remained invisible; managed identity is opt-in, since AZURE_COSMOS_AUTHENTICATION_TYPE defaults to key.

3. Deployer container lists have drifted from config.py

Even a full azd provision does not currently produce a complete schema:

Source	Containers	Missing
application/single_app/config.py (runtime truth)	67	—
deployers/bicep/modules/cosmosDb.bicep	63	4
deployers/terraform/main.tf	60	7
deployers/azurecli/deploy-simplechat.ps1	58	10

Missing from Bicep:

Container	Partition key
orchestration_runs	/conversation_id
orchestration_run_steps	/run_id
document_access_index	/scope_key
key_vault_secret_reminders	/scope_key

Terraform additionally omits custom_pages, governance_policies, governance_item_policies.

Partition keys on the containers that are present all match, so this is purely additive drift — containers were added to config.py without being added to the deployers.

orchestration_runs is the 13th create_container_if_not_exists call in config.py, so a fresh Bicep + managed identity deployment dies there on first boot with a correct-looking role assignment in place.

Impact

• Managed identity is effectively unusable for any deployment that upgrades, and unusable on fresh Bicep deployments today because of the drift.
• Failure mode is a crash loop with no port binding, which reads as a platform or networking fault rather than a permissions one.
• The visible evidence actively misleads: the sqlRoleAssignment is present and correct, so operators conclude permissions are fine.
• Only workarounds are switching to key auth or hand-creating containers, which blocks key-free/disableLocalAuth customer environments entirely.

Proposed fix

1. Provision through ARM at runtime when using managed identity

When AZURE_COSMOS_AUTHENTICATION_TYPE=managed_identity, create missing databases and containers via azure-mgmt-cosmosdb (CosmosDBManagementClient.sql_resources.begin_create_update_sql_container) instead of the data-plane create_container_if_not_exists. Continue using the data-plane client for all item reads and writes.

This is the only option that works on the azd deploy path, because it lives in application code rather than infrastructure.

Requires granting the App Service identity control-plane actions. Prefer a least-privilege custom role over the broad built-in Cosmos DB Operator:

Microsoft.DocumentDB/databaseAccounts/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/operationResults/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/operationResults/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings/read
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/throughputSettings/read

Deliberately excludes delete, listKeys, listConnectionStrings, and sqlRole* management so the identity cannot escalate itself. This must be added to all three deployers so new deployments receive it automatically.

Gating and schema versioning. A naive implementation issues 68 ARM calls per Gunicorn worker on every boot. ARM is rate-limited and materially slower than the data plane, so this needs to be bounded:

• Derive a deterministic hash of the expected container set (names, partition keys, TTLs, indexing policies) at import time.
• Persist the applied hash as a schema-version marker document in the settings container.
• Skip reconciliation entirely when the stored hash matches — the steady-state boot cost becomes one point read.
• Reconcile only when the hash differs, and take a short-lived lease or use optimistic concurrency on the marker so concurrent workers do not stampede ARM during a rolling restart.
• Reconciliation must be additive only. It must never delete or replace a container, so an unexpected hash mismatch can never destroy data.
• Failure to reconcile should surface a clear, actionable startup error naming the missing container and the required action, rather than a raw 403.

Open questions worth resolving during design:

• Should reconciliation be gated behind an explicit opt-in setting for operators who prefer infrastructure to own schema entirely?
• Should the key-auth path also move to ARM for consistency, or keep the existing data-plane behaviour to avoid requiring control-plane RBAC on key-based deployments?
• Where should the marker live before the settings container itself exists — bootstrap ordering needs care.

2. Close the deployer gap

Add the missing containers to all three deployer lists: Bicep +4, Terraform +7, azurecli +10. Still required for fresh provisions and for key-based deployments, independently of item 1.

3. Drift test

Add a functional test asserting all three deployer lists match config.py on names, partition keys, and TTLs. Without this, item 2 rots again on the next feature that adds a container — the drift is the root cause here, not the symptom.

4. Correct the misleading guidance

cosmosDb-postDeployPerms.sh and the equivalent block in azure.yaml grant the signed-in user, not the application identity. The comment on #693 recommends this script as the fix; that should be corrected, and the managed identity documentation updated to state plainly that container creation is a control-plane operation.

Acceptance criteria

• A managed identity deployment with an incomplete schema starts successfully and creates the missing containers.
• azd deploy of a build that adds a new container succeeds on a managed identity deployment without running azd provision and without key auth.
• Steady-state boot with an unchanged schema issues no container-provisioning ARM calls (verified by the schema-version marker short-circuit).
• Concurrent worker startup does not produce duplicate or conflicting provisioning calls.
• Reconciliation never issues a delete or replace against an existing container.
• Bicep, Terraform, and azurecli each provision all 67 containers with partition keys and TTLs matching config.py.
• A functional test fails when any deployer list drifts from config.py.
• The control-plane role is granted by all three deployers for managed identity deployments.
• A managed identity failure to provision produces an actionable startup error naming the container and required permission.
• Managed identity documentation states that container creation is control-plane, and the postDeployPerms guidance is corrected.

Notes

• deployers/Initialize-CosmosManagedIdentityAccess.ps1 (added in deployer 1.0.27) is the remediation path for deployments already broken by this. It grants both planes, validates that the supplied principal is an object ID rather than a client ID, and creates all 67 containers via ARM. It is a stopgap, not the fix.
• A frequent secondary cause of "the role is assigned but nothing works": Cosmos stores any GUID in a sqlRoleAssignment without validating it, so an application (client) ID pasted where the object ID belongs produces an assignment that looks correct in the portal and CLI but grants nothing. Worth calling out in the docs.
• Counts above were verified by parsing config.py, cosmosDb.bicep, main.tf, and deploy-simplechat.ps1 directly rather than by inspection.