Landing Zones for Model APIs

Most enterprises spent years building landing zones: a governed hierarchy of management groups, organizational units or folders, with private networking, preventive policy, central identity and centralized logging applied to every workload by default. Then generative AI arrived, and model endpoints were stood up in a hurry, in whatever subscription the first team had, with public network access enabled and an API key pasted into a pipeline variable. The landing zone was bypassed by the workload with the most sensitive inputs in the estate.

A model endpoint is not special. It is a regional PaaS service that receives your data, processes it and returns a result, which makes it structurally identical to a managed database. Azure OpenAI and Foundry, Amazon Bedrock and Vertex AI should land in the same pattern as everything else: private by construction, governed by policy at the hierarchy level, reachable only through a controlled path, and logged to a store the workload team cannot alter.

The landing zone already solved this#

A landing zone encodes decisions once, above the workload. Policies assigned at a management group, OU or folder apply to every subscription, account or project beneath it, including ones that do not exist yet. For model APIs, five of those decisions matter:

• No public data path. Model endpoints are reachable only over private connectivity from approved networks.
• No static keys. Callers authenticate with workload identity, and key-based auth is disabled at the resource.
• Approved models and regions only. Model families and deployment regions are constrained by policy, driven by data-residency and model-risk decisions.
• One front door. Applications call a central gateway, not the provider endpoint.
• Central, immutable logs. Invocation and control-plane logs land in a store outside the workload's control.

None of these are new ideas. The work is mapping them onto each provider's mechanisms, which differ more than the marketing suggests.

Private by construction#

On Azure, disabling public network access on the AI Services account and attaching a Private Endpoint is a two-property change. The part teams get wrong is DNS: the privatelink zones for OpenAI and Cognitive Services must be linked to the hub resolver, or clients silently resolve the public address and fail, prompting someone to "temporarily" re-enable public access.

AWS is different in a way that matters. Bedrock has no public-access switch; the regional endpoint is always resolvable. An interface VPC endpoint gives you a private path, and an endpoint policy constrains what flows over it, but only a service control policy denying bedrock:InvokeModel and related actions when the request did not arrive through an approved aws:SourceVpce actually closes the public route. Carve out calls AWS services make on your behalf using aws:ViaAWSService, or managed features such as knowledge bases will break.

On Google Cloud, Private Service Connect gives Vertex AI a private address inside your VPC, but the control that prevents exfiltration is a VPC Service Controls perimeter around the project, which rejects API calls from outside the perimeter regardless of credentials.

Policy as the perimeter#

Configuration drifts; preventive policy does not. The deny belongs at the hierarchy level, so that no workload team can create a public or key-authenticated endpoint even by accident. On Azure, one custom definition handles both.

resource "azurerm_policy_definition" "deny_public_ai" {
  name                = "deny-public-ai-services"
  policy_type         = "Custom"
  mode                = "Indexed"
  display_name        = "Deny AI Services accounts with public access or local keys"
  management_group_id = azurerm_management_group.platform.id

  policy_rule = jsonencode({
    if = {
      allOf = [
        { field = "type", equals = "Microsoft.CognitiveServices/accounts" },
        { anyOf = [
          { field = "Microsoft.CognitiveServices/accounts/publicNetworkAccess",
            notEquals = "Disabled" },
          { field = "Microsoft.CognitiveServices/accounts/disableLocalAuth",
            notEquals = true }
        ] }
      ]
    }
    then = { effect = "deny" }
  })
}

resource "azurerm_management_group_policy_assignment" "deny_public_ai" {
  name                 = "deny-public-ai"
  management_group_id  = azurerm_management_group.landing_zones.id
  policy_definition_id = azurerm_policy_definition.deny_public_ai.id
}

Because notEquals is also true when the property is absent, an account created without an explicit setting is denied rather than allowed. The same management group should carry the built-in allowed-locations policy and a definition on the deployments child resource that restricts model names and versions to an approved list. On AWS, the equivalent SCP pairs the aws:SourceVpce condition with an aws:RequestedRegion restriction and a resource list of approved foundation-model ARNs. Cross-Region inference profiles route requests to other regions within a geography, so decide explicitly whether residency permits them and write the SCP accordingly. On Google Cloud, gcp.resourceLocations constrains where resources are created, and Vertex AI offers an organization policy constraint for restricting which Model Garden models projects may use.

One gateway, many models#

Private connectivity and policy constrain where calls can go. A central model gateway governs what each call is allowed to do. Every application calls the gateway; only the gateway's identity holds data-plane roles on the model resources. That single hop gives you four things no per-application integration can.

Routing. Applications request a logical route, not a provider deployment. The gateway maps it to a pinned model version in an approved region, with fallback inside the same residency boundary. Model upgrades become a gateway change reviewed once, not a code change in every repository.

Quotas and token accounting. Tokens in and out are metered per route and attributed to a cost center at request time, so chargeback reflects actual consumption and a runaway batch job hits its own budget rather than exhausting shared capacity.

Key custody. The gateway authenticates to Azure with managed identity, to AWS with an IAM role, to Google Cloud with a service account. There are no provider keys to rotate, leak or paste into a pipeline variable, and the key paths themselves are closed by policy: disableLocalAuth on Azure, an SCP denying Bedrock API-key calls on AWS.

Logging with redaction. The gateway logs every request with prompt and response hashes, token counts, route, caller identity and latency, and redacts regulated values before anything leaves the request path.

gateway:
  auth_to_providers: workload_identity   # no static keys anywhere
  logging:
    sink: central-log-archive            # retention-locked, separate account
    redact: [pii, secrets]
    fields: [route, caller, model_version, tokens_in, tokens_out, latency_ms, prompt_sha256]
  routes:
    - name: claims-extraction
      cost_center: CC-4410
      allowed_callers: [claims-intake-prod]
      residency: us
      primary:  { provider: azure_openai, deployment: extraction-large-v3, region: eastus2 }
      fallback: { provider: azure_openai, deployment: extraction-large-v3, region: centralus }
      quotas: { tokens_per_minute: 120000, monthly_tokens: 900000000 }
    - name: policy-research
      cost_center: CC-2210
      allowed_callers: [research-assistant-prod]
      residency: eu
      primary:  { provider: bedrock, model_ref: approved/general-large, region: eu-central-1 }
      quotas: { tokens_per_minute: 40000, monthly_tokens: 150000000 }

The gateway can be Azure API Management with its AI gateway policies, a self-hosted proxy, or Apigee. The product matters less than the invariant: nothing reaches a model endpoint without passing through it. We describe the reference design on the platform page.

Experimentation is a separate zone#

Teams need room to try new models, and blocking that pushes experimentation onto personal accounts, which is worse. Give it a home: a sandbox management group, OU or folder with its own subscriptions, accounts or projects, permitted to use a broader model catalog, restricted to synthetic or approved non-production data, with hard budget caps and no network path to production data stores.

Production sits under a stricter branch of the hierarchy where only approved models, approved regions and private connectivity are allowed. Promotion is a deliberate act: a model moves from the sandbox catalog to the production allowlist through a review that produces a recorded decision, and only then does a gateway route point at it. That review record is the evidence a model risk function or auditor will ask for.

Logs that survive the incident#

Invocation logs, gateway logs and control-plane activity logs all flow to a central store in a separate logging subscription, account or project, with write-once retention: immutable blob storage on Azure, S3 Object Lock on AWS, locked log buckets on Google Cloud. Workload teams can read their own logs but cannot delete or shorten retention. When something goes wrong, the question is never whether the logs exist, only what they show. See cloud architecture for how this fits the wider logging topology.

What to do on Monday#

• Inventory every model endpoint across all tenants, accounts and projects, including public network access and key-auth status.
• Assign deny policies at the hierarchy level for public access and local keys, starting in audit mode to find violators.
• Fix private DNS before disabling public access, so nobody re-enables it under pressure.
• Stand up the gateway, move one production route onto it, and revoke the application's direct provider access.
• Encode approved model families and regions as policy, and document how cross-region routing fits your residency commitments.
• Create a sandbox branch of the hierarchy with budget caps and a written promotion path to production.
• Route all model and gateway logs to a retention-locked store in a separate logging account.