Stop Minting Static Credentials

- 18 mins read

Series: Terraform in Practice

Stop Minting Static Credentials

Introduction

Every static credential starts out reasonable. A pipeline needs to SSH into a server, so someone generates a key pair and stores the private half somewhere the pipeline can read it. A Terraform workspace needs to talk to AWS, so someone creates an access key and pastes it into a workspace variable. An application needs a database, so someone creates an application user, picks a password, and puts it in a config file.

Each of those is a credential that nobody asked to expire. It works on day one, and it keeps working long after the person who created it has moved on and nobody remembers which systems would break if it were rotated. So it doesn’t get rotated.

The fix has the same shape every time. Stop storing a secret that proves who you are, and start trusting an identity system to issue a short-lived one on demand. This post walks through three of those substitutions, each done for real in my lab environment, and the thing that bit on the way in for each one:

  1. A Vault SSH certificate authority instead of distributing key pairs.
  2. OIDC workload identity instead of long-lived cloud access keys, for AWS, Azure, and Google Cloud.
  3. Vault-managed database credentials instead of a shared application user.

The Azure and Google Cloud trust configurations are in terraform-dynamic-credentials, verified against both HCP Terraform and a self-hosted Terraform Enterprise. AWS dynamic credentials worked in my lab too, but an account admin created the IAM side, and I don’t have admin access to an AWS account to show every step here. The AWS module in the repo is written from the HashiCorp docs and passes terraform validate, but I haven’t applied it myself. Code excerpts elsewhere in the post are trimmed, with environment-specific names generalized.

TL;DR

  • SSH: Vault’s SSH secrets engine signs a certificate per job, good for thirty minutes. Hosts trust the CA’s public key, baked into the image. There is no private key to distribute because there is no long-lived private key that grants access.
  • Cloud: HCP Terraform and Terraform Enterprise sign a JWT for every plan and apply. AWS, Azure, and Google Cloud trade it for credentials that expire with the run. The workspace stores identifiers, not secrets.
  • Databases: a Vault static role owns the application user’s password and rotates it on a schedule. A dynamic role goes further and creates a new user per lease. Both work, and they fail in different ways.
  • The gotchas are almost never in the happy path. They’re in the second phase, the second week, or the second time you need to change something.

Problem 1: The SSH Key Pair Everyone Has a Copy Of

My lab’s CI pipeline needed to configure a lab server over SSH. The original setup was the common one: a private key stored in Vault’s key/value engine, read by the CI job, written to disk, and used directly.

secrets:
  SSH_PRIVATE_KEY:
    vault:
      engine:
        name: kv-v2
        path: secrets
      path: lab/lab-key
      field: private_key
    file: true
    token: $VAULT_ID_TOKEN

Storing the key in Vault beats storing it in the repo, but it doesn’t change what the key is. It’s one private key, valid indefinitely, copied onto every runner that executes the job, and rotating it means touching every host that trusts it.

The Fix: Sign a Certificate per Job

OpenSSH has supported certificates for years. Instead of putting a user’s public key in authorized_keys on every host, you put one CA public key in sshd_config, and any user certificate signed by that CA is accepted until it expires. Vault’s SSH secrets engine is that CA.

The Vault side is a mount, a CA, and a role that says what a certificate may contain:

resource "vault_mount" "aap_ssh" {
  path = "aap-ssh"
  type = "ssh"
}

# generate_signing_key keeps the CA private key inside Vault. It is never
# returned by the API, so it can't land in Terraform state.
resource "vault_ssh_secret_backend_ca" "aap_ssh" {
  backend              = vault_mount.aap_ssh.path
  generate_signing_key = true
  key_type             = "ed25519"
}

resource "vault_ssh_secret_backend_role" "aap_role" {
  name    = "aap-role"
  backend = vault_mount.aap_ssh.path

  key_type                = "ca"
  allow_user_certificates = true

  allowed_users = "ansible"
  default_user  = "ansible"

  allowed_extensions = "permit-pty,permit-port-forwarding"
  default_extensions = {
    "permit-pty" = ""
  }

  ttl                 = "1800" # 30m
  max_ttl             = "7200" # 2h
  not_before_duration = "30"
}

The host side lives in the image pipeline. CI reads the CA’s public key from Vault, and Packer drops it into the image along with one line of sshd_config:

CA_DEST=/etc/ssh/vault-ssh-ca.pub
cp /tmp/vault-ssh-ca.pub "$CA_DEST"
chmod 0644 "$CA_DEST"

if grep -q '^TrustedUserCAKeys' /etc/ssh/sshd_config; then
  sed -i "s|^TrustedUserCAKeys .*|TrustedUserCAKeys $CA_DEST|" /etc/ssh/sshd_config
else
  echo "TrustedUserCAKeys $CA_DEST" >> /etc/ssh/sshd_config
fi

On the client side, the lab server is now configured by Ansible Automation Platform rather than by the CI job. AAP has a HashiCorp Vault Signed SSH credential type, and linking it to a Machine credential makes AAP request a freshly signed certificate every time a job runs:

- name: Ensure signed-SSH input source on Machine credential
  ansible.controller.credential_input_source:
    input_field_name: ssh_public_key_data
    target_credential: "{{ machine_credential_name }}"
    source_credential: "{{ signed_ssh_lookup_name }}"
    metadata:
      secret_path: aap-ssh
      role: aap-role
      auth_path: approle
      valid_principals: ansible
      public_key: "{{ ssh_public_key }}"
    state: present

The end state: the lab’s instances launch with no EC2 key pair at all, port 22 only accepts connections from AAP’s egress range, and the static key is gone from the pipeline. The key pair AAP signs still exists, but on its own it opens nothing. A certificate is what gets you in, and each one is dead within two hours at most.

Gotcha: The CA Is Not a Normal Resource

I originally built the CA by hand, as RSA 4096, and brought it under Terraform later. That’s where the first surprise was. Every argument on vault_ssh_secret_backend_ca forces replacement, because the provider implements create, read, and delete but no update. And the provider’s read only returns public_key and backend, so after an import the other arguments are null in state. A configuration that matched the live CA exactly still planned a replacement.

There was no way to adopt the existing CA without regenerating it. So instead of importing it as RSA and rotating to ed25519 later, which would have regenerated it twice, I changed the key type in the same apply and regenerated it once.

Any change to the CA resource produces a new CA, and every host trusting the old public key stops accepting certificates. In a setup where the key is baked into images, rotation isn’t a Vault operation. It’s a Vault change followed by rebuilding every image. generate_signing_key = true makes this sharper: the private key never leaves Vault, which is exactly why it isn’t in state, and also why losing Vault means a new CA and a full image rebuild. I think that’s the right trade, but it should be a decision, not a discovery.

Gotcha: Write the TTLs in Seconds

The first version of the role said ttl = "30m". It applied cleanly, and then every plan after it showed a change. The API returns TTLs as integers in string form, the provider doesn’t normalize them, and "30m" is not equal to "1800". Writing seconds made the plan idempotent.

Gotcha: The Public Key Has a Space in It

The first image pipeline passed the CA public key to Packer’s shell provisioner as an environment variable, with this execute_command:

execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
environment_vars = ["VAULT_SSH_CA_PUBKEY=${var.vault_ssh_ca_pubkey}"]

An SSH public key is <type> <key>, with a space in the middle, and {{ .Vars }} expands inside the single-quoted sh -c string. The quoting broke.

The fix was to never let the key pass through a shell at all. GitLab CI can write a Vault secret to a temporary file instead of an environment variable:

VAULT_SSH_CA_PUBKEY:
  vault:
    engine:
      name: generic
      path: aap-ssh
    path: config/ca
    field: public_key
  file: true

Packer then uploads that file with a file provisioner, and the script reads it from disk. The key never goes through an environment variable, a command-line argument, or a shell expansion.

Problem 2: Cloud Keys in Workspace Variables

The classic Terraform setup for AWS is an IAM user with an access key, pasted into the workspace as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. It works, it never expires, and it’s usually more privileged than anything else in the account.

My lab had already moved past that, or so I thought. The workspace authenticated to Vault with workload identity, then read AWS credentials from Vault’s AWS secrets engine, which created a fresh IAM user for each run:

data "vault_aws_access_credentials" "creds" {
  backend = "aws"
  role    = "packer"
}

On paper, those are dynamic credentials. In practice, the first AWS call failed about 23 seconds after the credentials were minted, with 403 InvalidClientTokenId from STS. Adding skip_credentials_validation moved the same 403 to iam:ListRoles. Adding skip_requesting_account_id and a 30-second time_sleep moved it again, this time to a 401 AuthFailure from the first EC2 call, 56 seconds after minting. The error kept moving later instead of going away. A brand new IAM user takes a while to be recognized everywhere in AWS, and there was no sleep long enough to fix that reliably.

The HashiCorp docs say not to do exactly this: “data sources that use secrets engines to generate dynamic secrets must not be used with Vault dynamic credentials.” The documented route through Vault is Vault-backed dynamic credentials, which even has a TFC_VAULT_BACKED_AWS_SLEEP_SECONDS setting “to mitigate eventual consistency issues in AWS when using the iam_user auth type.”

I went the other way and dropped Vault out of the AWS path entirely. With the platform’s native dynamic credentials, the same apply created a key pair, a security group, an EC2 instance, and an Elastic IP with no delay at all.

The Fix: Let the Cloud Trust the Platform

HCP Terraform and Terraform Enterprise act as an OIDC identity provider. For every plan and every apply, the platform signs a JWT describing the run. The claim that matters is sub:

organization:my-org:project:my-project:workspace:my-workspace:run_phase:apply

You configure the cloud to trust tokens from the platform’s issuer URL, and to accept only tokens whose claims match your workspace. At run time, the provider trades the token for short-lived cloud credentials. The workspace holds a role ARN or a client ID, which are identifiers rather than secrets.

The docs are blunt about what happens if you skip the matching: “If you don’t match against at least the organization name, any organization or workspace on HCP Terraform will be able to access your cloud resources!”

Something still needs admin rights to create the trust in the first place. The difference is that it happens once, from a person’s own login, rather than being stored in every workspace.

AWS

The trust side is an IAM OIDC provider for the platform’s hostname and a role whose trust policy checks the audience and the subject:

resource "aws_iam_role" "this" {
  name = var.name

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = local.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.hostname}:aud" = "aws.workload.identity"
        }
        StringLike = {
          "${var.hostname}:sub" = "organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:run_phase:*"
        }
      }
    }]
  })
}

Note StringLike on the subject. The * after run_phase: is only a wildcard under StringLike, and the docs call this out explicitly for when plan and apply share a role. The workspace then needs two environment variables, TFC_AWS_PROVIDER_AUTH=true and TFC_AWS_RUN_ROLE_ARN, and the provider block should contain nothing but region.

In my lab, I specified the role and an account admin created it, so I can’t publish the live trust policy as tested. The module in the repo is written from the docs and passes terraform validate, but I haven’t applied it myself. Azure and Google Cloud I built and ran end to end.

Azure

Azure’s version is an app registration, its service principal, a role assignment, and federated identity credentials that tell Entra ID which tokens to accept:

resource "azuread_application_federated_identity_credential" "this" {
  for_each = toset(["plan", "apply"])

  application_id = azuread_application.this.id
  audiences      = ["api://AzureADTokenExchange"]
  display_name   = "${var.name}-${each.key}"
  issuer         = "https://${var.hostname}"
  subject        = "organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:run_phase:${each.key}"
}

The workspace gets TFC_AZURE_PROVIDER_AUTH=true, TFC_AZURE_RUN_CLIENT_ID, ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID.

The gotcha is the for_each. A federated credential’s subject is an exact string match, no wildcards, and the token’s subject ends in the run phase. So every workspace needs two credentials. I deleted the apply one to see what happens: the plan ran fine, because data sources authenticate during plan, and then the apply failed:

AADSTS700213: No matching federated identity record found for presented
assertion subject 'organization:my-org:project:dynamic-credentials-demo:workspace:dynamic-credentials-demo:run_phase:apply'.
Check your federated identity credential Subject, Audience and Issuer against
the presented assertion.

That’s a nasty failure mode, because a green plan is what gets approved. Combine it with Entra ID’s limit of 20 federated credentials per app registration, and one app registration can serve at most ten workspaces.

Microsoft has a preview feature, flexible federated identity credentials, that allows wildcard matching on the subject. I tested it both ways. On HCP Terraform, a single credential matching claims['sub'] matches '...:run_phase:*' covered both plan and apply. On Terraform Enterprise, Entra ID refused to create it:

Expression is not supported for applications in this cloud 'Public' using
issuer 'https://<tfe-hostname>'.

Microsoft’s docs list only https://app.terraform.io and its EU equivalent as supported Terraform issuers. If you run Terraform Enterprise, two credentials per workspace is the design.

Google Cloud

Google Cloud uses a workload identity pool, an OIDC provider in that pool, and a service account the workspace impersonates:

resource "google_iam_workload_identity_pool_provider" "this" {
  project                            = var.project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.this.workload_identity_pool_id
  workload_identity_pool_provider_id = var.pool_provider_id

  attribute_mapping = {
    "google.subject"                        = "assertion.sub"
    "attribute.terraform_organization_name" = "assertion.terraform_organization_name"
    "attribute.terraform_project_name"      = "assertion.terraform_project_name"
    "attribute.terraform_workspace_name"    = "assertion.terraform_workspace_name"
    "attribute.terraform_run_phase"         = "assertion.terraform_run_phase"
  }

  attribute_condition = "assertion.sub.startsWith(\"organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:\")"

  oidc {
    issuer_uri = "https://${var.hostname}"
  }
}

That startsWith condition covers both run phases, so Google Cloud doesn’t have Azure’s two-credential problem. It carries more weight than it looks like it does, though. The service account binding grants roles/iam.workloadIdentityUser to every identity in the pool:

member = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.this.name}/*"

So the condition is the only thing checking that the token came from your workspace and not from someone else’s. When I changed it to a different organization, the plan failed immediately:

oauth2/google: status code 400: {"error":"unauthorized_client",
"error_description":"The given credential is rejected by the attribute condition."}

The workspace gets TFC_GCP_PROVIDER_AUTH=true, TFC_GCP_WORKLOAD_PROVIDER_NAME, and TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL.

If You Run Terraform Enterprise

Everything above works the same on Terraform Enterprise, with the issuer set to your own hostname instead of app.terraform.io. Two things change.

First, the cloud has to be able to reach you. To verify a token, the cloud fetches /.well-known/openid-configuration and /.well-known/jwks from your Terraform Enterprise hostname. An instance that’s only reachable on a private network can’t do native dynamic credentials to a public cloud. Vault-backed dynamic credentials exist partly for this case.

Second, the certificate. HashiCorp’s Azure docs say “Custom and self-signed certificates are not supported due to restrictions in Azure,” and the Google Cloud docs say dynamic credentials “do not work if your Terraform Enterprise instance uses a custom or self-signed certificate.” My instance uses a publicly trusted certificate, so neither applied, but an instance behind an internal CA would have hit both.

Problem 3: The Shared Application User

The database version of a static credential is an application user whose password was set once, lives in a config file or a Kubernetes secret, and is shared by every instance of the application. Vault’s database secrets engine offers two ways out, and my lab has both configured against the same Postgres database for Terraform Enterprise.

The Fix, Part One: A Static Role

A static role is a one-to-one mapping between a Vault role and an existing database user. Vault stores the user’s password and rotates it on a schedule. Terraform Enterprise itself runs as tfe_app, whose password Vault rotates weekly:

resource "vault_database_secret_backend_static_role" "tfe_static" {
  backend         = vault_mount.tfe_psql.path
  name            = "tfe-static"
  db_name         = "tfe-psql"
  username        = "tfe_app"
  rotation_period = 604800 # 7 days
}

Two things to know before you create one. Vault rotates the password the moment the role is created, so the password you created the user with stops working immediately. And the Vault docs are explicit that the user Vault itself connects as should never be a static role: “Do not manage the same root database credentials that you provide to Vault in config/ with static roles.” In my lab, Vault connects as tfe, and the application runs as tfe_app.

Terraform Enterprise runs on Kubernetes, so the Vault Secrets Operator (VSO) delivers the password as a Kubernetes secret and restarts the deployment when it changes:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: tfe-database
spec:
  vaultAuthRef: tfe-auth
  mount: tfe-psql
  path: static-creds/tfe-static
  allowStaticCreds: true
  refreshAfter: 24h
  destination:
    name: tfe-database
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: terraform-enterprise

Gotcha: The First Rotation Is the Real Test

The first version of that manifest didn’t have allowStaticCreds. It deployed fine and Terraform Enterprise ran fine, for a week. Then Vault rotated the password on schedule, the Kubernetes secret still held the old one, and Terraform Enterprise started logging:

failed SASL auth: FATAL: password authentication failed for user "tfe_app" (SQLSTATE 28P01)

Without allowStaticCreds, VSO 1.4.0 treated the static credential response as a non-renewable lease, concluded there was nothing left to refresh, and never read it again. Its log said as much: “Vault secret does not support periodic renewal/refresh via reconciliation.” The VSO API reference describes the flag as something that “should be set when syncing credentials that are periodically rotated by the Vault server, rather than created upon request.”

A rotation setup that hasn’t survived its first rotation isn’t tested yet. If your rotation period is a week, you find out a week later, which is usually long after the change that caused it has been merged and forgotten.

The Fix, Part Two: A Dynamic Role

A dynamic role goes further. Instead of rotating one user’s password, Vault creates a brand new database user for each lease and drops it when the lease ends. Every instance of the application gets its own username, which the Vault docs point out makes auditing easier, because you can trace a query to a specific instance.

My lab has a dynamic role on the same database:

resource "vault_database_secret_backend_role" "tfe_psql" {
  backend = vault_mount.tfe_psql.path
  name    = "tfe-psql-role"
  db_name = "tfe-psql"

  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';\nGRANT ALL PRIVILEGES ON DATABASE tfe TO \"{{name}}\";"
  ]

  default_ttl = 3600  # 1 hour
  max_ttl     = 86400 # 24 hours
}

Nothing in the lab actually consumes it. It has only ever been read by hand. So before writing about it, I reproduced it against a throwaway Vault and Postgres 17, with the same creation statement and a table owned by a separate user, and tried to use it the way an application would.

Gotcha: GRANT ALL on a Database Doesn’t Reach the Tables

The generated user could log in, and that was all:

ERROR:  permission denied for table widgets

In Postgres, the privileges on a database are about connecting to it and creating things in it, not about the tables inside it. GRANT ALL PRIVILEGES ON DATABASE reads like it grants everything, and it grants none of the access an application needs. The example role in Vault’s own Postgres docs uses GRANT SELECT ON ALL TABLES IN SCHEMA public instead.

Gotcha: A User That Owns Something Can’t Be Dropped

The second problem is worse, because it shows up when the lease ends. I granted the dynamic user permission to create tables, had it create one, and revoked the lease. Vault queued the revocation, and the user was still there. The Vault log explained why:

[ERROR] expiration: failed to revoke lease: lease_id=db/creds/dyn/h3AscHe4Ki5S9HC096qXdMsL
error="failed to revoke entry: resp: (*logical.Response)(nil) err: ERROR: role
\"v-token-dyn-hOoD3GJA9yCYjC7PX4uC-1789066250\" cannot be dropped because some
objects depend on it (SQLSTATE 2BP01)" attempts=1 next_attempt=21.012025154s

Vault retries, and it keeps failing. Any application that runs its own schema migrations will create objects owned by whichever user ran them, which is a user that’s supposed to disappear.

The fix that worked was to give ownership to a role that never logs in, and make every dynamic user act as that role:

CREATE ROLE app_schema NOLOGIN;
ALTER TABLE widgets OWNER TO app_schema;
GRANT CREATE ON SCHEMA public TO app_schema;

Then the creation statement makes each generated user a member of app_schema and sets it as their default role:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'
  VALID UNTIL '{{expiration}}' IN ROLE app_schema;
ALTER ROLE "{{name}}" SET ROLE app_schema;

With that in place, the generated user could read the table, a table it created was owned by app_schema, and revoking the lease dropped the user cleanly.

Static or Dynamic?

They fail differently, and that’s the useful way to choose. A static role keeps one username and changes its password, so the risk is a consumer that doesn’t pick up the new one. A dynamic role changes the username every lease, so the risk is anything tied to the old username: permissions granted to it, and objects it owns.

Terraform Enterprise in my lab uses the static role. tfe_app owns the tfe database and its public schema, so the ownership problem never comes up, and the only thing that has to be right is the rotation. For an application that doesn’t own its schema, or where each instance should have its own identity, a dynamic role is the stronger option, as long as its creation statement grants access to the tables and not just the database.

Conclusion

Every substitution here follows the same pattern: remove the stored secret, trust an identity system instead, and make whatever it issues expire.

  • SSH certificates instead of key pairs - Vault signs a certificate per job with a thirty-minute TTL, and hosts trust the CA through TrustedUserCAKeys. Treat the CA resource as immutable: changing it means rebuilding every image that trusts it.
  • Workload identity instead of access keys - the platform signs a token per run phase, and the cloud trades it for credentials that expire with the run. On AWS, use StringLike for the wildcard. On Azure, create a federated credential for plan and apply. On Google Cloud, the attribute condition is the only check that the token came from your workspace.
  • Terraform Enterprise changes two things - the cloud has to reach its OIDC endpoints, and Azure and Google Cloud won’t accept a custom or self-signed certificate.
  • Vault-owned database credentials instead of a shared user - static roles rotate one user’s password, and nothing proves the delivery works until the first rotation. Dynamic roles create a user per lease, and need a creation statement that grants table access and keeps ownership off the ephemeral user.
  • Test past the first run - none of these failures showed up there. The Azure apply failed after a clean plan, the VSO outage waited a week for the first rotation, and the stuck revocation only appeared when the lease ended.

The Azure and Google Cloud configurations, with a workload you can run against your own HCP Terraform or Terraform Enterprise workspace, are in terraform-dynamic-credentials.

This is a post in the Terraform in Practice series.
Other posts in this series: