Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VaultStaticSecrets not resuming secret syncing post a Vault seal event #840

Open
MysticalMount opened this issue Jul 5, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@MysticalMount
Copy link

Describe the bug
VaultSaticSecret doesnt resume syncing post a Vault being sealed, and then unsealed. Vault instance is external to the cluster and was successfully unsealed.

Expected VaultSaticSecret to resume syncing post unseal.

To Reproduce

  1. Set up a VaultConnection and VaultAuth in the operator namespace.
  2. Create a VaultStaticSecrert using default connection and auth
  3. Confirm secret syncing
  4. Terminate a Vault instance
  5. Bring back up a Vault instance
  6. Unseal
  7. VaultStaticSecret will report 503 Vault is sealed permanenty

Application deployment:

    defaultVaultConnection:
      enabled: true
      address: "http://xxx:8200"
    defaultAuthMethod:
      enabled: true
      namespace: ""
      method: kubernetes
      mount: xxx
      kubernetes:
        role: "xxx"
        serviceAccount: vault-auth

Vault operator was restarted. No errors in the controller/operator logs pre or post the restart. Problem persisted for all VaultStaticSecrets.

Expected behavior
Expected VaultStaticSecret to resume post Vault becoming re-available, it seemed to stop after roughly 10 minutes but this is a best guess.

Environment

  • Kubernetes version: 1.30.2
    • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): Bare metal / Talos Linux
    • Other configuration options or runtime services (istio, etc.): Traefik
  • vault-secrets-operator version: v1.7.1

Additional context
Add any other context about the problem here.

@MysticalMount MysticalMount added the bug Something isn't working label Jul 5, 2024
@MysticalMount
Copy link
Author

I think whats happening here is the secret is being refreshed, but no event is generated if the secret already exists. Deletion of the target secret resource, post Vault's status becoming available again (i.e. successful connection and unsealed) - does seem to happen automatically.

However Ive only tested this post a connection error, whereupon I realised this, versus a 503/Vault is sealed but likely the behaviour is the same.

@alexthaii
Copy link

alexthaii commented Sep 6, 2024

I've also hit this bug during a seal/unseal event.

Restarting VSO didn't help.
Deleting either the target Secret or the VaultStaticSecret will make the VaultStaticSecret start syncing successfully again.

Environment:
Image: hashicorp/vault-secrets-operator:0.8.1
Kubernetes: 1.27.13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants