Unable to authenticate and read secrets in Hashicorp Vault using the argocd-vault-plugin command

[hdpdevops]

I’m working to setup ArgoCD to pull secrets out of Hashicorp Vault using the ArgoCD Vault plugin. Although I am able to read the secrets using the vault CLI in the approle I’ve created I’m having issues requesting secrets back from the Vault using this plugin.

What I’ve done:

I’ve created an approle (argocd) and assigned a policy to it (secret-ro) to ensure that it can read from secrets in the secret/test path.

$ vault read auth/approle/role/argocd
Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
policies                   [secret-ro]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         0
secret_id_ttl              0s
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [secret-ro]
token_ttl                  0s
token_type                 default

$ vault read sys/policy/secret-ro
Key      Value
---      -----
name     secret-ro
rules    path "secret/test" {
   capabilities = ["read", "list"]
}

I’ve created the secret with vault kv put secret/test ingress=my-ingress.domain.com

$ vault read secret/test
Key                 Value
---                 -----
refresh_interval    1h
ingress             my-ingress.domain.com

I’ve entered the role-id and secret-id (as well as the other required variable listed in the docs) into a vars.env file for argocd-vault-plugin to use.

I’ve created an test Kubernetes secret resource with the secret’s path specified in order for the argocd-vault-plugin command to do the replacement

kind: Secret
apiVersion: v1
metadata:
  name: example-annotation
type: Opaque
data:
  username: <path:secret/data/test#ingress>

Now when I run the cat secret.yaml | argocd-vault-plugin -c vars.env generate - command I expect to have successfully replaced the <path:secret/data/test#ingress> value with the secret stored in the vault. Instead I am getting the following error:

URL: PUT https://redacted.hashicorp.cloud:8200/v1/auth/approle/login
Code: 403. Errors:
* 1 error occurred:
	* permission denied

I have tried a different method of authenticating, by creating the token in the Vault and utilising that in the argocd-vault-plugin command (as documented here) which is providing a different error message:

Error: Replace: could not replace all placeholders in Template:
Error making API request.
URL: GET https://redacted.hashicorp.cloud:8200/v1/secret/data/test
Code: 403. Errors:
* 1 error occurred:
	* permission denied

So at this point I thought my policy configuration might have been incorrect, so I’ve logged into the vault CLI (vault login) using the same token that argocd-vault-plugin is using and verified that I can read the secret:

$ vault read secret/test
Key                 Value
---                 -----
refresh_interval    1h
ingress             my-ingress.domain.com

Now I’m coming up blank. I’m not sure what else I can look into to try and resolve this issue so any pointers or insight into this would be greatly appreciated.

[jkayani]

Reading over this, I'm a bit confused about which version of Vault KV engine you're using. You say you put the secrets into Vault using the command vault kv put <path>, yet you're able to read them back with vault read <path> without having to insert the /data after the prefix of the path where the KV-V2 engine is enabled.

Is the Vault path you're trying in the placeholder correct? What happens if you try to use vault kv get to read the secret out of Vault, does it error? I wonder if any of this has anything to do with the permissions you granted being incorrect, causing the 403. Beyond this it's hard to give advice.

[hdpdevops]

Thanks for the reply jkayani. For my sanity's sake I've gone through the steps once more and documented them as I've done so. From my reading of the documentation the below steps should work but it just doesn't seem to be happening for me.

Vault

• Enable approle authentication method

  vault auth enable approle

• Create new approle

  $ vault write auth/approle/role/argocd-vault-plugin token_num_uses=0 secret_id_num_uses=0 policies="argocd-vault-plugin-policy"
  Success! Data written to: auth/approle/role/argocd-vault-plugin

• Get approle role-id and secret-id and store in vars.env file

  $ vault read auth/approle/role/argocd-vault-plugin/role-id
  Key        Value
  ---        -----
  role_id    role-id-1234

  $ vault write -f auth/approle/role/argocd-vault-plugin/secret-id
  Key                   Value
  ---                   -----
  secret_id             secret-id-1234
  secret_id_accessor    dd18c3b3-b080-657d-1415-c5d240a91ad1
  secret_id_ttl         0s

• Enable secrets engine at argocd path

  $ vault secrets enable -version=2 -path=argocd kv
  Success! Enabled the kv secrets engine at: argocd/

• Create example secret

  $ vault kv put argocd/argocd-vault-plugin/test my-secret=123456789
  ============ Secret Path ============
  argocd/data/argocd-vault-plugin/test
  
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2022-06-16T04:13:11.541399045Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1

• Create policy for approle to use

  $ cat policy.hcl
  path "argocd/data/argocd-vault-plugin/test" {
   capabilities = ["read", "list"]
  }
  
  $ vault policy write argocd-vault-plugin-policy policy.hcl
  Success! Uploaded policy: argocd-vault-plugin-policy

ArgoCD

• Write values to vars.env file

  $ cat vars.env
  VAULT_ADDR="https://public-vault-address.hashicorp.cloud:8200"
  AVP_TYPE="vault"
  AVP_AUTH_TYPE="approle"
  AVP_ROLE_ID="role-id-1234"
  AVP_SECRET_ID="secret-id-1234"

• Source vars.env file with source vars.env

• Write example secret resource

  $ cat secret.yaml
  kind: Secret
  apiVersion: v1
  metadata:
    name: example-secret
  type: Opaque
  data:
    username: <path:argocd/data/argocd-vault-plugin/test#my-secret>

• Run argocd-vault-plugin command

  • Attempted in CLI on local machine

    $ cat secret.yaml | argocd-vault-plugin generate -
    
    Error: Error making API request.
    URL: PUT https://public-vault-address.hashicorp.cloud:8200/v1/auth/approle/login
    Code: 403. Errors:
    * 1 error occurred:
        * permission denied

Testing with token type now:

• Collect token

  $ vault write auth/approle/login role_id="role-id-1234" \
  >     secret_id="secret-id-1234"
  Key                     Value
  ---                     -----
  token                   token-1234
  token_accessor          lu5HEKMqApRLVzajbRaSkR6P.4rqi6
  token_duration          1h
  token_renewable         true
  token_policies          ["argocd-vault-plugin-policy" "default"]
  identity_policies       []
  policies                ["argocd-vault-plugin-policy" "default"]
  token_meta_role_name    argocd-vault-plugin

• Write token to vars.env and change auth type to token

  $ cat vars.env
  VAULT_ADDR="https://public-vault-address.hashicorp.cloud:8200"
  AVP_TYPE="vault"
  AVP_AUTH_TYPE="token"
  AVP_ROLE_ID="role-id-1234"
  AVP_SECRET_ID="secret-id-1234"
  VAULT_TOKEN="token-1234"

• source vars.env

• Run argocd-vault-plugin command

  $ cat secret.yaml | argocd-vault-plugin generate -
  
  Error: Replace: could not replace all placeholders in Template:
  Error making API request.
  URL: GET https://public-vault-address.hashicorp.cloud:8200/v1/argocd/data/argocd-vault-plugin/test
  Code: 403. Errors:
  * 1 error occurred:
      * permission denied

[via-justa]

I used this plugin with Argo-CD 2.3.4 and it worked, upgrading Argo-CD to 2.4.0 broke the entire installation and I needed to revert back to 2.3.4. @hdpdevops you did not mention what Argo-CD version you're using.
Looking at the compatibility table it looks like it's not been updated for a long time and the last confirmed "supported" Argo-CD version is 2.1.x. @jkayani have you had the chance to test the plugin on later versions and/or have plans to update the plugin to support Argo-CD 2.4.x?

[greenorchid]

Hi @via-justa ArgoCD v2.4.0 introduced an ARGOCD_ENV_ prefix for user-supplied environment variables, you likely need to update your plugin script(s) to cater for this. If the argocd-vault-plugin support matrix gets updated to include ArgoCD>=2.4.0 we should probably update the usage docs.

[hdpdevops]

Hi all.

I was using ArgoCD version 2.4.0 with the ArgoCD Vault plugin on version 1.11.0.

I've since downgraded the ArgoCD version to 2.3.4 and tested again but I'm receiving the same error messages.

[hdpdevops]

I've done some additional testing and I've figure out a resolution.

Firstly, as I'm using Hashicorp Vault Cloud I needed to provide the namespace that my secrets were creating in (the default namespace in non-cloud environments is root, while the cloud version is admin). I did this by specifying the VAULT_NAMESPACE=admin variable for the argocd-vault-plugin to use.

I tested that with ArgoCD's version 2.3.4 and everything worked as expected.

I then changed up to version 2.4.0 and my application's were unable to sync correctly. Likely because of the issue @greenorchid mentioned.

[jkayani]

We're tracking 2.4.0 compatibility in #356

Going to close since the immediate problem has been resolved. Feel free to re-open if you feel it's necessary.