Templating is not working for me

[elwood218]

Hi, I wanted to try templating but it not working for me.

I get following errors in master log:

2024-12-03 16:25:10,277 [saltext.vault.utils.vault.kv:356 ][ERROR   ][3655356] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secret/setups/setup1/"
...
2024-12-03 16:25:10,292 [salt.loaded.ext.pillar.vault:212 ][WARNING ][3655356] Could not resolve pillar path pattern secret/certificates/casigned/{pillar[vault][cert]}
2024-12-03 16:25:11,354 [salt.loaded.ext.pillar.vault:212 ][WARNING ][3655356] Could not resolve pillar path pattern secret/setups/{pillar[vault][setup]}
2024-12-03 16:25:11,603 [saltext.vault.utils.vault.kv:356 ][ERROR   ][3655356] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secret/certificates/casigned/bar/"

Salt version: 3007.1

Vault policy:

path "secret/data/setups/{{identity.entity.metadata.setup}}" {
    capabilities = ["read"]
}

path "secret/data/certificates/casigned/{{identity.entity.metadata.cert}}" {
    capabilities = ["read"]
}

path "secret/metadata/setups/{{identity.entity.metadata.setup}}" {
    capabilities = ["read"]
}

path "secret/metadata/certificates/casigned/{{identity.entity.metadata.cert}}" {
    capabilities = ["read"]
}

External Pillar:

ext_pillar:
  - vault: "secret/setups/{pillar[vault][setup]}"
  - vault: "secret/certificates/casigned/{pillar[vault][cert]}"

Salt Master - Vault config:

vault:
...
    metadata:
      entity:
        minion-id: '{minion}'
        role: '{pillar[role]}'
        setup: '{pillar[vault][setup]}'
        cert: '{pillar[vault][cert]}'

Kind regards
Mathias

[lkubb]

Vault policy templates should be formatted like this:

path "secret/data/setups/{{identity.entity.metadata.setup}}" {
    capabilities = ["read"]
}
# ...

Ref 1 Ref 2

I assume this is an artifact, but the pillar mentions casigned, while the policy is for foo.

Ensure you sync the entity metadata (salt-run vault.sync_entities) and clear the master cache (salt-run vault.clear_cache) before retrying.

[elwood218]

Ah sorry it is correct now - I accidentally copied it from the Ansible template :) Sorry. I have corrected it above. I also put the correct path because there was nothing really secret. So foo = casigned now.

[elwood218]

# salt-run vault.clear_cache
Exception occurred in runner vault.clear_cache: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/localfs.py", line 125, in flush
    shutil.rmtree(target)
  File "/opt/saltstack/salt/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/saltstack/salt/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/opt/saltstack/salt/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/opt/saltstack/salt/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/opt/saltstack/salt/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '__token.p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 383, in low
    data["return"] = func(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/runners/vault.py", line 888, in clear_cache
    cache.flush("vault")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/__init__.py", line 213, in flush
    return self.modules[fun](bank, key=key, **self._kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/localfs.py", line 132, in flush
    raise SaltCacheError(f'There was an error removing "{target}": {exc}')
salt.exceptions.SaltCacheError: There was an error removing "/salt/share/cache/vault": [Errno 13] Permission denied: '__token.p'

Mmh seems like there is also another problem.. will try to find out what was happening and come back to you.

[elwood218]

root@salt-master02:/salt/share/cache/vault/connection# ll
total 16
drwxr-xr-x 3 salt salt 4096 Dec  3 17:01 ./
drwxr-xr-x 3 salt salt 4096 Nov  5 18:01 ../
-rw------- 1 salt salt  366 Nov 24 13:37 secret_path_metadata.p
drwxr-xr-x 2 root root 4096 Dec  3 17:01 session/

Don't understand where that comes from and why everything belongs to salt but this session directory is root. Maybe I did something wrong or it is a problem of the not tested 3007.1 with Master Cluster - I don't really know.

[elwood218]

I probably found the error in Vault Audit Log. The new set entity metadata is not there.
I had run vault.sync_approles and vault.sync_entities already but so far it seems that this is not enough.

(The reason of the other permission error in vault/connection path on the salt server I did not found but it is solved after removing the directory and restart)

[elwood218]

Mmh wanted to run clear_cache again and then sync_entities again and now got the permission denied again..

root@salt-master01:~# salt-run vault.clear_cache
True
root@salt-master01:~# salt-run vault.sync_entities
Exception occurred in runner vault.sync_entities: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/localfs.py", line 86, in fetch
    with salt.utils.files.fopen(key_file, "rb") as fh_:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/files.py", line 388, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
PermissionError: [Errno 13] Permission denied: '/salt/share/cache/vault/connection/session/__token.p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 383, in low
    data["return"] = func(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/runners/vault.py", line 739, in sync_entities
    for minion in set(minions) & set(list_approles()):
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/runners/vault.py", line 687, in list_approles
    api = _get_approle_api()
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/runners/vault.py", line 1208, in _get_approle_api
    return vault.get_approle_api(__opts__, __context__, force_local=True)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/factory.py", line 871, in get_approle_api
    client, config = get_authd_client(opts, context, force_local=force_local, get_config=True)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/factory.py", line 78, in get_authd_client
    client, config, retry = try_build()
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/factory.py", line 51, in try_build
    client, config = _build_authd_client(opts, context, force_local=force_local)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/factory.py", line 299, in _build_authd_client
    cached_token = token_cache.get(10)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/cache.py", line 405, in get
    data = self._get_ckey(self.ckey, flush=flush)
  File "/opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/cache.py", line 120, in _get_ckey
    return self.cache.fetch(self.cbank, ckey) or None  # account for race conditions
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/__init__.py", line 168, in fetch
    return self.modules[fun](bank, key, **self._kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/localfs.py", line 92, in fetch
    raise SaltCacheError(
salt.exceptions.SaltCacheError: There was an error reading the cache file "/salt/share/cache/vault/connection/session/__token.p": [Errno 13] Permission denied: '/salt/share/cache/vault/connection/session/__token.p'

[lkubb]

Maybe one of the master daemons in your cluster does not run as the salt user? I don't use the master cluster mode myself, so I can only guess.

[elwood218]

If I got time I will try to debug that but all processes are running as user salt. Only the minion on the master is running as root. And it seems it only happens when running that vault.clear_cache.

[elwood218]

I was not able to update metadata of entity by vault.sync_entities. Only after removing the minion key, restart minion and accept key again there was the correct metadata and then it finally worked.