Refactor pre-processor syntax

[anbraten]

We currently use bash var syntax for pre-processing in our yaml config. So a var would be written by using ${CI_REPO_DEFAULT_BRANCH}. Its using https://github.com/drone/envsubst for this. This sometimes conflicts with users trying to write variables with normal bash syntax. To escape this the user is needed to write: $${CI_REPO_DEFAULT_BRANCH}.

Instead maybe changing to some other syntax like go-template / handlebar syntax with {{ CI_REPO_DEFAULT_BRANCH }} for the pre-processor could avoid confusion.

[qwerty287]

We should still provide all the features supported by https://github.com/drone/envsubst (https://github.com/drone/envsubst#supported-functions), this shouldn't be hard to implement using something like Go templates though.

[qwerty287]

Related: #1258

[anbraten]

As we can not drop the syntax directly we might need a roadmap like this:

• discuss a new pre-processor syntax
• add this syntax as early access to the ci
• improve the design
• deprecate old syntax, so users can migrate from $$ to $
• remove old syntax and run it as "normal" bash

[lafriks]

Looking at others gitlab ci uses same as we standard env ${ ... } syntax, github ci ${{ ... }} syntax, to be honest I would prefer to keep as we have now, maybe change to github syntax as that would at least indicate that this is not full go template syntax but something more specific (this would probably mean forking envsubst lib to support this

[qwerty287]

Maybe it could also be helpful to not run the templating on the whole yaml file, but only on some specific fields like env vars, settings images etc. Commands could be left out and then bash (or which shell your container has) parses it.

[zc-devs]

Commands could be left out and then shell parses it

I second this.
#3982, #3983.

---

Edit 1

specific fields like env vars

Do we need substitution there? Cannot imagine a use case. Even if I did, it would pretty useless, because syntax would be very simple, it would not support full-Bash substitution, I assume. We can do substitution in commands whatever we want.

[qwerty287]

Yes, but for plugin settings, we need some more advanced features (example: removing the v prefix of a tag) which is not directly done by the plugin. And if we do it for settings, we can also do it for env vars.

[zc-devs]

if we do it for settings, we can also do it for env vars

If we do it for env vars, we can also do it for commands, then 😄
Plugins is a different story. There is no envs now (you know).

I'm not pushing, just consider to leave out environment as well.