Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package Templating (va Liquid) #513

Merged
merged 12 commits into from
Apr 11, 2023
Merged

Package Templating (va Liquid) #513

merged 12 commits into from
Apr 11, 2023

Conversation

rydrman
Copy link
Collaborator

@rydrman rydrman commented Sep 24, 2022

An alternative templating language to #511

This implementation uses the liquid templating language. It has syntax more like jinja2, a much larger standard library of filters, control statements (if/else, unless/else, match/when), and other features. The extensions are more complex to write, but also arguably more powerful.

Example:

Yaml With Templating
# {% spk-min-version = "0.34.0" %}
# {% default version = "3.7.3" %}
# {% assign v = version | parse_version %}
# {% assign is_py3 = version | compare_version: ">=3" %}
pkg: python/{{ version }}
api: v0/package
sources:
  - git: https://github.com/python/cpython
    ref: v{{ version }}
build:
  options:
    - var: os
    - var: arch
    - var: centos
    - pkg: gcc
    - pkg: stdfs
    - pkg: bzip2
    # {% if is_py3 %}
    - pkg: ncurses
    - pkg: binutils
    - pkg: libffi
    - pkg: openssl
    - pkg: zlib/1.2
    # {% endif %}
    - var: abi
      default: "cp{{v.major}}{{v.minor}}m"
      choices: [
        "cp{{v.major}}{{v.minor}}m",
        "cp{{v.major}}{{v.minor}}dm",
    # {% unless is_py3 %}
        "cp{{v.major}}{{v.minor}}mu",
        "cp{{v.major}}{{v.minor}}dmu",
    # {% endunless %}
      ]
      inheritance: Strong
    - var: debug
      default: off
      choices: [on, off]
    - var: optimize
      default: on
      choices: [on, off]
  variants:
    - { gcc: 6.3, abi: "cp{{v.major}}{{v.minor}}m", debug: off }
  script:
    - |
      case "$SPK_OPT_debug" in
        on)
          if ! [[ "$SPK_OPT_abi" =~ ^cp{{v.major}}{{v.minor}}.*d ]]; then
            echo "Must use an abi with debug when building with debug enabled!"
            exit 1
          fi
          DEBUG="--with-pydebug"
          ;;
        off)
          if [[ "$SPK_OPT_abi" =~ ^cp{{v.major}}{{v.minor}}.*d ]]; then
            echo "Must not use an abi with debug when building with debug disabled!"
            exit 1
          fi
          DEBUG=""
          ;;
        *)
          echo "Unsupported debug: $SPK_OPT_debug"
          ;;
      esac
    # {% unless is_py3 %}
    - |
      case "$SPK_OPT_abi" in
        cp27m)
          UNICODE="--enable-unicode=ucs2"
          ;;
        cp27mu)
          UNICODE="--enable-unicode=ucs4"
          ;;
        *)
          echo "Unsupported abi: $SPK_OPT_abi"
          ;;
      esac
    # on systems where python3 is the default, we can
    # see syntax errors unless we ensure that 'python' runs python2
    - echo "#!/bin/bash" > /spfs/bin/python
    - echo 'exec python2 "$@"' >> /spfs/bin/python
    - chmod +x /spfs/bin/python
    # {% endunless %}
    # {% if is_py3 %}
    - |
      OPTIMIZE=""
      if [[ "${SPK_OPT_optimize}" == "on" ]]; then
          OPTIMIZE="--enable-optimiations"
      fi
    # {% endif %}
    - >
      ./configure
      --prefix=${PREFIX}
      CC=$CC
      CXX=$CXX
      LDFLAGS='-Wl,--rpath=/spfs/lib,-L/spfs/lib'
      PKG_CONFIG_PATH=/spfs/share/pkgconfig:/spfs/lib/pkgconfig
      CPPFLAGS='-I/spfs/include/ncurses'
      --enable-shared
      --with-ensurepip=no
      {% if is_py3 %}"$OPTIMIZE"{% endif %}
      {% unless is_py3 %}"$UNICODE"{% endunless %}
      $DEBUG
    - make -j$(nproc)
    - make install
    # remove test files that are just bloat
    - find /spfs/lib/python* -name "test" -type d | xargs -r rm -rv
    - find /spfs/lib/python* -name "*_test" -type d | xargs -r rm -rv
    - "ln -sf python{{ version | parse_version: major }} /spfs/bin/python"
    # python is best in spfs when pyc files are not used at all
    - find /spfs -type f -name "*.pyc" | xargs rm

tests:
  - stage: install
    script:
      # Verify we built a python with the requested ABI
      - python_abi=$(/spfs/bin/python -c 'import wheel.bdist_wheel;
        print(wheel.bdist_wheel.get_abi_tag())')
      - |
        if [ "$python_abi" != "$SPK_OPT_abi" ]; then
          echo "Python binary ABI does not match spk options: $python_abi != $SPK_OPT_abi"
          exit 1
        fi
  - stage: install
    script:
      # Verify bz2 support is available by importing and not getting a traceback
      - test -z "$(/spfs/bin/python -c 'import bz2' 2>&1)"
      # Verify zlib support is available by importing and not getting a traceback
      - test -z "$(/spfs/bin/python -c 'import zlib' 2>&1)"

install:
  environment:
    - set: PYTHONDONTWRITEBYTECODE
      value: 1
  requirements:
    - pkg: binutils
      fromBuildEnv: Binary
    - pkg: gcc
      fromBuildEnv: Binary
      include: IfAlreadyPresent
    - pkg: stdfs
    - pkg: libffi
    - pkg: ncurses
    - pkg: bzip2
      fromBuildEnv: Binary
    - pkg: zlib
      fromBuildEnv: Binary
    - pkg: openssl
      fromBuildEnv: Binary
  components:
    - name: run
      files:
        - /etc/
        - /bin/
        - /lib/
        - '!/lib/pkgconfig'
    - name: build
      uses: [run]
      files:
        - /include/
        - /lib/pkgconfig
    - name: man
      files:
        - /share/man
$ spk mkr packages/python/python.spk.yaml -o version=2.7.5
 INFO rendering template for python
 INFO using options {arch=x86_64, distro=fedora, fedora=35, os=linux, version=2.7.5}

Outputs:

See final yaml after rendering
# 
# 
# 
pkg: python/2.7.5
api: v0/package
sources:
  - git: https://github.com/python/cpython
    ref: v2.7.5
build:
  options:
    - var: os
    - var: arch
    - var: centos
    - pkg: gcc
    - pkg: stdfs
    - pkg: bzip2
    # 
    - var: abi
      default: "cp27m"
      choices: [
        "cp27m",
        "cp27dm",
    # 
        "cp27mu",
        "cp27dmu",
    # 
      ]
      inheritance: Strong
    - var: debug
      default: off
      choices: [on, off]
    - var: optimize
      default: on
      choices: [on, off]
  variants:
    - { gcc: 6.3, abi: "cp27m", debug: off }
  script:
    - |
      case "$SPK_OPT_debug" in
        on)
          if ! [[ "$SPK_OPT_abi" =~ ^cp27.*d ]]; then
            echo "Must use an abi with debug when building with debug enabled!"
            exit 1
          fi
          DEBUG="--with-pydebug"
          ;;
        off)
          if [[ "$SPK_OPT_abi" =~ ^cp27.*d ]]; then
            echo "Must not use an abi with debug when building with debug disabled!"
            exit 1
          fi
          DEBUG=""
          ;;
        *)
          echo "Unsupported debug: $SPK_OPT_debug"
          ;;
      esac
    # 
    - |
      case "$SPK_OPT_abi" in
        cp27m)
          UNICODE="--enable-unicode=ucs2"
          ;;
        cp27mu)
          UNICODE="--enable-unicode=ucs4"
          ;;
        *)
          echo "Unsupported abi: $SPK_OPT_abi"
          ;;
      esac
    # on systems where python3 is the default, we can
    # see syntax errors unless we ensure that 'python' runs python2
    - echo "#!/bin/bash" > /spfs/bin/python
    - echo 'exec python2 "$@"' >> /spfs/bin/python
    - chmod +x /spfs/bin/python
    # 
    # 
    - >
      ./configure
      --prefix=${PREFIX}
      CC=$CC
      CXX=$CXX
      LDFLAGS='-Wl,--rpath=/spfs/lib,-L/spfs/lib'
      PKG_CONFIG_PATH=/spfs/share/pkgconfig:/spfs/lib/pkgconfig
      CPPFLAGS='-I/spfs/include/ncurses'
      --enable-shared
      --with-ensurepip=no
      
      "$UNICODE"
      $DEBUG
    - make -j$(nproc)
    - make install
    # remove test files that are just bloat
    - find /spfs/lib/python* -name "test" -type d | xargs -r rm -rv
    - find /spfs/lib/python* -name "*_test" -type d | xargs -r rm -rv
    - "ln -sf python2 /spfs/bin/python"
    # python is best in spfs when pyc files are not used at all
    - find /spfs -type f -name "*.pyc" | xargs rm

tests:
  - stage: install
    script:
      # Verify we built a python with the requested ABI
      - python_abi=$(/spfs/bin/python -c 'import wheel.bdist_wheel;
        print(wheel.bdist_wheel.get_abi_tag())')
      - |
        if [ "$python_abi" != "$SPK_OPT_abi" ]; then
          echo "Python binary ABI does not match spk options: $python_abi != $SPK_OPT_abi"
          exit 1
        fi
  - stage: install
    script:
      # Verify bz2 support is available by importing and not getting a traceback
      - test -z "$(/spfs/bin/python -c 'import bz2' 2>&1)"
      # Verify zlib support is available by importing and not getting a traceback
      - test -z "$(/spfs/bin/python -c 'import zlib' 2>&1)"

install:
  environment:
    - set: PYTHONDONTWRITEBYTECODE
      value: 1
  requirements:
    - pkg: binutils
      fromBuildEnv: Binary
    - pkg: gcc
      fromBuildEnv: Binary
      include: IfAlreadyPresent
    - pkg: stdfs
    - pkg: libffi
    - pkg: ncurses
    - pkg: bzip2
      fromBuildEnv: Binary
    - pkg: zlib
      fromBuildEnv: Binary
    - pkg: openssl
      fromBuildEnv: Binary
  components:
    - name: run
      files:
        - /etc/
        - /bin/
        - /lib/
        - '!/lib/pkgconfig'
    - name: build
      uses: [run]
      files:
        - /include/
        - /lib/pkgconfig
    - name: man
      files:
        - /share/man

TODO:

  • add 'replace-re' filter to do regular expression replacements
  • process option map so that "python.version" works in templates
  • try to extract error positions for format_serde_error reuse
  • allow passing template options from a file
  • tie into SPK Future Proofing #491 (maybe later)
  • consider adding 'canned' includable templates for repetitive actions
    • os/arch options
  • Add something to documentation
  • add a way to do if-else for spk version (liquid version?)
  • consider a more advanced, structured set of data to be available in templates
    spk:
      version: 0.34.0
    # liquid:
    #  version: 0.12.0
    opt:
      version: 1.0.0
      arch: x86_64
    env:
      PATH: "..."
    # build:
    #   workdir: "/home/user/work/project"

Closes #137

@rydrman rydrman added the enhancement New feature or request label Sep 24, 2022
@rydrman rydrman self-assigned this Sep 24, 2022
@codecov
Copy link

codecov bot commented Sep 24, 2022

Codecov Report

Merging #513 (3b10510) into master (b10c261) will increase coverage by 0.13%.
The diff coverage is 70.16%.

@@            Coverage Diff             @@
##           master     #513      +/-   ##
==========================================
+ Coverage   57.47%   57.61%   +0.13%     
==========================================
  Files         224      232       +8     
  Lines       17297    17457     +160     
==========================================
+ Hits         9942    10058     +116     
- Misses       7355     7399      +44     
Impacted Files Coverage Δ
...tes/spk-cli/cmd-make-recipe/src/cmd_make_recipe.rs 0.00% <0.00%> (ø)
crates/spk-schema/src/error.rs 50.00% <ø> (ø)
crates/spk/src/cli.rs 0.00% <0.00%> (ø)
crates/spk-cli/common/src/flags.rs 26.15% <31.25%> (-1.39%) ⬇️
crates/spk-schema/src/spec.rs 80.46% <71.42%> (-0.84%) ⬇️
...schema/crates/liquid/src/filter_compare_version.rs 76.92% <76.92%> (ø)
crates/spk-schema/crates/liquid/src/tag_default.rs 81.81% <81.81%> (ø)
...k-schema/crates/liquid/src/filter_parse_version.rs 89.47% <89.47%> (ø)
crates/spk-schema/crates/liquid/src/lib.rs 91.66% <91.66%> (ø)
...spk-schema/crates/foundation/src/option_map/mod.rs 77.86% <93.75%> (+4.28%) ⬆️
... and 3 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@rydrman rydrman added the agenda item Items to be brought up at the next dev meeting label Sep 24, 2022
@rydrman
Copy link
Collaborator Author

rydrman commented Sep 27, 2022

It's worth noting that this package seems to create decently contextual errors on it's own, but out-of-the-box there's no easy way to reuse the same error type from #499 like with the handlebars package

@jrray
Copy link
Collaborator

jrray commented Sep 27, 2022

This gets my vote for being a more capable templating engine. Not having conditionals would be too limiting.

@rydrman rydrman mentioned this pull request Sep 28, 2022
3 tasks
@rydrman rydrman force-pushed the 508-templates-alt branch 2 times, most recently from b6b1d18 to d4b4cdd Compare October 3, 2022 04:46
@rydrman
Copy link
Collaborator Author

rydrman commented Oct 5, 2022

I've submitted cobalt-org/liquid-rust#477 after our discussion today to see if it's really a bug or not

@rydrman rydrman force-pushed the 508-templates-alt branch from 4ca9a81 to 98de95c Compare October 6, 2022 16:07
@rydrman
Copy link
Collaborator Author

rydrman commented Oct 8, 2022

It sounds like at some level, erroring when a value does not exist is desired in this implementation. This is still a little odd to me, but I've managed to rework the default tag to work for values at any level so this gives us a fairly clean workflow to move forward with.

Which makes this ready to review

@rydrman rydrman changed the title Wip: Package Templating Alternative (Liquid) Package Templating Alternative (Liquid) Oct 8, 2022
@rydrman rydrman changed the title Package Templating Alternative (Liquid) Package Templating (va Liquid) Oct 8, 2022
@rydrman rydrman force-pushed the 508-templates-alt branch from 5ca328a to 4757496 Compare October 8, 2022 01:07
@rydrman rydrman force-pushed the 508-templates-alt branch 3 times, most recently from 5433114 to 657c4d3 Compare November 15, 2022 20:41
@rydrman rydrman added this to the Version 1.0 milestone Nov 17, 2022
crates/spk-cli/common/src/flags.rs Outdated Show resolved Hide resolved
crates/spk-cli/common/src/flags.rs Outdated Show resolved Hide resolved
crates/spk-schema/crates/liquid/src/tag_default.rs Outdated Show resolved Hide resolved
docs/use/spec.md Outdated Show resolved Hide resolved
docs/use/spec.md Outdated Show resolved Hide resolved
docs/use/spec.md Outdated Show resolved Hide resolved
@rydrman rydrman requested a review from jrray November 19, 2022 05:08
@rydrman rydrman requested a review from dcookspi December 13, 2022 01:59
@rydrman rydrman requested review from jrray and removed request for jrray February 1, 2023 22:46
@rydrman rydrman requested a review from paxsonsa February 18, 2023 02:51
@rydrman rydrman force-pushed the 508-templates-alt branch from 0ce2a24 to 5956b8d Compare April 6, 2023 02:27
rydrman added 2 commits April 8, 2023 21:32
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
@rydrman rydrman force-pushed the 508-templates-alt branch 2 times, most recently from 429ba67 to 438cdf0 Compare April 9, 2023 15:07
@rydrman
Copy link
Collaborator Author

rydrman commented Apr 9, 2023

My last comment about the default tag was not true. Although I remember doing that work it looks like I never pushed it and had to redo the implementation. I also addressed the linked issue in the liquid implementation and moved the dependency to my fork while the PR is open. This will need one last review!

@rydrman rydrman requested a review from jrray April 9, 2023 15:41
crates/spk-schema/src/spec_test.rs Outdated Show resolved Hide resolved
rydrman and others added 10 commits April 10, 2023 16:12
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Additionally, refactor the way that options and requests are loaded
to use a consistent method of layering and de-duplication.

Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
Signed-off-by: Ryan Bottriell <ryan@bottriell.ca>
Signed-off-by: Ryan Bottriell <rbottriell@ilm.com>
@rydrman rydrman force-pushed the 508-templates-alt branch from 438cdf0 to 3b10510 Compare April 10, 2023 23:12
@rydrman rydrman requested a review from jrray April 10, 2023 23:12
@rydrman rydrman merged commit 2d3b2d8 into master Apr 11, 2023
@rydrman rydrman deleted the 508-templates-alt branch April 11, 2023 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agenda item Items to be brought up at the next dev meeting enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Build Option Expansion (aka "templating")
2 participants