Another JSONNET dependency management

[morlay]

Hi All. I have written another jsonnet dependency management.

Features

• Dependency management based on go modules
  • all dependency codes will download under $(go env GOMODCACHE)
  • GOPROXY supported to speed up downloading
  • JSONNET_PATH compatibility
• Includes all native functions of tanka
• Object YAML import supported
  • with hidden fields __dirname and __filename

see more https://github.com/jsonnetmod/jsonnetmod

Waiting your feedbacks 😃

Why not jsonnnet-bundler?

Network issue

jsonnet-bundler is downloading through github api from s3, witch will be broken without HTTPS_PROXY.
so i need something like GOPROXY to save me ( i could easy to setup prviate goproxy server too.)
and the jsonnet-bundler download repo codes into ${PROJECT_ROOT}/vendor/.tmp for each jsonnet projects.
using go modules, repo codes will under $(go env GOMODCACHE)/${repo}@${version}.
this feature will save disk usages, and avoid downloading repo twice if already cached.

[sh0rez]

Hey! I've found some time to play with this. I must say, great job, I really like this!

I have a couple of questions to better understand the project and it's relationship to other Jsonnet tools:

1. go mod relationship: It seems to be a descendant of go mod. Does it re-use code? Or only ideas? I'd love to hear some details on that
2. It appears to implement a custom importer at https://github.com/octohelm/jsonnetmod/blob/master/pkg/jsonnetmod/importer.go.
   • What is this for? Directly loading files form ~/go/pkg/mod?
   • Can this be fully replaced with the jpath directive in mod.jsonnet? That would be really awesome, because it would allow existing evaluators to work with your pkg manager without any changes
3. Can packages have dependencies themselves? How are incompatible versions resolved?

[morlay]

Let's start the tour from grafana.

When i found an example for grafana https://github.com/grafana/jsonnet-libs/blob/master/grafana/example/main.jsonnet

I touch a file named grafana.jsonnet

local grafana = import 'github.com/grafana/jsonnet-libs/grafana/grafana.libsonnet';
local k = import 'k.libsonnet';
{
  config+:: {
    prometheus_url: 'http://prometheus',
  },

  namespace: k.core.v1.namespace.new('grafana'),

  prometheus_datasource:: grafana.datasource.new('prometheus', $.config.prometheus_url, type='prometheus', default=true),

  grafana: grafana
           + grafana.withAnonymous()
           + grafana.addFolder('Example')
           + grafana.addDatasource('prometheus', $.prometheus_datasource),
}

Oops. I got an error.

$ jmod build ./grafana.jsonnet
execute failed: 
RUNTIME ERROR: resolve failed `k/main.libsonnet`: resolve `k` failed: unrecognized import path "k"
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/ksonnet-util/k-compat.libsonnet:3:11-31        thunk <k> from <$>
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/ksonnet-util/k-compat.libsonnet:5:1-2  $
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/ksonnet-util/grafana.libsonnet:2:2-29  $
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/ksonnet-util/kausal.libsonnet:4:2-28   $
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/grafana/configmaps.libsonnet:1:11-49   thunk <k> from <$>
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/grafana/configmaps.libsonnet:2:19-20   
        /Users/morlay/pkg/mod/github.com/grafana/[email protected]/grafana/configmaps.libsonnet:62:7-16   
        During manifestation    

Howerver. the mod.jsonnet already created.

$ cat mod.jsonnet
{
  module: '',
  require: {
    'github.com/grafana/jsonnet-libs':: 'v0.0.0-20210219224025-eae352a28812',
  },
}

We already found the the dep github.com/grafana/jsonnet-libs.

It is the custom importer do this.

In our codes, we import 'github.com/grafana/jsonnet-libs/grafana/grafana.libsonnet'. when we try to parse jsonnet code.
the importPath will call the custom importer.

if it can't find a local jsonnnetfile, it will resolve the dir github.com/grafana/jsonnet-libs/grafana.

First, we will try to find the RepoRoot of the import path like [go get did (https://pkg.go.dev/golang.org/x/tools/go/vcs#RepoRootForImportPath) github.com/grafana/jsonnet-libs

Then call cmd go mod download -json github.com/grafana/jsonnet-libs@upgrade to get go mod info

{
	"Path": "github.com/grafana/jsonnet-libs",
	"Version": "v0.0.0-20210219224025-eae352a28812",
	"Info": "/Users/morlay/pkg/mod/cache/download/github.com/grafana/jsonnet-libs/@v/v0.0.0-20210219224025-eae352a28812.info",
	"GoMod": "/Users/morlay/pkg/mod/cache/download/github.com/grafana/jsonnet-libs/@v/v0.0.0-20210219224025-eae352a28812.mod",
	"Zip": "/Users/morlay/pkg/mod/cache/download/github.com/grafana/jsonnet-libs/@v/v0.0.0-20210219224025-eae352a28812.zip",
	"Dir": "/Users/morlay/pkg/mod/github.com/grafana/[email protected]",
	"Sum": "h1:lIMcuJSGeDpWBdC0I2HaEGMIBJmHtKrghIj/nVopVeY=",
	"GoModSum": "h1:o+DeV1cDiTqtITZWsEUPG7i1z3njhqbzGE7JbJmWVNk="
}

The RepoRoot github.com/grafana/jsonnet-libs and the Version v0.0.0-20210219224025-eae352a28812 will be added to require of mod.jsonnet.

and the Dir will be replace the RepoRoot,
so the import github.com/grafana/jsonnet-libs/grafana/grafana.libsonnet will be resolved as /Users/morlay/pkg/mod/github.com/grafana/[email protected]/grafana/grafana.libsonnet

Now I have not enough time to implement go mod download -json (GOPROXY and multi vcs directly download supports) and the codes is under $GOROOT/src/go/cmd/internal/modcmd, the internal stop me to reuse directly.
I just call the cmd directly, and jmod have to require golang installed.

Let's continue our tour, to fix the k/main.libsonnet missing (the custom import only support dir resolving, so i hack k.libsonnet as k/main.libsonnet)

Same as go.mod, we could add a replace:

{
  replace: {
    k: 'github.com/jsonnet-libs/k8s-alpha/1.19',
  },
  jpath: 'vendor', // set jpath too, we could see the deps tree.
}

Run jmod build ./grafana.jsonnet again, we could get the rendered json codes.
The github.com/jsonnet-libs/k8s-alpha added to require of mod.jsonnet too.

And the vendor will be created with symlinks of all deps.

$ tree vendor
vendor
├── github.com
│   ├── grafana
│   │   └── jsonnet-libs -> /Users/morlay/pkg/mod/github.com/grafana/[email protected]
│   └── jsonnet-libs
│       └── k8s-alpha -> /Users/morlay/pkg/mod/github.com/jsonnet-libs/[email protected]
├── k -> /Users/morlay/pkg/mod/github.com/jsonnet-libs/[email protected]/1.19
└── ksonnet-util -> /Users/morlay/pkg/mod/github.com/grafana/[email protected]/ksonnet-util

Why ksonnet-util created?
because jmod now (v0.0.5+) is compatible with jsonnetfile.json.

Hope this could explain clear for Q1 & Q2.

The main purpose of custom importer is for auto import from codes.
Directly loading files form ~/go/pkg/mod just want to reduce disk usage.

[morlay]

Q3:

Can packages have dependencies themselves?

Sure. like go, we could use module in mod.jsonnet.

<mod.module>/path/to/sub/dir/main.jsonnet

How are incompatible versions resolved?

semver version.

jmod follow version rules of go modules.

Examples:

mod a dep github.com/prometheus-operator/[email protected]
mod b dep github.com/prometheus-operator/[email protected]

mod c dep a & b both, the github.com/prometheus-operator/[email protected] will be chosen.

if new version v0.7.1 release
We could use jmod get -u . to be easy to upgrade to new version

However, v0.7.0 and v0.6.0 may incompatible. We have to lock the version, like go mod did.

{ replace: { 'github.com/prometheus-operator/kube-prometheus': '@v0.6.0' }}

even, use the forked one

{ replace: { 'github.com/prometheus-operator/kube-prometheus': 'github.com/morlay/kube-prometheus@fork-fix' }}

VCS branch/tag/commitsha

We could define the special branch/tag/commitsha, too

{ replace: { 'github.com/kubernetes-monitoring/kubernetes-mixin': '@release-0.6' }}

require will record the version with two part.

{ require: { 'github.com/kubernetes-monitoring/kubernetes-mixin':: 'v0.1.1-0.20201207163533-1941868d86a7,release-0.6' }}

After first resolved, The first version v0.1.1-0.20201207163533-1941868d86a7 will be used until executing jmod get -u . to upgrade.

Now. replace always use the first defined. that means we could overwite it mod.jsonnet of root project

A WARNING will show like:

[WARNING] 'github.com/kubernetes-monitoring/kubernetes-mixin' already replaced to 
        'github.com/kubernetes-monitoring/[email protected]' (using by module 'github.com/prometheus-operator/kube-prometheus/jsonnet/[email protected]'), but another module want to replace
        'github.com/kubernetes-monitoring/kubernetes-mixin@master' (requested by module github.com/prometheus/node_exporter/docs/[email protected])

I prefer to use single source dir of one repo root like go modules.
I don't want to make jsonnet be another node_modules 😂

However, we have long journey to switch to semver repos.

[sh0rez]

Thanks for this detailed explanation!

Let me be clear right away, I think a custom importer that implements different import semantics from what google/jsonnet uses would be the worst thing that could possibly happen.

Luckily, that does not seem to be the case here.

The main purpose of custom importer is for auto import from codes.

That's totally fair because it adds nice functionality but does NOT change the import semantics

Directly loading files from ~/go/pkg/mod just want to reduce disk usage.

This actually does change the import semantics, but there totally are ways around it. Your jpath feature already creates symlinks, as afaict these (1) use a negligible amount of disk space and (2) have the exact same effect as directly importing from go/pkg/mod. Correct me if I'm wrong.

Let's continue our tour, to fix the k/main.libsonnet missing (the custom import only support dir resolving, so i hack k.libsonnet as k/main.libsonnet)

I kinda dislike the idea of treating a single library differently than the rest (https://github.com/jsonnetmod/jsonnetmod/blob/3c0b32ecca6ff48a3bb5162a15d4c21976c81175/pkg/jsonnetmod/importer.go#L42).

A generic solution I could think of:
If the key of the replace directive ends in .jsonnet|.libsonnet. Detect that and create import-stubs:

// mod.jsonnet
replace: {
    'k.libsonnet': 'github.com/jsonnet-libs/k8s-alpha/1.18/main.libsonnet',
}

// jpath/k.libsonnet:
(import "github.com/jsonnet-libs/k8s-alpha/1.18/main.libsonnet")

Import stubs are better than symlinks here, because these keep the semantics of the importer intact

Let me know what you think ;)

[morlay]

Nice suggestion. I'm new start with jsonnet, so I haven't consider google/jsonnet.

I did some refactor to implement the two features.

go install github.com/jsonnetmod/[email protected]

When jpath set, the custom import will return the foundAt with the symlinks of module or stub file under jpath, after downloading repo codes and resolving real repo dir by go modules.

[pmorch]

I'd just like to point out that https://github.com/jsonnetmod/jsonnetmod has this banner:

This repository has been archived by the owner. It is now read-only.