docs/concept: update modules guide for CUE v0.9.0+

This updates the "Modules, Packages and Instances" concept guide for the
"new" modules implementation that's the default in CUE v0.9.0 and later.
The term "module identifier" is updated to "module path" throughout, and
sections are added to explain the current package search algorithm.

This update is (mostly) intended to maintain the same document structure
and flow, whilst making sure that no incorrect prose is presented and
that deprecated features (such as cue.mod/{pkg,usr,gen}) aren't held out
as current and recommended. Some significant additional prose is added
to contextualise the deprecated features, and to expand on the multiple
disk locations at which a package might be found and their potential
interactions.

The previous version of the guide is archived at a different path,
clearly marked as relating to "old" modules, and with an explanatory
preamble that should prevent anyone making an active choice to use old
modules if they don't need to for legacy reasons.

This update doesn't perform a full rewrite of the guide. That will be
done later, when the guide is moved under the /docs/concept/modules/
hierarchy. Some rewording *is* included in this update (beyond that
required to meet this change's headline goal), but this isn't exhaustive
or complete. The guide's code blocks aren't modernised, as this will
also be done during the more significant rewrite.

Closes cue-lang/docs-and-content#179

Preview-Path: /docs/concept/modules-packages-instances/
Preview-Path: /docs/concept/old-modules-packages-instances/
Signed-off-by: Jonathan Matthews <[email protected]>
Change-Id: I01ef4b5e21a60f966919a593797b90b1a420cf86
Dispatch-Trailer: {"type":"trybot","CL":1200085,"patchset":22,"ref":"refs/changes/85/1200085/22","targetBranch":"master"}

content/docs/concept/modules-packages-instances/en.md

@@ -5,6 +5,17 @@ tags: [modules]
 toc_hide: true
 ---
 
+{{<info>}}
+This guide describes "new" modules, which are the current implementation of
+dependencies in CUE.
+
+[The previous version of this guide]({{< relref "old-modules-packages-instances" >}})
+covers "old" modules, which were the previous way to manage CUE dependencies,
+and is preserved for reference by folks using older versions of CUE.
+The FAQ "{{< linkto/inline "concept/faq/new-modules-vs-old-modules" >}}"
+explains their differences.
+{{</info>}}
+
 ## Overview
 
 CUE heavily relies on its order independence for package organization.
@@ -42,12 +53,11 @@ Here is how they differ:
 
 {{< /info >}}
 
-
 ## Modules
 
-A module contains a configuration layed out in a directory hierarchy.
+A module contains a configuration laid out in a directory hierarchy.
 It contains everything that is needed to deterministically
-determine the outcome of a CUE configuration.
+calculate the outcome of a CUE configuration.
 The root of this directory is marked by containing a `cue.mod`
 directory.
 The contents of this directory are mostly managed by the `cue` tool.
@@ -77,17 +87,35 @@ are considered to be part of this module.
 A module can be created by running the following command
 within the module root:
 
-{{{with code "en" "cmd-cue-mod-init"}}}
--- plain.txt --
-cue mod init [module name]
+{{{with script "en" "cue mod init #1"}}}
+cue mod init
+{{{end}}}
+{{{with _script_ "en" "HIDDEN: undo cue mod init"}}}
+rm -rf cue.mod
 {{{end}}}
 
-The module name is required if a package within the module needs to
-import another package within the module.
+A module path may be specified, but if it is omitted then a default value is
+used (currently `cue.example`). The module path is specified as an argument to
+the command:
 
-A module can also be created by setting up the `cue.mod` directory
-and `module.cue` file manually.
+{{{with script "en" "cue mod init #2"}}}
+cue mod init some.module.prefix/with/optional/path/components@v0
+{{{end}}}
 
+The module path must be used in CUE code if a package within the module needs
+to import another package defined within the module.
+Module paths start with a fully-qualified domain name, continue with optional
+forward-slash-separated path components, and end with an optional major version
+suffix.
+If the optional major version suffix is omitted then the resulting module is
+treated as if the major version suffix `@v0` were present.
+Ideally, the domain name and path components in a module path would map to a
+resource that's controlled by the CUE user (e.g. a website, a version control
+repository, or similar), but this is not looked up or enforced by the `cue`
+command.
+
+A module can also be created by setting up the `cue.mod` directory and
+`module.cue` file manually. **This is not recommended.**
 
 ### The cue.mod directory
 
@@ -97,38 +125,70 @@ The module directory has the following contents:
 -- plain.txt --
 cue.mod
 |-- module.cue  // The module file
-|-- pkg         // copies of external packages
-|-- gen         // files generated from external sources
-|-- usr         // user-defined constraints
+|-- pkg         // copies of external packages [DEPRECATED]
+|-- gen         // files generated from external sources [DEPRECATED]
+|-- usr         // user-defined constraints [DEPRECATED]
 {{{end}}}
 
-Aside from an occasional addition to the `usr` subdirectory or tweak
-to `module.cue`, this directory is
-predominantly managed by the `cue` tool.
-
-The `module.cue` file defines settings such as
-globally unique _module identifier_ (more on this in the
-[Import Path]({{< relref "#import-path" >}}) section).
-This information allows packages defined within the module to be importable
-within the module itself.
-In the future, it may hold version information of imported packages to determine
-the precise origin of imported files.
-
-The other directories hold packages that are facsimiles, derivatives, or
-augmentations of external packages:
-
-- *pkg*: an imported external CUE package,
-- *gen*: CUE generated from external definitions, such as protobuf or Go,
-- *usr*: user-defined constraints for the above two directories.
-
-These directories split files from the same package across different
+This directory is predominantly managed by the `cue` tool.
+
+The `module.cue` file defines settings such as the
+[*module path*]({{< relref "#import-path" >}}),
+which allows packages defined within the module to be imported from within the
+module itself.
+It also holds version information of imported packages to determine the precise
+origin of imported files. The modules reference documentation fully specifies
+[the contents of `module.cue`]({{< relref "docs/reference/modules" >}}#cue-mod-file).
+
+#### Deprecated directories
+
+The `pkg`, `gen` and `usr` directories inside the `cue.mod` directory can hold
+packages that are facsimiles, derivatives, or augmentations of external packages.
+Use of these directories is deprecated as they interact poorly with shared
+dependencies (those managed by the
+[`cue mod`]({{< relref "docs/reference/command/cue-help-mod" >}}) command), and
+are not accessible if their containing module is used as a dependency.
+
+<!-- TODO(jcm): some kind of diagram would really help this dense prose! -->
+(To demonstrate this inaccessibility, consider the example situation where
+module `A` depends on module `B`; module `B` depends on module `C`; and module
+`C` is stored in module `B`'s `pkg` directory. Because module `B` uses this
+deprecated method for storing its dependency, module `A` cannot use any
+packages from module `B`'s copy of module `C` without independently declaring
+its own dependency on module `C` through `cue mod`, or its own
+`pkg`/`usr`/`gen` directories. If module `B` had instead declared the
+dependency using the `cue mod` command and the `module.cue` file, then module
+`C` would be accessible to module `A` through the
+[shared module cache]({{< relref "docs/reference/modules" >}}#module-cache).)
+
+If a dependency is managed by the `cue mod` command but is also found in any of
+the `pkg`/`gen`/`usr` directories then
+[an error occurs]({{< relref "docs/concept/faq/new-modules-vs-old-modules" >}}#can-i-use-cuemodusr-with-new-modules).
+These directories are still supported as they can be useful in a limited set of
+circumstances, but only when CUE's current modules can't handle a particular
+use-case. They are intended to be used for the following:
+
+- `pkg`: manually managed and imported external packages
+- `gen`: CUE generated from external definitions, such as protobuf or Go
+- `usr`: user-defined constraints for external packages
+
+{{<info>}}
+Read the FAQ "{{< linkto/inline "concept/faq/new-modules-vs-old-modules" >}}" to
+learn about the differences between these two types of CUE module. Please help
+the CUE project by providing your feedback in {{<issue 2865/>}} **whenever you
+find yourself needing to use these deprecated directories** - this will help
+guide and shape the future of the modules implementation.
+{{</info>}}
+
+These three directories split files from the same package across different
 parallel hierarchies based on the origin of the content.
 But for all intent and purposes they should be seen as a single directory
 hierarchy.
 
-The `cue.mod/usr` directory is a bit special here.
-The `cue.mod/pkg` and `cue.mod/gen` directories are populated by the `cue` tool.
-The `cue.mod/usr` directory, on the other hand, holds user-defined
+The `usr` directory is a bit special here.
+The `gen` directory is populated by the `cue` tool and
+the `pkg` directory, whilst deprecated, can hold 3rd-party or external constraints.
+The `usr` directory, on the other hand, holds user-defined
 constraints for the packages defined in the other directories.
 
 User-defined constraints can be used to fill gaps in generated constraints;
@@ -138,7 +198,6 @@ to enforce that a certain API feature is still provided or of the desired form.
 The `usr` directory allows for a cleaner organization compared to storing
 such user-defined constraints directly in the `cue`-managed directories.
 
-
 ## Packages
 
 ### Files belonging to a package
@@ -160,15 +219,14 @@ know the name of the package as well
 
 {{{with code "en" "cmd-cue-eval-directory-and-package"}}}
 -- plain.txt --
-cue eval -p pkgname ./mypkg
+cue eval ./mypkg:packageName
 {{{end}}}
 
-If no module is defined, it will just load the files in this directory.
-If a module is defined, it will _also_ load all files with the same
-package name in its ancestor directories up till the module root.
-As we will see below,
-this strategy allows for defining organization-wide schemas and policies.
-
+If no module is defined then the `cue` command will only load the files in the
+current directory. If a module is defined then it will *also* load all files
+with the same package name in its ancestor directories up to the module root.
+As we will see below, this strategy allows for defining organization-wide
+schemas and policies.
 
 ### Import path
 
@@ -229,60 +287,84 @@ The relative position may not be within the `cue.mod` directory.
 
 ### Location on disk
 
-A `.cue` file can import a package by specifying its import path
+CUE code can import a package by specifying its import path
 with the import statement. For instance,
 
 {{{with code "en" "cue-import-example"}}}
 -- in.cue --
 import (
 	"list"
-
 	"example.com/path/to/package"
 )
 {{{end}}}
 
-Packages for which the first path component is not a fully qualified
-domain name are builtin packages and are not stored on disk.
-For other packages, CUE determines the location on disk as follows:
-
-1. If a module identifier is defined and is a prefix of the import path,
-   the package is located at the corresponding location relative to the
-   module root.
-2. Otherwise, the package contents looked up in
-   the `cue.mod/pkg`, `cue.mod/gen`, and `cue.mod/usr` subdirectores.
-
-In Step 2, an import path may match more than one directory.
-In that case, the contents of _all_ matched directories are used to build the
-package.
-Virtually, these directories should be seen as a single directory tree.
-
+Packages for which the first path component is **not** a fully qualified domain
+name are [builtin packages]({{< relref "#builtin-packages" >}}) and are not
+stored on disk, as with `list` in the example above. Any CUE code can import
+builtin packages, whether part of a module or not.
+
+In contrast, CUE code can only import non-builtin packages when that CUE code
+is part of a module. When a non-builtin package is imported, CUE determines its
+location on disk using several mechanisms in parallel:
+
+- If the module path is a prefix of the import path, then that prefix is
+  removed from the import path and the remainder of the import path is treated
+  as a directory path. If a package exists in that directory, relative to the
+  module root, then the package is located in that directory.
+- If a dependency of the current module declares a module path that has the
+  current module path as a prefix then:
+  - the dependency is fetched from the
+    [configured module registry]({{< relref "docs/reference/command/cue-help-registryconfig" >}})
+    if it's not already present in the
+    [shared module cache]({{< relref "docs/reference/modules" >}}#module-cache);
+  - taking the dependency's module path into account, the directory
+    inside the dependency that matches the import path is examined;
+  - if that directory contains a package then the package is located
+    in that directory.
+- The package contents are looked up in the deprecated `cue.mod/pkg`,
+  `cue.mod/gen`, and `cue.mod/usr` directories. If a package is found in any of
+  these deprecated directories then the package content is unified from the
+  relevant files across all of these directories.
+
+If **only one** of these three mechanisms discovers an appropriate directory
+location then that directory location (and its package content) is used.
+
+If **more than one** of these mechanisms discovers an appropriate directory
+location then
+[an error occurs]({{< relref "docs/concept/faq/new-modules-vs-old-modules" >}}#can-i-use-cuemodusr-with-new-modules).
+If **none** of these mechanisms discovers an appropriate directory location
+then an error occurs.
 
 ## Builtin Packages
 
-CUE has a collection of builtin packages that are compiled into the `.cue`
-binary.
+CUE has a standard library of builtin packages that are compiled into the `cue`
+command.
 
-A list of these packages form
-can be found here https://pkg.go.dev/cuelang.org/go/pkg.
+A list of these packages can be found at
+[pkg.go.dev/cuelang/go/pkg](https://pkg.go.dev/cuelang.org/go/pkg).
 The intention is to have this documentation in CUE format, but for now
 we are piggybacking on the Go infrastructure to host and present the CUE
-packages.
+standard library documentatation.
 
 To use a builtin package, import its path relative to `cuelang.org/go/pkg`
 and invoke the functions using its qualified identifier.
 For instance:
 
-{{{with code "en" "regexp-import-example"}}}
-#location top bottom
-
-exec cue eval regexp.cue
-cmp stdout out.golden
--- regexp.cue --
-import "regexp"
-
-matches: regexp.FindSubmatch(#"^([^:]*):(\d+)$"#, "localhost:443")
--- out.golden --
-matches: ["localhost:443", "localhost", "443"]
+{{{with code "en" "stdlib-import-example"}}}
+exec cue eval
+cmp stdout out
+-- stdlib.cue --
+package example
+
+import "strings"
+
+A: "Hello, world!"
+B: strings.ToUpper(A)
+C: strings.HasPrefix(B, "HELLO")
+-- out --
+A: "Hello, world!"
+B: "HELLO, WORLD!"
+C: true
 {{{end}}}
 
 ## File Organization

content/docs/concept/modules-packages-instances/gen_cache.cue

@@ -7,17 +7,36 @@ package site
 					page: {
 						cache: {
 							code: {
-								"cmd-cue-mod-init":                   "Ydb5kFV6sY3LP2tfOR+RzachntDOtluIkLnopAhhZks="
-								"cue-module-directory":               "T1AUhrbh1li5WSt5xW2C2Mz499xlqzKgmlT4VG0hy5U="
+								"cue-module-directory":               "uRCK+ofnTvuSGsbWM3GScVO46WT6FB+u2t20sebY/5Q="
 								"cmd-cue-eval-directory":             "0BXhTWVStDPiAJV6H4zXa2tju9nsIqi/JHJyPY8/h30="
-								"cmd-cue-eval-directory-and-package": "jZ1mwKv/SjAzGC2JWA859OD1k+7T/etiBmbdaJrDBEo="
+								"cmd-cue-eval-directory-and-package": "jCecsP2AmUVLfRZ0gKoZb06fEnU3pFyw9XEbtpXhzZM="
 								"import-path-explicit-package-name":  "dRiKimh0EZqPFbneq0RgWxJEh7vg1KkdIF1WrWyDLBs="
 								"import-path-implicit-package-name":  "KPyFU5Rq2TNBckmVwZoDXvbzKfqk0QH30hsV3BW3oXg="
 								"general-module-import-path":         "bBJ+frqks2U2MbBjKxwpYwPk+pCxTcPTkFN3NUfyVIU="
 								"example-module-import-path":         "/7PXfhfhUGdEZTOL9zsqCmHGqh0Zrvk9V/620dMjQqY="
 								"module-root-directory-contents":     "8/+fHjXuLrJ2C6CsKfPjj2CMuk42OyRFGimZSG6YkIw="
-								"cue-import-example":                 "7/tiuyUAqBUDiOM/GaGBt+2suUhtdKD1PqiekO+g29Y="
-								"regexp-import-example":              "1amJjaaoQKS9+o41IxrT9nB0kwYC2iAyaxkLFcuw/8Y="
+								"cue-import-example":                 "xV26pmQ58DXOAbhjvqNyLVGV6fCuHZeB368yMm0Fo6k="
+								"stdlib-import-example":              "NMKvWcj6zV4bZBcscs/DtEmwZv+xYP2Szk4Ia98wmI8="
+							}
+							multi_step: {
+								hash:       "8013F51U7KVC2CQHDJ78G1PHN8PPS9VHH0RVEI3CJ89L990HBMMG===="
+								scriptHash: "33TKFARPDRS0K4SESKA9PSPQMK0NJGIBVF6IBCMSIHOEUFBKUQHG===="
+								steps: [{
+									doc:      ""
+									cmd:      "cue mod init"
+									exitCode: 0
+									output:   ""
+								}, {
+									doc:      ""
+									cmd:      "rm -rf cue.mod"
+									exitCode: 0
+									output:   ""
+								}, {
+									doc:      ""
+									cmd:      "cue mod init some.module.prefix/with/optional/path/components@v0"
+									exitCode: 0
+									output:   ""
+								}]
 							}
 						}
 					}