diff --git a/_books/ion-1-1/src/SUMMARY.md b/_books/ion-1-1/src/SUMMARY.md index 03bd0988..6b18eeff 100644 --- a/_books/ion-1-1/src/SUMMARY.md +++ b/_books/ion-1-1/src/SUMMARY.md @@ -9,10 +9,11 @@ - [System macros](macros/system_macros.md) - [Modules](modules.md) - [Defining modules](modules/defining_modules.md) - - [The encoding module](modules/encoding_module.md) + - [Directives](modules/directives.md) + - [Local modules](modules/local_modules.md) + - [Encoding modules](modules/encoding_modules.md) + - [System module](modules/system_module.md) - [Shared modules](modules/shared_modules.md) - - [Inner modules](modules/inner_modules.md) - - [The system module](modules/system_module.md) - [Binary encoding](binary/encoding.md) - [Encoding primitives](binary/primitives.md) - [`FlexUInt`](binary/primitives/flex_uint.md) diff --git a/_books/ion-1-1/src/glossary.md b/_books/ion-1-1/src/glossary.md index 436d1612..7d1d3e98 100644 --- a/_books/ion-1-1/src/glossary.md +++ b/_books/ion-1-1/src/glossary.md @@ -1,8 +1,8 @@ # Glossary **active encoding module**
-The _encoding module_ whose symbol table and macro table are available in the current _segment_ of an Ion _document_. -The active encoding module is set by a _directive_. +An _encoding module_ whose symbol table and macro table are available in the current _segment_ of an Ion _document_. +The sequence of active encoding modules is set by an _encoding directive_. **argument**
The sub-expression(s) within a macro invocation, corresponding to exactly one of the macro's parameters. diff --git a/_books/ion-1-1/src/modules.md b/_books/ion-1-1/src/modules.md index 3d042f81..461d45c6 100644 --- a/_books/ion-1-1/src/modules.md +++ b/_books/ion-1-1/src/modules.md @@ -9,7 +9,8 @@ Ion 1.1 also introduces the concept of a _module_, an organizational unit that h > [!TIP] > You can think of an Ion 1.0 symbol table as a module with an empty macro table. -In Ion 1.1, each stream has an [encoding module](modules/encoding_module.md)—the active `(symbol table, macro table)` pair that is being used to encode the stream. +In Ion 1.1, each stream has an [encoding module sequence](modules/encoding_modules.md)— +a collection of modules whose symbols and macros are being used to encode the current segment. ## Module interface @@ -17,7 +18,7 @@ The interface to a module consists of: * its _spec version_, denoting the Ion version used to define the module * its _exported symbols_, an array of strings denoting symbol content -* its _exported macros_, an array of `` pairs, where all names are unique identifiers (or null). +* its _exported macros_, an array of `` pairs, where all names (where specified) are unique identifiers The spec version is external to the module body and the precise way it is determined depends on the type of module being defined. This is explained in further detail in [Module Versioning](#module-versioning). @@ -27,7 +28,7 @@ by the `symbols` field of a shared symbol table. The exported macro array is denoted by the module’s `macro_table` clause, with addresses allocated to macros or macro bindings in the order they are declared. -The exported symbols and exported macros are defined in the [module body](body.md). +The exported symbols and exported macros are defined in the [module body](modules/defining_modules.md). ## Types of modules @@ -35,22 +36,22 @@ The exported symbols and exported macros are defined in the [module body](body.m There are multiple types of modules. All modules share the same interface, but vary in their implementation in order to support a variety of different use cases. -| Module Type | Purpose | -|:----------------------------------------------|:---------------------------------------------------------------| -| [Encoding Module](modules/encoding_module.md) | Defining the local encoding context | -| [System Module](modules/system_module.md) | Defining system symbols and macros | -| [Inner Module](modules/inner_modules.md) | Organizing symbols and macros and limiting the scope of macros | -| [Shared Module](modules/shared_modules.md) | Defining symbols and macros outside of the data stream | +| Module Type | Purpose | +|:------------------------------------------------|:----------------------------------------------------------------| +| [Local Modules](modules/local_modules.md) | Organizing symbols and macros within a scope | +| [Shared Modules](modules/shared_modules.md) | Defining reusable symbols and macros outside of the data stream | +| [System Modules](modules/system_module.md) | Defining system symbols and macros | +| [Encoding Modules](modules/encoding_modules.md) | Encoding the current stream segment | ## Module versioning Every module definition has a _spec version_ that determines the syntax and semantics of the module body. -A module’s spec version is expressed in terms of a specific Ion version; the meaning of the module is as defined by that version of the Ion specification. +A module’s spec version is expressed in terms of a specific Ion version; +the meaning of the module is as defined by that version of the Ion specification. -The spec version for an encoding module is implicitly derived from the Ion version of its containing segment. +The spec version for a local module is inherited from its parent scope, which may be the stream itself. The spec version for a shared module is denoted via a required annotation. -The spec version of an inner module is always the same as its containing module. The spec version of a system module is the Ion version in which it was specified. To ensure that all consumers of a module can properly understand it, a module can only import @@ -62,16 +63,20 @@ This allows the module to be serialized using any version of Ion, and its meanin ```ion $ion_shared_module:: -$ion_1_1::("com.example.symtab" 3 - (symbol_table ...) - (macro_table ...)) +$ion_1_1:: +("com.example.symtab" 3 + (symbol_table ...) + (macro_table ...)) ``` -The spec version of an encoding module is always the same as the Ion version of its enclosing segment. +The spec version of a local module is always the same as the spec version of its enclosing scope. +If the local module is defined at the top level of the stream, +its spec version is the Ion version of the current segment. ```ion $ion_1_1 -$ion_encoding::( +$ion:: +(module foo // Module semantics specified by Ion 1.1 ... ) @@ -79,12 +84,14 @@ $ion_encoding::( // ... $ion_1_3 -$ion_encoding::( +$ion:: +(module foo // Module semantics specified by Ion 1.3 ... ) //... // Assuming no IVM -$ion_encoding::( +$ion:: +(module bar // Module semantics specified by Ion 1.3 ... ) diff --git a/_books/ion-1-1/src/modules/defining_modules.md b/_books/ion-1-1/src/modules/defining_modules.md index 6904036c..278ae25f 100644 --- a/_books/ion-1-1/src/modules/defining_modules.md +++ b/_books/ion-1-1/src/modules/defining_modules.md @@ -7,32 +7,27 @@ A module is defined by four kinds of subclauses which, if present, always appear 3. `symbol_table` - an exported list of text values 4. `macro_table` - an exported list of macro definitions +The lexical name given to a module definition must be an [identifier](../modules.md#identifiers). +However, it must not begin with a `$`--this is reserved for system-defined bindings like `$ion`. ### Internal environment The body of a module tracks an internal environment by which macro references are resolved. This environment is constructed incrementally by each clause in the definition and consists of: -* the _visible modules_, a map from identifier to module +* the _module bindings_, a map from identifier to module definition * the _exported symbols_, an array containing symbol texts * the _exported macros_, an array containing name/macro pairs -Before any clauses of the module definition are examined, the initial environment is as follows: - -* The visible modules map binds `$ion` to the system module for the appropriate spec version. - Inside an encoding directive, the visible modules map also binds `$ion_encoding` to the active encoding module - (the encoding module that was active when the encoding directive was encountered). - For an inner module, it also includes the modules previously made available by the enclosing - module (via `import` or `module`). -* The macro table and symbol table are empty. +Before any clauses of the module definition are examined, each of these is empty. Each clause affects the environment as follows: -* An `import` declaration retrieves a shared module from the implementation’s catalog, assigns - it a name in the visible modules, and makes its macros available for use. - An error must be signaled if the name already appears in the visible modules. -* A `module` declaration defines a new module and assigns it a name in the visible modules. - An error must be signaled if the name already appears in the visible modules. +* An `import` declaration retrieves a shared module from the implementation’s catalog and binds a name to it, + making its macros available for use. + An error must be signaled if the name already appears in the module bindings. +* A `module` declaration defines a new module and binds a name to it. + An error must be signaled if the name already appears in the module bindings. * A `symbol_table` declaration defines the exported symbols. * A `macro_table` declaration defines the exported macros. @@ -53,19 +48,22 @@ macro-addr ::= unannotated-uint Macro references are resolved to a specific macro as follows: -* An unqualified _macro-name_ is looked up within the exported macros, and if not found, then the - active encoding module's macro table. If it maps to a macro, that’s the resolution of the reference. - Otherwise, an error is signaled due to an unbound reference. -* An anonymous local reference (_macro-addr_) is resolved by index in the exported macro array. +* An unqualified _macro-name_ is looked up in the following locations: + 1. in the macros already exported in this module's `macro_table` + 2. in the [default_module](encoding_modules.md#the-default-module) + 3. in the [system module](system_module.md) + + If it maps to a macro, that’s the resolution of the reference. Otherwise, an error is signaled due to an unbound reference. + +* An anonymous local reference (_macro-addr_) is resolved by index in the exported macro array. If the address exceeds the array boundary, an error is signaled due to an invalid reference. * A qualified reference (_qualified-ref_) resolves solely against the referenced module. - If the module name does not exist in the visible modules, an error is signaled due to an unbound reference. - Otherwise, the name or address is resolved within that module’s exported macro array. + First, the module name must be resolved to a module definition. + * If the module name is in the module bindings, it resolves to the corresponding module definition. + * If the module name is not in the module bindings, resolution is attempted recursively upwards through the parent scopes. + * If the search reaches the top level without resolving to a module, an error is signaled due to an unbound reference. -> [!WARNING] -> An unqualified macro name can change meaning in the middle of an encoding module if you choose to shadow the -> name of a macro in the active encoding module. To unambiguously refer to the active encoding module, -> use the qualified reference syntax: `$ion_encoding::`. + Next, the name or address is resolved within that module definition’s exported macro table. ### `import` @@ -82,7 +80,8 @@ catalog-name ::= string catalog-version ::= int // positive, unannotated ``` -An import binds a lexically scoped module name to a shared module that is identified by a catalog key—a `(name, version)` pair. The `version` of the catalog key is optional—when omitted, the version is implicitly 1. +An import binds a lexically scoped module name to a shared module that is identified by a catalog key—a `(name, version)` pair. +The `version` of the catalog key is optional—when omitted, the version is implicitly 1. In Ion 1.0, imports may be substituted with a different version if an exact match is not found. In Ion 1.1, however, all imports require an exact match to be found in the reader's catalog; @@ -92,7 +91,7 @@ if an exact match is not found, the implementation must signal an error. ### `module` -The `module` clause defines a new module that is contained in the current module. +The `module` clause defines a new local module that is contained in the current module. ```bnf inner-module ::= '(module' module-name import* symbol-table? macro-table? ')' @@ -102,7 +101,7 @@ Inner modules automatically have access to modules previously declared in the co The new module (and its exported symbols and macros) is available to any following `module`, `symbol_table`, and `macro_table` clauses in the enclosing container. -See [inner modules](inner_modules.md) for full explanation. +See [local modules](local_modules.md) for full explanation. ### `symbol_table` diff --git a/_books/ion-1-1/src/modules/directives.md b/_books/ion-1-1/src/modules/directives.md new file mode 100644 index 00000000..21166ac9 --- /dev/null +++ b/_books/ion-1-1/src/modules/directives.md @@ -0,0 +1,81 @@ +# Directives + +_Directives_ are system values that modify the encoding context. + +Syntactically, a directive is a top-level s-expression annotated with `$ion`. +Its first child value is an operation name. +The operation determines what changes will be made to the encoding context and which clauses may legally follow. + +```ion +$ion:: +(operation_name + (clause_1 /*...*/) + (clause_2 /*...*/) + /*...more clauses...*/ + (clause_N /*...*/)) +``` + +In Ion 1.1, there are three supported directive operations: +1. [`module`](#module-directives) +2. [`import`](#import-directives) +3. [`encoding`](#encoding-directives) + +## Top-level bindings + +The `module` and `import` directives each create a stream-level binding to a module definition. +Once created, module bindings at this level endure until the file ends or another Ion version marker is encountered. + +Module bindings at the stream-level can be redefined. + +> [!TIP] +> The [`add_macros`](../macros/system_macros.md#add_macros) and [`add_symbols`](../macros/system_macros.md#add_symbols) +> system macros work by redefining the default module (`_`) in terms of itself. + +This behavior differs from module bindings created inside another module; +[attempting to redefine these will raise an error](defining_modules.md#internal-environment). + +### `module` directives +The `module` directive binds a name to a [local module](local_modules.md) definition at the top level of the stream. + +```ion +$ion:: +(module foo + /*...imports, if any...*/ + /*...submodules, if any...*/ + (macro_table /*...*/) + (symbol_table /*...*/) +) +``` + +### `import` directives + +The _import_ directive looks up the module corresponding to the given `(name, version)` pair in the catalog. +Upon success, it creates a new binding to that module at the top level of the stream. + +```ion +$ion:: +(import + bar // Binding + "com.example.bar" // Module name + 2) // Module version +``` +The `version` can be omitted. When it is not specified, it defaults to `1`. + +If the catalog does contain an exact match, this operation raises an error. + +## `encoding` directives + +An `encoding` directive accepts a sequence of module bindings to use as the following stream segment's +[encoding module sequence](encoding_modules.md). + +```ion +$ion:: +(encoding + mod_a + mod_b + mod_c) +``` + +The new encoding module sequence takes effect immediately after the directive and remains the same until the next `encoding` directive or Ion version marker. + +Note that the [default module](encoding_modules.md#the-default-module) is always implicitly at the head of the encoding module sequence. \ No newline at end of file diff --git a/_books/ion-1-1/src/modules/encoding_module.md b/_books/ion-1-1/src/modules/encoding_module.md deleted file mode 100644 index 3353d8f3..00000000 --- a/_books/ion-1-1/src/modules/encoding_module.md +++ /dev/null @@ -1,75 +0,0 @@ -# The encoding module - -The _encoding module_ is the module that is currently being used to encode the data stream. -When the stream begins, the encoding module is the [system module](system_module.md). - -The application may define a new encoding module by writing an _encoding directive_ at the top level of the stream. -An encoding directive is an s-expression annotated with `$ion_encoding`; its nested clauses define a new encoding module. - -When the reader advances beyond an encoding directive, the module it defined becomes the new encoding module. - -In the context of an encoding directive, the active encoding module is named `$ion_encoding`. -The encoding directive may preserve symbols or macros that were defined in the previous encoding directive by referencing `$ion_encoding`. -The `$ion_encoding` module may only be imported to an encoding directive, and it is done so automatically and implicitly. - -### Examples - -#### An encoding directive -A simple encoding directive—it defines a module that exports three symbols and two macros. -```ion -$ion_encoding::( - (symbol_table [ - "a", // $1 - "b", // $2 - "c" // $3 - ]) - (macro_table - (macro pi () 3.14159265) - (macro moon_landing_ts () 1969-07-20T20:17Z) - ) -) -``` - -#### Adding symbols to the encoding module -The implicitly imported `$ion_encoding` is used to append to the current symbol and macro tables. - -```ion -$ion_encoding::( - (symbol_table [ - "a", // $1 - "b", // $2 - "c", // $3 - ]) - (macro_table - (macro pi () 3.14159265) - (macro moon_landing_ts () 1969-07-20T20:17Z) - ) -) - -// ... - -$ion_encoding::( - // The first argument of the symbol_table clause is the module name '$ion_encoding', - // which adds the symbols from the active encoding module to the new encoding module. - // The '$ion_encoding' argument in the macro_table clause behaves similarly. - (symbol_table $ion_encoding - [ - "d", // $4 - "e", // $5 - "f", // $6 - ]) - (macro_table $ion_encoding - (macro e () 2.71828182)) -) - -// ... -``` - -#### Clearing the local symbols and local macros -```ion -$ion_encoding::() -``` -The absence of the `symbol_table` and `macro_table` clauses is interpreted as empty symbol and macro tables. - -Note that this is different from the behaviour of an IVM. -When an IVM is encountered, the encoding module is set to the system module. diff --git a/_books/ion-1-1/src/modules/encoding_modules.md b/_books/ion-1-1/src/modules/encoding_modules.md new file mode 100644 index 00000000..68bf1f09 --- /dev/null +++ b/_books/ion-1-1/src/modules/encoding_modules.md @@ -0,0 +1,238 @@ +# Encoding modules + +The encoding of each segment of a stream is shaped by the currently configured _encoding modules_, +an ordered sequence of modules that determine which symbols and macros are available for use in the stream. +A writer can modify this sequence by emitting an [encoding directive](directives.md#encoding-directives). + +By logically concatenating the encoding modules' symbol and macro tables respectively, +they can be viewed as unified local symbol and macro tables. + +For example, consider these module definitions and the subsequent encoding directive: +```ion +$ion:: +(module mod_a + (symbol_table ["a", "b", "c"]) + (macro_table + (macro foo () Foo) + (macro bar () Bar))) +$ion:: +(module mod_b + (symbol_table ["c", "d", "e"]) + (macro_table + (macro baz () Baz) + (macro quux () Quux))) +$ion:: +(module mod_c + (symbol_table ["f", "g", "h"]) + (macro_table + (macro quuz () Quuz) + (macro foo () Foo2))) + +$ion:: +(encoding + mod_a + mod_b + mod_c) +``` +It produces the encoding module sequence `_ mod_a mod_b mod_c`. +(The [default module](#the-default-module), `_`, is always implicitly at the head of the encoding sequence.) + +The segment's local symbol table, formed by logically concatenating the symbol tables of `mod_a`, +`mod_b`, and `mod_c` in that order, is: + +| Address | Symbol text | +|:-------:|-------------| +| `0` | `a` | +| `1` | `b` | +| `2` | `c` | +| `3` | `c` | +| `4` | `d` | +| `5` | `e` | +| `6` | `f` | +| `7` | `g` | +| `8` | `h` | + +Notice that no de-duplication takes place; `c` appears in both addresses `3` and `4`. + +The segment's macro table, formed by logically concatenating the macro tables of `mod_a`, +`mod_b`, and `mod_c` in that order, is: + +| Address | Macro | +|:-------:|---------------| +| `0` | `mod_a::foo` | +| `1` | `mod_a::bar` | +| `2` | `mod_b::baz` | +| `3` | `mod_b::quux` | +| `4` | `mod_c::quuz` | +| `5` | `mod_c::foo` | + +Notice that `mod_a::foo` and `mod_c::foo` can coexist in this unified view without issue. +Invocations of these macros require that they be qualified by their enclosing module's name. + +Because lower addresses take fewer bytes to encode than higher addresses, +writers should place the modules they anticipate referencing the most frequently at the beginning of the encoding module sequence. + +Modules in the current segment's encoding module sequence are said to be _active_, +while modules that are defined or imported but which are not in the encoding module sequence are _available_. +E-expressions can only invoke macros in an active module. + +For example: +```ion +$ion:: +(module mod_a + (macro_table + (macro foo () Foo))) + +// `mod_a` is now available + +$ion:: +(module mod_b + (macro_table + (macro bar () Bar))) + +// `mod_b` is now available + +$ion:: +(encoding mod_a) + +// `mod_a` is now active + +(:mod_a::foo) // Foo +(:mod_b::bar) // ERROR: `mod_b` is not in the encoding module sequence +``` + +## The default module + +The default module, `_`, is an empty top-level module that is implicitly defined at the beginning of every stream. + +When resolving an unqualified macro name, readers first look for the corresponding macro definition in `_`. +If it is not found in `_`, they will then look in [`$ion`](system_module.md). +If it is still not found, the reader will raise an error. + +This makes it possible to leverage macros in a lightweight way; +writers do not have to first name/define a custom module to house their macros, +and the macros themselves can be invoked in text without having to write out the module name. + +Macros and symbols can be added to the default module by redefining `_`. +Like all modules, `_` can be redefined in terms of itself, making appends and prepends straightforward. + +```ion +$ion_1_1 + +// `_` exists, but is empty + +$ion:: +(module _ + (macro_table + (macro foo () Foo))) + +// `_` now contains macro `foo` + +$ion:: +(module _ + (macro_table + _ // Add all macros in `_` to its redefinition + (macro bar () Bar))) + +// `_` now contains macros `foo` and `bar` + +(:foo) // Equivalent to `(:_::foo)` +(:bar) // Equivalent to `(:_::bar)` +``` + +System macros like [`add_symbols`](../macros/system_macros.md#add_symbols) +and [`add_macros`](../macros/system_macros.md#add_macros) apply their changes to `_`, +so we can rewrite the above more succinctly as: +```ion +$ion_1_1 + +// `_` exists, but is empty + +(:add_macros + (macro foo () Foo) + (macro bar () Bar)) + +// `_` now contains macros `foo` and `bar` + +(:foo) // Equivalent to `(:_::foo)` +(:bar) // Equivalent to `(:_::bar)` +``` + +`_` can also be redefined by an [`import`](directives.md#import-directives) directive. + +## Default encoding module sequence + +At the beginning of a stream, the encoding module sequence contains two modules: +1. the [default module](#the-default-module), `_` +2. the [system module](system_module.md), `$ion` + +Recall that a segment's symbol and macro tables are logical concatenations of those found in the segment's encoding modules. +Because `_` is empty at the beginning of the stream, +the stream's initial symbol and macro tables are identical to those of the system module, `$ion`. + +This is beneficial because it allows all system macros to be invoked from the stream's macro table +[in a single byte](../binary/e_expressions.md#e-expression-with-the-address-in-the-opcode) +rather than [the two-byte sequence](http://localhost:3000/binary/e_expressions.html#system-macro-invocations) +needed to invoke them from the system macro table. +In this way, a writer can define its macros and symbols in a maximally compact fashion at the head of the stream. + +## Modifying active modules + +If a module binding in the encoding module sequence is redefined, +the new module definition replaces the old one in the sequence. + +For example after these directives are evaluated: +```ion +$ion:: +(module mod_a + (macro_table + (macro foo () Foo)) + (macro bar () Bar))) + +$ion:: +(module mod_b) + +$ion:: +(module mod_c + (macro_table + (macro quux () Quux) + (macro quuz () Quuz))) + +$ion::(encoding mod_a mod_b mod_c) +``` +the encoding sequence is `_ mod_a mod_b mod_c`, and `mod_b` is empty. + +```ion +(.0) // => Foo +(.1) // => Bar +(.2) // => Quux +(.3) // => Quuz +``` + +If we then add macros to `mod_b`, those macros will immediately become available. + +```ion +$ion:: +(module mod_b + (macro_table + (macro baz () Baz))) + +(.0) // => Foo +(.1) // => Bar +(.2) // => Baz +(.3) // => Quux +(.4) // => Quuz +``` + +> [!IMPORTANT] +> Notice that modifying a module (in this case `mod_b`) can cause the addresses of all subsequent macros to be modified. + +## Clearing the symbol and macro tables +```ion +(module _) // Redefine `_` to be an empty module +// If other modules are in use, remove them from the encoding module sequence +$ion::(encoding) +``` + +Note that this is different from the behaviour of an IVM. +When an IVM is encountered, the encoding module is set to the system module. \ No newline at end of file diff --git a/_books/ion-1-1/src/modules/inner_modules.md b/_books/ion-1-1/src/modules/inner_modules.md deleted file mode 100644 index 5b040760..00000000 --- a/_books/ion-1-1/src/modules/inner_modules.md +++ /dev/null @@ -1,84 +0,0 @@ -# Inner modules - -Inner modules are defined within another module, and can be referenced only within the enclosing module. -Their scope is lexical; they can be referenced immediately following their definition, up until the end of the containing module. - -Inline modules always have a symbolic name given at the point of definition. -They inherit their spec version from the containing module, and they have no content version. -Inner modules automatically have access to modules previously declared in their containing module using `module` or `import`. -Inner modules may not contain their own nested inner modules. - -### Examples - -Inner modules can be used to define helper macros and use them by name in the definitions of other macros without -having to export the helper macro by name. -```ion -$ion_shared_module::$ion_1_1::( - "org.example.Foo" 1 - (module util (macro_table (macro point2d (x y) { x:(%x), y:(%y) }))) - (macro_table - (export util::0) - (macro y_axis_point (y) (.util::point2d 0 (%y))) - (macro poylgon (util::point2d::points+) [(%points)])) -) -``` -In this example, the macro `point2d` is declared in an inner module. -It is added to the shared module's macro table _without a name_, and subsequently referenced by name in the definition -of other macros. - -
- -Inner modules can also be used for grouping macros into namespaces (only visible within the outer module), and to declare -helper macros that are not added to the macro table of the outer module. -```ion -$ion_shared_module::$ion_1_1::( - "org.example.Foo" 1 - (module cartesian (macro_table (macro point2d (x y) { x:(%x), y:(%y) }) - (macro polygon (point2d::points+) [(%points)]) )) - - (module polar (macro_table (macro point2d (r phi) { r:(%r), phi:(%phi) }) - (macro polygon (point2d::points+) [(%points)]) )) - (macro_table - (export cartesian::polygon cartesian_poylgon) - (export polar::polygon polar_poylgon)) -) -``` -In this example, there are two macros named `point2d` and two named `polygon`. -There is no name conflict between them because they are declared in separate namespaces. -Both `polygon` macros are added to the shared module's macro table, each one given an alias in order to resolve the name conflict. -Neither one of the `point2d` macros needs to be added to the shared module's macro table because they can be referenced -in the definitions of both `polygon` macros without needing to be added to the shared module's macro table. - -
- -When grouping macros in inner modules, there are more than just organizational benefits. -By defining helper macros in an inner module, the order in which the macros are added to the macro table of the outer module does not have to be the same as the order in which the macros are declared: -```ion -$ion_shared_module::$ion_1_1::( - "org.example.Foo" 1 - // point2d must be declared before polygon... - (module util (macro_table (macro point2d (x y) { x:(%x), y:(%y) }))) - (macro_table - // ...because it is used in the definition of polygon - (macro poylgon (util::point2d::points+) [(%points)]) - // But it can be added to the macro table after polygon - util) -) -``` - -
- -Inner modules can also be used for organization of symbols. -```ion -$ion_encoding::( - (module dairy (symbol_table [cheese, yogurt, milk])) - (module grains (symbol_table [cereal, bread, rice])) - (module vegetables (symbol_table [carrots, celery, peas])) - (module meat (symbol_table [chicken, mutton, beef])) - - (symbol_table dairy - grains - vegetables - meat) -) -``` \ No newline at end of file diff --git a/_books/ion-1-1/src/modules/local_modules.md b/_books/ion-1-1/src/modules/local_modules.md new file mode 100644 index 00000000..b435779c --- /dev/null +++ b/_books/ion-1-1/src/modules/local_modules.md @@ -0,0 +1,132 @@ +# Local modules + +Local modules are lexically scoped. +They can be referenced immediately following their definition, up until the end of their enclosing scope. +They can be defined either: +1. **At the top level of a stream**, in which case the enclosing scope is the stream itself. +2. **Inside another module**, in which case the enclosing scope is the parent module. + The parent module can be a shared or local module. + +Local modules always have a symbolic name given at the point of definition, also known as a _binding_. +It is legal for a module binding to "shadow" a module binding in its parent scope by using the same name. + +```ion +$ion:: +(module foo // <-- Top-level module `foo` + (macro_table + (macro quux () Quux))) + +$ion:: +(module bar + (module foo // <-- Shadows the top-level module `foo` + (macro_table + (macro quuz () Quuz))) + (macro_table foo::quuz) // <-- Refers to the innermost `foo` +) +``` + +However, it is _not_ legal for a local module to use the same name as a module previously defined in the _same_ scope. + +```ion +$ion:: +(module bar + (module foo // <-- First definition of `foo` inside `bar` + (macro_table + (macro quux () Quux))) + (module foo // <-- ERROR: module `foo` already defined in this scope + (macro_table + (macro quuz () Quuz))) + /*...*/ +) +``` + +The only exception to this rule is at the top level. +Stream-level bindings are mutable, while bindings inside a module are immutable. + +```ion +$ion:: +(module foo // <-- Top-level module `foo` + (macro_table + (macro quux () Quux))) + +$ion:: +(module foo // <-- Redefines the top-level binding `foo` + (macro_table + (macro quuz () Quuz))) +``` + +Local modules inherit their spec version from the enclosing scope. +Local modules automatically have access to modules previously declared in their enclosing scope using `module` or `import`. + +### Examples + +Local modules can be used to define helper macros without having to export them. +```ion +$ion_shared_module::$ion_1_1::( + "org.example.Foo" 1 + (module util (macro_table (macro point2d (x y) { x:(%x), y:(%y) }))) + (macro_table + (macro y_axis_point (y) (.util::point2d 0 (%y))) + (macro poylgon (util::point2d::points+) [(%points)])) +) +``` +In this example, the macro `point2d` is declared in a local module. +The macro definitions being exported in the shared module's macro table are able to reference the helper macros by name. + +
+ +Local modules can also be used for grouping macros into namespaces (only visible within the parent scope). +```ion +$ion_shared_module::$ion_1_1::( + "org.example.Foo" 1 + (module cartesian (macro_table (macro point2d (x y) { x:(%x), y:(%y) }) + (macro polygon (point2d::points+) [(%points)]) )) + + (module polar (macro_table (macro point2d (r phi) { r:(%r), phi:(%phi) }) + (macro polygon (point2d::points+) [(%points)]) )) + (macro_table + (export cartesian::polygon cartesian_poylgon) + (export polar::polygon polar_poylgon)) +) +``` +In this example, there are two macros named `point2d` and two named `polygon`. +There is no name conflict between them because they are declared in separate namespaces. +Both `polygon` macros are added to the shared module's macro table, +with each one given an alias in order to resolve the name conflict. +Neither one of the `point2d` macros needs to be added to the shared module's macro table because they can be referenced +in the definitions of both `polygon` macros without needing to be added to the shared module's macro table. + +
+ +When grouping macros in local modules, there are more than just organizational benefits. +By first defining helper macros in an inner module, a module can export macros in a different order than they are declared: +```ion +$ion_shared_module::$ion_1_1::( + "org.example.Foo" 1 + // point2d must be declared before polygon... + (module util (macro_table (macro point2d (x y) { x:(%x), y:(%y) }))) + (macro_table + // ...because it is used in the definition of polygon + (macro poylgon (util::point2d::points+) [(%points)]) + // But it can be added to the macro table after polygon + util) +) +``` + +
+ +Local modules can also be used for organization of symbols. +```ion +$ion:: +(encoding + (module dairy (symbol_table [cheese, yogurt, milk])) + (module grains (symbol_table [cereal, bread, rice])) + (module vegetables (symbol_table [carrots, celery, peas])) + (module meat (symbol_table [chicken, mutton, beef])) + + (symbol_table dairy + grains + vegetables + meat) +) +``` \ No newline at end of file diff --git a/_books/ion-1-1/src/modules/shared_modules.md b/_books/ion-1-1/src/modules/shared_modules.md index 87a11a10..1eba79d7 100644 --- a/_books/ion-1-1/src/modules/shared_modules.md +++ b/_books/ion-1-1/src/modules/shared_modules.md @@ -4,7 +4,7 @@ Shared modules exist independently of the documents that use them. They are identified by a _catalog key_ consisting of a string name and an integer version. The self-declared catalog-names of shared modules are generally long, since they must be more-or-less globally unique. -When imported by another module, they are given local symbolic names by import declarations. +When imported by another module, they are given local symbolic names—a _binding_—by import declarations. They have a spec version that is explicit via annotation, and a content version derived from the catalog version. The spec version of a shared module must be declared explicitly using an annotation of the form `$ion_1_N`. diff --git a/_books/ion-1-1/src/whats_new.md b/_books/ion-1-1/src/whats_new.md index b19b7d24..7d5c1e6d 100644 --- a/_books/ion-1-1/src/whats_new.md +++ b/_books/ion-1-1/src/whats_new.md @@ -15,13 +15,35 @@ compact. ## Backwards compatibility -Ion 1.1 is backwards compatible with Ion 1.0. While their encodings are distinct, they share the same data model--any data that can be produced and read by an application in Ion 1.1 has an equivalent representation in Ion 1.0. +Ion 1.0 and Ion 1.1 share the same data model. +Any data that can be represented in Ion 1.0 can also be represented with full fidelity in Ion 1.1 and vice-versa. +This means that it is always possible to convert data from one version to the other without risk of data loss. -Ion 1.1 is *not* required to preserve Ion 1.0 binary encodings in Ion 1.1 encoding contexts (i.e., the type codes and -lower-level encodings are not preserved in the new version). The Ion Version Marker (IVM) is used to denote the -different versions of the syntax. Ion 1.1 does retain text compatibility with Ion 1.0 in that the changes are a strict -superset of the grammar, however due to the updated system symbol table, symbol IDs referred to using the `$n` syntax -for symbols beyond the 1.0 system symbol table are not compatible. +Ion 1.1 readers must be able to understand both Ion 1.0 and Ion 1.1 data. + +The text encoding grammar of Ion 1.1 is a superset of Ion 1.0's text encoding grammar. +Any Ion 1.0 text data can also be parsed by an Ion 1.1 text parser. +However, because Ion 1.1 has a different system symbol table, +symbol IDs in an Ion 1.0 stream do not always refer to the same text as the same symbol ID in an Ion 1.1 stream. +(For example: in an Ion 1.0 stream, `$4` is always the text `"name"`. However, it may or may not be `"name"` in an Ion 1.1 stream.) + +Ion 1.1's binary encoding is substantially different from Ion 1.0's binary encoding. +Many changes have been made to make values more compact, faster to read and/or faster to write. +Ion 1.0's [type descriptors](https://amazon-ion.github.io/ion-docs/docs/binary.html#typed-value-formats) +have been supplanted by Ion 1.1's more general +[opcodes](https://amazon-ion.github.io/ion-docs/docs/binary.html#typed-value-formats), +which have been organized to prioritize the most commonly used encodings and make leveraging macros as inexpensive as possible. + +In both text and binary Ion 1.1, the Ion Version Marker syntax is syntactically compatible with Ion 1.0's version marker syntax. +This means that an Ion 1.0-only reader can correctly identify when a stream uses Ion 1.1 (allowing it to report an error), +and an Ion 1.1 reader can correctly "downshift" to expecting Ion 1.0 data when it encounters a stream using Ion 1.0. + +Two streams using different Ion versions can be safely concatenated together provided that they are both text or both binary. +A concatenated stream containing both Ion 1.0 and Ion 1.1 can only be fully read by a reader that supports Ion 1.1. + +Upgrading an existing application to Ion 1.1 often requires little-to-no code changes, +as APIs typically operate at the data model level ("write an integer") +rather than at the encoding level ("write `0x64` followed by four Little-Endian bytes"). ## Text syntax changes @@ -30,7 +52,7 @@ Ion 1.1 text *must* use the `$ion_1_1` version marker at the top-level of the da The only syntax change for the text format is the introduction of *encoding expression* (*E-expression*) syntax, which allows for the invocation of macros in the data stream. This syntax is grammatically similar to S-expressions, except that these expressions are opened with `(:` and closed with `)`. For example, `(:a 1 2)` would expand the macro named `a` with the -arguments `1` and `2`. See the <> section for details. +arguments `1` and `2`. See the _[Macros, templates, and encoding expressions](#macros-templates-and-encoding-expressions)_ section for details. This syntax is allowed anywhere an Ion value is allowed: @@ -49,8 +71,11 @@ This syntax is allowed anywhere an Ion value is allowed: {c: (:bop d)} ``` -E-expressions are also grammatically allowed in the field name position of a struct and when used there, indicate that -the expression should expand to a struct value that is merged into the enclosing struct: + +E-expressions may also appear in the field name position of a struct. **E-Expression in field position of struct** ```ion @@ -62,9 +87,6 @@ the expression should expand to a struct value that is merged into the enclosing } ``` -In the above example, the E-expression `(:foo 1 2)` must evaluate into a struct that will be merged between the `b` -field and the `c` field. If it does not evaluate to a struct, then the above is an error. - ## Binary encoding changes Ion 1.1 binary encoding reorganizes the type descriptors to support compact E-expressions, make certain encodings @@ -74,9 +96,9 @@ sequence `0xE0 0x01 0x01 0xEA`. ### Inlined symbolic tokens In binary Ion 1.0, symbol values, field names, and annotations are required to be encoded using a symbol ID in the local -symbol table. For some use cases (e.g., as write-once, read-maybe logs) this creates a burden on the writer and may not +symbol table. For some use cases (e.g. write-once, read-maybe logs) this creates a burden on the writer and may not actually be efficient for an application. Ion 1.1 introduces optional binary syntax for encoding inline UTF-8 sequences -for these tokens which can allow an encoder to have flexibility in whether and when to add a given symbolic token to the +for these tokens which can allow an encoder to have flexibility in whether and when to add a given text value to the symbol table. Ion text requires no change for this feature as it already had inline symbolic tokens without using the local symbol @@ -118,12 +140,15 @@ symbol IDs and symbolic tokens with inline text. ### Type encoding changes -All Ion types use the new low-level encodings as specified in the previous section. Many of the opcodes used in Ion 1.0 -have been re-organized primarily to make E-expressions compact. +All Ion types use the new low-level encoding primitives described in the previous section. +Ion 1.0's [type descriptors](https://amazon-ion.github.io/ion-docs/docs/binary.html#typed-value-formats) +have been supplanted by Ion 1.1's more general +[opcodes](https://amazon-ion.github.io/ion-docs/docs/binary.html#typed-value-formats), +which have been organized to prioritize the most commonly used encodings and make leveraging macros as inexpensive as possible. -Typed `null` values are now [encoded in two bytes using the `0xEB` opcode]. +Typed `null` values are now [encoded in two bytes using the `0xEB` opcode](binary/values/null.md#nulls). -[Lists](binary/values/list.md) and [S-expressions](binary/values/eexp.md) have two encodings: +[Lists](binary/values/list.md) and [S-expressions](binary/values/sexp.md) have two encodings: a length-prefixed encoding and a new delimited form that ends with the `0xF0` opcode. [Struct](binary/values/struct.md) values have the option of encoding their field names as @@ -157,9 +182,9 @@ A non-biased, arbitrary length timestamp with packed bit encoding is defined for ### Encoding expressions in binary -In binary, [E-expressions](todo.md) are encoded with an opcode that includes the _macro identifier_ or an opcode that +In binary, [E-expressions](binary/e_expressions.md) are encoded with an opcode that includes the _macro identifier_ or an opcode that specifies a `FlexUInt` for the macro identifier. -The identifier is followed by the [encoding of the arguments to the E-expression](binary/e_expressions.md). +The identifier is followed by the [encoding of the arguments to the E-expression](binary/e_expressions.md#e-expression-argument-encoding). The macro's definition statically determines how the arguments are to be laid out. An argument may be a full Ion value with a leading opcode (sometimes called a "tagged" value), or it could be a lower-level encoding (e.g., a fixed width integer or `FlexInt`/`FlexUInt`). @@ -185,6 +210,7 @@ a (:values 1 2 3) b (:none) c a 1 2 3 b c ``` + Within a list or S-expression, the stream becomes additional child elements in the collection. **E-expressions in lists** @@ -230,46 +256,39 @@ field all together). In the following examples, let us define `(:make_struct c 5 } ``` -### Encoding context and modules +### Modules -In Ion 1.0, there is a single _encoding context_ which is the local symbol table. In Ion 1.1, the _encoding context_ -becomes the following: +Ion 1.0 uses symbol tables to group together related text values. +In order to also accommodate macros, Ion 1.1 also introduces _modules_, a named organizational unit that contains: +* **An exported symbol table**, a list of text values used to compactly encode symbol tokens like field names, annotations, and symbol values. +* **An exported macro table**, a list of macro definitions used to compactly encode complete values or partially populated containers. +* **An unexported nested modules map**, a set of unique module names and their associated module definitions. -* The local symbol table which is a list of strings. This is used to encode/decode symbolic tokens. +While Ion 1.0 does not have modules, it is reasonable to think of Ion 1.0's local symbol table as a module that only has symbols, +and whose macro table and nested modules map are permanently empty. -* The local macro table which is a list of macros. This is used to reference macros that can be invoked by -E-expressions. +Modules can be imported from the catalog (they subsume shared symbol tables) or defined locally. -* A mapping of a string name to *module* which is an organizational unit of symbol definitions and macro definitions. - Within the encoding context, this name is unique and used to address a module's contents either as the list of symbols - to install into the local symbol table, the list of macros to install into the local macro table, or to qualify the - name of a macro in a text E-expression or the definition of a macro. +### Directives -The *module* is a new concept in Ion 1.1. It contains: +_Directives_ modify the encoding context. +Syntactically, a directive is a top-level s-expression annotated with `$ion`. +Its first child value is an operation name. +The operation determines what changes will be made to the encoding context and which clauses may legally follow. -* A list of strings representing the symbol table of the module. - -* A list of macro definitions. - -Modules can be imported from the catalog (they subsume shared symbol tables), but can also be defined locally. Modules -are referenced as a group to allocate entries in the local symbol table and local macro table (e.g., the local symbol -table is initially, implicitly allocated with the symbols in the `$ion` module). - -Ion 1.1 introduces a new system value (an _encoding directive_) for the encoding context (see the *_TBD_* section for -details.) - -**Ion encoding directive example** ```ion -$ion_encoding::{ - modules: [ /* module declarations - including imports */ ], - install_symbols: [ /* names of declared modules */ ], - install_macros: [ /* names of declared modules */ ] -} +$ion:: +(operation_name + (clause_1 /*...*/) + (clause_2 /*...*/) + /*...*/ + (clause_N /*...*/)) ``` -
-This is still being actively worked and is provisional. -
+In Ion v1.1, there are three supported directive operations: +1. [`module`](modules/directives.md#module-directives) +2. [`import`](modules/directives.md#import-directives) +3. [`encoding`](modules/directives.md#encoding-directives) ### Macro definitions @@ -320,7 +339,7 @@ and _one or more_. Furthermore the template defines what type of argument can be * ["Tagless" values](todo.md), whose encodings do not begin with an opcode and are therefore both more compact and less flexible (For example: `flex_int`, `int32`, `float16`). * Specific [_macro shaped arguments_](todo.md) to allow for structural composition of macros and efficient encoding in binary. -The [macro definition](defining_macros.md) includes a *template body* that defines how the macro is expanded. In the language, system macros, macros defined in previously defined modules in the encoding context, and macros defined previously in the current module are available to be invoked with `(name ...)` syntax where `name` is +The [macro definition](macros/defining_macros.md) includes a *template body* that defines how the macro is expanded. In the language, system macros, macros defined in previously defined modules in the encoding context, and macros defined previously in the current module are available to be invoked with `(name ...)` syntax where `name` is the macro to be invoked. Certain names in the expression syntax are reserved for special forms (for example, `literal` and `if_none`). When a macro name is shadowed by a special form, or is ambiguous with respect to all macros visible, it can always be qualified with `(':module:name' ...)` syntax where `module` is the name of the module and `name` is the offset or name of the macro. Referring to a previously defined macro name _within_ a module may be diff --git a/_books/ion-1-1/theme/highlight.js b/_books/ion-1-1/theme/highlight.js index d0681f98..840c858b 100644 --- a/_books/ion-1-1/theme/highlight.js +++ b/_books/ion-1-1/theme/highlight.js @@ -1242,7 +1242,7 @@ hljs.registerLanguage("ion", function () { $pattern: /[\w$]+/, // Highlighted keywords/literals/etc can contain a `$` version_marker: "$ion_1_0 $ion_1_1", literal: "true false null +inf -inf nan", - keyword: 'module macro symbol_table macro_table $ion_symbol_table $ion_encoding flex_uint flex_int uint8 uint16 uint32 uint64 flex_int int8 int16 int32 int64 flex_sym', + keyword: 'import export encoding module macro symbol_table macro_table $ion $ion_symbol_table $ion_encoding flex_uint flex_int uint8 uint16 uint32 uint64 flex_int int8 int16 int32 int64 flex_sym', system_macro: "for annotate make_string none values set_macros add_macros set_symbols add_symbols use" }, comment_modes = [n.C_LINE_COMMENT_MODE, n.C_BLOCK_COMMENT_MODE]; @@ -1253,7 +1253,7 @@ hljs.registerLanguage("ion", function () { }; const e_expression_macro_name = { className: "e_expression_macro_name", - begin: /(?<=\(:)(\w+::)?\w+/, + begin: /(?<=\(:)(\w+::)*\w+/, }; const macro_definition_name = { className: "macro_definition_name", @@ -1261,11 +1261,11 @@ hljs.registerLanguage("ion", function () { }; const tdl_macro_invocation_address = { className: "tdl_macro_invocation_address", - begin: /(?<=\(\.)([a-zA-Z_$]+::)?\d+(?=\s|\))/, + begin: /(?<=\(\.)([a-zA-Z_$]+::)*\d+(?=\s|\))/, }; const tdl_macro_invocation_name = { className: "tdl_macro_invocation_name", - begin: /(?<=\(\.)([a-zA-Z_$]+::)?[a-zA-Z]\w*(?=\s|\))/, + begin: /(?<=\(\.)([a-zA-Z_$]+::)*[a-zA-Z]\w*(?=\s|\))/, }; // This is the same regex that highlight.js vends for numbers, but it disallows numbers without a digit