diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml index b66be1d4a..6cd92c94b 100644 --- a/.github/workflows/pytest.yaml +++ b/.github/workflows/pytest.yaml @@ -28,8 +28,7 @@ jobs: - ubuntu-latest - windows-latest python-version: - - "3.7" # Earliest version supported by message_ix - - "3.8" + - "3.8" # Earliest version supported by message_ix - "3.9" - "3.10" - "3.11" # Latest version supported by message_ix diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst index c5cf804ec..891027c96 100644 --- a/RELEASE_NOTES.rst +++ b/RELEASE_NOTES.rst @@ -4,6 +4,9 @@ Next release All changes ----------- +- Drop support for Python 3.7, which `reached end-of-life on 2023-06-27 `__ (:pull:`738`). + :mod:`message_ix` now requires Python version 3.8 or greater. +- New reporting operator :func:`.model_periods` and automatic keys ``y::model`` and ``y0`` (:pull:`738`). - Improve readability of LaTeX equations in docs (:pull:`721`). - Bugfix: :meth:`.Scenario.add_macro` would not correctly handle configuration that mapped a MESSAGE (commodity, level) to MACRO sector when the commodity and sector names were different (:pull:`719`). - Expand :doc:`macro` documentation, particularly code documentation (:issue:`315`, :pull:`719`). diff --git a/doc/conf.py b/doc/conf.py index 8ed8e4926..cbb13575f 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -73,6 +73,8 @@ .. role:: underline +.. role:: py(code) + :language: python """ # noqa: E501 # -- Options for HTML output ---------------------------------------------- diff --git a/doc/reporting.rst b/doc/reporting.rst index 365f7cb54..19d2e35d8 100644 --- a/doc/reporting.rst +++ b/doc/reporting.rst @@ -35,12 +35,11 @@ Each layer of this “stack” builds on the features in the level below: - — These features are accessible through :class:`.Reporter`, which can produce multiple **reports** from one or more Scenarios. -A report is identified by a **key** (usually a string), and may… +A report and the quantities that enter it is identified by a **key**, and may… - perform arbitrarily complex calculations while intelligently handling units; -- read and make use of data that is ‘exogenous’ to (not included in) a - Scenario; -- produce output as Python or R objects (in code), or to files or databases; +- read and make use of data that is ‘exogenous’ to (not included in) a Scenario; +- produce output as Python or R objects (in code), or write to files or databases; - calculate only a requested subset of quantities; and - much, much more! @@ -62,7 +61,7 @@ In :mod:`message_ix.reporting`: - For example, the |MESSAGEix| parameter ``resource_cost``, defined with the dimensions (node `n`, commodity `c`, grade `g`, year `y`) is identified by the key ``resource_cost:n-c-g-y``. When summed across the grade/`g` dimension, it has dimensions `n`, `c`, `y` and is identified by the key ``resource_cost:n-c-y``. - :meth:`.Reporter.from_scenario` automatically sets up keys and tasks (such as ``resource_cost:n-c-g-y``) that simply retrieve raw/unprocessed data from a :class:`~message_ix.Scenario` and return it as a :any:`genno.Quantity`. -- Computations are defined as functions in modules including: +- Operators are defined as functions in modules including: :mod:`message_ix.reporting.computations`, :mod:`ixmp.reporting.computations`, and :mod:`genno.computations`. @@ -114,7 +113,7 @@ The method :meth:`.Reporter.add` can be used to add *arbitrary* Python code that rep.get("custom") In this example, the function ``my_custom_report()`` *could* run to thousands of lines; read to and write from multiple files; invoke other programs or Python scripts; etc. -In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) computations. +In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) operators or functions. See the :mod:`genno` documentation for more. API reference @@ -146,11 +145,14 @@ Their documentation is repeated below for convenience. configure -:meth:`ixmp.Reporter.from_scenario ` automatically adds keys based on the contents of the :class:`.Scenario` argument. +:meth:`ixmp.Reporter.from_scenario ` automatically adds keys based on the contents of the :class:`.Scenario` argument; +that is, every :mod:`ixmp` set, parameter, variable, and equation available in the Scenario. :meth:`message_ix.Reporter.from_scenario <.Reporter.from_scenario>` extends this to add additional keys for derived quantities specific to the MESSAGEix model framework. These include: -- ``out`` - ``output`` × ``ACT``; that is, the product of ``output`` (output efficiency) and ``ACT`` (activity) +.. tip:: Use :meth:`~.Computer.full_key` to retrieve the full-dimensionality :class:`Key` for any of these quantities. + +- ``out`` = ``output`` × ``ACT``; that is, the product of ``output`` (output efficiency) and ``ACT`` (activity) - ``out_hist`` = ``output`` × ``ref_activity`` (historical reference activity) - ``in`` = ``input`` × ``ACT`` - ``in_hist`` = ``input`` × ``ref_activity`` @@ -158,11 +160,11 @@ These include: - ``emi_hist`` = ``emission_factor`` × ``ref_activity`` - ``inv`` = ``inv_cost`` × ``CAP_NEW`` - ``inv_hist`` = ``inv_cost`` × ``ref_new_capacity`` -- ``fom`` = ``fix_cost`` × ``CAP`` +- ``fom`` = ``fix_cost`` × ``CAP``; the name refers to "Fixed Operation and Maintenance costs" - ``fom_hist`` = ``fix_cost`` × ``ref_capacity`` -- ``vom`` = ``var_cost`` × ``ACT`` +- ``vom`` = ``var_cost`` × ``ACT``; "Variable Operation and Maintenance costs" - ``vom_hist`` = ``var_cost`` × ``ref_activity`` -- ``tom`` = ``fom`` + ``vom`` +- ``tom`` = ``fom`` + ``vom``; "Total Operation and Maintenance costs" - ``land_out`` = ``land_output`` × ``LAND`` - ``land_use_qty`` = ``land_use`` × ``LAND`` - ``land_emi`` = ``land_emission`` × ``LAND`` @@ -174,13 +176,14 @@ These include: - ``addon potential`` = ``addon up`` × ``addon ACT``, the maximum potential activity by add-on technology. - ``price emission``, the model variable ``PRICE_EMISSION`` broadcast across emission species (`e`) *and* technologies (`t`) rather than types (`type_emission`, `type_tec`). -.. tip:: Use :meth:`~.Computer.full_key` to retrieve the full-dimensionality :class:`Key` for any of these quantities. - Other added keys include: - :mod:`message_ix` adds the standard short symbols for |MESSAGEix| dimensions (sets) based on :data:`DIMS`. - Each of these is also available in a Reporter: for example ``rep.get("n")`` returns a list with the elements of the |MESSAGEix| set named "node". - These keys can be used as input + Each of these is also available in a Reporter: for example :py:`rep.get("n")` returns a list with the elements of the |MESSAGEix| set named "node"; + :py:`rep.get("t")` returns the elements of the set "technology", and so on. + These keys can be used as input to other computations. +- ``y0`` = the ``firstmodelyear`` or :math:`y_0` (:class:`int`). +- ``y::model`` = only the periods in the `year` set (``y``) that are equal to or greater than ``y0``. .. _default-reports: @@ -213,17 +216,13 @@ These automatic contents are prepared using: .. autosummary:: add - add_file - add_product add_queue add_single - aggregate apply check_keys configure - convert_pyam describe - disaggregate + eval finalize from_scenario full_key @@ -234,6 +233,13 @@ These automatic contents are prepared using: visualize write + .. autosummary:: + add_file + add_product + aggregate + convert_pyam + disaggregate + .. autodata:: DERIVED .. autodata:: DIMS @@ -247,21 +253,22 @@ These automatic contents are prepared using: :members: ComputationError, Key, KeyExistsError, MissingKeyError, Quantity, configure -Computations ------------- +Operators +--------- .. automodule:: message_ix.reporting.computations :members: - :mod:`message_ix.reporting` provides a small number of computations. + :mod:`message_ix.reporting` provides a small number of operators. Two of these (:func:`.plot_cumulative` and :func:`.stacked_bar`) are currently only used in the tutorials to produce simple plots; for more flexible plotting, :mod:`genno.compat.plotnine` is recommended instead. .. autosummary:: as_message_df + model_periods plot_cumulative stacked_bar - Other computations are provided by :mod:`ixmp.reporting`: + Other operators are provided by :mod:`ixmp.reporting`: .. autosummary:: ~ixmp.reporting.computations.data_for_quantity @@ -269,7 +276,8 @@ Computations ~ixmp.reporting.computations.store_ts ~ixmp.reporting.computations.update_scenario - …and by :mod:`genno.computation` and its compatibility modules. See the package documentation for details. + …and by :mod:`genno.computations` and its compatibility modules. + See the package documentation for details. .. autosummary:: ~genno.compat.plotnine.Plot @@ -280,21 +288,26 @@ Computations ~genno.computations.broadcast_map ~genno.computations.combine ~genno.computations.concat - ~genno.computations.disaggregate_shares ~genno.computations.div + ~genno.computations.drop_vars ~genno.computations.group_sum ~genno.computations.index_to ~genno.computations.interpolate ~genno.computations.load_file ~genno.computations.mul ~genno.computations.pow - ~genno.computations.ratio ~genno.computations.relabel ~genno.computations.rename_dims + ~genno.computations.round ~genno.computations.select + ~genno.computations.sub ~genno.computations.sum ~genno.computations.write_report + .. autosummary:: + ~genno.computations.disaggregate_shares + ~genno.computations.product + ~genno.computations.ratio Utilities --------- diff --git a/message_ix/reporting/__init__.py b/message_ix/reporting/__init__.py index eb0d001dc..660bf8e66 100644 --- a/message_ix/reporting/__init__.py +++ b/message_ix/reporting/__init__.py @@ -1,5 +1,6 @@ import logging from functools import lru_cache, partial +from operator import itemgetter from typing import Mapping, Sequence, Tuple, Union, cast import genno @@ -92,6 +93,8 @@ DERIVED = [ # Each entry is ('full key', (computation tuple,)). Full keys are not inferred and # must be given explicitly. + ("y::model", ("model_periods", "y", "cat_year")), + ("y0", (itemgetter(0), "y::model")), ("tom:nl-t-yv-ya", (genno.computations.add, "fom:nl-t-yv-ya", "vom:nl-t-yv-ya")), # Broadcast from type_addon to technology_addon ( @@ -196,7 +199,7 @@ def put(*args, **kwargs): # Callback function to further collapse other columns into IAMC columns collapse_cb = partial(collapse_message_cols, **collapse_kw) - put("convert_pyam", qty, "pyam", rename=rename, collapse=collapse_cb) + put("as_pyam", qty, "pyam", rename=rename, collapse=collapse_cb) # Standard reports for group, pyam_keys in REPORTS.items(): diff --git a/message_ix/reporting/computations.py b/message_ix/reporting/computations.py index 8f45b3e12..59458ba32 100644 --- a/message_ix/reporting/computations.py +++ b/message_ix/reporting/computations.py @@ -1,4 +1,4 @@ -from typing import Mapping, Union +from typing import List, Mapping, Union import pandas as pd from ixmp.reporting import Quantity @@ -7,6 +7,7 @@ __all__ = [ "as_message_df", + "model_periods", "plot_cumulative", "stacked_bar", ] @@ -57,6 +58,17 @@ def as_message_df( return {name: df} if wrap else df +def model_periods(y: List[int], cat_year: pd.DataFrame) -> List[int]: + """Return the elements of `y` beyond the firstmodelyear of `cat_year`.""" + return list( + filter( + lambda year: cat_year.query("type_year == 'firstmodelyear'")["year"].item() + <= year, + y, + ) + ) + + def plot_cumulative(x, y, labels): """Plot a supply curve. diff --git a/message_ix/tests/test_reporting.py b/message_ix/tests/test_reporting.py index b157bac6a..03b114847 100644 --- a/message_ix/tests/test_reporting.py +++ b/message_ix/tests/test_reporting.py @@ -5,6 +5,7 @@ import pandas as pd import pyam +import pytest import xarray as xr from genno import Quantity from genno.testing import assert_qty_equal @@ -73,7 +74,7 @@ def test_reporter_from_scenario(message_test_mp): # message_ix.Reporter pre-populated with additional, derived quantities # This is the same value as in test_tutorials.py - assert len(rep.graph) == 13074 + assert len(rep.graph) == 13076 # Derived quantities have expected dimensions vom_key = rep.full_key("vom") @@ -145,7 +146,7 @@ def test_reporter_from_westeros(test_mp): assert_allclose(obs, exp) -def test_reporter_convert_pyam(caplog, tmp_path, dantzig_reporter): +def test_reporter_as_pyam(caplog, tmp_path, dantzig_reporter): caplog.set_level(logging.INFO) rep = dantzig_reporter @@ -181,7 +182,9 @@ def add_tm(df, name="Activity"): return df.drop(["t", "m"], axis=1) # Use the convenience function to add the node - key2 = rep.convert_pyam(ACT, "iamc", rename=rename, collapse=add_tm) + with pytest.warns(DeprecationWarning): + key2 = rep.convert_pyam(ACT, "iamc", rename=rename, collapse=add_tm) + key2 = rep.add("as_pyam", ACT, "iamc", rename=rename, collapse=add_tm) # Keys of added node(s) are returned assert ACT.name + "::iamc" == key2 @@ -223,8 +226,16 @@ def add_tm(df, name="Activity"): # Use a name map to replace variable names replacements = {re.escape("Activity|canning_plant|production"): "Foo"} - key3 = rep.convert_pyam( - ACT, rename=rename, replace=dict(variable=replacements), collapse=add_tm + with pytest.warns(DeprecationWarning): + key3 = rep.convert_pyam( + ACT, rename=rename, replace=dict(variable=replacements), collapse=add_tm + ) + key3 = rep.add( + "as_pyam", + ACT, + rename=rename, + replace=dict(variable=replacements), + collapse=add_tm, ) df3 = rep.get(key3).as_pandas() @@ -236,16 +247,24 @@ def add_tm(df, name="Activity"): # Now convert variable cost cb = partial(add_tm, name="Variable cost") - key4 = rep.convert_pyam("var_cost", rename=rename, collapse=cb) + with pytest.warns(DeprecationWarning): + key4 = rep.convert_pyam("var_cost", rename=rename, collapse=cb) + key4 = rep.add("as_pyam", "var_cost", rename=rename, collapse=cb) + df4 = rep.get(key4).as_pandas().drop(["model", "scenario"], axis=1) # Results have the expected units assert all(df4["unit"] == "USD / case") # Also change units - key5 = rep.convert_pyam( - "var_cost", rename=rename, collapse=cb, unit="centiUSD / case" + with pytest.warns(DeprecationWarning): + key5 = rep.convert_pyam( + "var_cost", rename=rename, collapse=cb, unit="centiUSD / case" + ) + key5 = rep.add( + "as_pyam", "var_cost", rename=rename, collapse=cb, unit="centiUSD / case" ) + df5 = rep.get(key5).as_pandas().drop(["model", "scenario"], axis=1) # Results have the expected units diff --git a/message_ix/tests/test_tutorials.py b/message_ix/tests/test_tutorials.py index 29ebb0fb9..385c47df2 100644 --- a/message_ix/tests/test_tutorials.py +++ b/message_ix/tests/test_tutorials.py @@ -22,10 +22,6 @@ condition=GHA and sys.platform == "darwin", reason="Always fails on GitHub Action macOS runners", ), - pytest.mark.xfail( # 2 - condition=sys.version_info.minor >= 11, - reason="pyam-iamc does not support Python 3.11; see IAMconsortium/pyam#744", - ), ] # Affects all tests in the file @@ -87,7 +83,7 @@ def _t(group: Union[str, None], basename: str, *, check=None, marks=None): _t("w0", f"{W}_historical_new_capacity"), _t("w0", f"{W}_multinode"), # NB this is the same value as in test_reporter() - _t(None, f"{W}_report", check=[("len-rep-graph", 13074)], marks=MARK[2]), + _t(None, f"{W}_report", check=[("len-rep-graph", 13076)]), _t("at0", "austria", check=[("solve-objective-value", 206321.90625)]), _t("at0", "austria_single_policy", check=[("solve-objective-value", 205310.34375)]), _t("at0", "austria_multiple_policies"), diff --git a/pyproject.toml b/pyproject.toml index 636658fc6..dbd4ce760 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,6 @@ classifiers = [ "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", @@ -29,11 +28,11 @@ classifiers = [ "Topic :: Scientific/Engineering", "Topic :: Scientific/Engineering :: Information Analysis", ] +requires-python = ">=3.8" dependencies = [ "click", - 'importlib_metadata; python_version < "3.8"', "ixmp >= 3.7.0", - "genno[pyam]", + "genno[pyam] >= 1.18.1", "numpy", "pandas >= 1.2", "PyYAML", diff --git a/tutorial/westeros/westeros_report.ipynb b/tutorial/westeros/westeros_report.ipynb index 5a91f1c9f..9bd3379f1 100644 --- a/tutorial/westeros/westeros_report.ipynb +++ b/tutorial/westeros/westeros_report.ipynb @@ -475,7 +475,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Almost 12,700 nodes!\n", + "Over 13,000 nodes!\n", "\n", "Remember: `rep` simply *describes* these operations; none of them is executed until or unless you `get()` them.\n", "\n", @@ -535,7 +535,7 @@ "- This node returns the same Scenario object we passed to `Reporter.from_scenario()`.\n", "\n", "In short, if we run this cell, the Reporter will extract a 6-dimensional quantity from the Scenario object and return it.\n", - "The other >12,000 nodes will not be computed.\n", + "The other >13,000 nodes will not be computed.\n", "\n", "Let's try:" ] @@ -703,9 +703,10 @@ "\n", "\n", "# Add node(s) that convert data to pyam.IamDataFrame objects\n", - "new_key = rep.convert_pyam(\n", + "new_key = rep.add(\n", + " \"as_pyam\",\n", " # Quantity or quantities to convert\n", - " quantities=out.drop(\"h\", \"hd\", \"m\", \"nd\", \"yv\"),\n", + " out.drop(\"h\", \"hd\", \"m\", \"nd\", \"yv\"),\n", " # Dimensions to use for the \"region\" and \"year\" IAMC columns\n", " rename=dict(nl=\"region\", ya=\"year\"),\n", " # Use this function to collapse the \"l\", \"t\", and \"c\" dimensions\n",