Merge branch 'pypsa-meets-earth:main' into main

pypsa-meets-earth · Jan 16, 2025 · 2255a78 · 2255a78
2 parents adfaec0 + e393855
commit 2255a78
Show file tree

Hide file tree

Showing 14 changed files with 240 additions and 116 deletions.
diff --git a/README.md b/README.md
@@ -58,41 +58,22 @@ The diagram below depicts one representative clustered node for the sector-coupl
     <img src="https://api.star-history.com/svg?repos=pypsa-meets-earth/pypsa-earth,OSeMOSYS/osemosys_global,niclasmattsson/Supergrid,SGIModel/MUSE_OS,etsap-TIMES/TIMES_model&type=Date" width="60%">
 <a/>
 
-## Get involved
-
-There are multiple ways to get involved and learn more about our work. That's how we organise ourselves:
-
-- [**Discord NEW! (Open)**](https://discord.gg/AnuJBk23FU)
-  - chat with the community, team up on features, exchange with developers, code in voice channels
-  - registration and usage is for free
-        <p align="left">
-            <a href="https://discord.gg/AnuJBk23FU">
-            <img src="https://discord.com/assets/cb48d2a8d4991281d7a6a95d2f58195e.svg" width="150">
-            <a/>
-        </p>
-- **General initiative meeting (Open)**
-  - every forth Thursday each month Thursday 16-17:00 (UK time)
-    <a href="https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link">
-    `download .ics`
-    </a>
-  - join for project news and high-level code updates
-  - meeting hosted on Discord
-  - [open agenda](https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing). See what we will discuss. Invited members have edit rights.
-- **Buddy talk (Open)**
-  - book a 30min meeting with Max to discuss anything you like
-  - booking link: [calendly.com/pypsa-meets-earth](https://calendly.com/max-parzen/pypsa-meets-earth-exchange-30min)
-- **Specific code meeting (Open)**
-  - meeting hosted on Discord
-  - join updates, demos, Q&A's, discussions and the coordination of each work package
-    1. Demand creation and prediction meeting, on demand
-    2. AI asset detection meeting, on demand
-    3. Sector coupling meeting, every Thursday 09:00 (UK time), <a href="https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link" >`download .ics`</a>
-    4. PyPSA-Earth meeting, every Thursday 16:00 (UK time), <a href="https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link" >`download .ics`</a>
-- **Outreach meeting (Open)**
-  - every second week, Tuesday 17:00 (UK time)
-  - planning, discussing events, workshops, communication, community activities
-- [**Google Drive**](https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing)
-  - access to minutes, presentations, lists, documents (access to minutes)
+## How to get involved
+
+There are multiple ways to get involved and learn more about our work:
+1. **Join our forum** and communication platform on [**PyPSA-meets-Earth**](https://discord.gg/AnuJBk23FU) Discord Server
+2. **Chat on Discord with us** in the following open meetings:
+    - **General initiative meeting** for project news and [high-level code updates](https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing). Held every [fourth Thursday 16-17:00 (UK time)](https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link) and is a perfect place to meet the community and get a high-level update on PyPSA ecosystem relevant for PyPSA-Earth developments.
+    - **Weekly developers meetings**
+        - Eastern-Hemisphere friendly *Morning meeting* every [Thursday at 09:00 (UK time)](https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link).
+        - Western-Hemisphere friendly *Evening meeting* every [Thursday 16:00 (UK time)](https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link). Every forth Thursday is replaced by the General initiative meeting which has a more high-level perspective, but you can also join to discuss more particular questions.
+3. **Look at public materials** at [**google Drive**](https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing) to share to minutes, presentations, lists and documents. Feel gree to get a look!
+4. **Notify your interest** to on-demand meetings:
+    - On-demand meetings
+        - Demand creation and prediction meeting
+        - AI asset detection meeting
+        - Outreach meeting for planning, discussing events, workshops, communication, community activities
+5. Join us and **propose your stream**.
 
 ## Installation
 

diff --git a/Snakefile b/Snakefile
@@ -740,7 +740,10 @@ if config["augmented_line_connection"].get("add_to_snakefile", False) == False:
 
 
 rule add_extra_components:
+    params:
+        transmission_efficiency=config["sector"]["transmission_efficiency"],
     input:
+        overrides="data/override_component_attrs",
         network="networks/" + RDIR + "elec_s{simpl}_{clusters}.nc",
         tech_costs=COSTS,
     output:
@@ -814,6 +817,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == False:
             solving=config["solving"],
             augmented_line_connection=config["augmented_line_connection"],
         input:
+            overrides="data/override_component_attrs",
             network="networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
         output:
             "results/" + RDIR + "networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
@@ -880,6 +884,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == True:
             solving=config["solving"],
             augmented_line_connection=config["augmented_line_connection"],
         input:
+            overrides="data/override_component_attrs",
             network="networks/"
             + RDIR
             + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{unc}.nc",
@@ -909,7 +914,7 @@ if config["monte_carlo"]["options"].get("add_to_snakefile", False) == True:
         resources:
             mem_mb=memory,
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         script:
             "scripts/solve_network.py"
 
@@ -1149,7 +1154,7 @@ rule build_ship_profile:
 
 rule add_export:
     params:
-        gadm_level=config["sector"]["gadm_level"],
+        gadm_layer_id=config["build_shape_options"]["gadm_layer_id"],
         alternative_clustering=config["cluster_options"]["alternative_clustering"],
         store=config["export"]["store"],
         store_capital_costs=config["export"]["store_capital_costs"],
@@ -1634,7 +1639,7 @@ if config["foresight"] == "overnight":
             RESDIR
             + "postnetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.nc",
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         log:
             solver=RESDIR
             + "logs/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_solver.log",
@@ -1832,7 +1837,7 @@ rule prepare_db:
 rule build_industrial_distribution_key:  #default data
     params:
         countries=config["countries"],
-        gadm_level=config["sector"]["gadm_level"],
+        gadm_layer_id=config["build_shape_options"]["gadm_layer_id"],
         alternative_clustering=config["cluster_options"]["alternative_clustering"],
         industry_database=config["custom_data"]["industry_database"],
     input:
@@ -2100,7 +2105,7 @@ if config["foresight"] == "myopic":
             # config=RESDIR
             # + "configs/config.elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.yaml",
         shadow:
-            "shallow"
+            "copy-minimal" if os.name == "nt" else "shallow"
         log:
             solver=RESDIR
             + "logs/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_solver.log",

diff --git a/config.default.yaml b/config.default.yaml
@@ -602,6 +602,9 @@ sector:
   transmission_efficiency:
     electricity distribution grid:
       efficiency_static: 0.97 # efficiency of distribution grid (i.e. 3% loses)
+    H2 pipeline:
+      efficiency_per_1000km: 1
+      compression_per_1000km: 0.017 # DEA technology data. Mean of  Energy losses, lines 5000-20000 MW and lines >20000 MW for 2020, 2030 and 2050, [%/1000 km]
 
   dynamic_transport:
     enable: false # If "True", then the BEV and FCEV shares are obtained depending on the "Co2L"-wildcard (e.g. "Co2L0.70: 0.10"). If "False", then the shares are obtained depending on the "demand" wildcard and "planning_horizons" wildcard as listed below (e.g. "DF_2050: 0.08")
@@ -657,7 +660,6 @@ sector:
   co2_network: true
   co2_sequestration_potential: 200 #MtCO2/a sequestration potential for Europe
   co2_sequestration_cost: 10 #EUR/tCO2 for sequestration of CO2
-  hydrogen_underground_storage: true
   shipping_hydrogen_liquefaction: false
   shipping_average_efficiency: 0.4 #For conversion of fuel oil to propulsion in 2011
 
@@ -672,7 +674,6 @@ sector:
     NZ_2050: 0.36
     DF_2050: 0.12
 
-  gadm_level: 1
   h2_cavern: true
   marginal_cost_storage: 0
   methanation: true
@@ -695,7 +696,6 @@ sector:
     biomass: biomass
   keep_existing_capacities: true
 
-
 solving:
   options:
     formulation: kirchhoff

diff --git a/doc/index.rst b/doc/index.rst
@@ -78,39 +78,19 @@ The `website <https://pypsa-meets-earth.github.io/>`_ provides more context of t
 Get Involved
 ==============
 
-The PyPSA meets Earth team is currently running four types of meetings:
-
-
-- `**Discord NEW! (Open)** <https://discord.gg/AnuJBk23FU>`_
-
-  - Chat with the community, team up on features, exchange with developers, code in voice channels
-
-- **General code meeting (Open)**
-
-  - every forth Thursday each month 16-17:00 (UK time) `download .ics file <https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link>`_
-  - updates on overall project and code blocks
-  - meeting hosted on `Discord <https://discord.gg/AnuJBk23FU>`_; join us, we are waiting for you!
-  - `open agenda <https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing>`_. See what we will discuss. Invited members have edit rights.
-
-- **Specific code meeting (by invitation)**
-
-  - meeting hosted on Discord
-  - join updates, demos, Q&A's, discussions and the coordination of each work package
-
-    1. Demand creation and prediction meeting, on demand basis
-    2. AI asset detection meeting, on demand basis
-    3. Sector coupling meeting, every Thursday 09:00 (UK time), `download .ics file <https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link>`__
-    4. PyPSA-Earth meeting, every Thursday 16:00 (UK time), `download .ics file <https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link>`__
-
-- **Outreach meeting (by invitation)**
-
-  - every second week
-  - planning, discussing events, workshops, communication, community activities
-
-- **Buddy talk (Open)**
-
-  - book a 30min meeting with Max to discuss anything you like
-  - booking link: `calendly.com/pypsa-meets-earth <https://calendly.com/max-parzen/pypsa-meets-earth-exchange-30min>`_
+There are multiple ways to get involved and learn more about our work:
+    1. **Join our forum** and communication platform on `PyPSA-meets-Earth <https://discord.gg/AnuJBk23FU>`__ Discord Server
+    2. **Chat on Discord with us** in the following meetings:
+        - General initiative meeting for project news and `high-level code updates <https://docs.google.com/document/d/1r6wm2RBe0DWFngmItpFfSFHA-CnUmVcVTkIKmthdW3g/edit?usp=sharing>`__. Held every `fourth Thursday 16-17:00 (UK time) <https://drive.google.com/file/d/1naH4WwW9drkOkOJ3PLO4fyWdkZQi5-_w/view?usp=share_link>`__ and is a perfect place to meet the community and get a high-level update on PyPSA ecosystem relevant for PyPSA-Earth developments.
+        - Weekly developers meetings
+            - Eastern-Hemisphere friendly *Morning meeting* every `Thursday at 09:00 (UK time) <https://drive.google.com/file/d/1PDdmjsKhzyGRo0_YrP4wPQkn2XTNh6jA/view?usp=share_link>`__.
+            - Western-Hemisphere friendly *Evening meeting* every `Thursday 16:00 (UK time) <https://drive.google.com/file/d/1gaLmyV4qGPXsogkeRcAPWjC0ESebUxU-/view?usp=share_link>`__. Every forth Thursday is replaced by the General initiative meeting which has a more high-level perspective, but you can also join to discuss more particular questions.
+    3. **Look at public materials** at `google Drive <https://drive.google.com/drive/folders/13Z8Y9zgsh5IZaDNkkRyo1wkoMgbdUxT5?usp=sharing>`__ to share to minutes, presentations, lists and documents. Feel gree to get a look!
+    4. **Notify your interest** to on-demand meetings:
+        - Demand creation and prediction meeting
+        - AI asset detection meeting
+        - Outreach meeting for planning, discussing events, workshops, communication, community activities
+    5. Join us and **propose your stream**.
 
 =============
 Documentation

diff --git a/doc/release_notes.rst b/doc/release_notes.rst
@@ -17,6 +17,8 @@ This part of documentation collects descriptive release notes to capture the mai
 
 * In alternative clustering, generate hydro inflows by shape and avoid hydro inflows duplication for plants installed in the same node `PR #1120 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1120>`
 
+* Add a function to calculate length-based efficiencies and apply it to the H2 pipelines. `PR #1192 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1192>`__
+
 **Minor Changes and bug-fixing**
 
 * Prevent computation of powerplantmatching if replace option is selected for custom_powerplants `PR #1281 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1281>`__
@@ -25,6 +27,10 @@ This part of documentation collects descriptive release notes to capture the mai
 
 * Fix readthedocs by explicitly specifying the location of the Sphinx config `PR #1292 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1292>`__
 
+* Fix lossy bidirectional links, especially H2 pipelines, which would sometimes gain H2 instead of losing it.  `PR #1192 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1192>`__
+
+* Fix the need for administrative rights on Windows by changing all shadow directory settings for Windows in the Snakefile `PR #1295 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1295>`__
+
 PyPSA-Earth 0.6.0
 =================
 

diff --git a/scripts/_helpers.py b/scripts/_helpers.py
@@ -1273,7 +1273,7 @@ def get_GADM_layer(country_list, layer_id, update=False, outlogging=False):
 def locate_bus(
     coords,
     co,
-    gadm_level,
+    gadm_layer_id,
     path_to_gadm=None,
     gadm_clustering=False,
     col="name",
@@ -1305,8 +1305,8 @@ def locate_bus(
                         lambda name: three_2_two_digits_country(name[:3]) + name[3:]
                     )
         else:
-            gdf = get_GADM_layer(co, gadm_level)
-            col = "GID_{}".format(gadm_level)
+            gdf = get_GADM_layer(co, gadm_layer_id)
+            col = "GID_{}".format(gadm_layer_id)
 
         # gdf.set_index("GADM_ID", inplace=True)
     gdf_co = gdf[
@@ -1483,3 +1483,91 @@ def safe_divide(numerator, denominator, default_value=np.nan):
             f"Division by zero: {numerator} / {denominator}, returning NaN."
         )
         return np.nan
+
+
+def lossy_bidirectional_links(n, carrier):
+    """
+    Split bidirectional links of type carrier into two unidirectional links to include transmission losses.
+    """
+
+    # identify all links of type carrier
+    carrier_i = n.links.query("carrier == @carrier").index
+
+    if carrier_i.empty:
+        return
+
+    logger.info(f"Splitting bidirectional links with the carrier {carrier}")
+
+    # set original links to be unidirectional
+    n.links.loc[carrier_i, "p_min_pu"] = 0
+
+    # add a new links that mirror the original links, but represent the reversed flow direction
+    # the new links have a cost and length of 0 to not distort the overall cost and network length
+    rev_links = (
+        n.links.loc[carrier_i].copy().rename({"bus0": "bus1", "bus1": "bus0"}, axis=1)
+    )
+    rev_links["length_original"] = rev_links[
+        "length"
+    ]  # tracker for the length of the original links length
+    rev_links["capital_cost"] = 0
+    rev_links["length"] = 0
+    rev_links["reversed"] = True  # tracker for easy identification of reversed links
+    rev_links.index = rev_links.index.map(lambda x: x + "-reversed")
+
+    # add the new reversed links to the network and fill the newly created trackers with default values for the other links
+    n.links = pd.concat([n.links, rev_links], sort=False)
+    n.links["reversed"] = n.links["reversed"].fillna(False).infer_objects(copy=False)
+    n.links["length_original"] = n.links["length_original"].fillna(n.links.length)
+
+
+def set_length_based_efficiency(n, carrier, bus_suffix, transmission_efficiency):
+    """
+    Set the efficiency of all links of type carrier in network n based on their length and the values specified in the config.
+    Additionally add the length based electricity demand required for compression (if applicable).
+    The bus_suffix refers to the suffix that differentiates the links bus0 from the corresponding electricity bus, i.e. " H2".
+    Important:
+    Call this function AFTER lossy_bidirectional_links when creating links that are both bidirectional and lossy,
+    and have a length based electricity demand for compression. Otherwise the compression will not consistently take place at
+    the inflow bus and instead vary between the inflow and the outflow bus.
+    """
+
+    # get the links length based efficiency and required compression
+    if carrier not in transmission_efficiency:
+        raise KeyError(
+            f"An error occurred when setting the length based efficiency for the Links of type {carrier}."
+            f"The Link type {carrier} was not found in the config under config['sector']['transmission_efficiency']."
+        )
+    efficiencies = transmission_efficiency[carrier]
+    efficiency_static = efficiencies.get("efficiency_static", 1)
+    efficiency_per_1000km = efficiencies.get("efficiency_per_1000km", 1)
+    compression_per_1000km = efficiencies.get("compression_per_1000km", 0)
+
+    # indetify all links of type carrier
+    carrier_i = n.links.loc[n.links.carrier == carrier].index
+
+    # identify the lengths of all links of type carrier
+    # use "length_original" for lossy bidirectional links and "length" for any other link
+    if ("reversed" in n.links.columns) and any(n.links.loc[carrier_i, "reversed"]):
+        lengths = n.links.loc[carrier_i, "length_original"]
+    else:
+        lengths = n.links.loc[carrier_i, "length"]
+
+    # set the links' length based efficiency
+    n.links.loc[carrier_i, "efficiency"] = (
+        efficiency_static * efficiency_per_1000km ** (lengths / 1e3)
+    )
+
+    # set the links's electricity demand for compression
+    if compression_per_1000km > 0:
+        # connect the links to their corresponding electricity buses
+        n.links.loc[carrier_i, "bus2"] = n.links.loc[
+            carrier_i, "bus0"
+        ].str.removesuffix(bus_suffix)
+        # TODO: use these lines to set bus 2 instead, once n.buses.location is functional and remove bus_suffix.
+        """
+        n.links.loc[carrier_i, "bus2"] = n.links.loc[carrier_i, "bus0"].map(
+            n.buses.location
+        )  # electricity
+        """
+        # set the required compression demand
+        n.links.loc[carrier_i, "efficiency2"] = -compression_per_1000km * lengths / 1e3