Release hydroflow_lang v0.10.0, hydroflow_datalog_core v0.10.0, hydro…

…flow_datalog v0.10.0, hydroflow_deploy_integration v0.10.0, hydroflow_macro v0.10.0, lattices_macro v0.5.7, variadics v0.0.7, variadics_macro v0.5.5, lattices v0.5.8, multiplatform_test v0.3.0, pusherator v0.0.9, hydroflow v0.10.0, hydro_deploy v0.10.0, stageleft_macro v0.4.0, stageleft v0.5.0, stageleft_tool v0.4.0, hydroflow_plus v0.10.0, hydro_cli v0.10.0, safety bump 8 crates SAFETY BUMP: hydroflow_datalog_core v0.10.0, hydroflow_datalog v0.10.0, hydroflow_macro v0.10.0, hydroflow v0.10.0, hydro_deploy v0.10.0, stageleft v0.5.0, hydroflow_plus v0.10.0, hydro_cli v0.10.0
hydro-project · Nov 8, 2024 · dcd48fc · dcd48fc
1 parent 8d8b4b2
commit dcd48fc
Show file tree

Hide file tree

Showing 43 changed files with 1,405 additions and 130 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/datastores/gossip_kv/Cargo.toml b/datastores/gossip_kv/Cargo.toml
@@ -42,4 +42,4 @@ path = "cli/main.rs"
 
 [lib]
 name = "gossip_kv"
-path = "kv/lib.rs"
+path = "kv/lib.rs"
diff --git a/hydro_deploy/core/CHANGELOG.md b/hydro_deploy/core/CHANGELOG.md
@@ -5,8 +5,82 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v0.10.0 (2024-11-08)
+
+### Chore
+
+ - <csr-id-d5677604e93c07a5392f4229af94a0b736eca382/> update pinned rust version, clippy lints, remove some dead code
+
+### New Features
+
+ - <csr-id-afe78c343658472513b34d28658634b253148aee/> add ability to have staged flows inside unit tests
+   Whenever a Hydroflow+ program is compiled, it depends on a generated
+   `__staged` module, which contains the entire contents of the crate but
+   with every type / function made `pub` and exported, so that the compiled
+   UDFs can resolve local references appropriately.
+
+   Previously, we would not do this for `#[cfg(test)]` modules, since they
+   may use `dev-dependencies` and therefore the generated module may fail
+   to compile when not in test mode. To solve this, when running a unit
+   test (marked with `hydroflow_plus::deploy::init_test()`) that uses
+   trybuild, we emit a version of the `__staged` module with `#[cfg(test)]`
+   modules included _into the generated trybuild sources_ because we can
+   guarantee via trybuild that the appropriate `dev-dependencies` are
+   available.
+
+   This by itself allows crates depending on `hydroflow_plus` to have local
+   unit tests with Hydroflow+ logic inside them. But we also want to use
+   this support for unit tests inside `hydroflow_plus` itself. To enable
+   that, we eliminate the `hydroflow_plus_deploy` crate and move its
+   contents directly to `hydroflow_plus` itself so that we can access the
+   trybuild machinery without incurring a circular dependency.
+
+   Also fixes #1408
+ - <csr-id-8a809315cd37929687fcabc34a12042db25d5767/> add API for external network inputs
+   This is a key step towards being able to unit-test HF+ graphs, by being
+   able to have controlled inputs. Outputs next.
+
+### Style
+
+ - <csr-id-8442d1b524621a9f8b43372a9c25991efb33c25e/> fixes for latest nightly clippy
+
+### Commit Statistics
+
+<csr-read-only-do-not-edit/>
+
+ - 4 commits contributed to the release.
+ - 69 days passed between releases.
+ - 4 commits were understood as [conventional](https://www.conventionalcommits.org).
+ - 4 unique issues were worked on: [#1444](https://github.com/hydro-project/hydroflow/issues/1444), [#1449](https://github.com/hydro-project/hydroflow/issues/1449), [#1450](https://github.com/hydro-project/hydroflow/issues/1450), [#1537](https://github.com/hydro-project/hydroflow/issues/1537)
+
+### Commit Details
+
+<csr-read-only-do-not-edit/>
+
+<details><summary>view details</summary>
+
+ * **[#1444](https://github.com/hydro-project/hydroflow/issues/1444)**
+    - Update pinned rust version, clippy lints, remove some dead code ([`d567760`](https://github.com/hydro-project/hydroflow/commit/d5677604e93c07a5392f4229af94a0b736eca382))
+ * **[#1449](https://github.com/hydro-project/hydroflow/issues/1449)**
+    - Add API for external network inputs ([`8a80931`](https://github.com/hydro-project/hydroflow/commit/8a809315cd37929687fcabc34a12042db25d5767))
+ * **[#1450](https://github.com/hydro-project/hydroflow/issues/1450)**
+    - Add ability to have staged flows inside unit tests ([`afe78c3`](https://github.com/hydro-project/hydroflow/commit/afe78c343658472513b34d28658634b253148aee))
+ * **[#1537](https://github.com/hydro-project/hydroflow/issues/1537)**
+    - Fixes for latest nightly clippy ([`8442d1b`](https://github.com/hydro-project/hydroflow/commit/8442d1b524621a9f8b43372a9c25991efb33c25e))
+</details>
+
 ## v0.9.0 (2024-08-30)
 
+<csr-id-a2ec110ccadb97e293b19d83a155d98d94224bba/>
+<csr-id-11af32828bab6e4a4264d2635ff71a12bb0bb778/>
+<csr-id-a88a550cefde3a56790859127edc6a4e27e07090/>
+<csr-id-77246e77df47a0006dcb3eaeeb76882efacfd25c/>
+<csr-id-3fde68d0db0414017cfb771a218b14b8f57d1686/>
+<csr-id-0a465e55dd39c76bc1aefb020460a639d792fe87/>
+<csr-id-bb081d3b0af6dbce9630e23dfe8b7d1363751c2b/>
+<csr-id-a2147864b24110c9ae2c1553e9e8b55bd5065f15/>
+<csr-id-8856c8596d5ad9d5f24a46467690bfac1549fae2/>
+
 ### Chore
 
  - <csr-id-a2ec110ccadb97e293b19d83a155d98d94224bba/> manually set versions for crates renamed in #1413
@@ -15,6 +89,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
    ---------
 
+### Refactor (BREAKING)
+
+ - <csr-id-128aaecd40edce57dc254afdcd61ecd5b9948d71/> simplify process/cluster specs
+   ---
+   [//]: # (BEGIN SAPLING FOOTER)
+   Stack created with [Sapling](https://sapling-scm.com). Best reviewed
+   with
+   [ReviewStack](https://reviewstack.dev/hydro-project/hydroflow/pull/1394).
+   * #1395
+   * __->__ #1394
+
 ### Documentation
 
  - <csr-id-f5f1eb0c612f5c0c1752360d972ef6853c5e12f0/> cleanup doc comments for clippy latest
@@ -79,11 +164,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  - <csr-id-8856c8596d5ad9d5f24a46467690bfac1549fae2/> use `buildstructor` to handle excessive `Deployment` method arguments, fix #1364
    Adds new method `Deployment::AzureHost`
 
+<csr-id-128aaecd40edce57dc254afdcd61ecd5b9948d71/> simplify process/cluster specs
+   ---
+   [//]: # (BEGIN SAPLING FOOTER)
+   Stack created with [Sapling](https://sapling-scm.com). Best reviewed
+   with
+   [ReviewStack](https://reviewstack.dev/hydro-project/hydroflow/pull/1394).
+   * #1395
+   * __->__ #1394
+ - <csr-id-bb081d3b0af6dbce9630e23dfe8b7d1363751c2b/> end-to-end flamegraph generation, fix #1365
+   Depends on #1370
+ - <csr-id-a2147864b24110c9ae2c1553e9e8b55bd5065f15/> `Deployment.stop()` for graceful shutdown including updated `perf` profile downloading
+   * `perf` profile downloading moved from the `drop()` impl to `async fn
+   stop()`
+   * download perf data via stdout
+   * update async-ssh2-lite to 0.5 to cleanup tokio compat issues
+
+   WIP for #1365
+ - <csr-id-8856c8596d5ad9d5f24a46467690bfac1549fae2/> use `buildstructor` to handle excessive `Deployment` method arguments, fix #1364
+   Adds new method `Deployment::AzureHost`
+
 ### Commit Statistics
 
 <csr-read-only-do-not-edit/>
 
- - 19 commits contributed to the release.
+ - 20 commits contributed to the release.
+ - 38 days passed between releases.
  - 18 commits were understood as [conventional](https://www.conventionalcommits.org).
  - 17 unique issues were worked on: [#1313](https://github.com/hydro-project/hydroflow/issues/1313), [#1360](https://github.com/hydro-project/hydroflow/issues/1360), [#1366](https://github.com/hydro-project/hydroflow/issues/1366), [#1369](https://github.com/hydro-project/hydroflow/issues/1369), [#1370](https://github.com/hydro-project/hydroflow/issues/1370), [#1372](https://github.com/hydro-project/hydroflow/issues/1372), [#1378](https://github.com/hydro-project/hydroflow/issues/1378), [#1394](https://github.com/hydro-project/hydroflow/issues/1394), [#1396](https://github.com/hydro-project/hydroflow/issues/1396), [#1398](https://github.com/hydro-project/hydroflow/issues/1398), [#1403](https://github.com/hydro-project/hydroflow/issues/1403), [#1411](https://github.com/hydro-project/hydroflow/issues/1411), [#1413](https://github.com/hydro-project/hydroflow/issues/1413), [#1423](https://github.com/hydro-project/hydroflow/issues/1423), [#1428](https://github.com/hydro-project/hydroflow/issues/1428), [#1429](https://github.com/hydro-project/hydroflow/issues/1429), [#1431](https://github.com/hydro-project/hydroflow/issues/1431)
 
@@ -129,6 +235,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  * **[#1431](https://github.com/hydro-project/hydroflow/issues/1431)**
     - Only record usermode events in perf ([`c4683ca`](https://github.com/hydro-project/hydroflow/commit/c4683caca43f2927694c920b43ef35a6d1629eaa))
  * **Uncategorized**
+    - Release hydroflow_lang v0.9.0, hydroflow_datalog_core v0.9.0, hydroflow_datalog v0.9.0, hydroflow_deploy_integration v0.9.0, hydroflow_macro v0.9.0, lattices_macro v0.5.6, lattices v0.5.7, multiplatform_test v0.2.0, variadics v0.0.6, pusherator v0.0.8, hydroflow v0.9.0, stageleft_macro v0.3.0, stageleft v0.4.0, stageleft_tool v0.3.0, hydroflow_plus v0.9.0, hydro_deploy v0.9.0, hydro_cli v0.9.0, hydroflow_plus_deploy v0.9.0, safety bump 8 crates ([`0750117`](https://github.com/hydro-project/hydroflow/commit/0750117de7088c01a439b102adeb4c832889f171))
     - Manually set versions for crates renamed in #1413 ([`a2ec110`](https://github.com/hydro-project/hydroflow/commit/a2ec110ccadb97e293b19d83a155d98d94224bba))
 </details>
 
@@ -196,6 +303,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 <csr-read-only-do-not-edit/>
 
  - 11 commits contributed to the release.
+ - 59 days passed between releases.
  - 10 commits were understood as [conventional](https://www.conventionalcommits.org).
  - 10 unique issues were worked on: [#1334](https://github.com/hydro-project/hydroflow/issues/1334), [#1338](https://github.com/hydro-project/hydroflow/issues/1338), [#1339](https://github.com/hydro-project/hydroflow/issues/1339), [#1340](https://github.com/hydro-project/hydroflow/issues/1340), [#1343](https://github.com/hydro-project/hydroflow/issues/1343), [#1345](https://github.com/hydro-project/hydroflow/issues/1345), [#1346](https://github.com/hydro-project/hydroflow/issues/1346), [#1347](https://github.com/hydro-project/hydroflow/issues/1347), [#1348](https://github.com/hydro-project/hydroflow/issues/1348), [#1356](https://github.com/hydro-project/hydroflow/issues/1356)
 
@@ -245,6 +353,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 <csr-read-only-do-not-edit/>
 
  - 3 commits contributed to the release.
+ - 44 days passed between releases.
  - 2 commits were understood as [conventional](https://www.conventionalcommits.org).
  - 2 unique issues were worked on: [#1129](https://github.com/hydro-project/hydroflow/issues/1129), [#1157](https://github.com/hydro-project/hydroflow/issues/1157)
 
@@ -275,6 +384,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 <csr-read-only-do-not-edit/>
 
  - 4 commits contributed to the release.
+ - 38 days passed between releases.
  - 1 commit was understood as [conventional](https://www.conventionalcommits.org).
  - 1 unique issue was worked on: [#1090](https://github.com/hydro-project/hydroflow/issues/1090)
 
@@ -316,6 +426,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 <csr-read-only-do-not-edit/>
 
  - 4 commits contributed to the release.
+ - 32 days passed between releases.
  - 3 commits were understood as [conventional](https://www.conventionalcommits.org).
  - 3 unique issues were worked on: [#1015](https://github.com/hydro-project/hydroflow/issues/1015), [#1043](https://github.com/hydro-project/hydroflow/issues/1043), [#1084](https://github.com/hydro-project/hydroflow/issues/1084)
 

diff --git a/hydro_deploy/core/Cargo.toml b/hydro_deploy/core/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "hydro_deploy"
 publish = true
-version = "0.9.0"
+version = "0.10.0"
 edition = "2021"
 license = "Apache-2.0"
 documentation = "https://docs.rs/hydro_deploy/"
@@ -22,7 +22,7 @@ cargo_metadata = "0.18.0"
 dunce = "1.0.0"
 dyn-clone = "1.0.0"
 futures = "0.3.0"
-hydroflow_deploy_integration = { path = "../hydroflow_deploy_integration", version = "^0.9.0" }
+hydroflow_deploy_integration = { path = "../hydroflow_deploy_integration", version = "^0.10.0" }
 indicatif = "0.17.0"
 inferno = "0.11.0"
 itertools = "0.10.0" # TODO(mingwei): remove when `iter_intersperse` is stabilized.