The latest Rust Sitter drops the dependency on `tree-sitter-cli`, which
eliminates many transitive dependencies.

a couple few spurious `too_many_arguments` and a spurious
`zombie_processes` still on current nightly (`clippy 0.1.84 (4392847410
2024-10-21)`)

 #1458 (#1446)

consolidate into one task to prevent races

Generalized Hash Tries are part of the SIGMOD '23 FreeJoin
[paper](https://dl.acm.org/doi/abs/10.1145/3589295) by
Wang/Willsey/Suciu. They provide a compressed ("factorized")
representation of relations. By operating in the factorized domain, join
algorithms can defer cross-products and achieve asymptotically optimal
performance.

---------

Co-authored-by: Mingwei Samuel <[email protected]>
Co-authored-by: Andre Giron <[email protected]>

…module (#1485)

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/hydro-project/hydroflow/pull/1485).
* #1493
* #1492
* #1489
* #1488
* #1487
* #1486
* __->__ #1485

changed --server-addr flag to --address

Co-authored-by: Mingwei Samuel <[email protected]>

…hases (#1486)

git push --set-upstream origin clean

Co-authored-by: “Dee <[email protected]>

…r have to be maintained atomically (#1488)

…KV into separate module (#1516)

…type parameter (#1519)

Now, when the location is a top-level `Process` or `Cluster` that
corresponds to a `NoTick`, and for streams inside a tick we wrap the
location type (e.g. `Tick<Process<...>>`). This simplifies type
signatures for a lot of our example code.

…s in a tick, fix #1518 (#1520)

Adds the minimal reproducer test from @shadaj

Note this may have negative performance implications, as the singleton value now is stored in the state API (heap) instead of locally. If we use singleton syntax this duplicate allocation could probably be avoided.

> Confirmed that this fixed the bugs in our Paxos implementation, no noticeable performance impact. @shadj

…s directly in streams (#1523)

There are some TODOs in the code for which I have created issues for
0.10 milestone.

[P0 - Must Have for
0.10](https://github.com/hydro-project/hydroflow/issues?q=is:open+label:Datastores/Gossip-KV+milestone:0.10+label:P0)
[P1 - Good to have / stretch
goals](https://github.com/hydro-project/hydroflow/issues?q=is:open+label:Datastores/Gossip-KV+milestone:0.10+label:P1)
[P2 - Low
Priority](https://github.com/hydro-project/hydroflow/issues?q=is:open+label:Datastores/Gossip-KV+milestone:0.10+label:P2)

Closes #1530.

---------

Co-authored-by: Mingwei Samuel <[email protected]>

…rences (#1524)

)

…onal` (#1526)

Also renames `cross_singleton` to `zip` when both sides are
singleton-like.

…dd docs (#1522)

Fixes the issue by combining the all values generated per subgraph
execution into one, which effectively de-duplicates the values.

Adds basic docs.

BREAKING CHANGE: changes exact behavior of `lattice_bimorphism()`,
instead of emitting multiple items, will emit a single item which is the
merge of what would've been all the old outputs. `Output: Merge<Output>`
is a new required bound.

…used `Interval` IR node (#1540)

Closes #1533

…1541)

…1542)

Closes #1529 (P0), #1528 (P1).

Summary of changes:
1. Local Deployments use Terraform.
2. Both Local / AWS Terraform projects have been split into separate
"infra" and "application" sections because there are intervening steps
(building images, etc.) that need to be done before applications can be
deployed.
3. Independent documentation for each local/AWS deployment - should make
it less confusing because these are expected to diverge.

fixes #1531.

`Compiling pythonize v0.20.0` seems to take a long time

Took 36m 46s for the record,
https://github.com/MingweiSamuel/hydroflow/actions/runs/11728824729/job/32673130852

…#1550)

Previously, each location had a (semantic) global clock that drives
ticks, and so all streams in a tick domain were all in the same atomic
block. For future optimizations, we'd like developers to be able to
place streams on the same location into different clocks to eliminate
synchronization between them, which in turn would allow the computations
in those separate clocks to be potentially decoupled across machines.

…#1551)

When looking at a prefix in an IDE, the location type argument is
generally more useful.

)

This also changes the behavior of `with_default_optimize` to be
terminal, if users want to apply optimizations after these they should
explicitly invoke the optimizations.

…flow_datalog v0.10.0, hydroflow_deploy_integration v0.10.0, hydroflow_macro v0.10.0, lattices_macro v0.5.7, variadics v0.0.7, variadics_macro v0.5.5, lattices v0.5.8, multiplatform_test v0.3.0, pusherator v0.0.9, hydroflow v0.10.0, hydro_deploy v0.10.0, stageleft_macro v0.4.0, stageleft v0.5.0, stageleft_tool v0.4.0, hydroflow_plus v0.10.0, hydro_cli v0.10.0, safety bump 8 crates

SAFETY BUMP: hydroflow_datalog_core v0.10.0, hydroflow_datalog v0.10.0, hydroflow_macro v0.10.0, hydroflow v0.10.0, hydro_deploy v0.10.0, stageleft v0.5.0, hydroflow_plus v0.10.0, hydro_cli v0.10.0

Fix #1320 suggested in
#1318 (review)

Main breaking change is MDX parsing, which trips up on unescaped `<` in
the generated docs, so we have to adjust the generator logic.

Some rephrasing, extra details, after working thru with Rohit.

…riant (#1559)

Our lifetimes were accidentally made covariant when the lifetime `'a`
was removed from the process/cluster tag type. This fixes that typing
hole, and also loosens some restrictions on the lifetime of deploy
environments.

Reduces namespace pollution when wildcard-importing `hydroflow_plus`.

…e bounded (#1565)

Returning a `Stream` from `union` on unbounded streams was unsound,
since the order of outputs is not deterministic.

Also fixes compiler crashes when using `.enumerate()` on an un-batched
stream.

- Remove TODO.md (contents filed as an issue, #1538)
- Update stack diagram
- Update discussion of stack diagram

- Change `README.md` to reflect the project as a whole, not just
hydroflow.
- Copy Rust getting-started info from Hydroflow docs to HF+ docs

We only need it for examples, so this reduces compilation burden for
`trybuild`.

Significantly improves the appearance of Hydroflow+ logs when the
terminal causes wrapping.

I've decided to leave some more challenging TODOs as comments for now,
just so we can get the live site back into decent shape.

Also changes the "Get Started" button on the landing to point to the HF+
quickstart.

Docusaurus v3 broke Mermaid in the playground due to an API change.

…trengthen determinism guarantees (#1568)

Previously, sending data from a `Cluster` would return a stream assumed
to have deterministic contents **and** ordering, which is false. This
introduces another type parameter for `Stream` which tracks whether
element ordering is expected to be deterministic, and restricts
operators such as `fold` and `reduce` to commutative aggregations
accordingly.

…ced as a global constant (#1574)

This eliminates the need to store `cluster.self_id()` in a local
variable first, instead you can directly reference `CLUSTER_SELF_ID`.

…obal constant (#1575)

… introduce timestamped streams (#1584)

Big PR.

First big change is we introduce a `Timestamped` location. This is a bit
of a hybrid between top-level locations and `Tick` locations. The idea
is that you choose where timestamps are generated, and then have a
guarantee that everything after that will be atomically computed (useful
for making sure we add payloads to the log before ack-ing).

The contract is that an operator or module that takes a `Timestamped`
input must still be deterministic regardless of the stamps on messages
(which are hidden unless you `tick_batch`). But unlike a top-level
stream (which has the same constraints), you have the atomicity
guarantee. Right now the guarantee is trivial since we have one global
tick for everything. But in the future when we want to apply
@davidchuyaya's optimizations this will be helpful to know when there
are causal dependencies on when data can be sent to others.

Second change is we mark every non-deterministic operator (modulo
explicit annotations such as `NoOrder`) with Rust's `unsafe` keyword.
This makes it super clear where non-determinism is taking place.

I've used this to put `unsafe` blocks throughout our example code and
add `SAFETY` annotations that argue why the non-determinism is safe (or
point out that we've explicitly documented / expect non-determinism). I
also added `#![warn(unsafe_op_in_unsafe_fn)]` to the examples and the
template, since this forces good hygiene of annotating sources of
non-determinism even inside a module that is intentionally
non-deterministic.

Paxos changes are mostly refactors, and I verified that the performance
is the same as before.

… clock (#1588)

Basic first pass implementation, without changing the current scheduler,
no breaking behavior. Adds the flo syntax (`loop { ... }`), basic graph
structure checks, nested loop blocks, and two basic windowing operators
(`batch()` and `all_once()`).

Next steps:
- Needs scheduler changes:
  - Implement un-windowing operators
  - Implement windowing operator `repeat_n()` - 
- Pipeline flags: Needs checking of bounded vs unbounded for `batch()`
vs `all_once()`
- Needs checking that all inputs into a loop agree
- State type negotiations (for `batch()` and handoffs (?) - don't always
use `Vec`)
- Dag performance optimizations (lots of things)