Implement simple sequenced keys constraint

[devreal]

Here is a first implementation of constraints. The flow is as follows:

1. The application creates a constraint that it can pass to one or more TTs. Currently, all TTs must have the same key (like BcastA/B). We have to supply different mappers for BcastA/B because the keys contain different information (for A, k is in key[1], for B it's in key[0]). The mapper extracts the sequence number from the key and returns it as an ordinal to the constraint. The mapper can be either provided to the constraint directly or to the TT, which then passes it on to the constraint. That (and handling void :/) makes the code somewhat more complex than needed. Users probably will implement only one version for the configuration they need.
2. Once a task has all its inputs, it will query the constraint whether the key can be scheduled for execution. If not, the TT will store the task in a hash table that is used once the key is released.
3. When a task completes, the TT reports to constraint that a given key has completed. It is up to the constraint what to do with that information. In the example constraint, we decrement a counter and once it's zero we release the next wave of tasks.
4. TTs register themselves as a listener and are informed about a new set of tasks that is released. Releasing tasks in bulk gives us some room for optimization. Ultimately, that's why I diverted from the release callback per task that we talked about in Knoxville.

User-defined constraints should inherit from ConstraintBase to get the listener handling and have to provide the proper check and complete callbacks.

I still need to remove the control flow we have today in SPMM and check that it works as expected. Overall, the user-facing interface is quite clean and the internal interface is not terrible (a win already).

[therault]

I'm not sure I see the difference between constraints_check and constraints_complete. It looks like we always push_back a callback to both vectors (or to none), and sometimes we iterate over constraints_check only, but since we appear to never remove the callback from either, I'm not sure I follow why we need both (not saying I found a problem, just wanting more explanations :) )

[devreal]

I'll put some comments there. These are two different callbacks:

1. constraints_check hands the task over to the constraint, which will either tell us directly that it is not constrained (i.e., can be executed) or stores the key for later release through a callback (the listener action) and tells us that the key is indeed constrained.
2. constraints_complete tells the constraint that the task has executed (so that the constraint can check for new tasks to release through the listener action)

[therault]

That looks neat indeed.

I'm slightly worried about the limitation that the constraint needs to work on a single key type.

We don't need it in GEMM, because by constraining the broadcast tasks, we indirectly constrain the GEMM tasks, but let's say that I wanted to explicitly constrain the GEMM tasks too: I would need to define a new shared constraint with SequencedKeysConstraint<Key<3>> and I would have to release the constraint on both shared constraints to make them progress simultaneously?

Also, I don't see in the GEMM code any explicit release of constraints? Who makes them progress?

[devreal]

I agree. I have not found a good way to type pun the keys. Today, we couldn't include the gemm task in the constraint because the key types are different (Key<2> and Key<3>). I'm sure there is way to do that, I'm not sure about the complexity...

[therault]

This is not what I had in mind. Might work too, but let's go over it during the next meeting, to make sure it does what we want.

AFAICT, the SequencedKeyConstraint makes sure tasks are released in the order of the integer returned by the mapping function; release of tasks can be paused (calling the stop() method) and resumed (calling the start() method), but I'm not sure what calls stop or stat in the GEMM example. It looks to me that with this implementation, if we discover enough BCastA / BCastB tasks, we'll push them in the right order in the scheduler (which is already a progress), but there is no mechanism to actually prevent a BCastA with a high 'k' to be scheduled if it is discovered first.

[devreal]

In the SPMM example we attach the mapping function to the TT (the keys for BcastA/B are the same type but we need different information from them). We cannot pass one mapping here. The different check functions below are conditionally enabled on whether a mapper was provided to the constraint or whether the TT has to provide the ordinal.

[devreal]

The constraints need to be ported to the MADNESS backend. I'd like to refactor the common parts out into the TT base class to avoid duplication yet another bunch of code.

[devreal]

Somehow this breaks the element-wise SPMM. Not sure why, and how to dig into that. Will need some time or volunteers :)

[devreal]

@evaleev Something is off with the Boosts in the Ubuntu CI:

CMake Error at CMakeLists.txt:4 (find_package):
  Found package configuration file:

    /home/runner/work/ttg/ttg/install/lib/cmake/ttg/ttg-config.cmake

  but it set ttg_FOUND to FALSE so package "ttg" is considered to be NOT
  FOUND.  Reason given by package:

  The following imported targets are referenced, but are missing:
  Boost::callable_traits Boost::serialization Boost::iostreams

I've seen similar issues on one of our machines. This PR bumps the minimum required to 1.81. What was the reason for that?

[evaleev]

@evaleev Something is off with the Boosts in the Ubuntu CI:

Ubuntu provides too old of a Boost, which wreaks unseen levels of havoc (build succeeds but metadata in ttg-config.cmake is somehow all screwed up).

I've seen similar issues on one of our machines. This PR bumps the minimum required to 1.81. What was the reason for that?

tracking issues across the entirety of the stack is difficult. From what I recall some parts of boost 1.80 used std::unary_function which was removed in C++17, but most libs still shipped it ... but not Apple clang 15, so I bumped to boost to 1.81 across my stack without trying to fine-grain it. Building from source provides Boost 1.81 as it is a good-enough implementation of modularized Boost build, so it makes sense to demand at least that version to avoid diffs between system-provided boost and from-source boost.

Boost is hell. But reimplementing it is hell^2.