Skip to content

Commit

Permalink
Implement any/all evaluation in group.include.
Browse files Browse the repository at this point in the history
  • Loading branch information
joaander committed May 20, 2024
1 parent b82e6f2 commit 8dee128
Show file tree
Hide file tree
Showing 6 changed files with 160 additions and 76 deletions.
14 changes: 9 additions & 5 deletions doc/src/guide/howto/same.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@

You can submit the same action to different groups and resources. To do so,
create multiple elements in the action array *with the same name*. Each must use
[`group.include`](../../workflow/action/group.md#include) to select *non-overlapping
subsets*. You can use [`action.from`](../../workflow/action/index.md#from) to copy all
fields from one action and selectively override others.
[`action.group.include`] to select *non-overlapping subsets*. You can use
[`action.from`] to copy all fields from one action and selectively override others.

For example, this `workflow.toml` uses 4 processors on directories with small *N* and 8
those with a large *N*.
Expand All @@ -20,11 +19,16 @@ products = ["results.out"]
walltime.per_submission = "12:00:00"
processes.per_directory = 4
[action.group]
include = [["/N", "<=", "4096"]]
maximum_size = 32
[[action.group.include]]
condition = ["/N", "<=", "4096"]

[[action]]
from = "compute"
resources.processes.per_directory = 8
group.include = [["/N", ">", "4096"]]
[[action.group.include]]
condition = ["/N", ">", "4096"]
```

[`action.group.include`]: ../../workflow/action/group.md#include
[`action.from`]: ../../workflow/action/index.md#from
10 changes: 5 additions & 5 deletions doc/src/guide/tutorial/group.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,15 @@ This workflow will apply the `process_point` action to the directories where
`value/type == "point"` and the `process_letter` action to the directories where
`value/type == "letter"`.

`include` is an array. Each element is a length 3 array with the contents: `[JSON
pointer, operator, operand]`. Think of each element as an expression. The [*JSON
pointer*](../concepts/json-pointers.md) is a string that reads a particular value
`condition` is a length 3 array with the contents: `[JSON pointer, operator, operand]`.
Think of each element as an expression. The
[*JSON pointer*](../concepts/json-pointers.md) is a string that reads a particular value
from the directory's **value**. The *operator* is a comparison operator: `"<"`, `"<="`,
`"=="`, `">="`, or `">"`. The *operand* is the value to compare to. Together, these 3
elements make a *condition*.

**Row** applies these *conditions* to all directories in the workspace. When all
*conditions* are true, the directory is included in the action's **groups**.
**Row** applies the *condition* to all directories in the workspace. When the
*condition* is true, the directory is included in the action's **groups**.

> Note: This implies that every JSON pointer used in an `include` condition **MUST**
> be present in every value file.
Expand Down
30 changes: 20 additions & 10 deletions doc/src/workflow/action/group.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@ that it submits.
Example:
```toml
[action.group]
include = [["/subproject", "==", "project_one"]]
sort_by = ["/value"]
split_by_sort_key = true
maximum_size = 16
submit_whole = true
reverse_sort = true
[[action.group.include]]
condition = ["/subproject", "==", "project_one"]
```

> Note: You may omit `[action.group]` entirely.
Expand All @@ -21,27 +22,36 @@ groups of directories included in a given action.

## include

`action.group.include`: **array** of **arrays** - Define a set of conditions that must
all be true for a directory to be included in this group. Each condition is an **array**
of three elements: The *JSON pointer*, *the operator*, and the *operand*. The [JSON
pointer](../../guide/concepts/json-pointers.md) points to a specific element
from the directory's value. The operator may be `"<"`, `"<="`, `"=="`, `">="`, or `">"`.
`action.group.include`: **array** of **tables** - Define a set of selectors, *any* of
which may be true for a directory to be included in this group.

Each selector is a **table** with only one of the following keys:
* `condition`: An array of three elements: The *JSON pointer*, *the operator*, and the
*operand*. The [JSON pointer](../../guide/concepts/json-pointers.md) points to a
specific element from the directory's value. The operator may be `"<"`, `"<="`,
`"=="`, `">="`, or `">"`.
* `all`: Array of conditions (see above). All conditions must be true for this selector
to be true.

For example, select all directories where a value is in the given range:
```toml
include = [["/value", ">", 0.2], ["/value", "<", 0.9]]
[[action.group.include]]
all = [["/value", ">", 0.2], ["/value", "<", 0.9]]
```
Choose directories where an array element is equal to a specific value:
```toml
include = [["/array/1", "==", 12]]
[[action.group.include]]
condition = ["/array/1", "==", 12]
```
Match against strings:
```toml
include = [["/map/name", "==", "string"]]
[[action.group.include]]
condition = ["/map/name", "==", "string"]
```
Compare by array:
```toml
include = [["/array", "==", [1, "string", 14.0]]]
[[action.group.include]]
condition = ["/array", "==", [1, "string", 14.0]
```

Both operands **must** have the same data type. The JSON pointer must be present in the
Expand Down
2 changes: 1 addition & 1 deletion doc/src/workflow/action/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ action. The name may be set by [from](#from).
> the same name. All elements with the same name **must** have identical
> [`products`](#products) and [`previous_actions`](#previous_actions). All elements
> with the same name **must also** select non-intersecting subsets of directories with
> [`group.include`](group.md#include).
> [`action.group.include`](group.md#include).
## command

Expand Down
76 changes: 58 additions & 18 deletions src/project.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ use crate::scheduler::bash::Bash;
use crate::scheduler::slurm::Slurm;
use crate::scheduler::Scheduler;
use crate::state::State;
use crate::workflow::{Action, Workflow};
use crate::workflow::{Action, Selector, Workflow};
use crate::{Error, MultiProgressContainer};

/// Encapsulate the workflow, state, and scheduler into a project.
Expand Down Expand Up @@ -184,23 +184,55 @@ impl Project {

'outer: for name in directories {
if let Some(value) = self.state.values().get(&name) {
for (include, comparison, expected) in action.group.include() {
let actual = value
.pointer(include)
.ok_or_else(|| Error::JSONPointerNotFound(name.clone(), include.clone()))?;
if !expr::evaluate_json_comparison(comparison, actual, expected).ok_or_else(
|| {
Error::CannotCompareInclude(
actual.clone(),
expected.clone(),
name.clone(),
)
},
)? {
continue 'outer;
if action.group.include().is_empty() {
matching_directories.push(name);
} else {
for selector in action.group.include() {
let result = match selector {
Selector::Condition((include, comparison, expected)) => {
let actual = value.pointer(include).ok_or_else(|| {
Error::JSONPointerNotFound(name.clone(), include.clone())
})?;

expr::evaluate_json_comparison(comparison, actual, expected)
.ok_or_else(|| {
Error::CannotCompareInclude(
actual.clone(),
expected.clone(),
name.clone(),
)
})
}

Selector::All(conditions) => {
let mut matches = 0;
for (include, comparison, expected) in conditions {
let actual = value.pointer(include).ok_or_else(|| {
Error::JSONPointerNotFound(name.clone(), include.clone())
})?;

if expr::evaluate_json_comparison(comparison, actual, expected)
.ok_or_else(|| {
Error::CannotCompareInclude(
actual.clone(),
expected.clone(),
name.clone(),
)
})?
{
matches += 1;
}
}
Ok(matches == conditions.len())
}
};

if result? {
matching_directories.push(name);
continue 'outer;
}
}
}
matching_directories.push(name);
} else {
warn!("Directory '{}' not found in workspace.", name.display());
}
Expand Down Expand Up @@ -424,7 +456,8 @@ products = ["one"]
name = "two"
command = "c"
products = ["two"]
group.include = [["/i", "<", {}]]
[[action.group.include]]
condition = ["/i", "<", {}]
[[action]]
name = "three"
Expand Down Expand Up @@ -464,15 +497,22 @@ previous_actions = ["two"]
all_directories[0..6]
);

// Check all conditions.
let mut action = project.workflow.action[1].clone();
let include = action.group.include.as_mut().unwrap();
include.push(("/i".into(), Comparison::GreaterThan, Value::from(4)));
include.clear();
include.push(Selector::All(vec![
("/i".into(), Comparison::GreaterThan, Value::from(4)),
("/i".into(), Comparison::LessThan, Value::from(6)),
]));
assert_eq!(
project
.find_matching_directories(&action, all_directories.clone())
.unwrap(),
vec![PathBuf::from("dir5")]
);

// TODO, test any
}

#[test]
Expand Down
Loading

0 comments on commit 8dee128

Please sign in to comment.