-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/multi loader #559
base: main
Are you sure you want to change the base?
Feature/multi loader #559
Changes from 25 commits
3653fc8
3265056
32bbd7c
0f87b5d
7ff0094
a170021
7273c1d
482db6c
316e539
734b60c
ba7181e
56755f8
cd683f5
e53eecc
68d5000
94dca93
0b2733c
f4861ed
f6b3547
ab1c7cd
8b44690
6b58cc3
6d83ad8
3ff658f
99c1c63
0a96237
27fa881
dd17625
ee6a717
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
# Multi-Loader | ||
|
||
A wrapper around loader to run multiple experiments at once with additional features like validation, dry-run, log collection | ||
|
||
## Prerequisites | ||
As a wrapper around loader, multi-loader requires the initial cluster setup to be completed. See [vHive Loader to create a cluster](https://github.com/vhive-serverless/invitro/blob/main/docs/loader.md#create-a-cluster) | ||
|
||
## Configuration | ||
### Multi-Loader Configuration | ||
| Parameter name | Data type | Possible values | Default value | Description | | ||
|---------------------|--------------------|-----------------|---------------|------------------------------------------------------------| | ||
| Studies | []LoaderStudy | N/A | N/A | A list of loader studies with their respective configurations. See [LoaderStudy](#loaderstudy) | | ||
| BaseConfigPath | string | "tools/multi_loader/base_loader_config.json" | N/A | Path to the base configuration file | | ||
| PreScript | string | any bash command | "" | (Optional) A global script that runs once before all experiments | | ||
| PostScript | string | any bash command | "" | (Optional) A global script that runs once after all experiments | | ||
|
||
### LoaderStudy | ||
| Parameter name | Data type | Possible values | Default value | Description | | ||
|-----------------------|------------------------|-------------------------------|---------------|--------------------------------------------------------------------| | ||
| Config | map[string]interface{} | Any field in [LoaderConfiguration](https://github.com/vhive-serverless/invitro/blob/main/docs/configuration.md#loader-configuration-file-format) | N/A | The configuration for each loader experiment which overrides configurations in baseLoaderConfig | | ||
| Name | string | N/A | N/A | The name of the loader experiment | | ||
| TracesDir | string | N/A | N/A | Directory containing the traces for the experiment | | ||
| TracesFormat | string | "data/traces/example_{}" | N/A | Format of the trace files **The format string "{}" is required** | | ||
| TraceValues | []interface{} | ["any", 0, 1.1] | N/A | Values of the trace files Replaces the "{}" in TraceFormat | | ||
| OutputDir | string | any | data/out/{Name} | (Optional) Output directory for experiment results | | ||
| Verbosity | string | "info", "debug", "trace" | "info" | (Optional) Verbosity level for logging the experiment | | ||
| IatGeneration | bool | true, false | false | (Optional) Whether to Generate iats only and skip invocations | | ||
| Generated | bool | true, false | false | (Optional) if iats were already generated | | ||
| PreScript | string | any bash Command | "" | (Optional) Local script that runs this specific experiment | | ||
| PostScript | string | any bash Command | "" | (Optional) Local script that runs this specific experiment | | ||
|
||
> **_Important_**: Only one of the following is required: | ||
> 1. `TracesDir`, or | ||
> 2. `TracesFormat` and `TraceValues`, or | ||
> 3. `TracePath` within the `LoaderExperiment`'s `Config` field | ||
> | ||
> If more than one is defined, the order of precedence is as follows: | ||
> 1. `TracesDir`, | ||
> 2. `TracesFormat` and `TraceValues`, | ||
> 3. `TracePath` | ||
|
||
> **_Note_**: | ||
> The `Config` field follows the same structure as the [LoaderConfiguration](https://github.com/vhive-serverless/invitro/blob/main/docs/configuration.md#loader-configuration-file-format). | ||
> Any field defined in `Config` will override the corresponding value from the configuration in `BaseConfigPath`, but only for that specific experiment. | ||
> For example, if `BaseConfigPath` has `ExperimentDuration` set to 5 minutes, and you define `ExperimentDuration` as 10 minutes in `Config`, that particular experiment will run for 10 minutes instead. | ||
|
||
## Command Flags | ||
|
||
The multi-loader accepts the almost the same command-line flags as loader. | ||
|
||
> **_Note_**: These flags will subsequently be used during the execution of loader.go for **<u>every experiment</u>**. If you would like to define these flag for specific experiments only, define it in [LoaderStudy](#loaderstudy) | ||
|
||
Available flags: | ||
|
||
- **`--multiLoaderConfig`** *(default: `tools/multi_loader/multi_loader_config.json`)*: | ||
Specifies the path to the multi-loader configuration file. This file contains settings and parameters that define how the multi-loader operates [see above](#multi-loader-configuration) | ||
|
||
- **`--verbosity`** *(default: `info`)*: | ||
Sets the logging verbosity level. You can choose from the following options: | ||
- `info`: Standard information messages. | ||
- `debug`: Detailed debugging messages. | ||
- `trace`: Extremely verbose logging, including detailed execution traces. | ||
|
||
- **`--iatGeneration`** *(default: `false`)*: | ||
If set to `true`, the multi-loader will generate inter-arrival times (IATs) only and skip the invocation of actual workloads. This is useful for scenarios where you want to analyze or generate IATs without executing the associated load. | ||
|
||
- **`--generated`** *(default: `false`)*: | ||
Indicates whether IATs have already been generated. If set to `true`, the multi-loader will use the existing IATs instead of generating new ones. | ||
|
||
|
||
## Multi-loader Overall Flow | ||
|
||
1. **Initialization** | ||
- Flags for configuration file path, verbosity, IAT generation, and execution mode are parsed | ||
- Logger is initialized based on verbosity level | ||
|
||
3. **Experiment Execution Flow** | ||
- The multi-loader runner is instantiated with the provided configuration path. | ||
- A dry run is executed to validate the setup for all studies: | ||
- If any dry run fails, the execution terminates. | ||
- If all dry runs succeed, proceed to actual runs: | ||
- Global pre-scripts are executed. | ||
- Each experiment undergoes the following steps: | ||
1. **Pre-Execution Setup** | ||
- Experiment-specific pre-scripts are executed. | ||
- Necessary directories and folders are created. | ||
- Each sub-experiment is unpacked and prepared | ||
2. **Experiment Invocation** | ||
- The loader is executed with generated configurations and related flags | ||
3. **Post-Execution Steps** | ||
- Experiment-specific post-scripts are executed | ||
- Cleanup tasks are performed | ||
|
||
4. **Completion** | ||
- Global post-scripts are executed. | ||
- Run Make Clean | ||
|
||
### How the Dry Run Works | ||
|
||
The dry run mode executes the loader with the `--dryRun` flag set to true after the unpacking of experiments defined in the multi-loader configurations. | ||
|
||
In this mode, the loader performs the following actions: | ||
|
||
- **Configuration Validation**: It verifies the experiment configurations without deploying any functions or generating invocations. | ||
- **Error Handling**: If a fatal error occurs at any point, the experiment will halt immediately. | ||
|
||
The purpose is to ensure that your configurations are correct and to identify any potential issues before actual execution. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
package common | ||
|
||
type MultiLoaderConfiguration struct { | ||
Studies []LoaderStudy `json:"Studies"` | ||
BaseConfigPath string `json:"BaseConfigPath"` | ||
// Optional | ||
PreScript string `json:"PreScript"` | ||
PostScript string `json:"PostScript"` | ||
} | ||
|
||
type LoaderStudy struct { | ||
Name string `json:"Name"` | ||
Config map[string]interface{} `json:"Config"` | ||
// A combination of format and values or just dir should be specified | ||
TracesDir string `json:"TracesDir"` | ||
|
||
TracesFormat string `json:"TracesFormat"` | ||
TraceValues []interface{} `json:"TraceValues"` | ||
|
||
// Optional | ||
OutputDir string `json:"OutputDir"` | ||
Verbosity string `json:"Verbosity"` | ||
IatGeneration bool `json:"IatGeneration"` | ||
Generated bool `json:"Generated"` | ||
PreScript string `json:"PreScript"` | ||
PostScript string `json:"PostScript"` | ||
} |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think it is a good idea to create a separate file in common for node group. It is only used in the multi-loader, so it is already not common. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
package common | ||
|
||
type NodeGroup struct { | ||
MasterNode string | ||
AutoScalerNode string | ||
ActivatorNode string | ||
LoaderNode string | ||
WorkerNodes []string | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
package common | ||
|
||
import ( | ||
"bytes" | ||
"net" | ||
"os" | ||
"os/exec" | ||
"path" | ||
"slices" | ||
"strings" | ||
|
||
log "github.com/sirupsen/logrus" | ||
) | ||
|
||
func CheckNode(node string) { | ||
if !IsValidIP(node) { | ||
log.Fatal("Invalid IP address for node ", node) | ||
} | ||
cmd := exec.Command("ssh", "-oStrictHostKeyChecking=no", "-p", "22", node, "exit") | ||
// -oStrictHostKeyChecking=no -p 22 | ||
out, err := cmd.CombinedOutput() | ||
if bytes.Contains(out, []byte("Permission denied")) || err != nil { | ||
log.Error(string(out)) | ||
log.Fatal("Failed to connect to node ", node) | ||
} | ||
} | ||
|
||
func CheckPath(path string) { | ||
if(path) == "" { return } | ||
_, err := os.Stat(path) | ||
if err != nil { | ||
log.Fatal(err) | ||
} | ||
} | ||
|
||
func IsValidIP(ip string) bool { | ||
parsedIP := net.ParseIP(ip) | ||
return parsedIP != nil | ||
} | ||
|
||
// Check general multi-loader configuration that applies to all platforms | ||
func CheckMultiLoaderConfig(multiLoaderConfig MultiLoaderConfiguration) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is very multi-loader specific, please move out of common. |
||
log.Info("Checking multi-loader configuration") | ||
// Check if all paths are valid | ||
CheckPath(multiLoaderConfig.BaseConfigPath) | ||
// Check each study | ||
if len(multiLoaderConfig.Studies) == 0 { | ||
log.Fatal("No study found in configuration file") | ||
} | ||
for _, study := range multiLoaderConfig.Studies { | ||
// Check trace directory | ||
// if configs does not have TracePath or OutputPathPreix, either TracesDir or (TracesFormat and TraceValues) should be defined along with OutputDir | ||
if study.TracesDir == "" && (study.TracesFormat == "" || len(study.TraceValues) == 0) { | ||
if _, ok := study.Config["TracePath"]; !ok { | ||
log.Fatal("Missing one of TracesDir, TracesFormat & TraceValues, Config.TracePath in multi_loader_config ", study.Name) | ||
} | ||
} | ||
if study.TracesFormat != ""{ | ||
// check if trace format contains TRACE_FORMAT_STRING | ||
if !strings.Contains(study.TracesFormat, TraceFormatString) { | ||
log.Fatal("Invalid TracesFormat in multi_loader_config ", study.Name, ". Missing ", TraceFormatString, " in format") | ||
} | ||
} | ||
if study.OutputDir == "" { | ||
if _, ok := study.Config["OutputPathPrefix"]; !ok { | ||
log.Warn("Missing one of OutputDir or Config.OutputPathPrefix in multi_loader_config ", study.Name) | ||
// set default output directory | ||
study.OutputDir = path.Join("data", "out", study.Name) | ||
log.Warn("Setting default output directory to ", study.OutputDir) | ||
} | ||
} | ||
} | ||
log.Info("All experiments configs are valid") | ||
} | ||
|
||
func CheckCollectableMetrics(metrics string) { | ||
if !slices.Contains(ValidCollectableMetrics, metrics) { | ||
log.Fatal("Invalid metrics ", metrics) | ||
} | ||
} | ||
|
||
func CheckCPULimit(cpuLimit string) { | ||
if !slices.Contains(ValidCPULimits, cpuLimit) { | ||
log.Fatal("Invalid CPU Limit ", cpuLimit) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also quite multi-loader related, you can move it where it is actually use, instead of storing it in common.