Merge pull request #19 from bbodin/bruno-dev

Bruno dev

dse_experiments.md

@@ -0,0 +1,45 @@
+# Throughput-Buffering Trade-Off Analysis
+K-Iter has several design space exploration (DSE) algorithms available to perform throughput-buffering trade-off analysis on SDFGs/CSDFGs. Here, we list the algorithms available, including the parameters that can be passed to them. The algorithms run on SDF3-like XML files as input files.
+
+## Optimal methods
+### K-periodic-driven DSE (KDSE)
+Perform throughput-buffering DSE using K-periodic scheduling for throughput computation and critical cycles as a search heuristic for buffer size allocations. By default, the ```ThroughputBufferingDSE``` algorithm runs the DSE algorithm using K-periodic scheduling and critical cycles, without logging:
+
+`./Release/bin/kiter -f <input-file> -a ThroughputBufferingDSE`
+
+In order to log the data (search path and Pareto points), specify a log directory using the ```LOGDIR``` parameter:
+
+`./Release/bin/kiter -f <input-file> -a ThroughputBufferingDSE -p LOGDIR=./logs/`
+
+### ASAP scheduling driven DSE (SDF3)
+[SDF3](https://www.es.ele.tue.nl/sdf3/) has a throughput-buffering DSE algorithm using ASAP scheduling for throughput computation and storage dependencies as a heuristic for determining buffer size allocations. SDF3 can be installed within K-Iter. We added logging capabilities to SDF3, amongst other changes, that will be listed in the following sections. Note that `patch` is required to run `install_sdf3.sh`, so install that with your package manager of choice before running the script. In order to install and compile SDF3, run the following from the home directory of K-Iter:
+
+1. If the subdirectory doesn't already exist: `mkdir ./tools/sdf3`
+2. `./tools/install_sdf3.sh ./tools/sdf3/`
+
+Now that SDF3 has been installed, their throughput-buffering DSE algorithm can be run using the following command:
+
+`SDF3LOGDIR=./logs/ USE_SCC=false COARSE=false ./tools/sdf3/sdf3_custom/sdf3/build/release/Linux/bin/{sdf3analysis-sdf|sdf3analysis-csdf} --graph <input-file> --algo buffersize`
+
+Note that `SDF3LOGDIR` sets the log directory path, so set it accordingly and make sure that the log directory is made before running the DSE. Furthermore, either `sdf3analysis-sdf` or `sdf3analysis-csdf` will have to be used depending on whether the input file is a SDF or CSDF graph.
+
+### Corrected storage dependency detection using strongly connected components
+We identified an implementation error in SDF3's simple cycle detection algorithm which, in turn, caused a bug in their storage dependency detection algorithm. We corrected this by using strongly connected components to detect cycles in their dependency graph. This corrected version can be run by setting ```USE_SCC=true```.
+
+`SDF3LOGDIR=./logs/ USE_SCC=true COARSE=false ./tools/sdf3/sdf3_custom/sdf3/build/release/Linux/bin/{sdf3analysis-sdf|sdf3analysis-csdf} --graph <input-file> --algo buffersize`
+
+## Approximate methods
+### 1-periodic DSE (1DSE)
+Perform throughput-buffering DSE using 1-periodic scheduling and dichotomous search for buffer size allocations:
+
+`./Release/bin/kiter -f <input-file> -a PeriodicDSE -p LOGDIR=./logs/`
+
+### Coarse K-periodic-driven DSE (KDSE-C)
+KDSE-C multiplies the step size of the KDSE algorithm by a 2 to increase the coarseness of the DSE:
+
+`./Release/bin/kiter -f <input-file> -a ThroughputBufferingDSE -p LOGDIR=./logs/ -p COARSE=2`
+
+### Coarse K-periodic-driven DSE (SDF3-C)
+SDF3-C multiplies the step size of the SDF3 algorithm by a 2 to increase the coarseness of the DSE:
+
+`SDF3LOGDIR=./logs/ USE_SCC=false COARSE=true ./tools/sdf3/sdf3_custom/sdf3/build/release/Linux/bin/{sdf3analysis-sdf|sdf3analysis-csdf} --graph <input-file> --algo buffersize`

src/libkiter/algorithms/algorithms.h

@@ -21,6 +21,7 @@
 #include <algorithms/throughput/degroote.h>
 #include <algorithms/throughput/csdf_strictly_periodic.h>
 #include <algorithms/experiments.h>
+#include <algorithms/analysis/cycle_detection.h>
 
 
 

src/libkiter/algorithms/analysis/cycle_detection.cpp

@@ -0,0 +1,219 @@
+/*
+ * cycle_detection.cpp
+ *
+ *  Created on: 29 Dec 2021
+ *      Author: toky
+ */
+
+#include "cycle_detection.h"
+#include <commons/commons.h>
+#include <commons/verbose.h>
+#include <map>
+#include <vector>
+#include <algorithms/scc.h>
+#include <algorithms/throughput/symbolic_execution.h>
+#include <models/Dataflow.h>
+#include <limits>
+#include <stack>
+#include <set>
+
+std::vector<ARRAY_INDEX> dfs_stack;
+std::set<ARRAY_INDEX> blocked_set;
+std::map<ARRAY_INDEX, std::set<ARRAY_INDEX>> blocked_map;
+std::vector<std::vector<ARRAY_INDEX>> simple_cycles;
+
+bool algorithms::has_selfloops            (models::Dataflow* const  dataflow) {
+	for (auto t : dataflow->vertices()) {
+		if (dataflow->getReentrancyFactor(t) > 0) {
+			return true;
+		}
+	}
+	return false;
+}
+bool algorithms::has_cycles            (models::Dataflow* const  dataflow) {
+	std::map<int, std::vector<ARRAY_INDEX>> res = algorithms::computeSCCKosaraju(dataflow);
+	for (auto item : res) {
+		VERBOSE_INFO("Component " << item.first << " has " << item.second.size() << " members");
+	}
+	for (auto item : res) {
+		if (item.second.size() > 1) return true;
+	}
+	return false;
+}
+void algorithms::cycle_detection       (models::Dataflow* const  dataflow, parameters_list_t){
+	VERBOSE_INFO("has_cycles result: " << algorithms::has_cycles(dataflow));
+	std::cout << "has_cycles result: " << algorithms::has_cycles(dataflow) << std::endl;
+}
+
+void algorithms::find_simple_cycles(models::Dataflow* const  dataflow, parameters_list_t) {
+  ARRAY_INDEX start_id = 1; // NOTE probably better to check first vertex in dataflow graph rather than assume value of start ID (dataflow->getFirstVertex()?)
+  VERBOSE_DEBUG("original graph:" << std::endl);
+  VERBOSE_DEBUG(print_status(dataflow) << std::endl);
+  while (start_id <= dataflow->getVerticesCount()) {
+    models::Dataflow* subgraph = generate_subgraph(dataflow, start_id); // subgraph excludes all vertices less than the specified starting ID
+    std::map<int, std::vector<ARRAY_INDEX>> scc_map = algorithms::computeSCCKosaraju(subgraph);
+    // find the SCC that contains the vertex with the smallest ID
+    ARRAY_INDEX min_id = find_smallest_index(subgraph, scc_map);
+    if (min_id != INT_MAX) {
+      models::Dataflow* scc = generate_scc(subgraph, scc_map, min_id);
+      VERBOSE_DEBUG("smallest vertex ID amongst SCCs: " << min_id << std::endl);
+      VERBOSE_DEBUG(print_status(scc) << std::endl);
+      blocked_set.clear();
+      blocked_map.clear();
+      find_simple_cycles_scc(scc, min_id, min_id);
+      start_id = min_id + 1;
+    } else {
+      break;
+    }
+  }
+  // print detected cycles
+  std::cout << "Cycles detected:" << std::endl;
+  for (auto cycle : simple_cycles) {
+    int comp_count = 1;
+    int cycle_size = cycle.size();
+    for (auto id : cycle) {
+      if (comp_count < cycle_size) {
+        std::cout << id << " -> ";
+      } else {
+        std::cout << id << std::endl;
+      }
+      comp_count++;
+    }
+  }
+  std::cout << "Total number of cycles: " << simple_cycles.size() << std::endl;
+}
+
+// iterate through SCC map, ignore SCC of size == 1, track minimum ID value
+ARRAY_INDEX algorithms::find_smallest_index(models::Dataflow* const dataflow,
+                                            std::map<int, std::vector<ARRAY_INDEX>> scc_map) {
+  ARRAY_INDEX min_id = INT_MAX; // NOTE should be LDBL_MAX but i can't seem to get it to work
+  for (auto const& component : scc_map) {
+    if (component.second.size() > 1) {
+      if (*std::min_element(component.second.begin(), component.second.end()) < min_id) {
+        min_id = *std::min_element(component.second.begin(), component.second.end());
+      }
+    }
+  }
+
+  return min_id;
+}
+
+// generate SCC that contains start_id
+models::Dataflow* algorithms::generate_scc(models::Dataflow* const dataflow,
+                                           std::map<int, std::vector<ARRAY_INDEX>> scc_map,
+                                           ARRAY_INDEX start_id) {
+  models::Dataflow* scc = new models::Dataflow(*dataflow);
+  for (auto const& component : scc_map) {
+    if (std::find(component.second.begin(),
+                  component.second.end(),
+                  start_id) == component.second.end()) {
+      continue;
+    } else {
+      {ForEachVertex(dataflow, v) {
+          ARRAY_INDEX actor_id = dataflow->getVertexId(v);
+          if (std::find(component.second.begin(),
+                        component.second.end(),
+                        actor_id) == component.second.end()) {
+            scc->removeVertex(scc->getVertexById(actor_id));
+          }
+        }}
+      return scc; // return first SCC with start_id
+    }
+  }
+}
+// make subgraph excluding all vertices less than a specified ID
+models::Dataflow* algorithms::generate_subgraph(models::Dataflow* const dataflow,
+                                                ARRAY_INDEX lower_limit) {
+  models::Dataflow* subgraph = new models::Dataflow(*dataflow);
+  subgraph->reset_computation();
+  // remove all actors with IDs less than specified lower_limit
+  {ForEachVertex(dataflow, v) {
+      ARRAY_INDEX actor_id = dataflow->getVertexId(v);
+      if (actor_id < lower_limit) {
+        subgraph->removeVertex(subgraph->getVertexById(actor_id));
+      }
+    }}
+  return subgraph;
+}
+
+// rescursively remove specified vertex ID from blocked_set while taking into account blocked_map
+void algorithms::unblock(ARRAY_INDEX id) {
+  VERBOSE_DEBUG("removing " << id << " from blocked set" << std::endl);
+  blocked_set.erase(blocked_set.find(id));
+  VERBOSE_DEBUG("\tblocked set: ");
+  for (auto block : blocked_set) {
+    VERBOSE_DEBUG(block << " ");
+  }
+  VERBOSE_DEBUG(std::endl);
+  if (blocked_map.find(id) != blocked_map.end()) {
+    for (auto actor_id : blocked_map[id]) {
+      if (blocked_set.find(actor_id) != blocked_set.end()) {
+        unblock(actor_id);
+      }
+    }
+    VERBOSE_DEBUG("removing blocked mapping for all " << id << std::endl);
+    blocked_map.erase(blocked_map.find(id));
+  }
+}
+
+// find simple cycles starting at start_vertex from current_vertex
+bool algorithms::find_simple_cycles_scc(models::Dataflow* const scc,
+                                        ARRAY_INDEX start_id, ARRAY_INDEX current_id) {
+  bool found_cycle = false;
+  dfs_stack.push_back(current_id);
+  blocked_set.insert(current_id);
+
+  {ForOutputEdges(scc, scc->getVertexById(current_id), e) {
+      ARRAY_INDEX neighbour_id = scc->getVertexId(scc->getEdgeTarget(e));
+      VERBOSE_DEBUG("current_id: " << current_id << std::endl);
+      VERBOSE_DEBUG("neighbour_id: " << neighbour_id << std::endl);
+      if (neighbour_id == start_id) { // cycle found
+        VERBOSE_DEBUG("\tcurrent id == neighbour id; cycle found!" << std::endl);
+        dfs_stack.push_back(start_id);
+        std::vector<ARRAY_INDEX> cycle(dfs_stack);
+        VERBOSE_DEBUG("\tFound cycle: ");
+        for (auto i : cycle) {
+          VERBOSE_DEBUG(i << " ");
+        }
+        VERBOSE_DEBUG(std::endl);
+        simple_cycles.push_back(cycle);
+        // std::cout << "Cycles detected so far: " << simple_cycles.size() << std::endl;
+        dfs_stack.pop_back();
+        found_cycle = true;
+      } else if (blocked_set.find(neighbour_id) == blocked_set.end()) { // neighbouring vertex not in blocked_set
+        bool got_cycle = find_simple_cycles_scc(scc, start_id, neighbour_id);
+        found_cycle = found_cycle || got_cycle;
+      }
+    }}
+  if (found_cycle) {
+    unblock(current_id);
+  } else {
+    {ForOutputEdges(scc, scc->getVertexById(current_id), e) {
+        ARRAY_INDEX block_dep_id = scc->getVertexId(scc->getEdgeTarget(e));
+        VERBOSE_DEBUG("Adding " << block_dep_id << "->" << current_id << " to block map" << std::endl);
+        blocked_map[block_dep_id].insert(current_id);
+      }}
+  }
+  dfs_stack.pop_back();
+  return found_cycle;
+}
+
+// prints current status of dataflow graph
+std::string algorithms::print_status(models::Dataflow* const dataflow) {
+  std::stringstream output_stream;
+
+  output_stream << "\nActors:" << std::endl;
+  {ForEachVertex(dataflow, v) {
+      output_stream << dataflow->getVertexName(v) << std::endl;
+    }}
+  output_stream << "\nConnections:" << std::endl;
+  {ForEachEdge(dataflow, e) {
+      output_stream << "\tChannel " << dataflow->getEdgeName(e) << " ("
+                   << dataflow->getVertexName(dataflow->getEdgeSource(e))
+                   << "->"
+                   << dataflow->getVertexName(dataflow->getEdgeTarget(e))
+                   << "): " << std::endl;
+    }}
+
+  return output_stream.str();
+}

src/libkiter/algorithms/analysis/cycle_detection.h

@@ -0,0 +1,48 @@
+/*
+ * cycle_detection.h
+ *
+ *  Created on: 29 Dec 2021
+ *      Author: toky
+ */
+
+#ifndef SRC_LIBKITER_ALGORITHMS_ANALYSIS_CYCLE_DETECTION_H_
+#define SRC_LIBKITER_ALGORITHMS_ANALYSIS_CYCLE_DETECTION_H_
+
+
+#include <commons/KiterRegistry.h>
+#include <algorithms/schedulings.h>
+
+namespace models {
+	class Dataflow;
+}
+
+namespace algorithms {
+
+bool has_selfloops         (models::Dataflow* const  dataflow);
+bool has_cycles            (models::Dataflow* const  dataflow);
+void cycle_detection       (models::Dataflow* const  dataflow, parameters_list_t);
+  void find_simple_cycles(models::Dataflow* const  dataflow, parameters_list_t);
+  ARRAY_INDEX find_smallest_index(models::Dataflow* const dataflow,
+                                  std::map<int, std::vector<ARRAY_INDEX>> scc_map);
+  models::Dataflow* generate_scc(models::Dataflow* const dataflow,
+                                 std::map<int, std::vector<ARRAY_INDEX>> scc_map,
+                                 ARRAY_INDEX start_id);
+  models::Dataflow* generate_subgraph(models::Dataflow* const dataflow,
+                                      ARRAY_INDEX lower_limit);
+  void unblock(ARRAY_INDEX id);
+  bool find_simple_cycles_scc(models::Dataflow* const scc,
+                              ARRAY_INDEX start_id, ARRAY_INDEX current_id);
+  std::string print_status(models::Dataflow* const dataflow);
+
+
+}
+
+
+ADD_TRANSFORMATION(CycleDetection,
+                   transformation_t({ "CycleDetection" , "Returns 1 if there is a cycle in the graph and 0 if not..", algorithms::cycle_detection}));
+ADD_TRANSFORMATION(CycleCount,
+                   transformation_t({ "CycleCount" , "Count number of simple cycles in graph.", algorithms::find_simple_cycles}));
+
+
+
+#endif /* SRC_LIBKITER_ALGORITHMS_ANALYSIS_CYCLE_DETECTION_H_ */

src/libkiter/algorithms/dse/dse.cpp

@@ -8,7 +8,9 @@
 
 #include <map>
 #include <vector>
+#include <printers/stdout.h>
 #include <commons/verbose.h>
+#include <printers/SDF3Wrapper.h> // to write XML files
 #include <commons/commons.h>
 #include <models/Dataflow.h>
 #include <models/EventGraph.h>
@@ -31,6 +33,8 @@ void algorithms::throughput_buffering_tradeoff_dse(models::Dataflow* const dataf
   bool isBaseMonoOpt = false;
   bool thrTargetSpecified = false;
   bool modelBoundedBuffers = true;
+  bool useCoarse = false;
+  int coarseMultiplier = 2;
   std::string dirName = "./data/"; // default
 
   // parse parameters for KDSE
@@ -59,6 +63,10 @@ void algorithms::throughput_buffering_tradeoff_dse(models::Dataflow* const dataf
       parameters.find("SYMB_EXEC_CORRECTED") != parameters.end()) {
     modelBoundedBuffers = false;
   }
+  if (parameters.find("COARSE") != parameters.end()) { // use coarse step sizes (multiply by certain amount)
+    useCoarse = true;
+    coarseMultiplier = std::stoi(parameters["COARSE"]);
+  }
 
 
 
@@ -275,6 +283,9 @@ void algorithms::throughput_buffering_tradeoff_dse(models::Dataflow* const dataf
   else {
     VERBOSE_WARNING("No DSE supported method specified; ending search.");
   }
+  if (useCoarse) {
+    methodName += "_coarse";
+  }
 
 
   /* Initialise set of minimal storage distributions
@@ -403,8 +414,13 @@ void algorithms::throughput_buffering_tradeoff_dse(models::Dataflow* const dataf
           VERBOSE_DSE("\tFound storage dependency in channel "
                       << dataflow_prime->getEdgeName(*it) << std::endl);
           // make new modelled storage distribution according to storage dependencies
-          newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
-                                           minStepSizes[*it]));
+          if (!useCoarse) {
+            newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
+                                             minStepSizes[*it]));
+          } else {
+            newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
+                                             (minStepSizes[*it] * coarseMultiplier)));
+          }
           VERBOSE_DSE("\t\tIncreasing channel size of "
                       << dataflow_prime->getEdgeName(*it) << " to "
                       << newDist.getChannelQuantity(*it) << std::endl);
@@ -416,8 +432,13 @@ void algorithms::throughput_buffering_tradeoff_dse(models::Dataflow* const dataf
         VERBOSE_DSE("\tFound storage dependency in channel "
                     << dataflow->getEdgeName(*it) << std::endl);
         // make new modelled storage distribution according to storage dependencies
-        newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
-                                         minStepSizes[*it]));
+          if (!useCoarse) {
+            newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
+                                             minStepSizes[*it]));
+          } else {
+            newDist.setChannelQuantity(*it, (newDist.getChannelQuantity(*it) +
+                                             (minStepSizes[*it] * coarseMultiplier)));
+          }
         VERBOSE_DSE("\t\tIncreasing channel size of "
                     << dataflow->getEdgeName(*it) << " to "
                     << newDist.getChannelQuantity(*it) << std::endl);