Replace compile time configuration with runtime configuration

[Quincunx271]

We have a lot of configuration, especially for debug, which is gated by individual macros. I think that many of these could be made runtime configurable rather than compile-time configurable, possibly with a single macro to guard it rather than individual macros for each feature.

What I'm thinking:

• Stuff not in hot code should be runtime configurable (if (SomeConfigurationOption) rather than #ifdef SOME_CONFIGURATION_OPTION). This could be read from our sched.ini file or elsewhere.
• Stuff in hot code (the actual internals of our algorithms) should not be solely runtime configurable, as that could be too expensive when the option is turned off. I think making it runtime configurable but also guarded with #ifdef IS_DEBUG_EXTRA_INFO (or something) could be a good idea.

Why I want this:

• I frequently forget to change the compiletime configuration flags because they are in a different location than the runtime sched.ini options. This means that I often have to redo runs because I forgot to turn on something I needed, and it also means that some of my runs have much more information than is necessary.
• This would make it easier to reproduce results from the past. It's easier to have less places with configuration and instead have all the configuration in one place.

[vangthao95]

I agree with this. We can possibly create a new file called debug.ini or print.ini with all the configurations.

[kerbowa]

What about using LLVM's '-debug-only'. I'm already doing this some places. These flags are done at the module level so everything within the LLVM_DEBUG() macro gets executed when you pass -debug-only=foo.

[Quincunx271]

I don't know that this is the best way to do this. What I do know is that we have configuration in many different places (CMake configuration (build-time), sched.ini, SPEC .cfg files, etc.), and it would be best to come up with a more coherent configuration story.

#ifdefs generally make code much harder to understand, so I stand by removing them where possible. But now that the compilation times are much lower due to externalized LLVM builds, OptSched can be quickly rebuilt with different configuration options, so build-time configuration is not as cumbersome as before; each config can live in a different build location.