Automation of proto R/W in hierarchies

[brunoguindani]

Something that just crossed my mind. Do we have any way to automatize this process? maybe through a standardized interface for protos? The functions read/write_state_to_proto() and write_hypers_to_proto() are really the only low-level hierarchy operation left to the user's manual implementation. It would be amazing if they didn't have to do that.

[mberaha]

This is extremely interesting, but kind of far fetched at the moment. The problem is that a proto is only as good as its submessages, one could have a general proto like

message GeneralProto {
 map<string, double> scalarparams = 1;
 map<string, Vector> vectorparams = 2;
 map<string, Matrix> matrixparams = 3;
}

this is likely to fit all the possible states and hypers that we want to work with.
But then another problem comes along, suppose you have a simple struct like struct State {double a, b;} and want to write that struct to a GeneralProto using some call to a template function like

template<typename T>
void write_to_general_proto(const T& t, GeneralProto* out);

and here's where the fun begins: how do you do it? In Python, one can iterate through the elements of T (this is called reflection), but C++ has not such feature by default, i.e. in python I would do

def write_to_general_proto(obj, out: GeneralProto, prefix=""):
  for fieldname, fieldval in obj.__dict__.values():
     if isinstance(fieldval, double):
       out.scalarparms[prefix + fieldname] = fieldval # or something along this line
     elif....

     else:
       write_to_general_proto(fieldval, out, fieldname + ".")

where the last else should be able to handle nested objects

There are some tools to add reflection (one that I know of is https://github.com/garbageslam/visit_struct) but before settling for one, I would see what is the ecosystem available.

On the other hand, we could get rid of proto, and use some other C++ based serialization library only meant for C++. This would make life for pybmix a little harder, since then one needs to fine another workaround to pass the chains back and forth, but I'm open to discussion

[brunoguindani]

Yeah, I figured it all comes down to looping on a struct's fields. Python dictionaries really spoiled us huh.
Protobufs are kinda nice TBH, especially for iterfacing with Python, and I wouldn't get rid of them. Also, finding a reflection tool sounds promising. Possibly, could we outsource the writing to another programming language? Would it be too difficult, or computationally expensive?

[mberaha]

I'm not sure what you mean by "could we outsource the writing to another programming language?"

What is it you have in mind?

[brunoguindani]

Like, embedding lines of another programming language (Python?) to perform that operation. I don't have anything particular in mind, I don't even know if it's possible or efficient. Just brainstorming.

[mberaha]

I don't see how this could work... An object is a sequence of bytes, the programming language knows how to interpret those bytes.