Model list parsing

[bennahugo]

@JSKenyon I'm wondering how best to proceed with this. The current format for assigning models into directions is sort of

COL | LSM (+~? COL | LSM(@TAG)?)*(:COL | LSM (+~? COL | LSM(@TAG)?)*)*

where LSM must end in .lsm.html

This forces the user to use the html format for instance when using lsms, as it is done in cubical. Because backwards compatibility is needed for cubical this can't really be changed. However, as we are starting afresh with multiple models (including lsm, DicoModels and maybe even the wsclean standard) I propose that we use functional syntax to denote how the components must be loaded, e.g.

column("MODEL_DATA")
lsm("mylsm.txt") | lsm("mylsm.lsm.html")
ddfacet("mydico.DicoModel")
wsclean("mysky.txt")

with an additional specifier for tags, e.g.

tag("dE")
regions("myds9.reg")

and an additional specifier of how to predict, dft gives the current implementation and fft gives a targetted facetted implementation

dft(lsm("dft_components.lsm.html")@tag(dE))
fft(lsm("dft_components.lsm.html")@tag(dE))

where you can for instance do

fft(lsm("my_components.lsm.html")) 

to encompass all the components in the sky model into a rectangle split into multiple facets to degrid and FT, or use

fft(lsm("my_components.lsm.html")@tag(dE)) to degrid and FT each of the components on a minimum sized facet

simularly

fft(lsm("dft_components.lsm.html")@regions("dE.reg"))

to split components into regions which are carved into facets, degridded and FT'd

Then you can combine it more unambigiously, e.g.

column("MODEL_DATA")+~dft(lsm("dft_components.lsm.html")@tag(dE)):dft(lsm("dft_components.lsm.html")@tag(dE))
column("MODEL_DATA")+~fft(ddfacet("clean_components.DicoModel")@regions("directions.reg")):fft(ddfacet("clean_components.DicoModel")@regions("directions.reg")))
column("MODEL_DATA")+~fft(lsm("dft_components.txt")@regions("directions.reg")):fft(lsm("dft_components.txt")@regions("directions.reg")))

[bennahugo]

@o-smirnov maybe you will have comment on this as well?

[JSKenyon]

I do think that this is something worth considering. My greatest concern with the input model specification is how daunting it looks/feels for new users. I think your suggestion has merit @bennahugo as it is definitely clearer, and the expense of being less concise. I am not particularly against being verbose and explicit though. I think that we shouldn't rush to implement this right now though - lets think about it for a little while and maybe whiteboard some ideas.

[bennahugo]

maybe we could have defaults. I just don't like using the Windows convention of relying on extensions, esp if .txt can describe either a lsm csv or a wsclean csv model for instance

[JSKenyon]

Yeah, exactly. The current reliance on extensions is only there because I was basically just emulating CubiCal. @o-smirnov might have some bright ideas for us. We could also consider some shorthands for power users.

[o-smirnov]

Yeah the current simplistic syntax was meant to be just that, simplistic. The requirements of the predict are clearly outgrowing it.

I would still like to keep the common uses cases concise and simple. If the user just wants to peel off a source or use a couple of columns, they should not be needing to learn a whole syntax. I'm also a bit wary of developing a whole new mini-language for the purpose.

How about we provide a YaML specification for the model structure? Keep the current syntax for simple use cases (perhaps even paring down the more gothic and exotic bits), and let the power user provide a YaML file if they want to do really twisted stuff.

[bennahugo]

My concern about overloading options with other files is it makes the pipelining case harder than need be. What is one step becomes multiple steps to write out temporary argument files. We can perhaps make the dft/fft specification a list, one for each of the model types?

[o-smirnov]

Well you can always parse YaML from the command line, right -- in the pipelining case, nobody is around to see the ugliness...