Worfklow management through the AiiDA REST API

[ltalirz]

So far, the scope of the AiiDA REST API is limited to queries of the AiiDA database.

This already enables tools like the provenance explorer to be written in Javascript and run in your browser (now open source, by the way!). When building web interfaces around AiiDA that can also submit workflows, however, one currently has to find other ways, e.g.

1. use python dashboarding solutions like ipython widgets (as on AiiDAlab), panel (as e.g. on matscreen), plotly dash, ...
2. write your own specialized API for it like @flavianojs did with aiida-post for the AiiDA-Ginestra integration

The python dashboarding approach usually wins in terms of "time to first visualization/functionality", but does so at the cost of mixing data, visualization and controlling logic, which can make the components more difficult to reuse and test (see also some recent slides of mine on the topic).
The second approach via dedicated APIs offers potentially better separation of concerns and scalability - at the cost of having to implement your own API and client.

The purpose of this thread is to collect input on whether others have ongoing or planned work in this direction (thinking e.g. of @blokhin ) and whether these efforts would benefit from a generalized AiiDA REST API that supports workflow management.

On January 27th, 4pm Swiss time, @flavianojs and me will give a ~30 min presentation on the topic in the AiiDA team zoom channel. Flaviano will introduce AiiDA-post and I will outline a proposal of how a generalization of the AiiDA REST API towards workflow management could look like.
Let us know if you would like to join; you can already find the draft of the slides here (comments enabled).

@aiidateam/plugin-developers

[greschd]

Sounds interesting, I'd like to join the talk.

[JPchico]

This is a very interesting topic, is it possible to join this talk?

[ltalirz]

Links have been sent out; see you in ~1h

[ezpzbz]

Hi @ltalirz Sorry for the late sign up. Is there any free sits available to join ;) ?

[ltalirz]

absolutely!

[ltalirz]

Thanks to everyone for participating. Here comes a summary of the suggestions on the proposal

Here is a short summary of your input:

• Authentication: think about how fine-grained the access should be (is it enough to authenticate on an endpoint+method level?)
• While processes expose their inputs via the spec, this is not formalized for Data nodes - may need custom translators for different node types (or a refactor)
• Envisioned use cases:
  • Materials CLoud tools that want to launch a workflow
  • Exposing data (& the capability to run workflows) to other teams inside a company or research institute
• Client languages so far:
  • AiiDA-ginestra: Java
  • Aiida-explorer VSCode plugin: typescript

Will follow up with a short AEP (essentially outlining the points from the presentation) once I have time

[kjappelbaum]

@ltalirz great that you brought this up! I'd be happy to be involved in the discussion, the ELN-AiiDAlab integration might also be a nice use case

[ltalirz]

During this year's Google Summer of Code, our student @NinadBhat worked on implementing the aiida-restapi plugin, which adds support for workflow management through the REST API.
Thanks to work by @chrisjsewell , the package also contains a fully-fledged GraphQL endpoint (so far only queries supported; PR for mutations is open).

I should stress that this is a prototype implementation and that there are still a couple of open questions, but if you are interested in this topic, now would be a great time to take a look at the package; think about whether it addresses your use cases and how you would use it.

Slides showing basic usage & open questions here

[ltalirz]

Just wanted to mention that there's some renewed interest in this topic on our side.

aiida-restapi has just been made compatible with AiiDA 2.0.
For a tutorial on how to use it, see https://aiida-restapi.readthedocs.io/en/latest/user_guide/tutorial.html

If you are interested in this topic, contact @sphuber or me

[JPchico]

This is very nice indeed! I took a look at how it looks right now and I think there is quite a bit of potential.
One question though, previously there was a querybuild endpoint that allowed one to make requests for specific data, I guess that that will be taken over now by the grahpql endpoint right? Would it work in similar fashion? I.e. would we be able to send a normal get request and get a JSON response via curl or similar?

[ltalirz]

Yes, the implementation of the /graphql endpoint in aiida-restapi goes through the querybuilder and is very similar in functionality to the /querybuilder endpoint in aiida-core (you can already play around with it).

One benefit of GraphQL vs the /querybuilder endpoint would be that there is nice tooling around GraqhQL, e.g. plugins for VSCode that know your schema and help you write queries with autocompletion etc. Furthermore, GraphQL can be used to mutate data (not just query it), and @chrisjsewell already opened a pull request that would allow using the /graphql endpoint to create entities in the AiiDA database. There is just an outstanding question about how to handle file uploads for node types that have associated files/objects.

The benefit of a querybuilder/GraphQL approach vs a REST API is that the client decides which information to ask for, while in a REST API the implementer of the API has to decide which information to include. This is particularly relevant when querying graphs, where we are often interested in information from linked nodes as well.

For example: you want to build a dashboard for verdi process list, and for each process you would like to display if there are any reports. The /processes endpoint of the REST API does not include this information, so after querying the /processes endpoint to get the list of N processes you want to display, you then need to do N additional queries to get the reports for each of these (this is also known as the N+1 problem), and depending on how this query is implemented it may force you to download the content of the reports although you are only interested in their number. In GraphQL you can get this information in a single request, and it will give you exactly the information you asked for.

There are also downsides to graphql - since the server-side API is less prescriptive, simple requests involve a little bit more code (you need to tell it what you want). I read that, historically, error handling has been less granular (simply returns error code 200 if query fails; not sure whether this has changed in the meanwhile), and then there is the issue of handling files. However, I think the example in the aiida-restapi docs illustrates the power of this interface.

[JPchico]

This is very very good. I remember reading about some of these benefits of the graphql but I have not played too much with it. But I think it will be worth it to put some time on it, since as you describe some of those requests that would require a very large number of queries would be streamlined quite a bit. And of course this will mean that visualizing data will be much easier with this new approach.

[ltalirz]

P.S. Here are examples illustrating how to migrate aiida-core REST API queries to the /graphql endpoint in the plugin.

[ltalirz]

Thanks to @sphuber for adding an example demonstrating how to use aiida-restapi to submit a PwBaseWorkChain with aiida-quantumespresso!
https://github.com/aiidateam/aiida-restapi/tree/master/examples/submit_quantumespresso_pw

[sphuber]

Note that this requires the latest master branch of aiida-restapi to work.