From 206a9236799b669f68f91b0ebeb3454111718e51 Mon Sep 17 00:00:00 2001 From: richard gowers Date: Wed, 27 Mar 2024 14:02:29 +0000 Subject: [PATCH 1/7] docs: first pass at working with results page --- docs/guide/results/index.rst | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index 69761d850..a85f80374 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -3,6 +3,36 @@ Working with Results ==================== -This section is still under construction. In the meantime, please -contact us on our `issue tracker `_ -if you have any questions about analysing your results! +With :ref:`execution of your calculations ` completed, +we can now start looking at what has been produced. +At the very least, Protocols will produce estimates of free energy differences of the ``ChemicalSystem`` \s +(with the exception of MD Protocols which just simulate the behaviour of a single system). +Beyond this, the exact data produced by a given Protocol can vary significantly, +for example the HREX protocol will produce graphs to assess the quality of the simulation, +while the MD Protocol will produce a trajectory file for the simulated system. +For exact details on what is produced consult the :ref:`pages for each Protocol`. + +.. todo crossref to HREX and MD Protocol docs from issue 743 + +How you can inspect these results depends on whether you are doing this from the Python or command line interface. + +From a Python interface +----------------------- + +Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, +representing a single iteration of estimating the free energy difference. +One or more of these can be put into the ``.gather()`` method of the ``Protocol`` to form a :class:`.ProtocolResult`, +this class takes care of the averaging and concatenation of different iterations of the estimation process. +This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates +of free energy difference along with its uncertainty. + +From the command line +--------------------- + +If you had executed your calculation using the :ref:`quickrun ` command, +then a ``.json`` results log file as well as a directory of files will have been produced. +Most importantly, the ``.json`` results file has ``estimate`` and ``uncertainty`` keys, +which serve the same purpose as the ``get_estimate()`` and ``get_uncertainty()`` methods described above. + +The :ref:`openfe gather ` command offers a way to collate information across many different individual +simulations and prepare a table of results. From b98cfb413cf26e87588e738cea1cb09dd92523e8 Mon Sep 17 00:00:00 2001 From: richard gowers Date: Tue, 2 Apr 2024 09:58:15 +0100 Subject: [PATCH 2/7] docs: add details on loading json result files reorder to mention command line results first as this is the easier route to follow --- docs/guide/results/index.rst | 40 ++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index a85f80374..3196c2a83 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -14,17 +14,7 @@ For exact details on what is produced consult the :ref:`pages for each Protocol< .. todo crossref to HREX and MD Protocol docs from issue 743 -How you can inspect these results depends on whether you are doing this from the Python or command line interface. - -From a Python interface ------------------------ - -Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, -representing a single iteration of estimating the free energy difference. -One or more of these can be put into the ``.gather()`` method of the ``Protocol`` to form a :class:`.ProtocolResult`, -this class takes care of the averaging and concatenation of different iterations of the estimation process. -This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates -of free energy difference along with its uncertainty. +How you can inspect these results depends on whether you are doing this from the command line or a Python script. From the command line --------------------- @@ -34,5 +24,33 @@ then a ``.json`` results log file as well as a directory of files will have been Most importantly, the ``.json`` results file has ``estimate`` and ``uncertainty`` keys, which serve the same purpose as the ``get_estimate()`` and ``get_uncertainty()`` methods described above. +The ``json`` results file can be loaded into a Python session as:: + + >>> import gufe + >>> import json + >>> + >>> with open('././Transformation-97d7223f918bbdb0570edc2a49bbc43e_results.json', 'r') as f: + ... results = json.load(f, cls=gufe.tokenization.JSON_HANDLER.decoder) + >>> results['estimate'] + -19.889719513104342 + >>> results['uncertainty'] + 0.574685524681712 + + +See Also +~~~~~~~~ + The :ref:`openfe gather ` command offers a way to collate information across many different individual simulations and prepare a table of results. + + +From a Python interface +----------------------- + +Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, +representing a single iteration of estimating the free energy difference. +One or more of these can be put into the ``.gather()`` method of the ``Protocol`` to form a :class:`.ProtocolResult`, +this class takes care of the averaging and concatenation of different iterations of the estimation process. +This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates +of free energy difference along with its uncertainty. + From c65a33e01503199a47cc27f8e8ee234ba4f4f3c7 Mon Sep 17 00:00:00 2001 From: Richard Gowers Date: Thu, 4 Apr 2024 09:31:55 +0100 Subject: [PATCH 3/7] docs: add stub file for issue #803 --- docs/guide/results/working_with_networks.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 docs/guide/results/working_with_networks.rst diff --git a/docs/guide/results/working_with_networks.rst b/docs/guide/results/working_with_networks.rst new file mode 100644 index 000000000..b227fc6e7 --- /dev/null +++ b/docs/guide/results/working_with_networks.rst @@ -0,0 +1,10 @@ +.. _userguide_result_networks: + +Working with networks of results +================================ + +issue #803 + + +The :ref:`openfe gather ` command offers a way to collate information across many different individual +simulations and prepare a table of results. From 26c54adee3390fbb90f7c3f1e0e3a9ebc30559f5 Mon Sep 17 00:00:00 2001 From: Richard Gowers Date: Thu, 4 Apr 2024 09:32:55 +0100 Subject: [PATCH 4/7] docs: rework results docs to make it clearer on cli vs python execution add reference to working with results networks docs --- docs/guide/results/index.rst | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index 3196c2a83..5596a456d 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -6,25 +6,28 @@ Working with Results With :ref:`execution of your calculations ` completed, we can now start looking at what has been produced. At the very least, Protocols will produce estimates of free energy differences of the ``ChemicalSystem`` \s -(with the exception of MD Protocols which just simulate the behaviour of a single system). +(with the exception of PlainMDProtocol which just simulate the behaviour of a single system). Beyond this, the exact data produced by a given Protocol can vary significantly, -for example the HREX protocol will produce graphs to assess the quality of the simulation, +for example the RelativeHybridTopologyProtocol protocol will produce graphs to assess the quality of the simulation, while the MD Protocol will produce a trajectory file for the simulated system. For exact details on what is produced consult the :ref:`pages for each Protocol`. .. todo crossref to HREX and MD Protocol docs from issue 743 -How you can inspect these results depends on whether you are doing this from the command line or a Python script. +How you can inspect these results depends on whether you have executed your simulations +from the command line or a Python script. -From the command line ---------------------- +From command line execution +--------------------------- If you had executed your calculation using the :ref:`quickrun ` command, then a ``.json`` results log file as well as a directory of files will have been produced. +This directory will have various plots and results of analysis, the exact details of which are described +in the :ref:`pages for each Protocol`. + Most importantly, the ``.json`` results file has ``estimate`` and ``uncertainty`` keys, which serve the same purpose as the ``get_estimate()`` and ``get_uncertainty()`` methods described above. - -The ``json`` results file can be loaded into a Python session as:: +The full ``json`` results file can be reloaded into a Python session as:: >>> import gufe >>> import json @@ -37,15 +40,9 @@ The ``json`` results file can be loaded into a Python session as:: 0.574685524681712 -See Also -~~~~~~~~ - -The :ref:`openfe gather ` command offers a way to collate information across many different individual -simulations and prepare a table of results. - -From a Python interface ------------------------ +From Python execution +--------------------- Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, representing a single iteration of estimating the free energy difference. @@ -54,3 +51,8 @@ this class takes care of the averaging and concatenation of different iterations This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates of free energy difference along with its uncertainty. +See Also +-------- + +For how to deal with multiple results forming a network consult the :ref:`working with networks` +page. From 849a990526d0f793aca395027b939b36ffd1d74d Mon Sep 17 00:00:00 2001 From: Richard Gowers Date: Fri, 5 Apr 2024 11:50:34 +0100 Subject: [PATCH 5/7] Update docs/guide/results/index.rst Co-authored-by: Irfan Alibay --- docs/guide/results/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index 5596a456d..01e75e096 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -5,8 +5,8 @@ Working with Results With :ref:`execution of your calculations ` completed, we can now start looking at what has been produced. -At the very least, Protocols will produce estimates of free energy differences of the ``ChemicalSystem`` \s -(with the exception of PlainMDProtocol which just simulate the behaviour of a single system). +The majority of Protocols will produce estimates of free energy differences between two or more ``ChemicalSystem`` \s +(the current exception being the :class:`.PlainMDProtocol` which just simulates the dynamics of a single system). Beyond this, the exact data produced by a given Protocol can vary significantly, for example the RelativeHybridTopologyProtocol protocol will produce graphs to assess the quality of the simulation, while the MD Protocol will produce a trajectory file for the simulated system. From 8948e1c864bf98c448c0e3fb731b0b832dd7aae1 Mon Sep 17 00:00:00 2001 From: Richard Gowers Date: Fri, 5 Apr 2024 11:50:42 +0100 Subject: [PATCH 6/7] Apply suggestions from code review Co-authored-by: Irfan Alibay --- docs/guide/results/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index 01e75e096..1435c86d4 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -8,8 +8,8 @@ we can now start looking at what has been produced. The majority of Protocols will produce estimates of free energy differences between two or more ``ChemicalSystem`` \s (the current exception being the :class:`.PlainMDProtocol` which just simulates the dynamics of a single system). Beyond this, the exact data produced by a given Protocol can vary significantly, -for example the RelativeHybridTopologyProtocol protocol will produce graphs to assess the quality of the simulation, -while the MD Protocol will produce a trajectory file for the simulated system. +for example the :class:`.RelativeHybridTopologyProtocol` protocol will produce graphs to assess the quality of the simulation, alongside trajectory data files. +By comparison, the :class:`.PlainMDProtocol` will only produce the latter. For exact details on what is produced consult the :ref:`pages for each Protocol`. .. todo crossref to HREX and MD Protocol docs from issue 743 From 77c8ae2cd543a507580764dd1e5f61ca7992e989 Mon Sep 17 00:00:00 2001 From: richard gowers Date: Fri, 5 Apr 2024 11:56:43 +0100 Subject: [PATCH 7/7] docs: add toctree to userguide results rename current working with results page to inside new toctree --- docs/guide/results/index.rst | 60 +++------------------ docs/guide/results/working_with_results.rst | 57 ++++++++++++++++++++ 2 files changed, 64 insertions(+), 53 deletions(-) create mode 100644 docs/guide/results/working_with_results.rst diff --git a/docs/guide/results/index.rst b/docs/guide/results/index.rst index 1435c86d4..e539879e2 100644 --- a/docs/guide/results/index.rst +++ b/docs/guide/results/index.rst @@ -1,58 +1,12 @@ .. _userguide_results: Working with Results -==================== +===================== -With :ref:`execution of your calculations ` completed, -we can now start looking at what has been produced. -The majority of Protocols will produce estimates of free energy differences between two or more ``ChemicalSystem`` \s -(the current exception being the :class:`.PlainMDProtocol` which just simulates the dynamics of a single system). -Beyond this, the exact data produced by a given Protocol can vary significantly, -for example the :class:`.RelativeHybridTopologyProtocol` protocol will produce graphs to assess the quality of the simulation, alongside trajectory data files. -By comparison, the :class:`.PlainMDProtocol` will only produce the latter. -For exact details on what is produced consult the :ref:`pages for each Protocol`. +With simulations completed, +the results of individual simulations can be inspected, +and entire networks of results analysed. -.. todo crossref to HREX and MD Protocol docs from issue 743 - -How you can inspect these results depends on whether you have executed your simulations -from the command line or a Python script. - -From command line execution ---------------------------- - -If you had executed your calculation using the :ref:`quickrun ` command, -then a ``.json`` results log file as well as a directory of files will have been produced. -This directory will have various plots and results of analysis, the exact details of which are described -in the :ref:`pages for each Protocol`. - -Most importantly, the ``.json`` results file has ``estimate`` and ``uncertainty`` keys, -which serve the same purpose as the ``get_estimate()`` and ``get_uncertainty()`` methods described above. -The full ``json`` results file can be reloaded into a Python session as:: - - >>> import gufe - >>> import json - >>> - >>> with open('././Transformation-97d7223f918bbdb0570edc2a49bbc43e_results.json', 'r') as f: - ... results = json.load(f, cls=gufe.tokenization.JSON_HANDLER.decoder) - >>> results['estimate'] - -19.889719513104342 - >>> results['uncertainty'] - 0.574685524681712 - - - -From Python execution ---------------------- - -Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, -representing a single iteration of estimating the free energy difference. -One or more of these can be put into the ``.gather()`` method of the ``Protocol`` to form a :class:`.ProtocolResult`, -this class takes care of the averaging and concatenation of different iterations of the estimation process. -This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates -of free energy difference along with its uncertainty. - -See Also --------- - -For how to deal with multiple results forming a network consult the :ref:`working with networks` -page. +.. toctree:: + working_with_results + working_with_networks diff --git a/docs/guide/results/working_with_results.rst b/docs/guide/results/working_with_results.rst new file mode 100644 index 000000000..d7d4bfb43 --- /dev/null +++ b/docs/guide/results/working_with_results.rst @@ -0,0 +1,57 @@ +.. _userguide_individual_results: + +Working with individual results +=============================== + +With :ref:`execution of your calculations ` completed, +we can now start looking at what has been produced. +The majority of Protocols will produce estimates of free energy differences between two or more ``ChemicalSystem`` \s +(the current exception being the :class:`.PlainMDProtocol` which just simulates the dynamics of a single system). +Beyond this, the exact data produced by a given Protocol can vary significantly, +for example the :class:`.RelativeHybridTopologyProtocol` protocol will produce graphs to assess the quality of the simulation, alongside trajectory data files. +By comparison, the :class:`.PlainMDProtocol` will only produce the latter. +For exact details on what is produced consult the :ref:`pages for each Protocol`. + +.. todo crossref to HREX and MD Protocol docs from issue 743 + +How you can inspect these results depends on whether you have executed your simulations +from the command line or a Python script. + +From command line execution +--------------------------- + +If you had executed your calculation using the :ref:`quickrun ` command, +then a ``.json`` results log file as well as a directory of files will have been produced. +This directory will have various plots and results of analysis, the exact details of which are described +in the :ref:`pages for each Protocol`. + +Most importantly, the ``.json`` results file has ``estimate`` and ``uncertainty`` keys, +which serve the same purpose as the ``get_estimate()`` and ``get_uncertainty()`` methods described below. +The full ``json`` results file can be reloaded into a Python session as:: + + >>> import gufe + >>> import json + >>> + >>> with open('././Transformation-97d7223f918bbdb0570edc2a49bbc43e_results.json', 'r') as f: + ... results = json.load(f, cls=gufe.tokenization.JSON_HANDLER.decoder) + >>> results['estimate'] + -19.889719513104342 + >>> results['uncertainty'] + 0.574685524681712 + + +From Python execution +--------------------- + +Executing a :class:`.ProtocolDAG` using :func:`openfe.execute_DAG` will produce a :class:`.ProtocolDAGResult`, +representing a single iteration of estimating the free energy difference. +One or more of these can be put into the ``.gather()`` method of the ``Protocol`` to form a :class:`.ProtocolResult`, +this class takes care of the averaging and concatenation of different iterations of the estimation process. +This ``ProtocolResult`` class has ``.get_estimate()`` and ``.get_uncertainty()`` methods which return the estimates +of free energy difference along with its uncertainty. + +See Also +-------- + +For how to deal with multiple results forming a network consult the :ref:`working with networks` +page.