Independent function for starting the DM workflow after acquisition and nexus file writing is complete

[qzhang234]

We would like to have an independent function for starting the DM workflow after the acquisition is done and the nexus hdf metadata file is written.

We would prefer to have a plain python function independent from Bluesky RE (and potentially apstools, if possible) so that it is easier for beamline folks to develop, maintain and debug.

[qzhang234]

@banubot provided a template for starting DM from the DM's python environment. Looks like the only input variables needed are in argsDict and there are only 3 of them. Would Bluesky be able to communicate these variables with Hannah's DM environment, either via text files or String Registers? By the way all the fields Hannah needs are currently being written in the StrReg in my acquisition code

# This script is an example of how to start 
# the XPCS boost corr workflow using the DM python API
# Usage: source /home/dm/etc/dm.setup.sh; python dm_api_example.py

from dm.proc_web_service.api.procApiFactory import ProcApiFactory


# Get the DM workflow API object
dmWorkflowApi = ProcApiFactory.getWorkflowProcApi()

# Define the workflow arguments in a dict
argsDict = {
    "filePath": "H001_005_test_Feb_7-01000.h5",
    "qmap": "eiger4M_qmap_d36_s360.h5",
    "experimentName": "zhang202402",
}

workflowOwner = "8idiuser"
workflowName = "xpcs8-02-gladier-boost"
# Start the processing job
response = dmWorkflowApi.startProcessingJob(workflowOwner, workflowName, argsDict)
# Print the job ID
print(response.get('id'))

# To do the same thing using CLI instead of python, run:
# dm-start-processing-job --workflow-name xpcs8-02-gladier-boost filePath:H001_005_test_Feb_7-01000.h5 qmap:eiger4M_qmap_d36_s360.h5 experimentName:zhang202402

[prjemian]

Bluesky has a different set of Python package requirements than APS Data Management. The bluesky instrument package has provided an adaptation that provides access to the full set of Python software provided by APS DM. The adaptation is enabled by defining all the environment variables required by DM:

bluesky/src/instrument/utils/aps_data_management.py

Line 272 in 6bbcfec

def dm_source_environ():

Various DM APIs are already provided:

bluesky/src/instrument/utils/aps_data_management.py

Lines 8 to 14 in 6bbcfec

	~dm_api_cat
	~dm_api_daq
	~dm_api_dataset_cat
	~dm_api_ds
	~dm_api_file
	~dm_api_file_cat
	~dm_api_proc

For example, dm_api_cat:

bluesky/src/instrument/utils/aps_data_management.py

Lines 148 to 153 in 6bbcfec

	def dm_api_cat():
	"""Return the APS Data Management Catalog API object."""
	from dm import CatApiFactory
	dm_source_environ()
	return CatApiFactory.getRunCatApi()

[prjemian]

Since dm_source_environ() is run when starting a bluesky session, @banubot's example above can be run from a bluesky session.

[prjemian]

@qzhang234 Is there any additional work to be done to resolve this issue?

[qzhang234]

Hi Pete: Allow me to do some more research. In the meantime please keep the issue open. Thank you, QZ

…

________________________________ From: Pete R Jemian ***@***.***> Sent: Wednesday, November 6, 2024 1:29:16 PM To: aps-8id-dys/bluesky ***@***.***> Cc: Zhang, Qingteng ***@***.***>; Mention ***@***.***> Subject: Re: [aps-8id-dys/bluesky] Independent function for starting the DM workflow after acquisition and nexus file writing is complete (Issue #104) @ qzhang234 Is there any additional work to be done to resolve this issue? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <aps-8id-dys/bluesky/issues/104/2460602265@ github. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd @qzhang234<https://urldefense.us/v3/__https://github.com/qzhang234__;!!G_uCfscf7eWS!dDSMVF3PVwvwlKNN-d2VctH0QTmar6GBE4mx4qWllIZbPZfMt46MAaSrl_LSpfQPVe7xy6b-kL2_1YK1O1QDaPWeR4r6QQ$> Is there any additional work to be done to resolve this issue? — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/aps-8id-dys/bluesky/issues/104*issuecomment-2460602265__;Iw!!G_uCfscf7eWS!dDSMVF3PVwvwlKNN-d2VctH0QTmar6GBE4mx4qWllIZbPZfMt46MAaSrl_LSpfQPVe7xy6b-kL2_1YK1O1QDaPUQEEN2dw$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/ALPJBQVCBIFHWBVJKIV3EZDZ7JUYZAVCNFSM6AAAAABRFBYE5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRQGYYDEMRWGU__;!!G_uCfscf7eWS!dDSMVF3PVwvwlKNN-d2VctH0QTmar6GBE4mx4qWllIZbPZfMt46MAaSrl_LSpfQPVe7xy6b-kL2_1YK1O1QDaPWhA9St_A$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[prjemian]

A low-level plan (bluesky plan stub that wraps DMs startProcessingJob() function) is described in the bs_model_instrument. This is part of PR BCDA-APS/bs_model_instrument#20.