Replies: 2 comments 6 replies
-
2021-11: ASSESS |
Beta Was this translation helpful? Give feedback.
-
2024-08: CLOSED |
Beta Was this translation helpful? Give feedback.
-
|
Not currently being used, was used in very early pipelines. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe this should be set RETAIN/ REPLACE since we're still using them in some places . @davidbridgwood what's happening to the step functions in oasys/delius extraction? @matt-heery what about in EM extraction? |
Beta Was this translation helpful? Give feedback.
-
|
They don't exist anymore for delius/oasys as they pretty much exclusively covered the curation process which is now in CaDT. Extractions done by DMS and lambdas triggered by files landing in buckets. I'm not against step-functions in principal though - found them pretty useful and easy to setup when I needed them. |
Beta Was this translation helpful? Give feedback.
-
|
we are still using them to move data over to the AP (temporary solution as we will be using lake formation in the future), theyre very easy to set up. i dont think they should be used for data pipelines (Airflow is a much better orchestrator). if we are using lambda functions and chaining them together - i think they are really useful and can help split out the functionality of the lambda, so we can overcome the 15 min limit more easily. imo retain is good (but they are quite good to use for workflows eg in our hackathon project) |
Beta Was this translation helpful? Give feedback.
-
|
I'm wondering if the Tech Radar should be limited to what we use in production? I'll raise on slack channel :-) |
Beta Was this translation helpful? Give feedback.
-
https://aws.amazon.com/step-functions/
Beta Was this translation helpful? Give feedback.
All reactions