New agents guide and deployment config docs updates #4822

neverett · 2024-02-01T22:25:03Z

Why are the changes needed?

To prepare for Flyte agents GA.

What changes were proposed in this pull request?

New agents guide
Agents deployment docs updates
- New Airflow agent deployment doc
- New Snowflake agent deployment doc
- Minor updates to other agent deployment docs

Note: Links to new agents integrations docs are broken, since those docs are in a flytesnacks branch (see flyteorg/flytesnacks#1498).

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

flyteorg/flytesnacks#1498

Signed-off-by: nikki everett <[email protected]>

codecov · 2024-02-01T22:29:28Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6a2ce3f) 58.97% compared to head (3e05037) 58.93%.
Report is 34 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4822      +/-   ##
==========================================
- Coverage   58.97%   58.93%   -0.04%     
==========================================
  Files         644      645       +1     
  Lines       55146    55380     +234     
==========================================
+ Hits        32521    32638     +117     
- Misses      20050    20159     +109     
- Partials     2575     2583       +8

Flag	Coverage Δ
unittests	`58.93% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: nikki everett <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: nikki everett <[email protected]>

neverett · 2024-02-05T20:36:02Z

Not sure why the DCO check is failing, since all commits appear to be signed off.

Signed-off-by: nikki everett <[email protected]>

docs/flyte_agents/creating_an_agent.md

Signed-off-by: nikki everett <[email protected]>

neverett · 2024-02-13T23:03:22Z

docs/flyte_agents/using_agents_in_tasks.md

+
+If you need to connect to an external service in your workflow, we recommend using the corresponding agent rather than a web API plugin. Agents are designed to be scalable and can handle large workloads efficiently, and decrease load on FlytePropeller, since they run outside of it. You can also test agents locally without having to change the Flyte backend configuration, streamlining development.
+
+For a list of agents you can use in your tasks, see the [Agent Setup](https://docs.flyte.org/en/latest/deployment/agents/index.html) documentation.


Here it would be better to link to the Integrations section of the docs, specifically the External Service Backend Plugins section, but that section contains plugins that don't have a corresponding agent (e.g. AWS Athena). Should I add a new "Agents" section to that Integrations page?

Sounds good

docs/flyte_agents/creating_an_agent.md

neverett · 2024-02-13T23:05:55Z

docs/flyte_agents/creating_an_agent.md

+
+To create a new async agent, extend the `AgentBase` class in the `flytekit.backend` module and implement the `execute` methods. All calls must be idempotent.
+
+- `execute`: TK


Flagging this for more detail from eng.

Signed-off-by: nikki everett <[email protected]>

docs/flyte_agents/testing_agents_locally.md

* Add sync agent example Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: nikki everett <[email protected]>

* small update to start PR Signed-off-by: nikki everett <[email protected]> * add airflow and snowflake agent pages and update existing agents docs Signed-off-by: nikki everett <[email protected]> * small updates Signed-off-by: nikki everett <[email protected]> * [Doc] Airflow agent deployment (flyteorg#4831) Signed-off-by: Kevin Su <[email protected]> * [Doc] Snowflake agent deployment (flyteorg#4832) Signed-off-by: Kevin Su <[email protected]> * copyedits Signed-off-by: nikki everett <[email protected]> * move agents guide to flyte Signed-off-by: nikki everett <[email protected]> * rename directory and add guide to toctree Signed-off-by: nikki everett <[email protected]> * stub out async vs sync agents content Signed-off-by: nikki everett <[email protected]> * link to list of agents Signed-off-by: nikki everett <[email protected]> * edits for sync vs async agents Signed-off-by: nikki everett <[email protected]> * add visible TOC table to index page Signed-off-by: nikki everett <[email protected]> * update for async vs sync tasks Signed-off-by: nikki everett <[email protected]> * Update async agent example (flyteorg#4906) * Add sync agent example Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * formatting and copy updates Signed-off-by: nikki everett <[email protected]> * fix formatting in snowflake doc, small fixes in agents procedural docs Signed-off-by: nikki everett <[email protected]> * fix toctree Signed-off-by: nikki everett <[email protected]> * add link Signed-off-by: nikki everett <[email protected]> * fix links Signed-off-by: nikki everett <[email protected]> * update link Signed-off-by: nikki everett <[email protected]> * import blog content Signed-off-by: nikki everett <[email protected]> * rename agent creation doc, small content updates Signed-off-by: nikki everett <[email protected]> * fix article title Signed-off-by: nikki everett <[email protected]> * fix link Signed-off-by: nikki everett <[email protected]> * undo unnecessary formatting change in unrelated file Signed-off-by: nikki everett <[email protected]> --------- Signed-off-by: nikki everett <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]>

small update to start PR

4dd0bd0

Signed-off-by: nikki everett <[email protected]>

neverett added the documentation Improvements or additions to documentation label Feb 1, 2024

neverett requested review from ppiegaze, samhita-alla and pingsutw February 1, 2024 22:25

neverett self-assigned this Feb 1, 2024

dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Feb 1, 2024

neverett mentioned this pull request Feb 1, 2024

Agents docs updates flyteorg/flytesnacks#1475

Closed

add airflow and snowflake agent pages and update existing agents docs

d317113

Signed-off-by: nikki everett <[email protected]>

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Feb 2, 2024

nikki everett and others added 2 commits February 2, 2024 14:39

small updates

dd90f6b

Signed-off-by: nikki everett <[email protected]>

[Doc] Airflow agent deployment (#4831)

9aac0e8

Signed-off-by: Kevin Su <[email protected]>

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Feb 5, 2024

pingsutw and others added 2 commits February 5, 2024 14:06

[Doc] Snowflake agent deployment (#4832)

b1ea659

Signed-off-by: Kevin Su <[email protected]>

copyedits

eda8951

Signed-off-by: nikki everett <[email protected]>

nikki everett added 3 commits February 6, 2024 13:05

Merge branch 'master' into docs/agents-updates

e218d5a

move agents guide to flyte

3a251a7

Signed-off-by: nikki everett <[email protected]>

rename directory and add guide to toctree

f2de03e

Signed-off-by: nikki everett <[email protected]>

EngHabu reviewed Feb 8, 2024

View reviewed changes

docs/flyte_agents/creating_an_agent.md Outdated Show resolved Hide resolved

nikki everett added 3 commits February 9, 2024 16:01

stub out async vs sync agents content

543a44e

Signed-off-by: nikki everett <[email protected]>

link to list of agents

785491f

Signed-off-by: nikki everett <[email protected]>

edits for sync vs async agents

64c9dcd

Signed-off-by: nikki everett <[email protected]>

neverett commented Feb 13, 2024

View reviewed changes

docs/flyte_agents/creating_an_agent.md Outdated Show resolved Hide resolved

neverett commented Feb 13, 2024

View reviewed changes

docs/flyte_agents/creating_an_agent.md Outdated Show resolved Hide resolved

neverett commented Feb 13, 2024

View reviewed changes

nikki everett added 2 commits February 14, 2024 15:59

add visible TOC table to index page

658e5ce

Signed-off-by: nikki everett <[email protected]>

update for async vs sync tasks

e5cd118

Signed-off-by: nikki everett <[email protected]>

neverett mentioned this pull request Feb 14, 2024

Updates for agents integrations docs flyteorg/flytesnacks#1498

Merged

neverett commented Feb 15, 2024

View reviewed changes

docs/flyte_agents/testing_agents_locally.md Outdated Show resolved Hide resolved

neverett commented Feb 15, 2024

View reviewed changes

docs/flyte_agents/testing_agents_locally.md Show resolved Hide resolved

pingsutw and others added 11 commits February 16, 2024 10:25

Update async agent example (#4906)

30e0354

* Add sync agent example Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

formatting and copy updates

0f2a0d1

Signed-off-by: nikki everett <[email protected]>

fix formatting in snowflake doc, small fixes in agents procedural docs

330979a

Signed-off-by: nikki everett <[email protected]>

fix toctree

d4c4e8b

Signed-off-by: nikki everett <[email protected]>

add link

20931b8

Signed-off-by: nikki everett <[email protected]>

fix links

95b6eaa

Signed-off-by: nikki everett <[email protected]>

update link

25cb747

Signed-off-by: nikki everett <[email protected]>

import blog content

443df5e

Signed-off-by: nikki everett <[email protected]>

rename agent creation doc, small content updates

5b373aa

Signed-off-by: nikki everett <[email protected]>

fix article title

048b7a6

Signed-off-by: nikki everett <[email protected]>

fix link

dff3f9f

Signed-off-by: nikki everett <[email protected]>

neverett changed the title ~~Updates for agents docs~~ Agents guide and deployment config docs updates Feb 20, 2024

neverett changed the title ~~Agents guide and deployment config docs updates~~ New agents guide and deployment config docs updates Feb 20, 2024

undo unnecessary formatting change in unrelated file

3e05037

Signed-off-by: nikki everett <[email protected]>

pingsutw approved these changes Feb 20, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 20, 2024

neverett mentioned this pull request Feb 21, 2024

Add Airflow migration guide #4871

Merged

2 tasks

neverett merged commit 5f92f31 into master Feb 23, 2024
48 of 49 checks passed

neverett deleted the docs/agents-updates branch February 23, 2024 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New agents guide and deployment config docs updates #4822

New agents guide and deployment config docs updates #4822

neverett commented Feb 1, 2024 •

edited

Loading

codecov bot commented Feb 1, 2024 •

edited

Loading

neverett commented Feb 5, 2024

neverett Feb 13, 2024

EngHabu Feb 13, 2024

neverett Feb 13, 2024


		If you need to connect to an external service in your workflow, we recommend using the corresponding agent rather than a web API plugin. Agents are designed to be scalable and can handle large workloads efficiently, and decrease load on FlytePropeller, since they run outside of it. You can also test agents locally without having to change the Flyte backend configuration, streamlining development.

		For a list of agents you can use in your tasks, see the [Agent Setup](https://docs.flyte.org/en/latest/deployment/agents/index.html) documentation.


		To create a new async agent, extend the `AgentBase` class in the `flytekit.backend` module and implement the `execute` methods. All calls must be idempotent.

		- `execute`: TK

New agents guide and deployment config docs updates #4822

New agents guide and deployment config docs updates #4822

Conversation

neverett commented Feb 1, 2024 • edited Loading

Why are the changes needed?

What changes were proposed in this pull request?

Check all the applicable boxes

Related PRs

codecov bot commented Feb 1, 2024 • edited Loading

Codecov Report

neverett commented Feb 5, 2024

neverett Feb 13, 2024

Choose a reason for hiding this comment

EngHabu Feb 13, 2024

Choose a reason for hiding this comment

neverett Feb 13, 2024

Choose a reason for hiding this comment

neverett commented Feb 1, 2024 •

edited

Loading

codecov bot commented Feb 1, 2024 •

edited

Loading