Skip to content

Commit

Permalink
docs: blacken docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Lee-W committed Nov 16, 2023
1 parent f44f276 commit 6ae5da0
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions docs/airflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,19 +135,19 @@ Unpause these DAGs to see how they ingest and upsert any documents that have cha
- The Airflow ingest DAGs use [dynamic tasks](https://docs.astronomer.io/learn/dynamic-tasks) to iterate over multiple document sources. For instance, the bulk ingest DAG (`airflow/include/dags/ingest/ask-astro-load.py`) specifies markdown sources in github repositories.
```python
markdown_docs_sources = [
{'doc_dir': 'learn', 'repo_base': 'astronomer/docs'},
{'doc_dir': 'astro', 'repo_base': 'astronomer/docs'},
]
{"doc_dir": "learn", "repo_base": "astronomer/docs"},
{"doc_dir": "astro", "repo_base": "astronomer/docs"},
]
```

To ingest markdown documents from additional repositories simply add the source and re-run the DAG.
```python
markdown_docs_sources = [
{'doc_dir': 'learn', 'repo_base': 'astronomer/docs'},
{'doc_dir': 'astro', 'repo_base': 'astronomer/docs'},
{'doc_dir': 'base', 'repo_base': 'OpenLineage/docs'},
{'doc_dir': 'base', 'repo_base': 'OpenLineage/OpenLineage'}
]
{"doc_dir": "learn", "repo_base": "astronomer/docs"},
{"doc_dir": "astro", "repo_base": "astronomer/docs"},
{"doc_dir": "base", "repo_base": "OpenLineage/docs"},
{"doc_dir": "base", "repo_base": "OpenLineage/OpenLineage"},
]
```

- The 'extract' logic for each data source is specified in `airflow/include/tasks/extract.py`. These functions can be copied/pasted to select new sources.
Expand Down

0 comments on commit 6ae5da0

Please sign in to comment.