Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove deprecated functionality from Google provider #128

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airflow/serialization/serialized_objects.py
Original file line number Diff line number Diff line change
Expand Up @@ -1542,7 +1542,7 @@ def _serialize_operator_extra_links(cls, operator_extra_links: Iterable[BaseOper

Store the import path of the OperatorLink and the arguments passed to it.
For example:
``[{'airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink': {}}]``
``[{'airflow.providers.google.cloud.links.bigquery.BigQueryDatasetLink': {}}]``

:param operator_extra_links: Operator Link
:return: Serialized Operator Link
Expand Down
7 changes: 1 addition & 6 deletions docs/apache-airflow-providers-google/connections/gcp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -207,11 +207,6 @@ In order for this example to work, the account ``impersonated_account`` must gra
access token, which will allow to act on its behalf using its permissions. ``impersonated_account``
does not even need to have a generated key.

.. warning::
:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator` and
:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`
do not support direct impersonation as of now.

In case of operators that connect to multiple Google services, all hooks use the same value of
``impersonation_chain`` (if applicable). You can also impersonate accounts from projects
other than the project of the originating account. In that case, the project id of the impersonated
Expand Down Expand Up @@ -306,7 +301,7 @@ For example:
task_id="create-spreadsheet",
gcp_conn_id="google_cloud_default",
spreadsheet=SPREADSHEET,
delegate_to=f"projects/-/serviceAccounts/SA@{PROJECT_ID}.iam.gserviceaccount.com",
impersonation_chain=f"projects/-/serviceAccounts/SA@{PROJECT_ID}.iam.gserviceaccount.com",
)

Note that as domain-wide delegation is currently supported by most of the Google operators and hooks, its usage should be limited only to Google Workspace (gsuite) and marketing platform operators and hooks or by accessing these services through the GoogleDiscoveryAPI hook. It is deprecated in the following usages:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ returns the id of model in :ref:`XCom <concepts:xcom>` under ``model_id`` key.
This operator is deprecated when running for text, video and vision prediction and will be removed soon.
All the functionality of legacy AutoML Natural Language, Vision, Video Intelligence and new features are
available on the Vertex AI platform. Please use
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`,
:class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`,
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLImageTrainingJobOperator` or
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ There are several ways to run a Dataflow pipeline depending on your environment,
dependencies must be installed on the worker. For Java, worker must have the JRE Runtime installed.
For Python, the Python interpreter. The runtime versions must be compatible with the pipeline versions.
This is the fastest way to start a pipeline, but because of its frequent problems with system dependencies,
it may cause problems. See: :ref:`howto/operator:DataflowCreateJavaJobOperator`,
:ref:`howto/operator:DataflowCreatePythonJobOperator` for more detailed information.
it may cause problems. See: :ref:`howto/operator:JavaSDKPipelines`,
:ref:`howto/operator:PythonSDKPipelines` for more detailed information.
Developer can also create a pipeline by passing its structure in a JSON format and then run it to create
a Job.
See: :ref:`howto/operator:DataflowCreatePipelineOperator` and :ref:`howto/operator:DataflowRunPipelineOperator`
Expand Down Expand Up @@ -107,7 +107,7 @@ the create job operators. The source file can be located on GCS or on the local
or
:class:`~airflow.providers.apache.beam.operators.beam.BeamRunPythonPipelineOperator`

.. _howto/operator:DataflowCreateJavaJobOperator:
.. _howto/operator:JavaSDKPipelines:

Java SDK pipelines
""""""""""""""""""
Expand Down Expand Up @@ -141,7 +141,7 @@ Here is an example of creating and running a pipeline in Java with jar stored on
:start-after: [START howto_operator_start_java_job_local_jar]
:end-before: [END howto_operator_start_java_job_local_jar]

.. _howto/operator:DataflowCreatePythonJobOperator:
.. _howto/operator:PythonSDKPipelines:

Python SDK pipelines
""""""""""""""""""""
Expand Down
14 changes: 0 additions & 14 deletions docs/apache-airflow-providers-google/operators/cloud/gcs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -174,20 +174,6 @@ Also you can use deferrable mode in this operator if you would like to free up t
:start-after: [START howto_sensor_object_exists_task_defered]
:end-before: [END howto_sensor_object_exists_task_defered]

.. _howto/sensor:GCSObjectExistenceAsyncSensor:

GCSObjectExistenceAsyncSensor
-----------------------------

:class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceAsyncSensor` is deprecated and will be removed in a future release. Please use :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` and use the deferrable mode in that operator.

.. exampleinclude:: /../../providers/tests/system/google/cloud/gcs/example_gcs_sensor.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_object_exists_task_async]
:end-before: [END howto_sensor_object_exists_task_async]


.. _howto/sensor:GCSObjectsWithPrefixExistenceSensor:

GCSObjectsWithPrefixExistenceSensor
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ To create a Google Vertex AI Auto ML training jobs you have five operators
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLForecastingTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLImageTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTabularTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`
Each of them will wait for the operation to complete. The results of each operator will be a model
which was trained by user using these operators.
Expand Down Expand Up @@ -257,17 +257,6 @@ put dataset id to ``dataset_id`` parameter in operator.
:start-after: [START how_to_cloud_vertex_ai_create_auto_ml_tabular_training_job_operator]
:end-before: [END how_to_cloud_vertex_ai_create_auto_ml_tabular_training_job_operator]

How to run AutoML Text Training Job
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`

Operator is deprecated, the non-training (existing legacy models) AutoMLText API will be deprecated
on June 15, 2025.
There are 2 options for text classification, extraction, and sentiment analysis tasks replacement:
- Prompts with pre-trained Gemini model, using :class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.TextGenerationModelPredictOperator`.
- Fine tuning over Gemini model, For more tailored results, using :class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`.

Please visit the https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-tuning for more details.

How to run AutoML Video Training Job
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`

Expand Down

This file was deleted.

32 changes: 20 additions & 12 deletions docs/apache-airflow/howto/define-extra-link.rst
Original file line number Diff line number Diff line change
Expand Up @@ -138,24 +138,34 @@ Console, but if we wanted to change that link we could:
BIGQUERY_JOB_DETAILS_LINK_FMT = "http://console.cloud.google.com/bigquery?j={job_id}"


class BigQueryConsoleLink(BaseOperatorLink):
class BigQueryDatasetLink(BaseGoogleLink):
"""
Helper class for constructing BigQuery link.
Helper class for constructing BigQuery Dataset Link.
"""

name = "BigQuery Console"
operators = [BigQueryOperator]
name = "BigQuery Dataset"
key = "bigquery_dataset"
format_str = BIGQUERY_DATASET_LINK

def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
job_id = XCom.get_one(ti_key=ti_key, key="job_id")
return BIGQUERY_JOB_DETAILS_LINK_FMT.format(job_id=job_id) if job_id else ""
@staticmethod
def persist(
context: Context,
task_instance: BaseOperator,
dataset_id: str,
project_id: str,
):
task_instance.xcom_push(
context,
key=BigQueryDatasetLink.key,
value={"dataset_id": dataset_id, "project_id": project_id},
)


# Defining the plugin class
class AirflowExtraLinkPlugin(AirflowPlugin):
name = "extra_link_plugin"
operator_extra_links = [
BigQueryConsoleLink(),
BigQueryDatasetLink(),
]


Expand All @@ -171,9 +181,7 @@ by ``apache-airflow-providers-google`` provider currently:
.. code-block:: yaml

extra-links:
- airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink
- airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleIndexableLink
- airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink

- airflow.providers.google.cloud.links.bigquery.BigQueryDatasetLink
- airflow.providers.google.cloud.links.bigquery.BigQueryTableLink

You can include as many operators with extra links as you want.
Loading
Loading