How can I create a pandas dataframe without using Airflow?

[kebab-mai-haddi]

No description provided.

[mbasmanova]

@avisrivastava254084 Aviral, could you provide some context for this question? What are you trying to do?

[kebab-mai-haddi]

@mbasmanova , Hi!
I want to create a pandas dataframe from a hive table using Presto.
Currently, I am doing it in this manner:

def get_pandas_dataframe(self, hql, parameters=None):
        if not self.airflow_conn_reqd:
            import pandas
            cursor = self.presto_client.cursor()
            try:
                cursor.execute(self._strip_sql(hql), parameters)
                data = cursor.fetchall()
            except DatabaseError as e:
                raise PrestoException(self._get_pretty_exception_message(e))
            column_descriptions = cursor.description
            if data:
                df = pandas.DataFrame(data)
                df.columns = [c[0] for c in column_descriptions]
            else:
                df = pandas.DataFrame()
        else:
            df = self.get_pandas_df(query)
        return (
            df.replace('', np.nan)
        )

Are you able to understand it?

Note: Creating a pandas dataframe is provided in Airflow hooks of Presto but in normal Python client.

Can I create a PR and get this merged?

[mbasmanova]

@avisrivastava254084 Aviral, are you suggesting to add an Airflow hook for Presto (https://airflow.apache.org/concepts.html#hooks) ? Sure, let's try that. Go ahead and create a PR.

[mbasmanova]

@avisrivastava254084

Note: Creating a pandas dataframe is provided in Airflow hooks of Presto

You are referring to this piece of code, right?

https://github.com/apache/airflow/blob/7d904467d6523f80b8a441093bb71f6e77dcdd68/airflow/hooks/presto_hook.py#L99

Why can't you simply copy-paste the relevant parts into your Python program?

[mbasmanova]

@avisrivastava254084 Is this related to prestodb/presto-python-client#83 ?

[mbasmanova]

CC: @ggreg

[ggreg]

Following up on prestodb/presto-python-client#83

[mbasmanova]

@ggreg Thank you, Greg.

@avisrivastava254084 Aviral, I'm closing this issue in favor of prestodb/presto-python-client#83