External Tables: Remove creation of job status table

[ravjotbrar]

Remove the creation of the job status table when creating an external table.

[KevinAppelBofa]

The creation of the S2V_JOB_STATUS_USER_ table inside the Vertica has a comment ie
Persistent job status table showing all jobs, serving as permanent record of data loaded from Spark to Vertica. Creation time:Thu Dec 23 18:50:29 GMT 2021

This is occurring on both creating external tables and also when doing a write of a spark dataframe directly to the vertica

Once this is set the first time, a follow up write of either type is not changing this item; it remains with the comment associated to it and requires a manual drop of the table to get rid of it

I also had tested to see what happens and tried to drop this table while the external table was writing and that throws an exception.

Since this doesn't seem to handle the overwrite of the comments, what happens if the person actually submits two spark jobs at the same time to write into vertica 2 different things

This would be great if you can have this tabled dropped in both cases when the write is completed, this will make the DBAs happy that we are not leaving left over temp tables

[alexey-temnikov]

@KevinAppelBofa, thank you for the feedback! I understand the inconvenience of having these tables deleted manually. We are planning to work on this issue early next year.

[jeremyprime]

The status table can be useful to audit or debug when/what data was written to Vertica. Currently the table is always created if it doesn't exist, and subsequent writes will add another row to the table. However, as Kevin notes, there are cases when it is not needed and it only clutters the database. We can create another ticket to provide a flag, such as save_metadata_tables, so that creation of the status table can be enabled/disabled (I assume we'll want to disable creation by default, which also aligns with the existing prevent_cleanup flag for temporary HDFS data). This flag can also cover the creation of the rejected rows table in the future (see #293), which is also useful for debugging but could clutter the database.

So the scope of this ticket will be to not create the status table when creating an external table, while the other ticket will disable creation of status table by default but provide an option to enable its creation (mostly for development/debugging).

@KevinAppelBofa, would that work for you?

[KevinAppelBofa]

@jeremyp-bq that sounds great

[jeremyprime]

See #308 for the ticket to add the flag to enable/disable creation of the job status table.