when drop a non-Iceberg table , the directory associated with the table was not deleted

[lordk911]

Apache Iceberg version

1.6.1

Query engine

Spark

Please describe the bug 🐞

spark 3.4.4 with config:

spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive

when drop a orc table, the directory associated with the table was not deleted from hdfs, only delete metadata from HMS

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[lordk911]

Is this a Bug, or is it by design?

[MonkeyCanCode]

@lordk911 that is expected. Drop table only remove metadata refs and not the actual data files since 0.14. For actual data file removal, you will need to add purge at the end. This is documented in https://iceberg.apache.org/docs/latest/spark-ddl/#drop-table.

You can find the same from the code as well:

Entry point for drop table in spark: https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java#L284
Then it goes to

iceberg/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

Line 340 in ab6365d

public boolean dropTable(Identifier ident) {

which will tell us the following:

  @Override
  public boolean dropTable(Identifier ident) {
    return dropTableWithoutPurging(ident);
  }

and the content of dropTableWithoutPurging is following which also shows by default drop table xxx in spark won't remove data files but only metadata refs:

  private boolean dropTableWithoutPurging(Identifier ident) {
    if (isPathIdentifier(ident)) {
      return tables.dropTable(((PathIdentifier) ident).location(), false /* don't purge data */);
    } else {
      return icebergCatalog.dropTable(buildIdentifier(ident), false /* don't purge data */);
    }
  }

[lordk911]

Drop table only remove metadata refs and not the actual data files since 0.14. For actual data file removal, you will need to add purge at the end. This is documented in https://iceberg.apache.org/docs/latest/spark-ddl/#drop-table.

Thank you for your reply. I know about this change, but in fact, spark3.3.3 + iceberg 1.3.1, when using SparkSessionCatalog to execute the drop non-Iceberg table, purge keyword is not required.
If this is expected by design , it is acceptable for me.

[MonkeyCanCode]

Drop table only remove metadata refs and not the actual data files since 0.14. For actual data file removal, you will need to add purge at the end. This is documented in https://iceberg.apache.org/docs/latest/spark-ddl/#drop-table.

Thank you for your reply. I know about this change, but in fact, spark3.3.3 + iceberg 1.3.1, when using SparkSessionCatalog to execute the drop non-Iceberg table, purge keyword is not required. If this is expected by design , it is acceptable for me.

I haven't try this on older version of iceberg runtime. The one I currently using is spark 3.5.x + iceberg runtime 1.6.x and drop table without purge won't cleanup datafiles there. I would think that is by designed based on the doc. Wondering if some other bugs is causing data files to get drop in spark3.3.3 + iceberg 1.3.1.