Add IcebergDocument as one type of the operator result storage

[bobbai00]

Implement Apache Iceberg for Result Storage

How to Enable Iceberg Result Storage

1. Update storage-config.yaml:
   • Set result-storage-mode to iceberg.

Major Changes

• Introduced IcebergDocument: A thread-safe VirtualDocument implementation for storing and reading results in Iceberg tables.
• Introduced IcebergTableWriter: Append-only writer for Iceberg tables with configurable buffer size.
• Catalog and Data storage for Iceberg: Uses a local file system (file:/) via HadoopCatalog and HadoopFileIO. This ensures Iceberg operates without relying on external storage services.
• ProgressiveSinkOpExec with a new parameter workerId is added. Each writer of the result storage will take this workerId as one new parameter.

Dependencies

• Added Apache Iceberg-related libraries.
• Introduced Hadoop-related libraries to support Iceberg's HadoopCatalog and HadoopFileIO. These libraries are used for placeholder configuration but do not enforce runtime dependency on HDFS.

Overview of Iceberg Components

IcebergDocument

• Manages reading and organizing data in Iceberg tables.
• Supports iterator-based incremental reads with thread-safe operations for reading and clearing data.
• Initializes or overrides the Iceberg table during construction.

IcebergTableWriter

• Writes data as immutable Parquet files in an append-only manner.
• Each writer uniquely prefixes its files to avoid conflicts (workerIndex_fileIndex format).
• Not thread-safe—single-thread access is recommended.

Data Storage via Iceberg Tables

• Write:
  • Tables are created per storage key.
  • Writers append Parquet files to the table, ensuring immutability.
• Read:
  • Readers use IcebergDocument.get to fetch data via an iterator.
  • The iterator reads data incrementally while ensuring data order matches the commit sequence of the data files.

Data Reading Using File Metadata

• Data files are read using getUsingFileSequenceOrder, which:
  • Retrieves and sorts metadata files (FileScanTask) by sequence numbers.
  • Reads records sequentially, skipping files or records as needed.
  • Supports range-based reading (from, until) and incremental reads.
• Sorting ensures data consistency and order preservation.

Hadoop Usage Without HDFS

• The HadoopCatalog uses an empty Hadoop configuration, defaulting to the local file system (file:/).
• This enables efficient management of Iceberg tables in local or network file systems without requiring HDFS infrastructure.

[shengquan-ni]

LGTM