CoolDown to HDFS Lose Data

[fdf1779]

When I was using version 2.1.2 of Doris, I created an HDFS Resource through cold backup to cold store data on the HDFS surface. However, when creating the STORAGE POLICY, "cooldown_datetime"="2023-02-04 00:00:00" failed. Doris directly cold stored all 2.5PB of data in the table to HDFS. However, due to the limited write performance of HDFS, some partitions that were cold stored locally on HDFS now show that the data does not exist. When I searched these partitions, I found that the data was 0 and all were lost. I would like to ask if this is due to asynchronous writing, and if it cannot be queried before all cold backups are completed, or if there is any other reason. I hope for an answer.

[ixzc]

what's the meaning of creating the STORAGE POLICY failed? if you did this failed, how do you set STORAGE POLICY?
please provide your table schema.

[fdf1779]

Cold storage can be executed, but the cold storage expiration time is invalid. The set "cooldown_datetime" is set to "2023-02-04 00:00:00", but data after 2023-02-04 00:00:00 is also cold stored.

[maruyue]

You have misunderstood cooldown_datetime. It does not mean that the data before this time is cooled down, but that at this point in time, all the data in the table starts to be moved to cold storage.

[maruyue]

"cooldown_datetime" = "2022-06-08 00:00:00"

This means that at the time "2022-06-08 00:00:00", the data will cool down.