So, Iceberg basically stores all the historical versions of the table in different parquet files? I know parquet is efficient but won’t it still cause storage overload given how frequently updates are made to tables? Or is there a limit on the number of historical versions it stores at any given point?
So, Iceberg basically stores all the historical versions of the table in different parquet files? I know parquet is efficient but won’t it still cause storage overload given how frequently updates are made to tables? Or is there a limit on the number of historical versions it stores at any given point?
good point, the catalog should clean old snapshots over time. This is actually one of the features Snowflake catalog will propose.
That makes sense. Thank you!