Flink+iceberg

Flink iceberg is a feature in Apache Flink that enables efficient processing of incremental data changes in large data sets. It is based on the concept of an iceberg table, which is a table that contains both the current state of the data and a history of the changes made to the data over time.

With Flink iceberg, users can perform incremental updates on large data sets without having to reprocess the entire data set. This is achieved by storing the data set in a hierarchical format, with the current state of the data at the top level and the changes made to the data at lower levels.

Flink iceberg also provides support for schema evolution, which allows users to modify the schema of the data set over time without having to reprocess the entire data set.

Overall, Flink iceberg is a powerful tool that enables efficient processing of large data sets with incremental updates and schema evolution capabilities.