Knowledge Base
Apache Iceberg Reference
A definitive, authoritative reference for every major Apache Iceberg concept — from the core table format and metadata layer to catalogs, query engines, operational patterns, and agentic data architectures. Each entry is written to be a standalone resource, deeply interlinked across the knowledge base.
222 terms across 14 categories
Core Concepts
- ACID Transactions in Apache IcebergApache Iceberg delivers full ACID transaction guarantees on object storage through optimistic concurrency control and at…
- Apache Iceberg Spec v1 vs v2Apache Iceberg Spec v2 introduced row-level deletes (delete files), sequence numbers, required field tracking, and impro…
- Apache Iceberg Spec v3Apache Iceberg Spec v3 introduces deletion vectors for more efficient row-level deletes, the Variant data type for semi-…
- Apache Iceberg Spec v4 (Current State)Apache Iceberg Spec v4 is in early community discussion and proposal stages as of 2025, with potential features includin…
- Apache Iceberg Table FormatThe Apache Iceberg table format is a specification defining how data files, metadata files, manifests, and snapshots are…
- Apache Iceberg vs Apache HudiApache Iceberg and Apache Hudi are both open table formats for cloud lakehouses: Iceberg prioritizes multi-engine intero…
- Apache Iceberg vs Delta LakeApache Iceberg and Delta Lake are the two dominant open table formats for cloud lakehouses: Iceberg offers superior mult…
- Data LakehouseA data lakehouse is a modern data architecture that combines the low-cost, scalable storage of a data lake with the reli…
- Hidden Partitioning in Apache IcebergHidden partitioning in Apache Iceberg separates the physical partition layout from the logical table schema, allowing th…
- Iceberg Column MappingIceberg column mapping decouples the logical column names in the schema from the physical field names in data files usin…
- Iceberg Deletion VectorsDeletion vectors are a Spec v3 enhancement to Apache Iceberg's row-level delete mechanism, replacing positional delete f…
- Iceberg Open Table Format vs. Delta Lake vs. Apache HudiApache Iceberg, Delta Lake, and Apache Hudi are the three dominant open table formats competing to be the storage founda…
- Iceberg Sequence NumberThe Iceberg sequence number is a monotonically increasing integer assigned to each snapshot and each data/delete file, i…
- Iceberg Snapshot ReferencesIceberg snapshot references are named pointers (branches and tags) stored in the table metadata that reference specific …
- Iceberg Sort OrderAn Iceberg sort order is a table-level specification stored in metadata that defines how data should be physically order…
- Iceberg Table PropertiesIceberg table properties are key-value configuration settings stored in the table metadata that control write behavior, …
- Iceberg Table Statistics (Puffin)Iceberg table statistics are advanced column-level metrics: including NDV (number of distinct values) estimates using Ap…
- Iceberg ViewsApache Iceberg Views are named, stored SQL queries managed by the Iceberg catalog that appear as virtual tables to downs…
- Open Table Format Comparison (Iceberg, Delta Lake, Hudi, Paimon)A comprehensive comparison of the four major open table formats: Apache Iceberg, Delta Lake, Apache Hudi, and Apache Pai…
- Partition Evolution in Apache IcebergPartition evolution in Apache Iceberg lets you change a table's partitioning scheme at any time without rewriting existi…
- Schema Evolution in Apache IcebergSchema evolution in Apache Iceberg allows you to safely add, drop, rename, reorder, and widen columns in a table without…
- Time Travel in Apache IcebergTime travel in Apache Iceberg lets you query a table as it existed at any past snapshot or timestamp, enabling reproduci…
- What is Apache Iceberg?Apache Iceberg is an open, high-performance table format for huge analytic datasets stored in data lakes, enabling ACID …
File & Metadata Layer
- Apache Iceberg ORC FormatApache ORC (Optimized Row Columnar) is an alternative columnar storage format supported by Apache Iceberg alongside Parq…
- Apache Iceberg SnapshotAn Iceberg snapshot is an immutable, point-in-time view of a table's complete data state, recorded as a manifest list th…
- Apache Parquet and IcebergApache Parquet is the default and recommended data file format for Apache Iceberg tables, providing columnar storage, ri…
- Iceberg Avro Metadata FormatApache Avro is the metadata file format used for all Apache Iceberg manifest files and manifest lists, providing schema-…
- Iceberg Data FilesIceberg data files are the immutable columnar files (Parquet, ORC, or Avro) that store the actual table data in object s…
- Iceberg Delete FilesIceberg delete files record row-level deletions without rewriting data files, enabling efficient UPDATE, DELETE, and MER…
- Iceberg EncryptionApache Iceberg supports column-level and file-level encryption through its encryption specification, enabling sensitive …
- Iceberg Equality DeletesEquality delete files in Apache Iceberg record column values identifying rows to be deleted, enabling row-level deletes …
- Iceberg FileIO APIThe Iceberg FileIO API is an abstraction layer that decouples the Iceberg table format from specific storage system impl…
- Iceberg Manifest FileAn Iceberg manifest file is an Avro metadata file that tracks a subset of an Iceberg table's data files, recording each …
- Iceberg Manifest ListAn Iceberg manifest list is a file associated with each snapshot that lists all the manifest files making up that snapsh…
- Iceberg Metadata FileThe Iceberg metadata file (metadata.json) is the top-level entry point for an Iceberg table, recording the full history …
- Iceberg Positional DeletesPositional delete files in Apache Iceberg record the exact file path and row position of deleted rows, enabling efficien…
- Iceberg Puffin FilesPuffin is the Apache Iceberg file format for storing advanced table statistics and indexes beyond the basic min/max boun…
Catalogs
- Apache GravitinoApache Gravitino is an open-source multi-source metadata hub that provides unified metadata management across heterogene…
- Apache Polaris CatalogApache Polaris is an open-source implementation of the Apache Iceberg REST Catalog specification, co-created by Dremio a…
- AWS Glue Catalog for Apache IcebergAWS Glue Data Catalog is Amazon's managed metadata catalog service with native support for Apache Iceberg tables via the…
- Hive Metastore Catalog for IcebergThe Hive Metastore (HMS) is the original Iceberg catalog implementation, using a relational database to store Iceberg ta…
- Iceberg Catalog MigrationIceberg catalog migration moves tables between catalog implementations (HMS to Polaris, Glue to Nessie, JDBC to REST Cat…
- Iceberg JDBC CatalogThe Iceberg JDBC Catalog uses any JDBC-compatible relational database (PostgreSQL, MySQL, SQLite) as a persistent metada…
- Iceberg Multi-Catalog ArchitectureMulti-catalog architectures in Apache Iceberg use multiple catalog instances to achieve environment isolation, domain se…
- Iceberg REST CatalogThe Iceberg REST Catalog is a standardized HTTP API specification for Apache Iceberg catalog operations, enabling any en…
- Iceberg REST Catalog API ReferenceThe Apache Iceberg REST Catalog specification defines a standardized HTTP API for catalog operations: namespace manageme…
- Project NessieProject Nessie is an open-source transactional metadata catalog for Apache Iceberg with Git-like branching and merging s…
- Snowflake Open CatalogSnowflake Open Catalog is a managed Apache Polaris service offered by Snowflake that provides a vendor-neutral Iceberg R…
- What is an Iceberg Catalog?An Apache Iceberg catalog is the service responsible for tracking the current metadata file location for each Iceberg ta…
Operations & Optimization
- Copy-on-Write (CoW) in IcebergCopy-on-Write (CoW) is an Iceberg write mode where UPDATE and DELETE operations rewrite entire affected data files to pr…
- Expire Snapshots in Apache IcebergExpiring snapshots in Apache Iceberg is the maintenance operation that removes old snapshot metadata (and optionally the…
- Iceberg Bloom FiltersBloom filter indexes in Apache Iceberg enable probabilistic row-level skipping by allowing query engines to determine wi…
- Iceberg Branching and TaggingIceberg table branches and tags are named references to specific snapshots or independent snapshot chains, enabling Git-…
- Iceberg Concurrent Write HandlingApache Iceberg uses optimistic concurrency control with atomic catalog commits to safely handle multiple simultaneous wr…
- Iceberg Cost OptimizationCost optimization for Apache Iceberg lakehouses targets storage costs (snapshot expiration, compression, tiering), compu…
- Iceberg Data SkippingData skipping in Apache Iceberg is the multi-level mechanism by which query engines eliminate irrelevant files and row g…
- Iceberg Incremental ReadsIceberg incremental reads enable processing only the new or changed data between two snapshots by using the snapshot dif…
- Iceberg Maintenance SchedulingProduction Apache Iceberg maintenance requires scheduling compaction, snapshot expiration, orphan file cleanup, and mani…
- Iceberg Orphan FilesOrphan files in Apache Iceberg are data files written to object storage during failed transactions that were never commi…
- Iceberg Performance Tuning GuideA comprehensive guide to optimizing Apache Iceberg query and write performance, covering partition pruning effectiveness…
- Iceberg Predicate PushdownPredicate pushdown in Apache Iceberg propagates WHERE clause filter conditions from the query layer down through the man…
- Iceberg Rewrite ManifestsRewriting Iceberg manifests is a maintenance operation that consolidates many small manifest files into fewer, larger on…
- Iceberg Table ClusteringTable clustering in Apache Iceberg co-locates related rows within the same data files to maximize column statistics sele…
- Iceberg Table CompactionIceberg compaction is the maintenance process of merging small data files into optimally sized files, applying pending d…
- Iceberg Table Design Best PracticesIceberg table design best practices cover partition strategy, sort order selection, file format and compression choices,…
- Iceberg Table RollbackRolling back an Apache Iceberg table reverts its current state to a prior snapshot, effectively undoing all writes since…
- Iceberg Upsert (MERGE INTO)Iceberg upsert operations using MERGE INTO enable atomic insert-or-update workflows against Iceberg tables, implementing…
- Iceberg Write Distribution ModesIceberg write distribution modes control how data is distributed across parallel write tasks before being written to out…
- Merge-on-Read (MoR) in IcebergMerge-on-Read (MoR) is an Iceberg write strategy where UPDATE and DELETE operations write small delete files instead of …
- Row-Level Deletes in Apache IcebergRow-level deletes in Apache Iceberg enable precise removal or modification of individual rows within existing data files…
- Small File Problem in Apache IcebergThe small file problem in Apache Iceberg occurs when frequent write transactions generate many small Parquet files, degr…
- Z-Order Clustering in Apache IcebergZ-Order (or Z-curve) clustering in Apache Iceberg is a multi-dimensional data layout optimization that co-locates rows w…
Engines & Integrations
- Apache Airflow and Apache IcebergApache Airflow is the most widely used workflow orchestration platform for Iceberg data pipelines, providing scheduling,…
- Apache Doris and Apache IcebergApache Doris is a high-performance real-time analytical database with native Iceberg external catalog support, enabling …
- Apache Flink and Apache IcebergApache Flink is the leading stream processing engine for Apache Iceberg, enabling real-time data ingestion with exactly-…
- Apache Kafka and Apache IcebergApache Kafka and Apache Iceberg form the foundation of real-time lakehouse pipelines: Kafka provides the event streaming…
- Apache Spark and Apache IcebergApache Spark is the most feature-complete query engine for Apache Iceberg, providing full DDL, DML, time travel, stored …
- Apache Superset and Apache IcebergApache Superset is the leading open-source business intelligence tool that queries Apache Iceberg tables through SQL con…
- Databricks and Apache IcebergDatabricks supports Apache Iceberg through UniForm (Delta-to-Iceberg automatic metadata generation) and native Iceberg c…
- dbt and Apache Icebergdbt (data build tool) transforms raw Iceberg table data into clean, tested, documented analytical models using SQL, with…
- Dremio and Apache IcebergDremio is an Agentic Lakehouse platform that provides a fully integrated Iceberg experience through its Intelligent Quer…
- DuckDB and Apache IcebergDuckDB is an embedded analytical database with a native Apache Iceberg extension that enables direct, high-performance S…
- Hive and Apache IcebergApache Hive 4.x has native Iceberg support, enabling Hive SQL to read and write Iceberg tables as first-class objects wh…
- Presto and Apache IcebergPrestoDB is the Meta-maintained fork of the original Presto query engine with an Iceberg connector that supports Iceberg…
- PyIceberg: Python Library for Apache IcebergPyIceberg is the official Python library for Apache Iceberg, providing a pure-Python client for reading, writing, and ma…
- Snowflake Iceberg TablesSnowflake Iceberg Tables let organizations store Iceberg data in their own object storage (external volumes) while using…
- StarRocks and Apache IcebergStarRocks is a high-performance OLAP query engine with native Apache Iceberg external table support via its Multi-Catalo…
- Trino and Apache IcebergTrino (formerly PrestoSQL) is a distributed SQL query engine with native Apache Iceberg support, optimized for interacti…
Agentic & AI
- Agentic LakehouseAn Agentic Lakehouse is a data lakehouse architecture purpose-built for AI agents and autonomous analytics, combining op…
- Iceberg AI ReadinessIceberg AI readiness describes the architectural properties that make Apache Iceberg tables ideal for AI and machine lea…
- Iceberg AI Semantic LayerThe AI Semantic Layer on Apache Iceberg translates raw Iceberg table data into AI-understandable business context throug…
- Iceberg Apache Arrow FlightApache Arrow Flight provides a high-throughput, low-latency RPC protocol for transferring Apache Arrow columnar data fro…
- Iceberg Feature StoreApache Iceberg is used as the offline storage layer in ML feature stores, providing point-in-time correct feature retrie…
- Iceberg LLM Grounding and RAG for Structured DataLLM grounding with Apache Iceberg uses governed, versioned Iceberg tables as the authoritative data source for LLM respo…
- Iceberg Natural Language AnalyticsNatural language analytics on Apache Iceberg enables business users and AI agents to ask questions in plain English and …
- LangChain and Apache IcebergLangChain agents can query Apache Iceberg lakehouses using SQL tools and Arrow Flight connections, enabling natural lang…
- MCP and Apache IcebergModel Context Protocol (MCP) servers for Apache Iceberg enable AI agents and LLMs to discover, query, and reason over Ic…
Cloud-Specific Integrations
- Amazon EMR and Apache IcebergAmazon EMR (Elastic MapReduce) is AWS's managed Spark and Flink cluster service that supports Apache Iceberg as a first-…
- Amazon S3 Tables for Apache IcebergAmazon S3 Tables is an AWS managed service that provides Apache Iceberg table storage and catalog directly within Amazon…
- AWS Athena and Apache IcebergAmazon Athena is a serverless SQL query engine with native Apache Iceberg support via the AWS Glue Data Catalog, enablin…
- BigQuery and Apache IcebergGoogle BigQuery supports Apache Iceberg tables through BigLake managed tables and Biglake Metastore, enabling BigQuery S…
- Google Cloud and Apache IcebergGoogle Cloud's Apache Iceberg stack integrates BigQuery, Cloud Storage, Biglake Metastore, and Cloud Dataplex to provide…
- Microsoft Fabric and Apache IcebergMicrosoft Fabric supports Apache Iceberg tables through OneLake's open format integration and mirrored Fabric tables, en…
Modern Lakehouse Concepts & Interoperability
- Apache DataFusionA highly extensible Rust-native SQL query engine that supports native querying and management of Apache Iceberg tables.
- Apache XTable TranslationsAn open-source translation layer that converts table metadata bidirectionally among Apache Iceberg, Delta Lake, and Apac…
- BauplanA Python-native serverless lakehouse engine that uses Apache Iceberg as its core storage format to run pipelines with Gi…
- CDC Log Ingestion PipelinesData pipelines that capture database transaction logs and apply those insert, update, and delete events to lakehouse tab…
- ClickHouseA fast, open-source columnar database management system that supports native reading, querying, and writing of Apache Ic…
- Columnar Memory LayoutsA memory architecture that groups data values sequentially by columns rather than rows, enabling efficient vectorized qu…
- Debezium CDC EnginesThe standard open-source framework built on Apache Kafka Connect that monitors database transaction logs and translates …
- Decoupled Compute and StorageAn architecture separating the compute resources used to execute queries from the storage systems where data files resid…
- Delta Lake UniForm MetadataA features in Delta Lake (Universal Format) that automatically generates Apache Iceberg metadata alongside Delta logs du…
- Dynamic Filter PushdownA query optimization technique that uses runtime join results to dynamically filter scan operations on the opposing tabl…
- Google Cloud BigLakeA storage virtualization and governance engine in Google Cloud that unifies data lakes and warehouses, supporting manage…
- Microsoft Fabric OneLakeA unified SaaS data lake for Microsoft Fabric that supports Apache Iceberg through metadata shortcuts and table format v…
- Modern Data Stack (MDS)A cloud-native suite of analytical tools built around a central cloud data warehouse or lakehouse, emphasizing modularit…
- Object Storage Prefix HashingA layout strategy that prepends random hash prefixes to object paths to distribute storage request workloads across diff…
- PuppyGraphA real-time, zero-ETL graph query engine that allows users to query relational data lakes, including Apache Iceberg, as …
- Read AmplificationA storage performance metric representing the ratio of physical data read from storage compared to the logical data requ…
- Space AmplificationA storage metric representing the ratio of physical disk space occupied by a table compared to the size of its active lo…
- Spice.aiA data acceleration platform that enables application and AI developers to query, federate, and write to Apache Iceberg …
- Split Planning LoopsThe execution phase where query engine coordinators analyze table partition statistics to group data files into parallel…
- Time To First Byte (TTFB)A latency metric measuring the duration from when a client engine initiates an object storage read request to when the s…
- VeloDBA commercial data warehouse built on Apache Doris that offers native integration with Apache Iceberg tables for fast que…
- Vortex File FormatA high-performance columnar storage format succeeding Parquet, optimized for GPU-native decompression, in-memory Arrow c…
- Write AmplificationA performance metric measuring the ratio of physical data bytes written to storage compared to the logical data bytes up…
- Zero-Copy CloningA metadata-only operation that duplicates a table state instantly by pointing the new table clone to the existing data f…
Lakehouse Catalogs & Governance
- Catalog FederationThe capability of a query engine to connect to multiple disjoint metadata catalogs simultaneously, presenting them as a …
- Catalog NamespacesHierarchical logical containers (similar to databases, schemas, or folders) used within an Iceberg catalog to organize t…
- Data Lineage TrackingThe practice of documenting and visualizing the lifecycle, transformations, and flow of data from its source to its fina…
- Glue Catalog IAM PoliciesAWS Identity and Access Management policies that regulate user and compute engine access to AWS Glue Data Catalog databa…
- Glue Catalog Lake FormationA security service layer built on the AWS Glue Data Catalog that enables fine-grained, column-level, row-level, and cell…
- Iceberg DynamoDB CatalogAn AWS serverless catalog implementation that stores Iceberg table pointers in Amazon DynamoDB, using conditional writes…
- Iceberg Hive Catalog Lock ManagerA coordination system that uses the Hive Metastore lock APIs to serialize concurrent table updates, ensuring commit safe…
- Iceberg JDBC Catalog LocksA database row-level locking mechanism used by SQL-backed Iceberg catalogs to coordinate concurrent commits and ensure t…
- Nessie Git-like BranchingA version control model in Project Nessie that allows teams to manage database catalog tables using branches, tags, and …
- Nessie MergingThe operation in Project Nessie that merges commits from an isolated staging branch back into the main production branch…
- Nessie TaggingA version control feature in Project Nessie that creates named, immutable reference pointers to freeze the state of the …
- Polaris Catalog SharingA secure metadata sharing feature in Apache Polaris that allows external accounts and partner engines to access catalog …
- Polaris RBAC ModelThe Role-Based Access Control framework in Apache Polaris that secures catalog metadata resources by mapping service pri…
- Polaris Service PrincipalsProgrammatic identities created within Apache Polaris to authenticate query engines, ETL pipelines, and applications usi…
- REST Catalog Credential VendingA security architecture where the Iceberg REST catalog dynamically issues short-lived, scoped storage credentials to que…
- REST Catalog OAuth2 Token FlowThe standardized authentication protocol used by Iceberg REST clients to exchange credentials for secure Bearer tokens t…
- Single Source of Truth (SSOT)A data design principle where a central repository serves as the definitive reference for the state and metadata definit…
- Snowflake External Table Catalog SyncThe automated or manual synchronization process that refreshes Snowflake metadata pointers to match updates made to Iceb…
- Snowflake Managed Iceberg TablesIceberg tables where Snowflake serves as the catalog manager, executing all DML writes and writing physical Parquet file…
- Unity Catalog Delta-Iceberg CompatibilityThe capabilities in Unity Catalog that allow Delta Lake tables to be read as Apache Iceberg tables by generating complia…
Dremio-Specific Engine & Optimizations
- Dremio Acceleration EngineThe Dremio Acceleration Engine is the dedicated resource allocation mechanism within Dremio executors responsible for bu…
- Dremio Aggregation ReflectionsDremio Aggregation Reflections are pre-computed data structures that store pre-aggregated metrics and measures grouped b…
- Dremio Arrow Flight SQLDremio Arrow Flight SQL is a high-performance database connectivity protocol based on Apache Arrow and gRPC, transferrin…
- Dremio Column-Level MaskingDremio Column-Level Masking is a dynamic data security feature that obfuscates or replaces sensitive column values (such…
- Dremio Columnar Cloud Cache (C3)Dremio Columnar Cloud Cache (C3) is a block-level storage caching mechanism on executor nodes that stores copies of data…
- Dremio Coordinator NodeThe control plane node in a Dremio cluster responsible for query parsing, optimization, metadata catalog management, and…
- Dremio Data Reflections MatchingDremio Data Reflections Matching is the query planner compiler logic that automatically identifies, evaluates, and subst…
- Dremio Engine Auto-scalingThe automated process of dynamically scaling Dremio execution engine pods in a Kubernetes cluster to optimize performanc…
- Dremio External QueriesDremio External Queries are SQL pass-through commands that allow users to execute native database queries directly on un…
- Dremio Iceberg Metadata SyncDremio Iceberg Metadata Sync is the background coordination process that updates Dremio's catalog pointer to reference t…
- Dremio Join Co-segmentationDremio Join Co-segmentation is a query planning optimization that aligns join keys with matching table partition structu…
- Dremio LDAP IntegrationThe configuration of Dremio authentication to validate user credentials and map groups against LDAP or Active Directory …
- Dremio Metadata CachingDremio Metadata Caching is the process of storing table metadata (such as schemas, partition statistics, and file lists)…
- Dremio Parquet Vectorized ReaderDremio Parquet Vectorized Reader is an engine component that reads columnar Parquet data directly into in-memory Apache …
- Dremio Physical Datasets (PDS)Dremio Physical Datasets (PDS) are physical tables, views, or raw file directories registered from external data sources…
- Dremio Raw ReflectionsDremio Raw Reflections are pre-computed data layouts that preserve raw, row-level columns from a dataset, optimized with…
- Dremio ReflectionsDremio Reflections are pre-computed query acceleration structures that automatically optimize and speed up SQL queries u…
- Dremio Row-Level Security (RLS)Dremio Row-Level Security (RLS) is an access control mechanism that dynamically filters table rows returned by a query b…
- Dremio Sabot EngineThe Sabot Engine is Dremio's highly optimized, vectorized query execution engine built on Apache Arrow, designed to proc…
- Dremio SpacesDremio Spaces are logical namespace containers in the Dremio catalog where data teams organize, collaborate on, and secu…
- Dremio SQL RunnerDremio SQL Runner is the integrated web-based query editor and SQL interface inside the Dremio console used for writing …
- Dremio User Defined Functions (UDFs)Dremio User Defined Functions (UDFs) are custom calculations created in SQL that allow data teams to encapsulate reusabl…
- Dremio Virtual Datasets (VDS)Dremio Virtual Datasets (VDS) are logical views defined in the semantic layer that allow data teams to clean, join, and …
Governance & Security
- Iceberg Access Control PatternsIceberg access control is implemented at the catalog layer through the Iceberg REST Catalog RBAC model, providing namesp…
- Iceberg Audit LoggingIceberg audit logging captures a complete record of all catalog interactions, table reads, write commits, schema changes…
- Iceberg Data LineageIceberg data lineage is the ability to trace the origin, transformation history, and downstream consumption of data in I…
- Iceberg Data MaskingData masking in Apache Iceberg protects sensitive column values from unauthorized consumers by applying masking function…
- Iceberg Multi-Tenancy PatternsMulti-tenancy in Apache Iceberg isolates multiple tenants, teams, or environments in a shared lakehouse using namespace …
Table Format Maintenance & Operations
- Iceberg Bin-Packing CompactionA fast, non-sorting compaction strategy in Apache Iceberg that combines small data files into larger files to reduce rea…
- Iceberg Manifest MergingAn automatic, write-path optimization in Apache Iceberg that merges small manifest files during commits to prevent metad…
- Iceberg Metadata PruningAn automated housekeeping process in Apache Iceberg that limits the accumulation of historical table metadata JSON files…
- Iceberg Orphan Files PenaltyThe performance and cost overhead incurred when unreferenced, abandoned files accumulate in object storage due to failed…
- Iceberg Partition-Level CompactionA targeted maintenance strategy that runs file compaction only on specific partitions, optimizing high-churn data subset…
- Iceberg Rollback Snapshot ProceduresSQL procedures used to revert the active state of an Apache Iceberg table to a specific snapshot ID or historical timest…
- Iceberg Sort-Based CompactionA compaction strategy in Apache Iceberg that combines small data files and sorts the records globally by specified colum…
- Iceberg Spark Procedure add_filesA Spark SQL procedure in Apache Iceberg used to register existing Parquet or ORC data files directly into an Iceberg tab…
- Iceberg Spark Procedure expire_snapshotsA Spark SQL procedure in Apache Iceberg used to remove expired snapshots and physically delete their unreferenced data a…
- Iceberg Spark Procedure register_tableA Spark SQL procedure in Apache Iceberg used to register an existing metadata JSON file as a table within the active cat…
- Iceberg Spark Procedure remove_orphan_filesA Spark SQL procedure in Apache Iceberg used to identify and delete files in table storage that are not referenced in an…
- Iceberg Spark Procedure rewrite_data_filesA Spark SQL metadata procedure in Apache Iceberg used to consolidate small files and apply sorting or clustering strateg…
- Iceberg Spark Procedure rewrite_manifestsA Spark SQL maintenance procedure in Apache Iceberg used to reorganize and merge small manifest files to optimize query …
- Iceberg Spark Procedure rewrite_position_deletesA Spark SQL maintenance procedure in Apache Iceberg used to consolidate small position delete files to improve query rea…
- Iceberg Z-Order CompactionA multi-dimensional clustering compaction strategy in Apache Iceberg that sorts data along a Z-order space-filling curve…
Iceberg Specification, Schema & Internals
- Iceberg Bucket Partition TransformAn Iceberg partition transform that hashes values of a source column into a user-specified number of buckets to distribu…
- Iceberg Column ProjectionThe process of reading only the specific fields requested by a query from physical storage files using unique field IDs …
- Iceberg Date/Time Partition TransformsA set of native partition transforms in Apache Iceberg that partition data by year, month, day, or hour using source dat…
- Iceberg Decimal Type WideningA metadata-only schema evolution rule in Apache Iceberg that permits promoting the precision of a decimal field while ke…
- Iceberg File Content TypeA field in the manifest entry schema that indicates the type of data stored in a file, distinguishing between data files…
- Iceberg File Path SpecThe set of structural rules in Apache Iceberg that defines how metadata files, manifest files, and physical data files a…
- Iceberg Identity Partition TransformThe default partition transform in Apache Iceberg that partitions data directly by the values of a source column without…
- Iceberg Lock ManagerA catalog helper interface in Apache Iceberg used to manage transactional locks and secure pointer updates on catalogs t…
- Iceberg Manifest Entry SchemaThe Avro schema definition that specifies how Iceberg tracks data or delete files within manifest files, detailing colum…
- Iceberg Manifest Entry StatusAn integer status code in the Iceberg manifest entry schema that defines the state of a file in a snapshot: 0 (EXISTING)…
- Iceberg Manifest List SchemaThe Avro schema definition that tracks the set of manifest files comprising an Iceberg snapshot, including partition sum…
- Iceberg Metadata Table FilesA virtual system table in Apache Iceberg that lists all active data and delete files, exposing their file paths, sizes, …
- Iceberg Metadata Table HistoryA virtual system table in Apache Iceberg that exposes the historical timeline of committed snapshots, timestamps, and li…
- Iceberg Metadata Table SnapshotsA virtual system table in Apache Iceberg that lists all active snapshots, exposing their creation times, parent IDs, ope…
- Iceberg Metrics ModeA configuration property in Apache Iceberg that controls the depth of column-level statistics stored in manifest files t…
- Iceberg Nested Type SystemThe set of structural data types - structs, lists, and maps - defined by the Iceberg specification that use unique field…
- Iceberg Optimistic Concurrency Control (OCC)A concurrency model in Apache Iceberg that assumes low conflict rates, allowing concurrent writers to prepare changes in…
- Iceberg Parent Snapshot IDA reference property stored in snapshot metadata that links a snapshot to its immediate ancestor, establishing the table…
- Iceberg Sequence Number InheritanceThe mechanism in Apache Iceberg where data and delete files inherit their commit sequence numbers dynamically from the m…
- Iceberg Snapshot Expiration AgeA table configuration property that defines the maximum age of snapshots before they become eligible for deletion during…
- Iceberg Snapshot SummaryA map of key-value metadata properties embedded inside a snapshot that describes the write operation and counts of affec…
- Iceberg Spec V3 File EncryptionThe specification for securing Iceberg tables using envelope encryption and AES-GCM to encrypt data and metadata files.
- Iceberg Spec V3 Object-Store Storage LayoutA storage layout configuration in Apache Iceberg that uses random prefix hashes to distribute data files evenly across c…
- Iceberg Table Metadata SchemaThe JSON schema specification for the Iceberg table metadata file, tracking schemas, partition specs, snapshots, and tab…
- Iceberg Truncate Partition TransformAn Iceberg partition transform that truncates string values to a specific width or groups numeric values into fixed-size…
Patterns & Architecture
- Iceberg CDC (Change Data Capture)CDC with Apache Iceberg enables real-time synchronization of operational database changes (inserts, updates, deletes) in…
- Iceberg Data Mesh ArchitectureA data mesh on Apache Iceberg uses Iceberg tables as the storage standard for domain-owned data products, with the Icebe…
- Iceberg Lakehouse FederationIceberg lakehouse federation enables querying Iceberg tables across multiple catalogs, cloud environments, and storage p…
- Iceberg Streaming IngestionIceberg streaming ingestion is the pattern of continuously writing data from event streams, Kafka topics, and CDC feeds …
- Iceberg Table Migration from HiveMigrating from Apache Hive tables to Apache Iceberg converts existing Parquet files into Iceberg-managed tables with ful…
- Medallion Architecture with Apache IcebergThe Medallion Architecture (Bronze/Silver/Gold) is a multi-layer data organization pattern where raw data flows through …
- Write-Audit-Publish (WAP) PatternThe Write-Audit-Publish (WAP) pattern is a data pipeline quality assurance workflow using Apache Iceberg branches to wri…
📚 Go Deeper on Apache Iceberg
Alex Merced has authored three hands-on books covering Apache Iceberg, the Agentic Lakehouse, and modern data architecture. Pick up a copy to master the full ecosystem.