All posts

Iceberg views: a versioned view definition, and the cross-engine promise tested

An Iceberg view is a metadata file with versions, schemas and one SQL representation per dialect - a view that is tracked the way a table is. Here is what that file contains, what a replace actually does to it, and what happened when a view written by Spark was read from Trino.

12 min read Iceberg

TL;DR

  • An Iceberg view is not SQL text in a metastore column. It is a metadata JSON file with schemas, versions, a version-log and one SQL representation per dialect — the same treatment a table gets.
  • CREATE OR REPLACE VIEW does not overwrite. Three replaces produced three versions and two schemas, all retained, with the old definitions still readable in the metadata.
  • The engine marks its work: Spark wrote table_type = ICEBERG-VIEW into the Hive Metastore alongside a metadata_location. That marker is how an engine tells an Iceberg view from a Hive view.
  • The cross-engine promise did not hold in this setup, in either direction. Trino listed the Spark-created view as table_type = VIEW and then failed to query it with Not an Iceberg table. Spark could not see the view Trino created at all.
  • The reason is visible in the metadata: Spark stored one representation, dialect: "spark". An engine can only use a representation it understands — and over a Hive Metastore, Trino created a plain Hive view rather than an Iceberg one.
  • Views are a catalog capability, not a format feature you get everywhere. What works depends on the catalog and the engine’s support for it, which is worth testing before designing around it.

A view is the oldest abstraction in analytics: name a query, let people use the name. What Iceberg adds is that the definition stops being an untracked string in a metastore and becomes a versioned object with a schema, exactly like a table.

Whether that buys you the thing it is usually sold for — one view definition, many engines — is a separate question, and one worth answering by running it.

What a Hive view could not do

A Hive view is a row in the metastore holding SQL text. That has four consequences that show up in practice:

  • No versioning. Replacing it destroys the previous definition. There is no history, no way to see what it used to say, and no way back.
  • No tracked schema. The columns are whatever the text happens to produce today. A change upstream silently changes the view’s shape.
  • One dialect. The text is written in whichever engine’s SQL created it. Another engine parses it and hopes.
  • No identity. Nothing distinguishes a view definition from any other metastore entry beyond its type.

Iceberg’s view spec answers the first three directly, and the fourth turns out to matter more than it sounds.

Creating one, and what lands where

The syntax is unremarkable:

CREATE TABLE ice.viewdemo.t (id INT, city STRING, amt DOUBLE) USING iceberg;
INSERT INTO ice.viewdemo.t VALUES (1,'paris',10.0),(2,'lyon',20.0);

CREATE OR REPLACE VIEW ice.viewdemo.v AS
  SELECT city, amt FROM ice.viewdemo.t WHERE amt > 5;

What lands in the Hive Metastore is not:

TBL_TYPE        = VIRTUAL_VIEW
table_type      = ICEBERG-VIEW
metadata_location = s3a://warehouse/viewdemo.db/v/metadata/00000-....gz.metadata.json
current-schema  = {"type":"struct","schema-id":0,"fields":[...]}
engine_version  = Spark 3.5.9

Two things to notice. The metastore entry is a pointer, exactly as it is for an Iceberg table — the real definition lives in a metadata file on object storage. And the marker is ICEBERG-VIEW, not ICEBERG. An engine reading the metastore uses that string to decide what it is looking at, which becomes the whole story further down.

The metastore also keeps VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT populated with the SQL, for the benefit of tools that only know how to read a Hive view.

What is in the metadata file

This is the part worth reading in full, because the structure is the feature:

{
  "view-uuid": "242992fd-307c-4fcb-96bf-82ee51527149",
  "format-version": 1,
  "location": "s3a://warehouse/viewdemo.db/v",
  "schemas": [
    { "schema-id": 0, "type": "struct", "fields": [
        { "id": 0, "name": "city", "required": false, "type": "string" },
        { "id": 1, "name": "amt",  "required": false, "type": "double" } ] }
  ],
  "current-version-id": 1,
  "versions": [
    {
      "version-id": 1,
      "timestamp-ms": 1790567461060,
      "schema-id": 0,
      "summary": {
        "engine-name": "spark",
        "engine-version": "3.5.9",
        "iceberg-version": "Apache Iceberg 1.11.0 (commit 6976e020...)"
      },
      "default-catalog": "spark_catalog",
      "default-namespace": ["default"],
      "representations": [
        { "type": "sql",
          "sql": "SELECT city, amt FROM ice.viewdemo.t WHERE amt > 5",
          "dialect": "spark" }
      ]
    }
  ],
  "version-log": [ { "timestamp-ms": 1790567461060, "version-id": 1 } ]
}

Five things are doing work here:

  • schemas with field ids, the same mechanism that makes column renames free in a table. A view’s shape is recorded, not inferred.
  • versions, each a complete definition. Replacing adds one rather than overwriting.
  • representations, a list — and each carries a dialect. This is the cross-engine mechanism: one view can hold the same query written for several engines.
  • default-catalog and default-namespace, captured at creation, so that unqualified names in the SQL resolve the way they did when it was written rather than the way the current session happens to be configured.
  • summary, naming the engine and Iceberg version that produced it — useful when a definition behaves differently than expected.

Replacing a view keeps the old one

CREATE OR REPLACE VIEW reads as destructive and is not. Running it twice more, the second time also changing the column list:

CREATE OR REPLACE VIEW ice.viewdemo.v AS SELECT city, amt FROM ice.viewdemo.t WHERE amt > 15;
CREATE OR REPLACE VIEW ice.viewdemo.v AS SELECT city      FROM ice.viewdemo.t WHERE amt > 15;

The metadata afterwards:

current-version-id: 3
versions: 3   schemas: 2
version-log: [1, 2, 3]
  v1  schema-id=0  dialect=spark: SELECT city, amt FROM ice.viewdemo.t WHERE amt > 5
  v2  schema-id=0  dialect=spark: SELECT city, amt FROM ice.viewdemo.t WHERE amt > 15
  v3  schema-id=1  dialect=spark: SELECT city      FROM ice.viewdemo.t WHERE amt > 15

Every definition is still there. A second schema appeared only when the column list actually changed — versions 1 and 2 differ in their predicate and share schema-id 0, while version 3 dropped a column and got schema-id 1.

So a view carries the same kind of audit trail a table does: what the definition was, when it changed, which engine changed it, and what shape it had at the time. That alone is a real improvement on a metastore string, regardless of how the cross-engine story turns out.

The cross-engine test

The headline claim for Iceberg views is that one definition serves many engines. The stack here has Spark and Trino over the same Hive Metastore and the same storage, which is the setup that claim describes. So:

-- in Trino
SHOW TABLES FROM iceberg.viewdemo;

"t"
"v"

Trino sees it. It even classifies it correctly:

SELECT table_name, table_type FROM iceberg.information_schema.tables
WHERE table_schema = 'viewdemo';

"t","BASE TABLE"
"v","VIEW"

And then:

SELECT * FROM iceberg.viewdemo.v;

Query ... failed: Not an Iceberg table: viewdemo.v

The other direction is no better. A view created by Trino:

-- in Trino: succeeds, and Trino can query it
CREATE OR REPLACE VIEW iceberg.viewdemo.tv AS
  SELECT city, amt FROM iceberg.viewdemo.t WHERE amt > 5;

-- in Spark
[TABLE_OR_VIEW_NOT_FOUND] The table or view `ice`.`viewdemo`.`tv` cannot be found
View created by Stored as Readable in Spark Readable in Trino
Spark Iceberg view (table_type = ICEBERG-VIEW) yes no
Trino Hive view (no table_type at all) no yes

Neither engine could read the other’s view.

Why it failed, precisely

Two distinct causes, and it is worth separating them because they have different answers.

Trino did not create an Iceberg view. Checking what each engine wrote into the metastore:

v   ->  table_type = ICEBERG-VIEW     (Spark)
tv  ->  table_type = <none>           (Trino)

Trino’s Iceberg connector here is configured with iceberg.catalog.type=hive_metastore, and in that configuration it created a plain Hive view — no Iceberg view metadata file, no versions, no dialect. So Spark’s Iceberg catalog, which looks for the ICEBERG-VIEW marker, does not consider it one of its objects at all. That is not a bug in either engine; it is what the Hive-metastore catalog implementation does.

Spark stored one dialect. Look again at the representation list:

"representations": [
  { "type": "sql", "sql": "...", "dialect": "spark" }
]

One entry, dialect: "spark". The spec’s design is that a view holds several representations and an engine picks the one it can parse. A view written by Spark has only Spark’s, so a reader that will not execute Spark SQL has nothing it can use — even once it recognises the object.

That is the honest shape of the feature. The format defines a multi-dialect container; filling it with more than one dialect is something an engine or a tool has to do, and neither engine does it for you.

So when are they worth using?

They are still worth using, but for the versioning rather than the portability — unless you have tested the portability on your own catalog.

  • Single-engine, today. If Spark writes and Spark reads, an Iceberg view gives you version history, a tracked schema and an audit trail that a Hive view does not. That is a real gain with no cross-engine assumption.
  • Test your catalog before designing around portability. The result above is specific to a Hive Metastore catalog with these engine versions. A REST catalog is the configuration where cross-engine view support is actively developed, and it is the first thing to try if portability is the goal.
  • Treat the dialect as part of the contract. If two engines must read one view, something has to write a representation for each. Check what is in representations rather than assuming.
  • A shared table is a safer contract than a shared view. Both engines read the same Iceberg table in this stack without any of this trouble. If you need one definition consumed by several engines today, materialising it — a table refreshed by a job — moves the problem to ground where interoperability already works.

Views and materialized views

They solve different problems and are easy to conflate. A view stores a query and runs it on every read; it costs nothing to keep current because there is nothing to keep. A materialized view stores the result and therefore needs a refresh strategy and a staleness policy.

Iceberg’s view spec covers the former. Materialized views exist in some engines on top of Iceberg tables, with the refresh handled by the engine rather than the format, which is why their behaviour varies much more between engines than views do.

Common misconceptions

“An Iceberg view is portable across engines.” It is designed to be. Whether it is depends on your catalog and both engines’ support for it, and in the setup measured here it was not portable in either direction.

“CREATE OR REPLACE VIEW overwrites the definition.” It adds a version. The previous definitions stay in the metadata file and the version-log.

“The view’s SQL lives in the metastore.” The metastore holds a pointer plus a compatibility copy of the text. The definition of record is the metadata file.

“A view has no schema until you query it.” It has a recorded schema with field ids, and a new one is added when the column list changes.

“Trino cannot do Iceberg views.” Trino created a view happily — just a Hive one, because of how this catalog is configured. That is a configuration outcome, not a capability statement.

Frequently asked questions

How do I tell an Iceberg view from a Hive view? Check table_type in the metastore properties. ICEBERG-VIEW means there is a metadata file; nothing means it is a Hive view.

Can I see a view’s previous definitions? Yes — read the metadata file the metadata_location property points at. Every version and its SQL is in versions, and the order of changes is in version-log.

Does dropping a column from the underlying table break the view? The view keeps its recorded schema, but the query inside it will fail at read time if it references a column that no longer exists. The schema is a record, not a guarantee about the base tables.

Do views support time travel? The view is versioned, not the data. Reading through a view reads the current state of the underlying tables. For data-level time travel, the time travel post applies to the tables themselves.

Which catalogs support views? It varies, and this is the question to answer for your own setup first. The Hive Metastore catalog stored them from Spark in this test; a REST catalog is where cross-engine support is most actively developed.

Conclusion

The versioning half of Iceberg views delivers exactly what it claims. Three replaces left three definitions and two schemas in a metadata file with an audit trail, where a Hive view would have left one string and no history. If you write views from one engine, that is reason enough to prefer them.

The portability half is a design, not yet a guarantee. The spec’s representations list is built for many dialects, and what actually got written was one — so the same view that Spark reads fine was, to Trino, an object it could list, classify as a view, and refuse to query.

Which is the useful lesson, and it generalises past views: the format defining something does not mean your catalog and your engines implement it. Between two engines over one metastore, the reliable shared object is still a table.

References

Trademarks

Apache Iceberg, Apache Spark, Apache Hive, Apache and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Trino is a trademark of the Trino Software Foundation.

Found this useful?

These posts and tools are free. If one saved you an afternoon, you can buy me a coffee.

Buy me a coffee