cdelmonte.dev

How delta-explain turns one predicate and the Delta log into a trustworthy explanation of file pruning, through shared representation, safe rewrites, and conservative measurement.

On the JVM, optimizing a hot kernel is not only about writing faster code: it is also about understanding how much the result depends on the machine code HotSpot derives from the scalar loop.

Run df.explain on a Parquet scan and you find ColumnarToRow, an operator whose only job is to change the data’s shape.

How Spark’s internal rewriting framework, Catalyst, exposes an embedded DSL with a public extension surface, the same one Delta Lake and Iceberg use to plug into the optimizer pipeline.

Six concrete anti-patterns I encountered building a real Catalyst extension: from the wrong rule type for throws, to mutable state under AQE and Spark Connect, to JVM bootstrap traps in PySpark.

Partition pruning and data skipping are invisible by default. delta-explain reads the Delta log directly and shows, step by step, how a WHERE predicate narrows down candidate files, with no engine required.

How Delta Kernel, Arrow, and pluggable execution are disaggregating the lakehouse stack.

Many of the most surprising performance pathologies in modern data systems are semiotic failures: structural divergences between what an interface signifies and what the underlying system does.

In large-scale Spark pipelines, skew can occur when a single key carries a disproportionately large nested payload.

Why calling Spark ’lazy’ is technically reductive, and how thinking of it as a dataflow compiler changes the way you design pipelines.

Parsing arithmetic expressions looks simple… until precedence enters the picture.

GKE Behind the Scenes: Understanding the Interaction Between Kubernetes and GCP Service Accounts Through The Metadata Server.