Miles Cole

Leased a blob?

Liquid Clustering was already a better abstraction than static partitioning due to its flexible nature.

Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo.

Schema inference is convenient.

I’m guilty.

Coming from a notebook-first Spark background, I wanted to write the introduction to Spark Job Definitions (SJDs) that I wish I had when I first encountered them.

I’m excited to formally announce LakeBench, now in version v0.3, the first Python-based multi-modal benchmarking library that supports multiple data processing engines on multiple benchmarks.

Last December (2024) I published a blog seeking to explore the question of whether data engineers in Microsoft Fabric should ditch Spark for DuckDb or Polars.

This is part 2 of my prior post that continues where I left off.

If there anything that data engineers agree about, it’s that table compaction is important.