The Sympathetic Ink Blog

In the past few years, we’ve seen a Cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future).

This is just me sharing my experience trying out AI coding.

The topic of software architecture has become a bit cringe.

As I looked back at my experience working in data engineering for this post, I realized I never really consciously decided to specialize in data.

Lineage has long been a requirement for anyone processing data - whether for complying with regulations, ensuring data reliability or, to quote Marvin Gaye, plainly just knowing what’s going on from provenance to impact analysis.

In 2018, I wanted to describe how the components of databases, distributed or not, were being commoditized as individual parts that anyone could recombine into use-case specific engines.

Recently I’ve signed up for an improv class and it’s been a lot of fun.

Much the same as there was a common need for a columnar file format and a columnar in-memory representation, there's a common need for lineage across the data ecosystem.

In 2015, a discussion started in the Parquet community around the need for an in-memory columnar format.

15 years ago (2007-2011) I was at Yahoo!