Luminousmen Blog - Python, Data Engineering & Machine Learning

I've read a few posts this past week – every single one promises something incredible, each with a big cool number: real-time queries 16x faster, storage 40x faster, point lookups at O(1).

Imagine the following scenario: a family credit account has a limit of $1000 a month, shared across both cards.

We usually talk about engineering like it's a technical discipline.

A teammate finds a bug that's been quietly corrupting the last month of data.

I look at technically perfect AI art and feel nothing.

In software engineering, the term "design pattern" (which, honestly, is kind of redundant — aren't all patterns designs?)

I'm not gonna tell you what to do.

Somewhere, right now, a VP of Engineering is standing in front of her data team giving The Speech.

I've made every dumb Spark mistake at least once.

In the last post, we covered the landscape of data sketches and why they matter.

You know that feeling when your COUNT DISTINCT query has been running for three hours and your manager is breathing down your neck?

I see it everywhere now.

I've watched too many talented senior data engineers quit over the years.

BigQuery pricing is weirdly easy to misunderstand.

⚠️ Warning: This isn't a "what is BigQuery?"

There's an old psychology experiment that still messes with my head when I think about it.