Blogs on Data Engineering Blog & Second Brain

This is my Part 3 about Omarchy, after using it for more than a year full time, as my main daily driver first on Lenovo ThinkBook 14 AMD, and now TUXEDO InfinityBook Pro 14 Gen10, for my business and personal use, replacing my MacBook Pro M1 after using m…

I’ve been using a personal second brain for many years.

In Part 1, we discovered the grammar for data: a way to define a data project with its complex requirements and how we define it declaratively as a grammar in one sentence with nouns (sources), transformations (verbs), templates, and modifiers, essentially

There’s so much talk about new ways of working with agent engineering supported workflows.

I had the pleasure to discuss and analyze the workflow of none other than Benn Stancil, legendary writer and storyteller in the data space and beyond, who started writing back in 2013 (with a seven-or-eight-year startup break in between).

[[Writing]] has become my passion over time; I have built a whole company based on technical writing services (even though most say that’s impossible).

It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them?

Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious.

Grammars for languages or any other field are a beautiful thing.

With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily.

I’ve taken notes over my whole life.

Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines.

This series interviews real practitioners to extract the patterns behind how they actually use AI in their data work today.

Writing SQL was never the hard part.

I have read the “Poor Charlie’s Almanack” by Charlie Munger and thought about what it would take to write one for Dagster.

This series interviews (see #1 with Mark Freeman) real practitioners to extract the patterns behind how they actually use AI in their data work today.

When I started my career as a data warehouse engineer and business intelligence engineer in 2003, external tables with materialized views were the standard.

Ask a BI engineer what they actually spend their time on: it’s not building dashboards.

There’s so much going on in the AI space, and how to work with AI agents is changing every day.

Imagine going from a 100-million-row dataset to an interactive analytics app with just a few prompts.

I’ve been online twenty years, and blogging for ten of them.

Continuing from Part 1, where we learned what git for data is, how the architecture and use cases work, how you can achieve git-like functionality with different approaches, and how the key is to avoid moving data as much as possible to keep state that can

I always wanted a personal knowledge assistant based on my notes.

This is a follow-up to my part 1 of Switching macOS to Arch Linux with Omarchy, where I documented my first months with Arch Linux and [[Omarchy]], after switching from 15 years of using macOS and Windows on and off at work since 2003.

Why is StarRocks gaining popularity among data engineers who need fast analytics on large-scale data?

You ingest data.

To be able to live well, to avoid anxiety and depression, it’s not that easy anymore in 2025.

With the recent consolidation in the area of Modern Data Stack, the question arises: are open data stacks, a more flexible but labor-intensive way of building data platforms, out of fashion?

A constant struggle in data is to make everything fast.

Following up on Part 1, where we created an entire end-to-end FinOps project to analyze cloud costs from different hyperscalers with dlt and local parquet files.

Remember the 2 AM on-call duty when a recent data pipeline broke the production environment?

Companies often use multiple platforms simultaneously: Lovable for apps, OpenAI and Claude for AI, AWS and GCP for infrastructure.

Today, I sent out a newsletter that is distributed only via email.

Every day, thousands of data engineers scroll through r/dataengineering (174K members strong) looking for answers to the same fundamental questions: How do I prepare for interviews in this market?

In data analytics, we’re facing a paradox.

Most data teams spend 80% of their time wrestling with infrastructure—writing custom UPSERT logic, building incremental loading from scratch, or debugging why their Airflow DAGs failed again at 2 AM.

When I [[Writing|write]], I deliberately don’t research.

Querying billions of weather records and getting results in under 200 milliseconds isn’t theory; it’s what real-time analytics solutions provide.

Many ask themselves, “Why would I use a semantic layer?

I switched my five-year-old MacBook Pro M1 Max for a cheap (comparable) Lenovo ThinkBook 14 G7 ARP (AMD) laptop, running Linux (Arch btw, or better, Omarchy.

With this summer edition, you’ll have a roadmap for your vacation time to learn the basics of being a full-stack data engineer.

In the quest for happiness, it’s very important to know why we are on this earth, right?

Remember when data scientists spent 80% of their time wrestling with data wrangling instead of building models?

Two decades ago, we talked about self-serve business intelligence when we could filter dashboards compared to pixel-perfect reports.

Moving from orchestration theory to the enterprise level is a real challenge.

Wouldn’t it be great to build a data warehouse on top of affordable storage and scattered files?

After watching the two PewDiePie videos where he learned about installing Arch (something considered quite hard, even for Linux enthusiasts) and building three products (camera for the dog, weather/drinking/meditation device, and who knows what comes next)

Wondering why open table formats are suddenly booming?

Imagine building enterprise data infrastructure where you write 90% less code but deliver twice the value.

As data engineers, we spend countless hours combing through logs - tracking pipeline states, monitoring Spark cluster performance, reviewing SQL queries, investigating errors, and validating data quality.