Blogs on Data Engineering Blog & Second Brain

Data orchestrators have been essential since the inception of data workloads, because you need something to orchestrate your tasks and your business logic.

Shifting left is an interesting concept that’s gaining momentum in modern data engineering.

The database landscape has reached 394 ranked systems across multiple categories—relational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases.

Many data engineers and analysts start their journey with Postgres.

Geospatial data is everywhere in modern analytics.

Johann Hari says in Stolen Focus that rats and pigeons can be manipulated as we want.

While everyone’s talking about AI revolutionizing business, there’s a quiet renaissance happening with one of the most influential business tools created: the pivot table.

To be proficient as a data engineer, you need to know various toolkits—from fundamental Linux commands to different virtual environments and optimizing efficiency as a data engineer.

What started as a straightforward implementation guide for a declarative data stack quickly evolved into something more fundamental.

Data-driven decision-making is crucial for business success, but organizations face a growing challenge of complexity and data governance.

Modern data teams face growing complexity: multiple data sources, various BI tools, and ever-increasing self-service analytics.

Do you remember the good old times of Twitter?

MVC is an interesting concept from the late 70s that separates the View (presentation) from the Controller via the Model.

From Fortune 500 companies processing trillions of security records to innovative startups building interactive data tools, DuckDB is revolutionizing how organizations handle analytical workloads.

Imagine creating business dashboards by simply describing what you want to see.

DuckDB has a significant share1 and is frequently featured in the latest data engineering news.

Data stacks have come a long way, evolving from monolithic, one-fits-all systems like Oracle/SAP to today’s modular open data stacks.

I’m currently on vacation, and it is time to dive into one of my favorite topics: knowledge workflow management.

In my journey, detailed in why Vim is more than an editor, I’ve discovered the impact of integrating Vim and its motions into my entire computer workflow.

Welcome to the third and final installment of our series “Data Modeling: The Unsung Hero of Data Engineering.”

In case you missed Part 1, An Introduction to Data Modeling, make sure to check first, where we discussed the importance of data modeling in data engineering, the history, and the increasing complexity of data.

Amidst the excitement and hype surrounding artificial intelligence, the significance of data engineering and its critical foundation—data modeling—can often be overlooked.

Data manipulation and analysis can be challenging and involve working with large datasets.

As I sit down to write this article, I’m filled with a sense of vulnerability and excitement.

In part I, The Open Data Stack Distilled into Four Core Tools, we discussed how to quickly set up a data stack, tackling end-to-end data analytics challenges.

Throughout my time as a developer, I’ve used VS Code, Sublime, Notepad++, TextMate, and others.

In this article, we are going to explore core open-source tools that are needed for any company to become data-driven.

I tried the new [[Dagster]] feature to configure the Airbyte with Python code.

Will Rust kill Python for Data Engineers?

A [[semantic layer]] is something we use every day.

Image by Rachel Claire on Pexels Ever wanted or been asked to build an open-source [[Data Lake]] offloading data for analytics?

Data consumers, such as data analysts, and business users, care mostly about the production of data assets.

With burnout and mental stress at every level of our lives, I find my Personal Knowledge Management (PKM) system even more valuable.

Image by Mohammad Bagher Adib Behrooz on Unsplash Why GraphQL for data engineers, you might ask?

Taking notes helps you not to forget things, teaches you to express yourself, brainstorms your thoughts, research a topic, and so many more things.

A brief of history When I began my journey with websites, I started as many of us with HTML, CSS, HTML & MySQL running on an Apache Webserver.

Project Updates on 2024-03-17 After three years, recognizing the evolving landscape of data engineering tools and the ongoing relevance of this project, I’ve made several updates to Practical Data Engineering.

Today we have more requirements with ever-growing tools and framework, complex cloud architectures, and with data stack that is changing rapidly.

The first example of email can be found on computers at MIT in a program called MAILBOX, all the way back in 1965.

These days, everyone talks about open-source.

Are you on the lookout for a replacement for the Microsoft Analysis Cubes, are you looking for a big data OLAP system that scales ad libitum, do you want to have your analytics updated even real-time?

Today’s Office is something I started doing while I was working on the road and working remotely.

Tools I use part III will focus on increasing your productivity in communication, grammar, deep work or study, little notes and in Google Chrome.

Today, there are 6,500 people on LinkedIn who call themselves data engineers according to stitchdata.com.

There is a bit of a confusion between Data Warehouse vs Data Lake or ETL vs ELT.

As promised in part I here is the part II of tools I use and in this part, I will entirely focus on one of my most used and favourite tool called Microsoft OneNote.

The 5 top most searched Data Warehouse Automation tools on the market compared with GoogleTrends is telling you that WhereScape is first before TimeXtender and BiReady (new Attunity Compose) over the last year.

This article is for you if you considering to use Data Warehouse Automation (DWA) and asking yourself why you should use Data Warehouse Automation tools what does it do for you.

I was working with a Data Warehouse Automation (DWA) tool for a little more than a year, and I have to say I loved it.

Very long time ago, 1986, 31 years ago to be precise, IBM in Europe created the very first architecture of a data warehouse.