Sem Sinchenko

Blogpost about an implementation of KMeans clustering algorithm using Apache DataFusion DataFrame APIs and operations on relations.

GraphFrames is distributed graph algorithms using Apache Spark as a backend.

A SKILL.md that makes my harness write an onboarding ticket instead of code: what to change, where, and why instead of copy-pasteable diffs.

In a previous post, I was sceptical about using Apache DataFusion for graph analytics.

A small Java + Apache Arrow + Vector API benchmark against native library code.

Yes, I really ended up running headless Emacs in Docker Compose.

About a year ago, I got involved in reviving and maintaining GraphFrames, a 10-year-old OSS project with a lot of history and not enough active maintenance.

In this blog post, I will share my thoughts on the current hype around "agentic coding" and why I still use Aider for human-in-the-loop pair programming in 2026.

In this blog post, I will recount my experience working on the addition of the graph embeddings API to the GraphFrames library.

I don’t think anyone besides me has considered using Apache DataFusion to write graph algorithms.

In this blog post, I will provide an end-to-end example of how to integrate JMH benchmarks for an Apache Spark-based library into an SBT build.

While Open Lakehouse platforms now natively support tables, geospatial data, vectors, and more, property graphs are still missing.

The question about why Apache Spark is "slow" is one of the most often questions I'm hearing from junior engineers and peoples I'm mentoring.

In this blog post, I will provide a brief high-level overview of projects designed to accelerate Apache Spark by the native physical execution, including Databricks Photon, Apache Datafusion Comet, and Apache Gluten (incubating).

Preface I would like to express my gratitude to Matthew Powers for testing my project and providing feedback, and to Steve Russo for offering a valuable review of my code and drawing my attention to avoiding the use of unwrap.

Open table formats like Apache Iceberg and Delta are evolving rapidly today.

Summary This blog post is a detailed story about how I ported a popular data quality framework, AWS Deequ, to Spark-Connect.

Databricks recently open-sourced Unitycatlog, a unified data catalog that aims to provide a single source of truth for data discovery, governance, and access control across multiple systems.

Leveraging Time-Based Feature Stores for Efficient Data Science Workflows In our previous post, we briefly touched upon the concept of ML feature stores and their significance in streamlining machine learning workflows.

In this blog post, I will share my experience in building an ML Feature Store using PySpark.

This blog post presents a very detailed step-by-step guide on how to create a SparkConnect protocol extension in Java and call it from PySpark.

I recently had the opportunity to work on an open source project that implements a custom Apache Spark data source and associated logic for working with graph data.

In this post, I want to discuss the groundbreaking changes in the latest LTS release of the Databricks runtime.

In this post, I will show you how to use information from the spark plan to track data lineage at the column level.

Sometimes it is an important question, how much memory does our DataFrame use?

I would like to tell you about my bicycle trip through Eastern Serbia.

The problem Recently I decided to switch from Ubuntu to NixOS.

Generating Python docstrings with GPT and Emacs Motivation There is an open source library in which I'm a maintainer.

Working with File System from PySpark Motivation Any of us is working with File System in our work.