rmoff's random ramblings

The Kafka Connect JDBC Sink can be used to stream data from a Kafka topic to a database such as Oracle, Postgres, MySQL, DB2, etc.

I got the error SQLSyntaxErrorException: BLOB/TEXT column 'MESSAGE_KEY' used in key specification without a key length with Kafka Connect JDBC Sink connector (v10.0.2) and MySQL (8.0.23)

ksqlDB is a fantastically powerful tool for processing and analysing streams of data in Apache Kafka.

Kibana’s map functionality is a powerful way to visualise data that has a location element in it.

Whilst Apache Kafka is an event streaming platform designed for, well, streams of events, it’s perfectly valid to use it as a store of data which perhaps changes only occasionally (or even never).

Some people learn through doing - and for that there’s a bunch of good ksqlDB tutorials here and here.

One of the fun things about working with data over the years is learning how to use the tools of the day—but also learning to fall back on the tools that are always there for you - and one of those is bash and its wonderful library of shell tools.

tl;dr: specify the --user root argument: docker exec --interactive \ --tty \ --user root \ --workdir / \ container-name bash

Confluent Cloud is not only a fully-managed Apache Kafka service, but also provides important additional pieces for building applications and pipelines including managed connectors, Schema Registry, and ksqlDB.

When Kafka Connect ingests data from a source system into Kafka it writes it to a topic.

KIP-66 was added in Apache Kafka 0.10.2 and brought new functionality called Single Message Transforms (SMT).

Apache Kafka ships with many Single Message Transformations (SMT) included - but the great thing about it being an open API is that people can, and do, write their own transformations.

Apache Kafka 2.6 included KIP-585 which adds support for defining predicates against which transforms are conditionally executed, as well as a Filter Single Message Transform to drop messages - which in combination means that you can conditionally drop me…

The ReplaceField Single Message Transform has three modes of operation on fields of data passing through Kafka Connect: Include only the fields specified in the list (include) Include all fields except the ones specified (exclude) Rename field(s) (renames)

Over the years I’ve used various blogging platforms; after a brief dalliance with Blogger I started for real with the near-inevitable Wordpress.com.

The Cast Single Message Transform lets you change the data type of fields in a Kafka message, supporting numerics, string, and boolean.

The TimestampConverter Single Message Transform lets you work with timestamp fields in Kafka messages.

Just like the RegExRouter, the TimeStampRouter can be used to modify the topic name of messages as they pass through Kafka Connect.

We kicked off this series by seeing on day 1 how to use InsertField to add in the timestamp to a message passing through the Kafka Connect sink connector.

If you want to mask fields of data as you ingest from a source into Kafka, or write to a sink from Kafka with Kafka Connect, the MaskField Single Message Transform is perfect for you.

If you want to change the topic name to which a source connector writes, or object name that’s created on a target by a sink connector, the RegExRouter is exactly what you need.

The Flatten Single Message Transform (SMT) is useful when you need to collapse a nested message down to a flat structure.

Setting the key of a Kafka message is important as it ensures correct logical processing when consumed across multiple partitions, as well as being a requirement when joining to messages in other topics.

You can use the InsertField Single Message Transform (SMT) to add the message timestamp into each message that Kafka Connect sends to a sink.

Back in March 2020 the western world came to somewhat of a juddering halt, thanks to COVID-19.

Is a blog even a blog nowadays if it doesn’t include a "Here is my home office setup"?

Very short & sweet this post, but Google turned up nothing when I was stuck so hopefully I’ll save someone else some head scratching by sharing this.

As you may already realise, Kafka is not just a fancy message bus, or a pipe for big data.

Streaming data from Kafka to Elasticsearch is easy with Kafka Connect - you can see how in this tutorial and video.

There was a good question on StackOverflow recently in which someone was struggling to find the appropriate ksqlDB DDL to model a source topic in which there was a variable number of fields in a STRUCT.

Let’s imagine we have XML data on a queue in IBM MQ, and we want to ingest it into Kafka to then use downstream, perhaps in an application or maybe to stream to a NoSQL store like MongoDB.

👉 Ingesting XML data into Kafka - Introduction We saw in the first post how to hack together an ingestion pipeline for XML into Kafka using a source such as curl piped through xq to wrangle the XML and stream it into Kafka using kafkacat, optionally using…

We previously looked at the background to getting XML into Kafka, and potentially how [not] to do it.

👉 Ingesting XML data into Kafka - Introduction What would a blog post on rmoff.net be if it didn’t include the dirty hack option?

XML has been around for 20+ years, and whilst other ways of serialising our data have gained popularity in more recent times (such as JSON, Avro, and Protobuf), XML is not going away soon.

Short & sweet to help out future Googlers.

Running IBM MQ in a Docker container and the client connecting to it was throwing repeated Channel was blocked errors.

One of my favourite hacks for getting data into Kafka is using kafkacat and stdin, often from jq.

Readers of a certain age and RDBMS background will probably remember northwind, or HR, or OE databases - or quite possibly not just remember them but still be using them.

Here’s a collection of Kafka-related talks, just for you.

Prompted by a question on StackOverflow I thought I’d take a quick look at setting up ksqlDB to ingest CDC events from Microsoft SQL Server using Debezium.

I use Hugo for my blog, hosted on GitHub pages.

Kafka Connect is the integration API for Apache Kafka.

There’s ways, and then there’s ways, to count the number of records/events/messages in a Kafka topic.

Google Chrome automagically adds sites that you visit which support searching to a list of custom search engines.

I had the pleasure of presenting at DataEngBytes recently, and am delighted to share with you the 🗒️ slides, 👾 code, and 🎥 recording of my ✨brand new talk✨: 🤖Building a Telegram bot with Apache Kafka, Go, and ksqlDB

A tiny snippet since I wasted 10 minutes going around the houses on this one… tl;dr: If you try to create a command that is not in lower case (e.g.

The server sends Transfer-Encoding: chunked data, and you want to work with the data as you get it, instead of waiting for the server to finish, the EOF to fire, and then process the data?

At the beginning of all this my aim was to learn something new (Go), and use it to write a version of a utility that I’d previously hacked together in Python that checks your Apache Kafka broker configuration for possible problems with the infamous advert…

So far I’ve been running all my code either in the Go Tour sandbox, using Go Playground, or from a single file in VS Code.