blog.nisshi.io - Engineering

Achieving 7000 msg/s (~6.7MB/s) with just 27MB of memory on a free t3.micro EC2 in AWS

Using flame graphs to remove a hot path, stop copying data and switching to a fast CRC32

Writing 585 MB/s with just 13MB of memory usage.

Using SQLite as a storage engine with Nisshi, a Kafka compatible streaming platform, producing and consuming Protobuf messages using generated test data.

Learn how Nisshi uses the Service and Layer traits to route, layer and process Apache Kafka messages in a modular and composable way.

Learn how to use Nisshi to validate and automatically convert Kafka messages into Apache Parquet format, simplifying data processing.

Nisshi, a Kafka-compatible broker using S3 or PostgreSQL, undergoes automated smoke testing with GitHub workflows and BATS.

In this article we deploy Nisshi on Fly using Tigris Data S3 compatible storage.

Using serde, quote, syn and proc_macro2 to implement the Kafka protocol in Rust.