Mark Needham

A problem that I’m currently trying to solve is how to work out whether a given sentence is a question.

I’m taking some tentative steps into the world of batch data pipelines and I’ve been following Dagster’s DuckDB tutorial when I ran into a dependency issue that I had to work around.

A few times this week I’ve run into an interesting problem with Python version requirements when trying to install various packages.

I’m using the Poetry package manager for all my Python projects these days and wanted to upgrade as library that I installed a few weeks ago.

Last week Mistral AI announced the release of their first Large Language Model (LLM), trained with 7 billion parameters, and better than Meta’s Llama 2 model with 13 billion parameters.

I recently wanted to get a list of the views that I’d created in a DuckDB database and it took me a while to figure out how to do it.

I’ve been building a little demo with dbt and DuckDB to transform CSV files from Jeff Sackmann’s tennis dataset and ran into an error that initially puzzled me.

Last week OpenAI sent out the following email introducing the gpt-3.5-turbo-instruct large language model: Figure 1.

I’ve been learning about vector search in recent weeks and I came across FaceBook’s FAISS library.

I’ve been updating the StarTree Kafka SASL recipe to use Pinot 0.12 and ran into an error while trying to have it use kcat to ingest data into Kafka.

I was recently helping Tim get a Pinot data-loading Kotlin script working and it took me a while to figure out the best way to run it.

In a recent demo, I wanted to use Quix Streams to process a specified number of messages from a Kafka topic, write a message to another stream, and then exit the Quix app.

In the latest version of Jupyter Notebook/Lab (at least), the output of each cell is shown in full, regardless of how long it is.

I wanted to filter a table in pyarrow table recently and ran into troubles when trying to use the filter syntax that I’m used to from DuckDB.

I’ve been playing around with ChromaDB and I wanted to programatically get a list of the embedding functions, which was a little trickier thna I expected.

I’ve been using Jupyter Lab notebooks in some of my recent videos on Learn Data with Mark and I wanted to show cell execution timings so that viewers would have an idea of how long things were taking.

My colleagues Sandeep Dabade and Kulbir Nijjer recently wrote a three part blog post series about the StarTree index, an Apache Pinot indexing technique that dynamically builds a tree structure to maintain aggregates across a group of dimensions.

As I mentioned in a previous blog post, I’ve been using Python’s Poetry library, but today it stopped working!

When using OpenAI it can be tricky to get it to return a consistent response for a prompt.

A few years ago I wrote a blog post showing how to delete a Kafka topic when running on Docker and while that approach still works, I think I’ve now got a better way.

I’ve been creating a demo showing how to ingest Avro-encoded data from Apache Kafka into Apache Pinot and ran into a deprecation warning.

I’ve been trying out Python’s Poetry dependency management tool recently and I really like it, but couldn’t figure out how to get it setup as VSCode’s Python interpreter.

I use Docker for pretty much every demo I create and this sometimes results in me running out of IP addresses to serve all those networks.

I’ve been playing around with Microsoft’s TrueSkill algorithm, which attempts to quantify the skill of a player using the Bayesian inference algorithm.

I wanted to test how Apache Pinot deals with data being truncated from the underlying stream from which it’s consuming, so I’ve been trying to work out how to prune data in Redpanda.

I’m still playing around with Puppeteer, a Nodejs library that provides an API to control Chrome/Chromium.

In many of the StarTree recipe videos that I’ve worked on, I show how to write queries in the Pinot UI.

Redpanda supports consumer groups, which are sets of consumers that cooperate to consume data from topics.

I’ve been meaning to take Quix Streams for a spin for a while and got the chance while building a recent demo.

I often write little Python scripts that import code from other local modules and a common problem I have when using the Python REPL is that I update the code in the other modules and then can’t use the new functionality without restarting the REPL and re…

I recently came across clickhouse-local via this article in the MotherDuck monthly newsletter and I wanted to give it a try on my expected goals dataset.

When editing videos for my YouTube channel, Learn Data with Mark, I spend a bunch of time each week chopping up a screencast into scenes that I then line up with a separately recorded voice-over.

Yesterday I wrote a blog post about some different ways to take the first element from a Python list.

When writing Python scripts, I often find myself wanting to take the first item from a collection and ignore the rest of the values.

I’ve been playing around with a bunch of Large Language Models (LLMs) on Hugging Face and while the free inference API is cool, it can sometimes be busy, so I wanted to learn how to run the models locally.

I surely can’t be the first to make the mistake that I’m about to describe and I expect I won’t be the last!

I’m starting to realise that things move insanely fast in the world of LLMs (Large Language Models) and you will run into issues because you aren’t using the latest version of libraries.

Following on from a blog post that I wrote yesterday about doing similarity search with ChromaDB, I noticed an odd error message being printed as the script was exiting.

Somewhat belatedly, I’ve been playing around with LangChain and HuggingFace to spike a tool that lets me ask question about Tim Berglund’s Real-Time Analytics podcast.

I’ve been working with a data set that represents dates as strings in the format 'YYYYmmdd' and I wanted to convert those values to Dates in DuckDB.

I’ve been trying out some of the Hugging Face tutorials and came across an interesting warning message while playing around with the google/flan-t5-large model.

I’ve been playing around with the Kaggle European Soccer dataset, which contains, amongst other things, players and their stats in the FIFA video game.

I wanted to get a list of my GitHub pull requests (PRs) and commits, which was surprisingly difficult to figure out how to do.

If you’ve been using LinkedIn recently, you’ve likely seen those posts where people post slides in a kind of carousel that you can horizontally scroll.

I’ve been learning all about the PIVOT function that was recently added in DuckDB and I ran into an issue where lots of the cells in my post PIVOT table were null values.

I’m still playing around with data returned by Apache Pinot’s HTTP API and I wanted to sort a dictionary of segment names by partition id and index.

I’ve been trying out the Running Pinot in Kubernetes tutorial and ran into a problem trying to write data to Kafka.

I’ve been writing some scripts to parse data from Apache Pinot’s HTTP API and I wanted to format the values stored in a map to make them more readable.

In the 0.8 release of DuckDB, they added functionality that lets you add your own functions when using the Python package I wanted to see if I could use it to generate dummy data so that’s what we’re going to do in this blog post.

I’ve been working on a Real-Time Analytics workshop that I’m going to be presenting at the ODSC Europe conference in June 2023 and I wanted to have Debezium publish records from a MySQL database without including the schema.