Feep! blog

In my last post about the Stack Overflow data dumps, I mentioned that, while StackExchange Inc. was no longer uploading dumps to archive.org, that task had been taken up by a member of the community.

Until this week, search.feep.dev was served via Cloudflare and Cloudflare Tunnel.

Because Feep!

In my last post, I wrote about cleaning up URL processing.

Over time, the indexing and ranking code has gotten hard to follow, in part because I was using the term “URL” for what turned out to be several different things.

The branding for Feep!

I said in my last post that I was going to try out OpenHands; but then somebody showed me Aider Chat and I decided to try that instead, on the strength of a personal recommendation.

Like almost everybody, I’ve been using LLMs more and more recently.

Feep!

I run a personal GitLab instance on a local machine.

Since I just updated to a newer copy of the Stack Exchange data dumps, I’ve been shepherding that process through my CI pipelines to get the updated data into production.

My search engine got its start from StackExchange data dumps, and they’re still one of the major data sources I use.

The title kind of spoils the story, but I ran into a confusing bug recently and it seemed interesting enough to write about.

There’s been a sudden explosion of advancements in Large Language Models in the past couple of weeks.

Someone emailed me a few weeks ago and asked me to add an RSS feed to this blog.

As I mentioned briefly in my post on the CDB file format, I had some trouble understanding how the JavaScript bitwise operators worked.

For a new feature I’m building (“quick results”, coming Soon™), I needed a simple key-value file format, ideally one that was fast and easy work with.

In my previous post, I talked about crawling the web and mentioned that I planned to start with the ability to download a list of URLs and write them to a WARC file.

I’m currently relying on various bulk exports (Stack Exchange XML dumps, Zim files from Wikipedia, and so on) to populate the search engine.

A 2004 article on Why Writing Your Own Search Engine is Hard was posted on Hacker News today, and I wrote a comment comparing it with my experiences writing my own search engine 18 years later.

To prepare for setting up a production environment, I just added a some code to handle the situation when the server gets overloaded.

I’m using docker-compose to deploy Feep!

Despite my previous enthusiasm about redo, I’ve just finished tearing the last of my .do files out and replacing them with shell scripts again.

I’ve spent the last week or so writing down all the ideas I have for Feep!

Almost every time I update StackOverflow's data files (by far the largest single data source I have, at 40 GB), I find out halfway through the process that I don’t have enough disk space and need to clear out some old data.

I've been writing blog posts, but so far they've just been markdown files in a folder and there hasn't been any way to read them.

Previously, I introduced the concept of "chunks" for use in generating link data.

Today I added favicons the search results, to make them more visually interesting.

In my last post I showed a big diagram of all the steps to get a StackExchange post from a dump file into the search index, and drew some boxes around different parts.

I've gone through several different schemes for coordinating everything that needs to be done to get pages into the index.

I've been managing the process of importing StackExchange posts with an increasingly hacky and annoying shell script.

I've built a data processing system for working with streams of pages; it seems like it's pretty solid now so let's take a look at how it works.

For a search engine to be of any use, it has to have a collection of data to search over.