Weblog on marginalia.nu

I currently have 17 GB free space on my root drive, out of 0.47 TB usable space, that’s 3% free space.

There’s been some changes to Marginalia Search: Migrated to systemd from docker New unranked query endpoint Wide domains got their own index for faster crawls In brief, this has removed a lot of operational headaches, reduced expensive queries leaving more

I was on the bus on my way to an important business meeting about the exponential growth of my MRR, when suddenly I noticed some troublemakers were hassling an old lady.

You can learn new things.

… optional, that is.

This post is an elaboration on a comment I made on Hacker News recently, on a blog post that showed an increase in volume and decline in quality among the “Show HN” submissons.

The Marginalia Search index has recently seen some design tweaks to make it perform better, primarily the introduction of postings list compression.

The Marginalia Search default ranking algorithm recently saw a fairly radical improvement, due to a new domain trust system that drastically reduces the number of content farm results, as long as there are human results it usually finds them across all the

Being clear about what your blog posts are about lets people who are interested in what you have to say find your writing more easily.

The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI.

One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish.

I’ve been using the CoPilot plugin for IntelliJ on and off for the last few years, and while initially pretty enthusiastic, I’ve come to first disable it and then delete it altogether along with JetBrains’ local AI-completions, and generally felt this has…

How you engage with the world changes how you experience the world, and how the world experiences you.

The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware.

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change.

The most recent change to the search engine is a system that profiles websites based on their rendered DOM.

You wake at 05:30 in the morning, feeling somewhat groggy.

The search engine has recently gained the ability to index the PDF file format.

Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client.

A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains.

I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant.

This text is satirical in nature.

This update is a few days late, the canonical birth date of the project is Feb 26.

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days.

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before.

Marginalia Search now properly supports phrase matching.

I’ve been on a bit of a frustration-driven quest to solve a problem I frequently encounter working on the search engine, that is, reading data from disk.

As an experiment, I’ve reduced my coffee-intake to a single cup a day for about a week now.

A year ago I walked out of the office for the last time.

The best description of my problem solving process is the Feynman algorithm, which is sometimes presented as a joke where the hidden subtext is “be smart”, but I disagree.

I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search.

A neat property of the parquet file format is that it’s designed with block I/O in mind, so that when you are interested in only parts of the contents of a file, it’s possible to some extent to only read that data.

Been working on improving Marginalia Search query parsing and understanding.

The project has been haunted by a mysterious bug since sometime February.

I set out a little over a week ago to add a service registry to Marginalia Search, primarily to reduce its dependence on docker.

It’s been three years since the inception of Marginalia Search, then a dinky experiment to find where the heck the cool Internet has gone, now my full time job.

One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time.

I get significantly more work done when I unplug my computer from the Internet.

You have a hobby you’ve been into for a decade or more.

This is a bit of a retrospective of every project I’ve worked on, as far as I remember them.

Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS.

The Marginalia Crawler has seen improvements!

A simple guide to reading in 9 simple steps Navigate to the desired article.

I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage.

So a bit of an update on what I’ve been working on.

As a general observation, I tend to be more productive when I know what to do next at any given moment.

So the search engine is moving to a new server soon, thanks to the generous grant mentioned recently.

I ran into a bit of a puzzling situation yesterday, testing some of the new index construction changes before they’re going live in a few days.

I’m happy to announce that the generous people at FUTO have granted the project $15,000 with no strings attached to help the search engine out with some more server power.

So… I’ve had the most unreal week of coding.