Tony Wang, A Software Engineer | RSS Feed

Five real incidents from a distributed crawler fleet where nothing crashed, no alert fired, and the system quietly stopped doing its job.

I scanned the Tranco top 1,000,000 domains to measure how much of the web runs managed bot defense.

An in-depth look at how AI-driven anti-bot defenses, SERP reshuffling, and legal/ethical constraints are reshaping Google search crawling after 2025 — and what indie developers and small teams must change in their architecture, ops, and business models to…

A comprehensive guide to setting up a highly efficient, scalable infrastructure to process 10 million requests in 10 minutes at a minimal cost.

An analysis of the top 10 million websites reveals that over a quarter are inactive, highlighting the web's shifting landscape.

Dive into the world of distributed web crawling with Golang, Docker, and Redis.

Unlock the potential of the web with a Google-inspired distributed web crawler.

Explore building a powerful distributed crawler using Crawlee, a JavaScript-based headless browser, for efficient web scraping of Google Maps.

Embark on a comprehensive journey to construct a powerful TikTok scraper using Golang, Docker, and Kubernetes.

Automate global deployment of a scalable crawler with Celery, RabbitMQ, and Fabric.

Learn to build a scalable Python web crawler using Docker, Celery, and RabbitMQ.

Build a Docker cluster with Celery and RabbitMQ for efficient distributed tasks.