The DataNinja

UPI Daily Spend Tracker: Turning Daily Bank Alerts into Real-Time Analytics A real-life problem, solved with data engineering, APIs, Python, ClickHouse, and Grafana.

🧩 The Problem: Queries Stuck in “Opening/Closing Tables” During peak traffic hours on our production MySQL primary (prod-shard3-db01), we started observing a sudden spike in query latency across multiple application services.

We recently faced a subtle but critical disk space bloat issue on one of our MySQL production nodes, and the root cause turned out to be how Percona’s pt-heartbeat handles logging during log rotation.

Background After upgrading some of our production servers to Ubuntu 24.04 (Noble), we began noticing unexpected MySQL restarts during routine package updates.

During a recent performance test on one of my MySQL instances, I noticed the Disk I/O Utilization graph pegged at 100%.

Setting up a disaster recovery (DR) pipeline between your on-premise MySQL database and Amazon RDS can be a game-changer for data resilience.

During a scheduled maintenance window on a Saturday, I promoted one of our MySQL replicas to act as the new primary.

Primary promotion before the introduction of GTID used to be a manual and highly error-prone task.