Jack's home on the web

The investigation timeline: From false DNS timeout leads to discovering the real culprit—a serial console bottleneck causing 3+ minute network outages.

Abstract Link to heading We run a very large AWS EKS cluster with lots of interesting challenges.

Setup Link to heading Most cloud providers allow you to set commands to run when a host comes online.

Fighting with hard drives TL;DR Link to heading Changing kubelet’s root directory can break CSI drivers and other components.

Do you ever need to debug a node on kubernetes?

Image from UnSplash Preface Link to heading This post describes how I’ve been conducting sprint planning on my current team.

Image from UnSplash I’ve led infrastructure at a startup for the past 4 years that has had to scale quickly.

Context Link to heading We are using AWS Auora to manager our databases and I want to setup automatic rotation of database passwords.

An unambiguous best practice for systems is to codify your infrastructure in a repeatable text format, rather than clicks in a UI.

Table driven tests are a popular model for testing in Go and a clear improvement to traditional testing of one function per case.

Histograms are an important part of metrics and monitoring.

A recent vulnerability in the go-redis library allowed correctly configured clients to return the results of other redis commands in the event of redis failures.

Problem Link to heading You have a constructor function that creates an object with a background goroutine.

Problem Link to heading You’ve written an object or class.

Problem Example Link to heading I commonly see bugs appending to slices in Go in a way that is not thread safe.

I had an integration test that was very complex (most are, sadly) and took too long to start.

A short checklist of what I wish for in a good Go library, in no particular order.

What I wanted to do Link to heading Today’s pkg/errors Link to heading Package errors is a popular go error wrapping package.

Problem Link to heading The go language spec has no way to simultaneously add or augment behavior to an interface while maintaining commonly accepted best programming practices around minimal interface design.

I get woken up to an alert about read capacity on one of my DynamoDB tables.

Go’s pprof package is frequently used for heap and CPU profiles, but a little used feature of the package is being able to create custom profiles.

Not lacking nuance Link to heading I’ve mentioned a general guideline of accept interfaces, return structs in a previous post and multiple times on code reviews to coworkers, but often get the question “Why”.

I recently ran into a -race error message that stumped me for a while, so I’ll retell it here in phases to see if you can catch it.

An interesting property of Go’s built-in data structures is that read operations work even when they are nil!

Package httptrace is a new addition in Go 1.7 that allows users to trace the steps of a HTTP request.

This post will talk about a new library in Go 1.7, the context library, and when or how to correctly use it.

This post describes what I feel are best practices when creating a client library for a service.

This post will describe common approaches to organizing Go code into packages and what I feel is the ideal directory structure for Go repositories.

In this post I will describe the Preemptive Interface pattern often used in code and why I think it is generally an incorrect pattern to follow in Go.