https://tdhopper.com/index.xml

I routinely SSH into a company machine on Openstack to do work.

I routinely run Impala queries on a remote machine and want the results to save to a CSV file on that machine.

I’m trying out the beautiful Fish shell.

This is more of a Today I Taught.

I needed to concatenate a bunch of CSV files while skipping the header row.

If you want to check whether a Python string is an integer, you can try casting to an int with int() and catching the ValueError if it’s not an integer: 1 2 3 4 5 6 def is_integer(value: str, *, base: int=10) -> bool: try: int(value, base=base) return…

I created a single page website to collect notes on one of my other hobbies: ultralight backpacking.

I was using [tee](http://man7.org/linux/man-pages/man1/tee.1.html) with a long running Python process, but I wasn’t seeing any output.

Listening to Russ Roberts’ Econtalk podcast for the last 5 years has given me a whole new perspective on the world.

Joining two Pandas DataFrames with an equal number of rows is slightly harder than it appears.

Jupyter notebooks nicely render Pandas data frames if they’re the last line in a cell.

Impala Doesn’t Have a histogram() function.

Wes Mckinney’s Ibis, a Pythonic interface to Impala, has functionality for creating Impala tables from Python Pandas dataframes. 1 2 3 4 5 6 7 8 9 import pandas as pd import ibis hdfs = ibis.hdfs_connect(host=webhdfs_host, port=webhdfs_port) client = ib…

I don’t use Ruby much, but I wanted to use Jekyll for this blog.

Galileo’s conflict with the Church was not as it is often portrayed: In January of 1616, the month before before the Roman Inquisition would infamously condemn the Copernican theory as being “foolish and absurd in philosophy”, Monsignor Francesco Ingoli a…

I enjoy the Stanford Encyclopedia of Philosophy, so I created a Twitter account that tweets links to random articles from it.

People mention John Cook’s blog a lot in Github repos.

I'll never forget the great words of one of my best math professors: "If all I wanted was the answer, I sure as hell wouldn't ask you." — Tim Hopper (@tdhopper) December 9, 2013 Another great former professor quote: "I'm not smarter than you.

@tdhopper my father wrote his PhD on this question, and I believe his answer was "depends" — Fred Benenson (@fredbenenson) December 7, 2015

If you enjoy this post, check out my Python Developer Tooling Handbook!

In my nonlinear optimization class in grad school at North Carolina State University, I wrote a paper on the famed SMO algorithm for support vector machines.

True story: I’m a closet knitter.

When I worked at RTI International, I worked on an exploratory analysis of Twitter discussion of electronic cigarettes.

On MCMC: Any sufficiently advanced MCMC is indistinguishable from magic.

Today is my last day at Qadium.

rand_antoniak draws a sample from the distribution of tables created by a Chinese restaurant process with parameter alpha after n patrons are seated.

In an earlier notebook, I showed how we can fit the parameters of a Bayesian mixture model using a Gibbs sampler.

Much of the literature on Dirichlet Processes makes assertions similar to the following: “DP is the Dirichlet process, a distribution over distributions.”

The interactive visualizations in this analysis require a live notebook.

Even though I've been coding for 12 years, I'd still just consider myself a grammer. — Tim Hopper (@tdhopper) November 10, 2014

Yee Whye Teh et al’s 2005 paper Hierarchical Dirichlet Processes describes a nonparametric prior for grouped clustering problems.

I’m excited to see tools developed for the web being applied to offline domains like agriculture and health.

I appreciated James Hague’s post on Computer Science Courses that Don’t Exist, But Should.

From Testicular volume is inversely correlated with nurturing-related brain activity in human fathers in PNAS: One participant’s testes volume measurement was excluded because his value was 2.8 SDs above the mean (mean = 38,064; SD = 11,183) and was more…

1 2 3 4 5 6 7 import pandas as pd import numpy as np import random import matplotlib.pyplot as plt from scipy import stats from collections import namedtuple, Counter Suppose we receive some data that looks like the following: 1 2 data = pd.Series.fro…

I recently had the honor of being interviewed by Michael Swenson for his interview series called “Profiles in Computational Imagination”.

I'll never forget the great words of one of my best math professors: "If all I wanted was the answer, I sure as hell wouldn't ask you." — Tim Hopper (@tdhopper) December 9, 2013

I am convinced that a programming student hoping to get a job in that field should be actively building a portfolio online.

Latent Dirichlet Allocation is a generative model for topic modeling.

As we saw earlier the Dirichlet process describes the distribution of a random probability distribution.

I love this new post on Quora’s engineering blog.

Dirichlet Distribution The symmetric Dirichlet distribution (DD) can be considered a distribution of distributions.

How to be a 10x engineer: Incur technical debt fast enough to appear 10x as productive as the ten engineers tasked with cleaning it up. — Brian Degenhardt (@bmdhacks) January 29, 2015

I’m starting to love single-page informational websites.

Having worked from home for the last few years, I have a hard time understanding how people get anything done in open-floor plan offices.

Caution: the following post is laden with qualitative extrapolation of anecdotes and impressions.

Start Using Landsat on AWS: “The Landsat program has been running since 1972 and is the longest ongoing project to collect such imagery.

TIL `cat` will correctly join MP3 files http://t.co/mmc4NN2Kmp — Tim Hopper (@tdhopper) March 15, 2015 From Stackoverflow: An MP3 file is nothing more than the raw MPEG2-Layer 3 (audio) stream data, there is no file level header structure with, for exampl…

Dynamically Update a Plot in IPython: One thing I miss about Mathematica is Animate and Manipulate.

I gave a talk at the Research Triangle Analysts meetup about Pyspark.