Shane Caldwell

Put down that ruler and go build a microscope.

What to expect as we enter the Year of The Judge

Towards measuring alignment with human taste in autoformalization with judge agents.

METR's SWE-bench analysis shows us taste isn't verifiable.

Getting comfortable with the hardware on a quest for more MFU.

The practical realities of devestatingly high communication cost in training.

Looking at the data and letting it look back at us.

Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.

If you contribute a public benchmark, are you giving free capability to your competitors?

Distributed training sans datacenter.

We tried RL once.

An attempt to explain why benchmarks are either bad or secret, and why the bar charts don't matter so much.

An exploration of functional programming through Haskell, motivated by trying to understand the near-religious devotion its practitioners have for the language.

An analysis of prompt injection vulnerabilities in large language models and why they represent a fundamental security challenge.

Exploring the fundamental data sharing challenges that limit machine learning progress in information security, and why the field needs its own ImageNet moment.

An exploration of using deep reinforcement learning to create autonomous penetration testing agents, examining the challenges and potential solutions for automating cybersecurity assessments.

A comprehensive review of the Offensive Security Certified Professional (OSCP) certification from the perspective of a machine learning engineer entering the security field.