Sebastian Raschka, PhD

OpenAI just released their new open-weight LLMs this week: gpt-oss-120b and gpt-oss-20b, their first open-weight models since GPT-2 in 2019.

It has been seven years since the original GPT architecture was developed.

The latest in LLM research with a hand-curated, topic-organized list of over 200 research papers from 2025.

KV caches are one of the most critical techniques for efficient inference in LLMs in production.

Why build an LLM from scratch?

A lot has happened this month, especially with the releases of new flagship models like GPT-4.5 and Llama 4.

As you know, I've been writing a lot lately about the latest research on reasoning in LLMs.

This article explores recent research advancements in reasoning-optimized LLMs, with a particular focus on inference-time compute scaling that have emerged...

Overview of four ways to build reasoning-capable LLMs, including inference-time scaling, supervised finetuning, reinforcement learning, and search-based methods.

This article covers 12 influential AI research papers of 2024, ranging from mixture-of-experts models to new LLM scaling laws for precision.

Implements byte pair encoding (BPE) tokenization from scratch: tokenizer training, GPT-style merge rules, and step-by-step Python examples.

I want to share my running bookmark list of many fascinating (mostly LLM-related) papers I stumbled upon in 2024.

There has been a lot of new research on the multimodal LLM front, including the latest Llama 3.2 vision models, which employ diverse architectural...

This article shows you how to transform pretrained large language models (LLMs) into strong text classifiers.

Three-hour coding workshop that builds the core pieces of a GPT-style large language model from the ground up for developers who want implementation intuition.

There are hundreds of LLM papers each month proposing new techniques and approaches.

This article covers a new, cost-effective method for generating data for instruction finetuning LLMs; instruction finetuning from scratch; pretraining LLMs...

This article covers three new papers related to instruction finetuning and parameter-efficient finetuning with LoRA in large language models (LLMs).

This is an overview of the LLM development process.

What a month!

Guide to using and finetuning pretrained transformers, comparing feature extraction, prompt-based use, full finetuning, and parameter-efficient LLM adaptation.

It's another month in AI research, and it's hard to pick favorites.

Once again, this has been an exciting month in AI research.

Technical tutorial on LoRA and DoRA for parameter-efficient finetuning, with from-scratch PyTorch code and intuition for weight-decomposed low-rank adaptation.

Practical guide to improving LLM finetuning with better instruction datasets, covering data curation, prompt-output pairs, synthetic data, and experiment ideas.

Large language models (LLMs) offer one of the most interesting opportunities for developing more efficient training methods.

Peak memory consumption is a common bottleneck when training deep learning models such as vision transformers and LLMs.

Finetuning allows us to adapt pretrained LLMs in a cost-efficient manner.

Training and using large language models (LLMs) is expensive due to their large compute requirements and memory footprints.

Pretrained large language models are often referred to as foundation models for a good reason: they perform well on various tasks, and we can use them as a...

In the rapidly evolving field of artificial intelligence, utilizing large language models in an efficient and effective manner has become increasingly...

Previously, I shared an article using multi-GPU training strategies to speed up the finetuning of large language models.

When it comes to productivity workflows, there are a lot of things I'd love to share.

This blog post outlines techniques for improving the training performance of your PyTorch model without compromising its accuracy.

A deep-dive implementation of self-attention, multi-head attention, causal attention, and cross-attention — the mechanisms behind modern transformer LLMs.

Step-by-step tutorial explaining scaled dot-product self-attention for large language models, with Python code that builds the mechanism from scratch.

Curated reading list for understanding large language models, from attention and transformers to BERT, GPT, scaling laws, instruction tuning, and RLHF.

Since the release of the AI Classifier by OpenAI made big waves yesterday, I wanted to share a few details about the different approaches for detecting...

Data augmentation is a key tool in reducing overfitting, whether it's for images or text.

Conversational chat bots such as ChatGPT probably will not be able replace traditional search engines and expert knowledge anytime soon.

Imagine you want to quickly train a few machine learning or deep learning models on the cloud but don't want to deal with cloud infrastructure.

Recently, I shared the top 10 papers that I read in 2022.

Every day brings something new and exciting to the world of machine learning and AI, from the latest developments and breakthroughs in the field to emerging...

About monthly machine learning musings, and other things I am currently workin on ...

Occasionally, I share research papers proposing new deep learning approaches for tabular data on social media, which is typically an excellent discussion...

Regarding neural network training, I think we are all guilty of doing this: we choose our batch sizes as powers of 2, that is, 64, 128, 256, 512, 1024, and...

Deploys a Super Resolution GAN (SRGAN) app to the cloud with Lightning.ai — covers containerization, model serving, and production deployment steps.

Build a Super Resolution GAN app with Lightning.ai: interactive UI, model serving, and a modern workflow for sharing deep learning research.

The PyTorch team recently announced TorchData, a prototype library focused on implementing composable and reusable data loading utilities for PyTorch.

Today, PyTorch officially introduced GPU support for Apple's ARM M1 chips.