Keep the gradient flowing

In Part 1, we saw that the asymptotic variance of the vanilla REINFORCE estimator scales cubically ($O(T^3)$) with trajectory length $T$.

I have a dirty secret.

Differentiating through optimization is a fundamental problem in hyperparameter optimization, dataset distillation, meta-learning and optimization as a layer, to name a few.

This blog post discusses the convergence rate of the Stochastic Gradient Descent with Stochastic Polyak Step-size (SGD-SPS) algorithm for minimizing a finite sum objective.

The stochastic Polyak step-size (SPS) is a practical variant of the Polyak step-size for stochastic optimization.

The Langevin algorithm is a simple and powerful method to sample from a probability distribution.

The idea for what was later called Monte Carlo method occurred to me when I was playing solitaire during my illness.

Backtracking step-size strategies (also known as adaptive step-size or approximate line-search) that set the step-size based on a sufficient decrease condition are the standard way to set the step-size on gradient descent and quasi-Newton methods.

Six: All of this has happened before.

When an optimization problem has multiple global minima, different algorithms can find different solutions, a phenomenon often referred to as the implicit bias of optimization algorithms.

This is the first of a series of blog posts on short and beautiful proofs in optimization (let me know what you think in the comments!).

While the most common accelerated methods like Polyak and Nesterov incorporate a momentum term, a little known fact is that simple gradient descent –no momentum– can achieve the same rate through only a well-chosen sequence of step-sizes.

I've seen things you people wouldn't believe.

We can tighten the analysis of gradient descent with momentum through a cobination of Chebyshev polynomials of the first and second kind.

There's a fascinating link between minimization of quadratic functions and polynomials.

A naive implementation of the logistic regression loss can results in numerical indeterminacy even for moderate values.

This blog post extends the convergence theory from the first part of these notes on the Frank-Wolfe (FW) algorithm with convergence guarantees on the primal-dual gap which generalize and strengthen the convergence guarantees obtained in the first part. $$…

$$ \def\aa{\boldsymbol a} \def\bb{\boldsymbol b} \def\cc{\boldsymbol c} \def\xx{\boldsymbol x} \def\zz{\boldsymbol z} \def\uu{\boldsymbol u} \def\vv{\boldsymbol v} \def\yy{\boldsymbol y} \def\ss{\boldsymbol s} \def\TT{\boldsymbol T} \def\CC{\boldsymbol C}…

This blog post is the first in a series discussing different theoretical and practical aspects of the Frank-Wolfe algorithm. $$ \def\xx{\boldsymbol x} \def\yy{\boldsymbol y} \def\ss{\boldsymbol s} \def\dd{\boldsymbol d} \DeclareMathOperator*{\argmin}{{arg…

Most proofs in optimization consist in using inequalities for a particular function class in some creative way.

My friend Rémi Leblond has recently uploaded to ArXiv our preprint on an asynchronous version of the SAGA optimization algorithm.

TL;DR: I describe a method for hyperparameter optimization by gradient descent.

Announce: first public release of lightning!

Together with other scikit-learn developers we've created an umbrella organization for scikit-learn-related projects named scikit-learn-contrib.

Recently I've implemented, together with Arnaud Rachez, the SAGA[1] algorithm in the lightning machine learning library (which by the way, has been recently moved to the new scikit-learn-contrib project).

My latests work (with Francis Bach and Alexandre Gramfort) is on the consistency of ordinal regression methods.

Cross-validation iterators in scikit-learn are simply generator objects, that is, Python objects that implement the __iter__ method and that for each call to this method return (or more precisely, yield) the indices or a boolean mask for the train and test

Due to lack of time and interest, I'm no longer maintaining this project.

Last Friday was PyData Paris, in words of the organizers, ''a gathering of users and developers of data analysis tools in Python''.

My latest research paper[1] deals with the estimation of the hemodynamic response function (HRF) from fMRI data.

:og_image: http://fa.bianp.net/blog/images/2014/mprof_example.png One of the lesser known features of the memory_profiler package is its ability to plot memory consumption as a function of time.

TL; DR These are some notes on calibration of surrogate loss functions in the context of machine learning.

As part of the development of memory_profiler I've tried several ways to get memory usage of a program from within Python.

In this post I compar several implementations of Logistic Regression.

TL;DR: I've implemented a logistic ordinal regression or proportional odds model.

My latest contribution for scikit-learn is an implementation of the isotonic regression model that I coded with Nelle Varoquaux and Alexandre Gramfort.

Householder matrices are square matrices of the form $$ P = I - \beta v v^T$$ where $\beta$ is a scalar and $v$ is a vector.

** Note: this post contains a fair amount of LaTeX, if you don't visualize the math correctly come to its original location ** In machine learning it is common to formulate the classification task as a minimization problem over a given loss function.

Besides performing a line-by-line analysis of memory consumption, memory_profiler exposes some functions that allow to retrieve the memory consumption of a function in real-time, allowing e.g. to visualize the memory consumption of a given function over t…

SciPy contains two methods to compute the singular value decomposition (SVD) of a matrix: scipy.linalg.svd and scipy.sparse.linalg.svds.

This tutorial introduces the concept of pairwise preference used in most ranking problems.

My newest project is a Python library for monitoring memory consumption of arbitrary process, and one of its most useful features is the line-by-line analysis of memory usage for Python code.

A little experiment to see what low rank approximation looks like.

In scipy's development version there's a new function closely related to the QR-decomposition of a matrix and to the least-squares solution of a linear system.

Last week we released a new version of scikit-learn.

I've been working lately in improving the scikit-learn example gallery to show also a small thumbnail of the plotted result.

Today's coding sprint was a bit more crowded, with some notable scipy hackers such as Ralph Gommers, Stefan van der Walt, David Cournapeau or Fernando Perez from Ipython joining in.

As a warm-up for the upcoming EuroScipy-conference, some of the scikit-learn developers decided to gather and work together for a couple of days.

Ridge coefficients for multiple values of the regularization parameter can be elegantly computed by updating the thin SVD decomposition of the design matrix: import numpy as np from scipy import linalg def ridge(A, b, alphas): """ Return coefficie…

I haven't worked in the manifold module since last time, yet thanks to Jake VanderPlas there are some cool features I can talk about.