The Gradient Field

Slopes in Two Directions

Let's start with the simplest possible surface: $f(x,y) = x^2 + y^2$, a bowl opening upward. Standing at any point $(x_0, y_0)$, we can ask: how steep is it heading east? How steep heading north?

Those are exactly our two partial derivatives. If we hold $y$ fixed and walk east, the slope we feel is $\partial f/\partial x = 2x$. Hold $x$ fixed and walk north: the slope is $\partial f/\partial y = 2y$.

The red plane below slices the surface at constant $y = y_0$, revealing the east-west cross-section. The blue plane slices at constant $x = x_0$, revealing the north-south cross-section. Drag the sliders to move the point and watch the slopes change.

So at any point we have two slope measurements - one per direction. But what if we want to walk northeast? Or at some arbitrary angle? We need to combine these measurements into a single object.

Assembling the Gradient Vector

The key idea: pack both partials into a single vector. For $f(x,y)$, we define

The symbol $\nabla$ (read "del" or "nabla") is a vector differential operator - by itself it means nothing, but applied to $f$ it produces this gradient vector. In 3D, it picks up a $z$-component too: $\nabla f = f_x\,\mathbf{i} + f_y\,\mathbf{j} + f_z\,\mathbf{k}$.

For our bowl $f = x^2 + y^2$, the gradient is $\nabla f = 2x\,\mathbf{i} + 2y\,\mathbf{j}$. At the point $(1, 0.5)$ that gives $\langle 2, 1 \rangle$ - pointing mostly east with some north. Notice something important: the gradient lives in the $xy$-plane, not on the surface itself. It tells you which horizontal direction to face to climb fastest.

Move the point and watch the orange gradient arrow.

Perpendicular to Level Curves

Here's something beautiful. The level curves of $f(x,y) = x^2 + y^2$ are circles - the curves where $f = c$ for each constant $c$. Along a level curve, $f$ doesn't change; you're walking at constant altitude.

It turns out the gradient is always perpendicular to the level curves. Think of it this way: if you're standing on a hillside, the steepest path goes straight up the slope - that's perpendicular to the contour lines, which run across the slope. The gradient points in that steepest direction.

Move the point below. The orange arrow (gradient) always meets the contour lines at a right angle.

Magnitude Equals Steepness

The gradient's direction tells us where to climb. Its magnitude tells us how steep:

Where the contour lines are packed closely together, a small horizontal step causes a large change in $f$ - so the gradient magnitude is large there. Where contours are far apart, the terrain is gentle and $|\nabla f|$ is small.

The arrows below encode both pieces of information: their direction is $\nabla f / |\nabla f|$ (the steepest direction), and their color encodes $|\nabla f|$ (blue = gentle, red = steep). For $f = x^2 + y^2$, the gradient grows as we move away from the origin - the bowl's walls get steeper.

The Directional Derivative

Now we can answer the original question: if we walk in the direction of an arbitrary unit vector $\mathbf{u}$, how fast does $f$ change? The answer is the dot product:

where $\theta$ is the angle between $\mathbf{u}$ and $\nabla f$. Let's read off the three key cases:

• $\theta = 0$: $\mathbf{u}$ aligned with $\nabla f$. We get $D_{\mathbf{u}} f = |\nabla f|$ - maximum climb.
• $\theta = \pi/2$: $\mathbf{u}$ perpendicular to $\nabla f$. We get $D_{\mathbf{u}} f = 0$ - we're walking along a level curve.
• $\theta = \pi$: $\mathbf{u}$ opposite to $\nabla f$. We get $D_{\mathbf{u}} f = -|\nabla f|$ - steepest descent.

Common trap: $D_{\mathbf{u}} f \neq |\nabla f|$ unless $\mathbf{u}$ happens to be aligned with $\nabla f$. The gradient magnitude is the maximum possible directional derivative, achieved only in one direction.

Rotate the blue unit vector with the slider and watch the projection (green) change.

Properties of the Gradient

The gradient obeys familiar-looking algebraic rules. For differentiable scalars $f$ and $g$:

These follow directly from the corresponding rules for partial derivatives. For the product rule, just look at one component: $$\frac{\partial(fg)}{\partial x} = f \frac{\partial g}{\partial x} + g \frac{\partial f}{\partial x}.$$ Doing the same for $y$ and $z$, then assembling, gives $\nabla(fg) = f\,\nabla g + g\,\nabla f$.

There's also a chain rule. If $f = F(u)$ where $u = g(x,y,z)$, then the chain rule on each component gives: $$\frac{\partial f}{\partial x} = F'(u)\frac{\partial g}{\partial x}, \quad \frac{\partial f}{\partial y} = F'(u)\frac{\partial g}{\partial y}, \quad \frac{\partial f}{\partial z} = F'(u)\frac{\partial g}{\partial z}.$$ Factoring out $F'(u)$:

As a quick check: $f = e^{x^2+y^2}$ has $u = x^2 + y^2$, $F(u) = e^u$, $F'(u) = e^u$. So $\nabla f = e^{x^2+y^2}\langle 2x, 2y \rangle$ - which you can verify directly.

Why Machine Learning Cares About Gradients (additional material)

Everything we've built - the gradient pointing uphill, its magnitude telling us how steep - turns out to be the core engine behind how AI learns.

Imagine you're training a neural network to recognize photos of cats. The network has millions of adjustable knobs (called weights). There's a function $L(w_1, w_2, \ldots, w_n)$ that measures how badly the network is doing - it's called the loss function. Big loss = lots of mistakes. Small loss = good predictions.

Training the network means finding the knob settings that make $L$ as small as possible. That's a minimization problem in millions of dimensions - exactly the kind of landscape we've been studying, just with way more axes than two.

The algorithm is beautifully simple: compute $\nabla L$ (the gradient of the loss), then take a small step in the opposite direction, because the gradient points uphill and we want to go downhill:

The parameter $\eta$ (called the learning rate) controls step size. Too big and you overshoot the valley; too small and training takes forever. This update rule is called gradient descent, and it's the same idea as rolling a ball downhill on our surface plots - the ball follows the steepest path down.

Every time ChatGPT writes a sentence, every time your phone recognizes your face, every time a self-driving car spots a stop sign - somewhere underneath, gradients got computed and weights got nudged downhill. The math is exactly what we've been doing here, just scaled up.