Lagrange multipliers: follow the constraint

Find the minimum of

\[f(x,y)=(x-2)^2+4y^2\]

subject to

\[g(x,y)=x^2+2y^2=1.\]

Both functions have elliptical contours, with different centres and axis ratios. Partial derivatives and the chain rule are enough to follow the argument.

An unconstrained attempt

Without the constraint, \(f\) has minimum zero at \(A=(2,0)\). But \(g(A)=4\), so this point is inadmissible. Setting \(\nabla f=0\) does not solve our problem: a direction that decreases \(f\) may leave the allowed curve.

The gradients are

\[\nabla f=\binom{2(x-2)}{8y}, \qquad \nabla g=\binom{2x}{4y}.\]

Height, contours, and steepest ascent

The graph \(z=h(x,y)\) of a scalar function is a height field. A horizontal cut \(z=c\) gives a contour, the inputs satisfying \(h(x,y)=c\). Our constraint is the height-one contour of \(g\). Lifting those same inputs onto the graph of \(f\) shows the values we may attain.

For a unit input direction \(v\) through \(P\), the chain rule gives the slope

\[\left.\frac{d}{dt}h(P+tv)\right|_{t=0} =\nabla h(P)\cdot v.\]

In Surfaces, contours, and gradients, move the point and change the direction. Follow each gradient to compare the chosen arrow with steepest ascent.

Surfaces, contours, and gradients

Map3D surfacesContours from above

Objective f

Constraint function g

Objectiveg = 1Black: chosen direction

Point P

≈ (0, 0.707)

Objective f(P)

6

∇f(P)

≈ (−4, 5.66)

∇g(P)

≈ (0, 2.83)

Position angle θ

90°

Minimum point Maximum point

Direction from tangent

0° · tangent

Follow ∇fFollow ∇g

Undo Redo Reset

Settings

Rotation

45°

Elevation

35°

Reset view

Info

Drag either map to rotate both; Settings provides camera sliders. Position angle moves P along g = 1. The top-down map also lets you drag P.

Dashed coloured arrows show unit gradient directions in the input plane. Solid coloured arrows lift those directions into the surface’s tangent plane; their height shows the maximum upward slope. Black arrows lift your chosen direction. Both maps use the same scales, with height compressed by 0.14.

Direction from tangent: 0° is tangent, +90° is outward, −90° is inward. Follow ∇f or Follow ∇g selects steepest ascent. Reset restores the starting point and direction while keeping your view.

At the starting point \(P=(0,1/\sqrt2)\), the unit tangent \(T=(-1,0)\) gives objective slope \(4\) and constraint slope zero. Yet the largest upward slopes are \(4\sqrt3\) for \(f\) and \(2\sqrt2\) for \(g\). The tangent is not the steepest ascent direction.

Why does a gradient give the largest slope? By Cauchy–Schwarz, for every unit \(v\),

\[\nabla h(P)\cdot v\leq\lVert\nabla h(P)\rVert.\]

When the gradient is nonzero, equality holds at

\[v=\frac{\nabla h(P)}{\lVert\nabla h(P)\rVert}.\]

Thus the gradient’s direction gives steepest first-order ascent, and its magnitude gives the maximum slope. The solid coloured arrows lift these unit directions into the graph’s tangent plane. The gradient itself is a vector in the input plane.

What the constraint permits

If a path \(\gamma(s)\) stays on \(g=1\), differentiating gives

\[\nabla g(\gamma(s))\cdot\gamma'(s)=0.\]

The gradient is therefore perpendicular to the constraint’s tangent. At an ordinary point, one of the two tangent directions decreases \(f\). At a constrained extremum, neither does: \(\nabla f\cdot T=0\) too. Both gradients then lie on the same normal line.

Equivalently, the objective contour and constraint have the same tangent where they touch. At the minimum, the two gradients point oppositely; at the maximum, they point the same way. Both satisfy the following theorem.

The Lagrange multiplier theorem

The regularity assumption lets the implicit function theorem describe the constraint locally as a smooth surface. Its tangent vectors are exactly those perpendicular to \(\nabla g(a)\), and each is the velocity of a path on the surface. At an extremum, differentiating \(f\) along every such path gives zero. Thus \(\nabla f(a)\) is perpendicular to the whole tangent space and is a multiple of \(\nabla g(a)\). See also MIT’s tangent-space proof.

The theorem gives candidates for extrema; we must still determine which is the minimum.

Solve the ellipse problem

The constraint gradient \((2x,4y)\) vanishes only at the origin, which is not on \(g=1\). Applying (1) gives

\[\begin{aligned} x-2&=\lambda x,\\ 8y&=4\lambda y,\\ x^2+2y^2&=1. \end{aligned}\]

If \(y\ne0\), the second equation forces \(\lambda=2\). The first then gives \(x=-2\), impossible on the ellipse. Therefore \(y=0\), so \(x=\pm1\). The two candidates are

\[\begin{array}{c|c|c} (x,y)&\lambda&f(x,y)\\ \hline (1,0)&-1&1\\ (-1,0)&3&9 \end{array}\]

The ellipse is compact and \(f\) is continuous, so a global minimum and maximum exist. Both must satisfy the multiplier equations, and these are the only candidates. Hence the unique minimum is \(1\) at \((1,0)\), and the unique maximum is \(9\) at \((-1,0)\).

Select these points in the maps. At the minimum, \(\nabla f=(-2,0)\) and \(\nabla g=(2,0)\); at the maximum, they are \((-6,0)\) and \((-2,0)\). The multiplier records their direction and relative magnitude.

A problem about discrete entropy

Let \(q=(q_1,\ldots,q_n)\) be a fixed probability distribution with \(q_i>0\) and \(n\geq2\). Choose \(p\) with \(p_i>0\) and \(\sum_i p_i=1\). Using natural logarithms, define the cross entropy

\[C_q(p)=\sum_{i=1}^n q_i\log\frac1{p_i}.\]

Use Lagrange multipliers to find its stationary point, then prove that it is the unique global minimum, attained at \(p=q\). What is the minimum value?