Everything About Linear Algebra

1 Introduction

This is the public reading view for three source-preserving collections:

introductory linear algebra notes and personal homework submissions;
a short advanced linear algebra notebook based on LADR and GTM 135;
numerical linear algebra notes on norms, factorisations, conditioning, and stability.

The original course directories, source manifests, and migration receipts remain the authority for provenance. This file only groups those sources into one continuous reading path; source-authored omissions and explicit TODOs are not silently completed here.

2 Linear equations, vectors, and matrices

2.1 WS 1, p. 1: vectors and vector spaces

(complement: “MathHygine”) Principle of Mathematical Induction: $[𝑆 (𝑛) \land (\forall 𝑘 \in ℕ, 𝑆 (𝑘) \to 𝑆 (𝑘 + 1))] \to (\forall 𝑚 \in ℕ, 𝑆 (𝑚))$ .

For the system

{\begin{cases} 3 𝑥 + 21 𝑦 - 3 𝑧 = 0 \\ - 6 𝑥 - 2 𝑦 - 𝑧 = 62 \\ 2 𝑥 - 3 𝑦 + 8 𝑧 = 32 \end{cases}

place numbers in the columns of the augmented matrix $(\begin{matrix} 3 & 21 & - 3 & 0 \\ - 6 & - 2 & - 1 & 62 \\ 2 & - 3 & 8 & 32 \end{matrix})$ .

Definition 2.1 : Vector space

A matrix with only one column is called a column vector, or simply a vector. The entries of a vector are called its components. The set of all column vectors with $𝑛$ components is denoted by $ℝ^{𝑛}$ . We will refer to $ℝ^{𝑛}$ as a vector space.

A matrix with only one row is a row vector. In this text, we refer to vectors as column vectors unless otherwise stated. 下一章会说 preference for column vectors 的 apparent reason.

For example, $(\begin{matrix} 1 \\ 2 \\ 9 \\ 1 \end{matrix})$ is a vector in $ℝ^{4}$ ; $(\begin{matrix} 1 & 5 & 5 & 3 & 7 \end{matrix})$ is a row vector with $5$ components.

$𝐴 = (\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots \\ 𝑎_{21} & \dots \\ \dots & 𝑎_{34} \end{matrix})$ is called a $3 \times 4$ matrix (row, col; three by four).
Matrix $𝐴 = 𝐵$ if same size and $\forall 𝑖, 𝑗, 𝑎_{𝑖 𝑗} = 𝑏_{𝑖 𝑗}$ .
If $𝐴$ is $𝑛 \times 𝑛$ , $𝐴$ is called a square matrix, and the entries $𝑎_{11}, 𝑎_{22}, \dots, 𝑎_{𝑛 𝑛}$ form the main diagonal of $𝐴$ .
A square matrix $𝐴$ is called diagonal provided all its entries above and below the diagonal are $0$ , i.e. $𝑎_{𝑖 𝑗} = 0$ whenever $𝑖 \neq 𝑗$ .
$𝐴$ is called upper triangular provided all its entries below the main diagonal are $0$ ; lower triangular: entries above the main diagonal are $0$ .

Note that the $𝑚$ columns of an $𝑛 \times 𝑚$ matrix are vectors in $ℝ^{𝑛}$ but not $ℝ^{𝑚}$ : each vector in the $𝑚$ vectors has $𝑛$ components.

Standard representation of vectors: $𝑣 = (\begin{matrix} 𝑥 \\ 𝑦 \end{matrix})$ (in Cartesian plane; in $ℝ^{3}$ defined analogously, and likewise in $ℝ^{𝑛}$ ). When considering an infinite set of vectors, arrow representation becomes impractical. One may represent $𝑣 = (\begin{matrix} 𝑥 \\ 𝑦 \end{matrix})$ simply by the point $(𝑥, 𝑦)$ , the head of the standard arrow representation of $𝑣$ . Example: the set of all vectors $(\begin{matrix} 𝑥 \\ 𝑥 + 1 \end{matrix})$ where $𝑥$ is arbitrary is represented as the line $𝑦 = 𝑥 + 1$ ; for a few special values of $𝑥$ we may still use arrow representation. The source sketch labels the vectors $(\begin{matrix} 1 \\ 2 \end{matrix})$ for $𝑥 = 1$ and $(\begin{matrix} - 2 \\ - 1 \end{matrix})$ for $𝑥 = - 2$ on that line.

Consider the system

{\begin{cases} 2 𝑥 + 8 𝑦 + 4 𝑧 = 2 \\ 2 𝑥 + 5 𝑦 + 𝑧 = 5 \\ 4 𝑥 + 10 𝑦 - 𝑧 = 1 \end{cases} .

The matrix which contains the coefficients is called its coefficient matrix, $(\begin{matrix} 2 & 8 & 4 \\ 2 & 5 & 1 \\ 4 & 10 & - 1 \end{matrix})$ . By contrast, $(\begin{matrix} 2 & 8 & 4 & 2 \\ 2 & 5 & 1 & 5 \\ 4 & 10 & - 1 & 1 \end{matrix})$ , which displays all numerical information, is called the augmented matrix.

2.2 WS 1, p. 2: augmented matrices and RREF

For the sake of clarity, we will often indicate the position of the equal signs in the equations by a dotted line: $(\begin{matrix} 2 & 8 & 4 & | & 2 \\ 2 & 5 & 1 & | & 5 \\ 4 & 10 & - 1 & | & 1 \end{matrix})$ .

我们可以把之前的对 equation 的操作用在 matrix 上，并且将 answer represented as a vector. Thus the displayed system has answer $(\begin{matrix} 𝑥 \\ 𝑦 \\ 𝑧 \end{matrix}) = (\begin{matrix} 1 \\ 4 \\ 3 \end{matrix})$ .

Example: $(\begin{matrix} 1 & - 1 & 0 & 0 & 4 & 2 \\ 0 & 0 & 1 & 0 & - 1 & 2 \\ 0 & 0 & 0 & 1 & - 1 & 3 \end{matrix})$ gives $(\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \\ 𝑥_{4} \\ 𝑥_{5} \end{matrix}) = (\begin{matrix} 2 + 𝑡 + 4 𝑟 \\ 𝑡 \\ 2 + 𝑟 \\ 3 + 𝑟 \\ 𝑟 \end{matrix})$ .

这个 equation 容易解是因为：

The leading coefficient is always $1$ .
The leading variable in each equation 在其他 equation 中不出现.
The leading variables in natural order 出现.

当一个 linear system 有这些三条性质后就非常容易解，因而我们希望将 linear system reduce 至满足 $𝑃_{1}, 𝑃_{2}, 𝑃_{3}$ .

For example, row operations reduce the displayed augmented matrix to $(\begin{matrix} 1 & 2 & 0 & 0 & 3 & 2 \\ 0 & 0 & 1 & 0 & - 1 & 4 \\ 0 & 0 & 0 & 1 & - 2 & 3 \\ 0 & 0 & 0 & 0 & 0 & 0 \end{matrix})$ . 只要朝一个 down，从第三条原则就可以完成这个获得满足 $𝑃_{1}, 𝑃_{2}, 𝑃_{3}$ 的 reduced matrix 而解 linear system 的 algorithm.

From top to down, move on to the $𝑖$ th equation $𝑐 𝑥_{𝑗} + \dots = 𝑏$ :

Divide by $𝑐$ , so $𝑥_{𝑗} + \dots = \frac{𝑏}{𝑐}$ .
Eliminate $𝑥_{𝑗}$ from all other equations above and below.
Proceed to next equation.
Check: if $0 =$ non- $0$ , inconsistent.
Rearrange equations so the leading variables are in natural order.

The reduced row-echelon form (行阶梯矩阵 or Rref) satisfies:

若一 row 有 non- $0$ entries, the first non- $0$ entry must be $1$ , called the leading or pivot.
若一 col 中有 pivot，则 col 中其他 entries 必须为 $0$ .
若一 row 中有 pivot，则它后每个 row 必须有 pivot 在它右边（and rows of $0$ s must be at the bottom of matrix）.

Definition 2.2 : Elementary row operation

之前我们对 linear system 中 equations 的三种 operations 用在 matrix 上，这三种 operation 统称 elementary row operations:

Divide a row by a non-zero scalar.
Subtract a multiple of a row from another.
Swap two rows.

这种使用 elementary row operations 将 matrix 化为 rref 的解 linear system 的 algorithm 叫做 Gauss–Jordan elimination.

3 Linear systems and matrices

The worksheet starts with a system of linear equations:

{\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 39 \\ 𝑥 + 3 𝑦 + 2 𝑧 = 34 \\ 3 𝑥 + 2 𝑦 + 𝑧 = 26 \end{cases} .

To solve for $𝑥, 𝑦, 𝑧$ , we need to transform the system into the form ${\begin{cases} 𝑥 = \dots \\ 𝑦 = \dots \\ 𝑧 = \dots \end{cases}$ . In other words:

Eliminate terms that are off the diagonal.
Make the coefficients of the variables along the diagonal equal to $1$ .

The row-reduction calculation on the page is

{\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 39 \\ 𝑥 + 3 𝑦 + 2 𝑧 = 34 \\ 3 𝑥 + 2 𝑦 + 𝑧 = 26 \end{cases} \to {\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 39 \\ 𝑦 - 𝑧 = - 5 \\ 3 𝑥 + 2 𝑦 + 𝑧 = 26 \end{cases} \to {\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 39 \\ 𝑦 - 𝑧 = - 5 \\ - 4 𝑦 - 8 𝑧 = - 91 \end{cases},

where the first arrow subtracts the first equation and the next arrow uses $- 3 \times$ the first equation. Then

{\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 39 \\ 𝑦 - 𝑧 = - 5 \\ - 4 𝑦 - 8 𝑧 = - 91 \end{cases} \to {\begin{cases} 𝑥 + 5 𝑧 = 49 \\ 𝑦 - 𝑧 = - 5 \\ - 12 𝑧 = - 111 \end{cases} \to {\begin{cases} 𝑥 + 5 𝑧 = 49 \\ 𝑦 - 𝑧 = - 5 \\ 𝑧 = 9.25 \end{cases} \to {\begin{cases} 𝑥 = 2.75 \\ 𝑦 = 4.25 \\ 𝑧 = 9.25 \end{cases} .

The intervening operations are $- 2 \times$ the second equation and $+ 4 \times$ the second equation; then divide by $12$ ; finally use $- 5 \times$ the third equation and add the third equation.

Finally, we check the sol by substituting $𝑥, 𝑦, 𝑧$ into the original linear system. Happily, in Linear Algebra it is easy to check.

4 Linear combinations

4.1 WS 3, p. 1: geometry and number of solutions

Geometric Interpretation: three planes can have a point of intersection $(𝑥, 𝑦, 𝑧)$ , three planes having a line in common, or three planes with no common intersection. 这是 common situation. 然而还有下面这两种 situation：

$3$ planes having a line in common: a system with infinitely many sols.
$3$ planes with no common intersection: a system without sols.

For

{\begin{cases} 2 𝑥 + 4 𝑦 + 6 𝑧 = 0 \\ 4 𝑥 + 5 𝑦 + 6 𝑧 = 3 \\ 7 𝑥 + 8 𝑦 + 9 𝑧 = 6 \end{cases}

reduction gives $𝑥 - 𝑧 = 2$ , $𝑦 + 2 𝑧 = - 1$ . Generally choose $𝑧 = 𝑡$ ; then $𝑥 = 𝑡 + 2$ , $𝑦 = - 2 𝑡 - 1$ , and the general solution is $(𝑥, 𝑦, 𝑧) = (𝑡 + 2, - 2 𝑡 - 1, 𝑡) = (2, - 1, 0) + 𝑡 (1, - 2, 1)$ . The general sol represents a line in space; the source sketch marks $𝑡 = 0 : (2, - 1, 0)$ , $𝑡 = 1 : (3, - 3, 1)$ , and $𝑡 = 2 : (9, - 5, 2)$ .

For

{\begin{cases} 𝑥 + 2 𝑦 + 3 𝑧 = 0 \\ 4 𝑥 + 5 𝑦 + 6 𝑧 = 3 \\ 7 𝑥 + 8 𝑦 + 9 𝑧 = 0 \end{cases}

reduction yields $𝑥 - 𝑧 = 2$ , $𝑦 + 2 𝑧 = - 1$ , $0 = - 6$ . Whatever value we choose, $0 = - 6$ cannot be satisfied; this system is inconsistent and has no sol.

Complement (Joy of sets): To say a set is close under some operation $◇$ is to mean that $𝑎, 𝑏 \in 𝑆 \to 𝑎 ◇ 𝑏 \in 𝑆$ .

4.2 Number of solutions of a linear system

本章：① examine how many sols a system of linear equations can possibly can. ② Then we will present some definitions and rules of matrix algebra.

A system of equa. is said to be consistent if it has at least $1$ sol; inconsistent: no sol (iff rref of its augmented matrix contains $(\begin{matrix} 0 & \dots & 0 & | & 𝑐 \end{matrix})$ with $𝑐 \neq 0$ ).

If consistent, either infinitely many sols (at least one free variable) or exactly one sol (if all variables are leading).

Definition 4.3 : Matrix rank

The number of leading

1

′s in

rref (𝐴)

is denoted

rank (𝐴)

. For

𝐴 = (\begin{matrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{matrix})

rref (𝐴) = (\begin{matrix} 1 & 0 & - 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix})

, so

rank (𝐴) = 2

For an $𝑛 \times 𝑚$ coefficient matrix $𝐴$ and the $𝑛 \times (𝑚 + 1)$ augmented matrix $[𝐴 | 𝑏]$ :

$rank (𝐴) \leq 𝑚$ and $rank (𝐴) \leq 𝑛$ .
If the system is inconsistent, $rank (𝐴) < 𝑛$ .
If the system has one sol, $rank (𝐴) = 𝑚$ .
If the system has infin sols, $rank (𝐴) < 𝑚$ .

If system inconsistent, rref of augmented matrix contains a row $[0 \dots 0 | 1]$ , so no leading $1$ in that row for the coefficient part. Number of variables = total number of variables $-$ number of leading variables $= 𝑚 - rank (𝐴)$ . Thus exactly one sol gives $𝑚 - rank (𝐴) = 0$ , whereas infinitely many sols gives $𝑚 - rank (𝐴) > 0$ .

As contrapositive: (1) if $rank (𝐴) = 𝑛$ , the system has a sol; (2) if $rank (𝐴) < 𝑚$ , the system has no sol or infin sols; (3) if $rank (𝐴) = 𝑚$ , the system has no sol or exactly one sol.

Theorem 4.1 : Number of equations vs. number of unknowns

If a linear system has exactly one sol, then it has at least as many equations as variables (i.e.

𝑚 \leq 𝑛

for coefficient matrix

𝐴^{𝑛 \times 𝑚}

). Its contrapositive: a linear system with fewer equations than variables (

𝑛 < 𝑚

) has either no sol or infin sols.

To illustrate it: consider $2$ linear equations in $3$ variables. 每个 $𝑎 𝑥 + 𝑏 𝑦 + 𝑐 𝑧 = 𝑑$ 都表示一个 plane，而两个 plane 要么平行无交点，要么交于一直线 (infin sols)，不可能只有一个交点。

4.3 WS 3, pp. 2–3: matrix algebra and linear combinations

Definition 4.4 : Sum and scalar multiples of matrices

For same-size matrices, addition is entrywise; for

𝑘 \in ℝ

, multiplication by

𝑘

multiplies every entry by

𝑘

Definition 4.5 : Dot products of vectors

For

𝑣 = < 𝑣_{1}, \dots, 𝑣_{𝑛} >

and

𝑤 = < 𝑤_{1}, \dots, 𝑤_{𝑛} >

𝑣 \cdot 𝑤 = 𝑣_{1} 𝑤_{1} + 𝑣_{2} 𝑤_{2} + \dots + 𝑣_{𝑛} 𝑤_{𝑛}

. Note that the definition is not row-column-sensitive; it does not distinguish between row and col vectors.

Definition 4.6 : The product of $𝐴 𝑥$

Let

𝐴

be an

𝑛 \times 𝑚

matrix with row vectors

𝑤_{1}, \dots, 𝑤_{𝑛}

and let

𝑥 \in ℝ^{𝑚}

. Then

𝐴 𝑥 = (\begin{matrix} 𝑤_{1} \cdot 𝑥 \\ \dots \\ 𝑤_{𝑛} \cdot 𝑥 \end{matrix})

. In words, the

𝑖

th component of

𝐴 𝑥

is the dot product of the

𝑖

th row of

𝐴

with

𝑥

Example: $(\begin{matrix} 1 & 2 & 3 \\ 1 & 0 & - 1 \end{matrix}) (\begin{matrix} 3 \\ 1 \\ 2 \end{matrix}) = (\begin{matrix} 3 + 2 + 6 \\ 3 + 0 - 2 \end{matrix}) = (\begin{matrix} 11 \\ 1 \end{matrix})$ . The product $𝐴 𝑥$ is defined only when the num of col of $𝐴$ equals the num of components in $𝑥$ .

Theorem 4.2 : $𝐴 𝑥$ in terms of columns of $𝐴$

Let the columns of

𝐴

𝑣_{1}, \dots, 𝑣_{𝑚}

. Then

𝐴 𝑥 = 𝑥_{1} 𝑣_{1} + 𝑥_{2} 𝑣_{2} + \dots + 𝑥_{𝑚} 𝑣_{𝑚}

Definition 4.7 : Linear combination

A vector

𝑏 \in ℝ^{𝑛}

is called a linear combination of vectors

𝑣_{1}, \dots, 𝑣_{𝑚} \in ℝ^{𝑛}

if there are scalars

𝑥_{1}, \dots, 𝑥_{𝑚}

such that

𝑏 = 𝑥_{1} 𝑣_{1} + \dots + 𝑥_{𝑚} 𝑣_{𝑚}

Example: is $𝑏 = (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix})$ a linear combination of $𝑣 = (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})$ and $𝑤 = (\begin{matrix} 4 \\ 5 \\ 6 \end{matrix})$ ? Solve $(\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}) = 𝑥 (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) + 𝑦 (\begin{matrix} 4 \\ 5 \\ 6 \end{matrix}) = (\begin{matrix} 𝑥 + 4 𝑦 \\ 2 𝑥 + 5 𝑦 \\ 3 𝑥 + 6 𝑦 \end{matrix})$ . Row reduction gives $𝑥 = - \frac{1}{3}$ , $𝑦 = \frac{1}{3}$ , hence $𝑏 = - \frac{1}{3} 𝑣 + \frac{1}{3} 𝑤$ .

Theorem 4.3 : Algebraic rule for $𝐴 𝑥$

𝐴

is an

𝑛 \times 𝑚

matrix,

𝑥, 𝑦 \in ℝ^{𝑚}

, and

𝑘

is a scalar, then

𝐴 (𝑥 + 𝑦) = 𝐴 𝑥 + 𝐴 𝑦

and

𝐴 (𝑘 𝑥) = 𝑘 (𝐴 𝑥)

Theorem 4.4 : Matrix form of a linear system

With augmented matrix $[𝐴 | 𝑏]$ , a linear system can be written $𝐴 𝑥 = 𝑏$ . Its $𝑖$ th component is $𝑎_{𝑖 1} 𝑥_{1} + \dots + 𝑎_{𝑖 𝑚} 𝑥_{𝑚} = 𝑏_{𝑖}$ , the $𝑖$ th equation. For $3 𝑥_{1} + 𝑥_{2} = 7$ , $𝑥_{1} + 2 𝑥_{2} = 4$ : $(\begin{matrix} 3 & 1 \\ 1 & 2 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) = (\begin{matrix} 7 \\ 4 \end{matrix})$ , equivalently $𝑥_{1} (\begin{matrix} 3 \\ 1 \end{matrix}) + 𝑥_{2} (\begin{matrix} 1 \\ 2 \end{matrix}) = (\begin{matrix} 7 \\ 4 \end{matrix})$ .

For $2 𝑥_{1} - 3 𝑥_{2} + 5 𝑥_{3} = 7$ , $9 𝑥_{1} + 4 𝑥_{2} - 6 𝑥_{3} = 8$ : $(\begin{matrix} 2 & - 3 & 5 \\ 9 & 4 & - 6 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix}) = (\begin{matrix} 7 \\ 8 \end{matrix})$ . 如果我们可以 divide by matrix $𝐴$ ， $𝑥 = \frac{𝑏}{𝐴}$ ，就能直接解 $𝑥$ 。换言之我们是否能够找到 $𝐴^{- 1}$ 呢？ Chapter 2.

5 Linear transformations

5.1 WS 4, p. 1: coordinate encoding and standard matrices

现在你的位置为 $5^{○} E, 42^{○} N$ 。用一个 vector $(\begin{matrix} 5 \\ 42 \end{matrix}) \in ℝ^{2}$ 表示方位。现在你使用一个 encode 来加密你的方位：

{\begin{cases} 𝑦_{1} = 𝑥_{1} + 3 𝑥_{2} \\ 𝑦_{2} = 2 𝑥_{1} + 5 𝑥_{2} \end{cases}, (\begin{matrix} 𝑦_{1} \\ 𝑦_{2} \end{matrix}) = (\begin{matrix} 1 & 3 \\ 2 & 5 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) .

这个加密方位的码为 $(\begin{matrix} 131 \\ 220 \end{matrix})$ . 我们做了一个 transformation，使得 the same $(𝑦_{1}, 𝑦_{2})$ but 坐标不变。例如 $(0, 1)$ 从 $(𝑥_{1}, 𝑥_{2})$ 坐标 map 到 $(𝑦_{1}, 𝑦_{2})$ 坐标后仍是 $(0, 1)$ ；但 $(𝑦_{1}, 𝑦_{2})$ 下的 $(3, 5)$ 同样在 $(𝑥_{1}, 𝑥_{2})$ 下为 $(3, 5)$ ，而 $(𝑦_{1}, 𝑦_{2})$ 下的 $(5, 42)$ 在 $(𝑥_{1}, 𝑥_{2})$ 下为 $(131, 220)$ 。The source sketch marks these corresponding points and axes.

Definition 5.8 : Linear transformation

A function $𝑇 : ℝ^{𝑚} \to ℝ^{𝑛}$ is called a linear transformation if for every $𝑥 \in ℝ^{𝑚}$ there is an $𝑛 \times 𝑚$ matrix $𝐴$ such that $𝑇 (𝑥) = 𝐴 𝑥$ .

Example: $𝑦 = 𝑥_{1}^{2} + 𝑥_{2}^{2} + 𝑥_{3}^{2}$ has input $(\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix})$ and output $[𝑦]$ ; $𝐴 = (\begin{matrix} 𝑥_{1} & 𝑥_{2} & 𝑥_{3} \end{matrix})$ . 可以发现 $𝑦 = 𝑥 \cdot 𝑥$ is not a linear transformation of $𝑥$ .

Definition 5.9 : Identity matrix

Identity matrix, denoted by $𝐼_{𝑛}$ : $𝐼_{2} = (\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix})$ and $𝐼_{3} = (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix})$ .

For $𝑇 (𝑥) = (\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix}) 𝑥$ , $𝑇 ((\begin{matrix} 1 \\ 1 \end{matrix})) = (\begin{matrix} - 1 \\ 1 \end{matrix})$ and $𝑇 ((\begin{matrix} 0 \\ - 2 \end{matrix})) = (\begin{matrix} 2 \\ 0 \end{matrix})$ ; it is a counterclockwise rotation by $90$ degrees. Also $𝑥$ and $𝑇 (𝑥)$ have the same length: $\sqrt{𝑥_{1}^{2} + 𝑥_{2}^{2}} = \sqrt{{(- 𝑥_{2})}^{2} + 𝑥_{1}^{2}}$ . The source diagram labels the two arrows $𝑥$ and $𝑇 (𝑥)$ .

For $𝑇 (𝑥) = 𝐴 𝑥$ with $𝐴 = (\begin{matrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{matrix})$ , $𝑇 ((\begin{matrix} 1 \\ 0 \\ 0 \end{matrix})) = (\begin{matrix} 1 \\ 4 \\ 7 \end{matrix})$ and $𝑇 ((\begin{matrix} 0 \\ 1 \\ 0 \end{matrix})) = (\begin{matrix} 2 \\ 5 \\ 8 \end{matrix})$ .

5.2 WS 4, p. 2: linearity, bases, and transition matrices

Theorem 5.5 : Standard matrix

Consider a linear transformation

𝑇 : ℝ^{𝑚} \to ℝ^{𝑛}

. Let

𝑒_{𝑖}

be the vector whose

𝑖

th component is

1

and all other components are

0

. Then

𝐴 = (\begin{matrix} 𝑇 (𝑒_{1}) & 𝑇 (𝑒_{2}) & \dots & 𝑇 (𝑒_{𝑚}) \end{matrix})

. Equivalently,

𝑒_{1}, 𝑒_{2}, \dots, 𝑒_{𝑚}

is the standard vector of

ℝ^{𝑛}

(而

ℝ^{3}

中的

𝑒_{1}, 𝑒_{2}, 𝑒_{3}

都用

𝑖, 𝑗, 𝑘

来 denote).

Theorem 5.6 : Linearity test

A transformation

𝑇 : ℝ^{𝑚} \to ℝ^{𝑛}

is linear iff (a)

\forall 𝑣, 𝑤 \in ℝ^{𝑚}

𝑇 (𝑣 + 𝑤) = 𝑇 (𝑣) + 𝑇 (𝑤)

; and (b)

\forall 𝑣 \in ℝ^{𝑚}

and scalar

𝑘

𝑇 (𝑘 𝑣) = 𝑘 𝑇 (𝑣)

Definition 5.10 : Distribution vectors and transition matrices

A vector

𝑥 \in ℝ^{𝑛}

is said to be a distribution vector if its components (1) sum to

1

and (2) are all

\geq 0

. A square matrix

𝐴

is a transition matrix if its col vectors are distribution vectors.

6 Geometry of linear transformations

6.1 WS 5, p. 1: examples, scaling, and projection

在 2-1 我们知道 $(\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix})$ 是 $ℝ^{2}$ 中 counter clockwise 转 $90$ 度的 linear transformation。现在再看几个：

𝐴 = (\begin{matrix} 2 & 0 \\ 0 & 2 \end{matrix}), 𝐵 = (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}), 𝐶 = (\begin{matrix} - 1 & 0 \\ 0 & 1 \end{matrix}),

𝐷 = (\begin{matrix} 0 & 1 \\ - 1 & 0 \end{matrix}), 𝐸 = (\begin{matrix} 1 & 0.2 \\ 0 & 1 \end{matrix}), 𝐹 = (\begin{matrix} 1 & - 1 \\ 1 & 1 \end{matrix}) .

The source diagrams apply them to $(1, 2)$ and $(1, - 1)$ :

$𝐴$ : 放大一倍, sending them to $(2, 4)$ and $(- 2, - 2)$ .
$𝐵$ : orthogonal projection onto $𝑥$ -axis, sending both to $(1, 0)$ .
$𝐶$ : reflection about $𝑦$ -axis.
$𝐷$ : clockwise 转 $90$ degrees, sending them to $(- 2, - 1)$ and $(1, - 1)$ .
$𝐸$ : 向右 shear, sending them to $(1.4, 2)$ and $(0.8, - 1)$ .
$𝐹$ : counterclockwise shift + 放大 $\sqrt{2}$ 倍, sending them to $(- 1, 3)$ and $(2, 0)$ .

Theorem 6.7 : Scalings

(\begin{matrix} 𝑘 & 0 \\ 0 & 𝑘 \end{matrix})

defines a scaling by

𝑘

, since

(\begin{matrix} 𝑘 & 0 \\ 0 & 𝑘 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) = (\begin{matrix} 𝑘 𝑥_{1} \\ 𝑘 𝑥_{2} \end{matrix}) = 𝑘 (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) = 𝑘 𝑥

Definition 6.11 : Orthogonal projection

Consider a line $𝐿$ in coordinate plane, 经过 $(0, 0)$ . 任何 $𝑥 \in ℝ^{2}$ 都可以写成 $𝑥 = 𝑥^{∥} + 𝑥^{⟂}$ , where $𝑥^{∥} ‖ 𝐿$ and $𝑥^{⟂} ⟂ 𝐿$ . The transformation $𝑇 (𝑥) = 𝑥^{∥}$ is the orthogonal projection of $𝑥$ onto $𝐿$ , denoted ${proj}_{𝐿 (𝑥)}$ .

随意取 $𝑤 ‖ 𝐿$ , ${proj}_{𝐿 (𝑥)} = \frac{𝑥 \cdot 𝑤}{𝑤 \cdot 𝑤} 𝑤$ . 特别地，如果 $𝑤$ 是 unit vector $𝑢 = (\begin{matrix} 𝑢_{1} \\ 𝑢_{2} \end{matrix}) ‖ 𝐿$ , then ${proj}_{𝐿 (𝑥)} = (𝑥 \cdot 𝑢) 𝑢$ . 这一个 transformation 是 linear 的；with matrix $𝑃 = (\begin{matrix} 𝑢_{1}^{2} & 𝑢_{1} 𝑢_{2} \\ 𝑢_{1} 𝑢_{2} & 𝑢_{2}^{2} \end{matrix})$ .

Definition 6.12 : Reflection

Consider a line

𝐿

in coordinate plane, 过

(0, 0)

, and write

𝑥 = 𝑥^{∥} + 𝑥^{⟂}

. Then

𝑇 (𝑥) = 𝑥^{∥} - 𝑥^{⟂}

is reflection of

𝑥

about

𝐿

, denoted

{ref}_{𝐿 (𝑥)}

. 由 Definition 2.2.1,

{ref}_{𝐿 (𝑥)} = 𝑥^{∥} - (𝑥 - 𝑥^{∥}) = 2 {proj}_{𝐿 (𝑥)} - 𝑥 = (2 𝑃 - 𝐼_{2}) 𝑥

The matrix of $𝑇$ has form $(\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & - 𝑎 \end{matrix})$ where $𝑎^{2} + 𝑏^{2} = 1$ : $𝑆 = 2 𝑃 - 𝐼_{2} = (\begin{matrix} 2 𝑢_{1}^{2} - 1 & 2 𝑢_{1} 𝑢_{2} \\ 2 𝑢_{1} 𝑢_{2} & 2 𝑢_{2}^{2} - 1 \end{matrix}) = (\begin{matrix} 𝑢_{1}^{2} - 𝑢_{2}^{2} & 2 𝑢_{1} 𝑢_{2} \\ 2 𝑢_{1} 𝑢_{2} & 𝑢_{2}^{2} - 𝑢_{1}^{2} \end{matrix})$ .

6.2 WS 5, p. 2: rotations and shearing

Theorem 6.8 : Rotations

𝑇 (𝑥) = 𝐴 𝑥

is a counterclockwise rotation in

ℝ^{2}

(rotate by

𝜃

) iff

𝐴 = (\begin{matrix} cos (𝜃) & - sin (𝜃) \\ sin (𝜃) & cos (𝜃) \end{matrix}) = (\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix})

, where

𝑎^{2} + 𝑏^{2} = 1

For a nonzero vector $𝑥$ , write its coordinate in polar form $(𝑟 cos (𝜑), 𝑟 sin (𝜑))$ . Rotation gives $𝑥^{'} = (𝑟 cos (𝜑 + 𝜃), 𝑟 sin (𝜑 + 𝜃))$ , so $𝑥_{1^{'}} = 𝑥 cos (𝜃) - 𝑦 sin (𝜃)$ and $𝑦^{'} = 𝑥 sin (𝜃) + 𝑦 cos (𝜃)$ , which gives the displayed matrix.

Theorem 6.9 : Rotations combined with a scaling

将

𝑣 \in ℝ^{2}

counterclockwise rotate

𝜃

并放大至

𝑟

倍. Let

𝑇 (𝑎, 𝑏) = (\begin{matrix} 𝑟 cos (𝜃) \\ 𝑟 sin (𝜃) \end{matrix})

. Then

𝑇 (𝑥) = 𝐴 𝑥

𝐴 = (\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix}) = 𝑟 (\begin{matrix} cos (𝜃) & - sin (𝜃) \\ sin (𝜃) & cos (𝜃) \end{matrix})

6.3 5. Shearing

Vertical shear:

𝑇 ((\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix})) = (\begin{matrix} 𝑥_{1} \\ 𝑘 𝑥_{1} + 𝑥_{2} \end{matrix}) = (\begin{matrix} 1 & 0 \\ 𝑘 & 1 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) .

Horizontal shear:

𝑇 ((\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix})) = (\begin{matrix} 𝑥_{1} + 𝑘 𝑥_{2} \\ 𝑥_{2} \end{matrix}) = (\begin{matrix} 1 & 𝑘 \\ 0 & 1 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) .

Theorem 6.10 : Horizontal and vertical shearing

Horizontal shearing of slope

𝑘 𝑥_{2}

has matrix

(\begin{matrix} 1 & 𝑘 \\ 0 & 1 \end{matrix})

; vertical shearing of slope

𝑘 𝑥_{1}

has matrix

(\begin{matrix} 1 & 0 \\ 𝑘 & 1 \end{matrix})

The source’s concluding table records:

Scaling by $𝑘$ : $𝑘 𝐼_{2} = (\begin{matrix} 𝑘 & 0 \\ 0 & 𝑘 \end{matrix})$ .
Orthogonal projection onto line $𝐿$ : $(\begin{matrix} 𝑢_{1}^{2} & 𝑢_{1} 𝑢_{2} \\ 𝑢_{1} 𝑢_{2} & 𝑢_{2}^{2} \end{matrix})$ , for $𝑢 ‖ 𝐿$ and $‖ 𝑢 ‖ = 1$ .
Reflection about line $𝐿$ : $(\begin{matrix} 2 𝑢_{1}^{2} - 1 & 2 𝑢_{1} 𝑢_{2} \\ 2 𝑢_{1} 𝑢_{2} & 2 𝑢_{2}^{2} - 1 \end{matrix})$ .
Rotation through angle $𝜃$ (逆时针): $(\begin{matrix} cos (𝜃) & - sin (𝜃) \\ sin (𝜃) & cos (𝜃) \end{matrix})$ .
Rotation through $𝜃$ with scaling by $𝑟$ : $𝑟 (\begin{matrix} cos (𝜃) & - sin (𝜃) \\ sin (𝜃) & cos (𝜃) \end{matrix})$ .
Shear: horizontal $(\begin{matrix} 1 & 𝑘 \\ 0 & 1 \end{matrix})$ ; vertical $(\begin{matrix} 1 & 0 \\ 𝑘 & 1 \end{matrix})$ .

7 Gram–Schmidt and QR factorization

7.1 WS 17, p. 1: proof of QR factorization QR factorization

Let $𝑀 = (\begin{matrix} 𝑚_{1} & \dots & 𝑚_{𝑑} \end{matrix})$ be $𝑛 \times 𝑑$ . Gram–Schmidt produces $𝑄 = (\begin{matrix} 𝑞_{1} & \dots & 𝑞_{𝑑} \end{matrix})$ :

𝑞_{1} = \frac{𝑚_{1}}{‖ 𝑚_{1} ‖},

𝑞_{𝑖} = \frac{𝑚_{𝑖} - \sum_{𝑘 = 1}^{𝑖 - 1} (𝑚_{𝑖} \cdot 𝑞_{𝑘}) 𝑞_{𝑘}}{‖ 𝑚_{𝑖} - \sum_{𝑘 = 1}^{𝑖 - 1} (𝑚_{𝑖} \cdot 𝑞_{𝑘}) 𝑞_{𝑘} ‖} .

The $(𝑞_{1}, \dots, 𝑞_{𝑑})$ are orthonormal. 因而 $𝑄$ is an orthogonal matrix, and

𝑆_{𝑀 \to 𝑄} = (\begin{matrix} {[𝑚_{1}]}_{𝑄} & \dots & {[𝑚_{𝑑}]}_{𝑄} \end{matrix}) .

The worksheet computes

𝑄^{𝑡} 𝑀 = (\begin{matrix} 𝑚_{1} \cdot 𝑞_{1} & 𝑚_{2} \cdot 𝑞_{1} & \dots & 𝑚_{𝑑} \cdot 𝑞_{1} \\ 𝑚_{1} \cdot 𝑞_{2} & 𝑚_{2} \cdot 𝑞_{2} & \dots \\ \dots & \dots & 𝑚_{𝑑} \cdot 𝑞_{𝑑} \end{matrix}) .

Because $𝑞_{𝑖} ⟂ span (𝑚_{1}, \dots, 𝑚_{𝑖 - 1})$ , the entry $𝑞_{𝑖} \cdot 𝑚_{𝑗}$ is $0$ when $𝑖 > 𝑗$ ; the result is upper triangular. Thus $𝑀 = 𝑄 𝑅$ , with $𝑅 = 𝑄^{𝑡} 𝑀$ upper triangular.

7.2 Proof of matrix product theorem

For ordered bases $𝐵 = (𝑏_{1}, \dots, 𝑏_{𝑑})$ and $𝐴 = (𝑎_{1}, \dots, 𝑎_{𝑑})$ of $𝑊 \subseteq ℝ^{𝑛}$ :

(\begin{matrix} 𝑏_{1} & \dots & 𝑏_{𝑑} \end{matrix}) = (\begin{matrix} 𝑎_{1} & \dots & 𝑎_{𝑑} \end{matrix}) 𝑆_{𝐵 \to 𝐴},

where

𝑆_{𝐵 \to 𝐴} = (\begin{matrix} {[𝑏_{1}]}_{𝐴} & \dots & {[𝑏_{𝑑}]}_{𝐴} \end{matrix}) .

Note that if $𝑏_{𝑖} = 𝑐_{1} 𝑎_{1} + \dots + 𝑐_{𝑑} 𝑎_{𝑑}$ , then ${[𝑏_{𝑖}]}_{𝐴} = (\begin{matrix} 𝑐_{1} \\ \dots \\ 𝑐_{𝑑} \end{matrix})$ ; hence

𝐴 𝑆_{𝐵 \to 𝐴} = (\begin{matrix} {𝐴 [𝑏_{1}]}_{𝐴} & \dots & {𝐴 [𝑏_{𝑑}]}_{𝐴} \end{matrix}) = (\begin{matrix} 𝑏_{1} & \dots & 𝑏_{𝑑} \end{matrix}) = 𝐵 .

8 Orthogonal transformations

8.1 WS 18, p. 1: definition and matrix criteria

Definition 8.13 : Orthogonal transformation

𝑇 : ℝ^{𝑛} \to ℝ^{𝑛}

is orthogonal (正交变换) if it preserves dot products:

\forall 𝑥, 𝑦 \in ℝ^{𝑛}

𝑥 \cdot 𝑦 = 𝑇 (𝑥) \cdot 𝑇 (𝑦)

Theorem 8.11 : Equivalent characterizations

𝑇 : ℝ^{𝑛} \to ℝ^{𝑛}

is orthogonal iff it preserves the length of vectors:

\forall 𝑥 \in ℝ^{𝑛}

‖ 𝑇 (𝑥) ‖ = ‖ 𝑥 ‖

因而 linear trans 保留 dot product iff 保留 length. The proof expands ${‖ 𝑇 (𝑥 + 𝑦) ‖}^{2} = {‖ 𝑥 + 𝑦 ‖}^{2}$ :

{‖ 𝑇 (𝑥) ‖}^{2} + 2 𝑇 (𝑥) \cdot 𝑇 (𝑦) + {‖ 𝑇 (𝑦) ‖}^{2} = {‖ 𝑥 ‖}^{2} + 2 𝑥 \cdot 𝑦 + {‖ 𝑦 ‖}^{2},

then cancels equal norm terms.

(a) An orthogonal trans $𝑇$ is injective: $𝑇 (𝑥) = 0 \to ‖ 𝑇 (𝑥) ‖ = 0 \to ‖ 𝑥 ‖ = 0$ , so $ker 𝑇 = {0}$ and $𝑇$ is inj. (b) It is an isomorphism because a same-dimension linear trans is inj iff surj. (c) The standard matrix of $𝑇$ has orthonormal columns. (d) The composition of orthogonal trans is orthogonal, since

‖ 𝑇_{𝑘} \cdot 𝑇_{𝑘 - 1} \cdot \dots \cdot 𝑇_{1} (𝑥) ‖ = \dots = ‖ 𝑥 ‖ .

Definition 8.14 : Orthogonal matrix

A square matrix

𝐴

is orthogonal if

𝐴^{𝑡} 𝐴 = 𝐼_{𝑛}

(即

𝐴^{𝑡} = 𝐴^{- 1}

If $𝐴 = (\begin{matrix} 𝑣_{1} & \dots & 𝑣_{𝑛} \end{matrix})$ , then the $𝑖 𝑗$ th entry of $𝐴^{𝑡} 𝐴$ is $𝑣_{𝑖} \cdot 𝑣_{𝑗}$ . Therefore $𝐴$ is orthogonal iff its cols are orthonormal. 我们也由此知道：由 orthonormal basis 组成的 matrix $𝐴$ ， $𝐴^{- 1}$ 就是 $𝐴^{𝑡}$ .

The worksheet proves ${(𝐴 𝐵)}^{𝑡} = 𝐵^{𝑡} 𝐴^{𝑡}$ : the $𝑖 𝑗$ th entry of $𝐴 𝐵$ is the $𝑖$ th row of $𝐴$ dot the $𝑗$ th col of $𝐵$ , while the $𝑖 𝑗$ th entry of $𝐵^{𝑡} 𝐴^{𝑡}$ is $𝑏_{𝑖} \cdot 𝑎_{𝑗}$ . Thus they agree.

8.2 WS 18, p. 2: products and change of basis

If $𝐴$ is orthogonal, then $𝐴^{- 1}$ (也是 $𝐴^{𝑡}$ ) is orthogonal, since $𝔸^{𝑡} = 𝐼_{𝑛}$ implies $(𝐴^{𝑡}) {(𝐴^{𝑡})}^{𝑡} = 𝔸^{𝑡} = 𝐼_{𝑛}$ . 结论：这意味着对于 $𝐴$ 的 cols 是 orthonormal 的，那 $𝐴$ 的 rows 也是 orthonormal 的.

Theorem 8.12 : Products of orthogonal matrices

𝐴, 𝐵

are orthogonal

𝑛 \times 𝑛

matrices, then

𝐴 𝐵

is orthogonal.

Indeed,

(𝐴 𝐵) {(𝐴 𝐵)}^{𝑡} = 𝐴 (𝐵 𝐵^{𝑡}) 𝐴^{𝑡} = 𝐼_{𝑛},

so ${(𝐴 𝐵)}^{𝑡} = {(𝐴 𝐵)}^{- 1}$ .

Theorem 8.13 : Orthogonal maps and their matrices

𝑇 : ℝ^{𝑛} \to ℝ^{𝑛}

is orthogonal iff

{[𝑇]}_{𝜀}

is orthogonal, where

𝜀

is the standard basis. More generally,

𝑇

is orthogonal iff

{[𝑇]}_{𝛽}

is orthogonal for any orthonormal basis

𝛽

For the reverse direction,

𝑇 (𝑎) \cdot 𝑇 (𝑏) = ({[𝑇]}_{𝜀} 𝑎) \dot{{[𝑇]}_{𝜀} 𝑏} = 𝑎_{𝜀_{𝜀}^{𝑡 [𝑇]}}^{𝑡 [𝑇]} 𝑏 = 𝑎 \cdot 𝑏 .

Claim 1: If $𝐴, 𝐵$ are orthonormal basis matrices, their change-of-basis matrices are orthogonal. On WS 16, the entries are

𝑆_{𝐴 \to 𝐵} = (\begin{matrix} 𝑎_{1} \cdot 𝑏_{1} & 𝑎_{2} \cdot 𝑏_{1} & \dots & 𝑎_{𝑛} \cdot 𝑏_{1} \\ 𝑎_{1} \cdot 𝑏_{2} & \dots \\ \dots & \dots & 𝑎_{𝑛} \cdot 𝑏_{𝑛} \end{matrix})

and

𝑆_{𝐵 \to 𝐴} = (\begin{matrix} 𝑏_{1} \cdot 𝑎_{1} & 𝑏_{2} \cdot 𝑎_{1} & \dots \\ \dots & \dots & 𝑏_{𝑛} \cdot 𝑎_{𝑛} \end{matrix}) = 𝑆_{𝐴 \to 𝐵}^{𝑡} .

Thus $𝑆_{𝐴 \to 𝐵} = 𝑆_{𝐵 \to 𝐴}^{- 1}$ .

8.3 WS 18, p. 3: conclusion

Claim 2: orthogonal matrices 的 product 也是 orthogonal matrix. 这是因为 orthogonal transformations 的 composition 也是 orthogonal transformation，且它的自身的 matrix 代表为 orthogonal matrix。

Claim 3: 如果 $𝑇$ orthogonal，则 ${[𝑇]}_{𝛽}$ orthogonal for 任意 orthonormal basis $𝛽$ :

{[𝑇]}_{𝛽} = 𝑆_{𝜀 \to 𝛽} {[𝑇]}_{𝜀} 𝑆_{𝛽 \to 𝜀} .

All three factors are orthogonal, hence so is ${[𝑇]}_{𝛽}$ .

Claim 4: $𝛽$ 为任意 orthonormal basis; 如果 ${[𝑇]}_{𝛽}$ orthogonal，则 $𝑇$ orthogonal. Since

{[𝑇]}_{𝛽} = 𝑆_{𝜀 \to 𝛽} {[𝑇]}_{𝜀} 𝑆_{𝛽 \to 𝜀},

we obtain

{[𝑇]}_{𝜀} = 𝑆_{𝛽 \to 𝜀} {[𝑇]}_{𝛽} 𝑆_{𝜀 \to 𝛽},

a product of orthogonal matrices; hence ${[𝑇]}_{𝜀}$ is orthogonal and therefore $𝑇$ is orthogonal.

总结： $𝑇$ is orthogonal (保留 dot product) $= 𝑇$ 保留 length $= 𝑇$ 保留 distance $= 𝑇$ 把 $ℝ^{𝑛}$ 的某个 orthonormal basis map 到另一个 orthonormal basis $= {[𝑇]}_{𝛽}$ 为 orthogonal 的， $𝛽$ 为任意 orthonormal basis $= {[𝑇]}_{𝛽}$ 的 rows/cols 为一个 orthonormal basis of $ℝ^{𝑛}$ .

9 Least squares

9.1 WS 19, p. 1: projection and image/kernel theorem

Theorem 9.14 : Projection and Least squares method

对于任意 subspace

𝑉 \subseteq ℝ^{𝑛}

\forall 𝑥 \in ℝ^{𝑛}

{proj}_{𝑉 (𝑥)}

𝑉

中离

𝑥

最近的 vector, i.e.

𝑣 \in 𝑉

‖ 𝑥 - {proj}_{𝑉 (𝑥)} ‖ \leq ‖ 𝑥 - 𝑣 ‖

The source diagram labels $‖ 𝑥 - {proj}_{𝑉 (𝑥)} ‖$ as the 垂直距离.

(a) $𝐴 𝑥 = 𝑏$ is consistent iff $𝑏 \in im 𝐴$ : if $\exists 𝑥$ such that $𝐴 𝑥 = 𝑏$ , then $𝑏 \in im 𝐴$ , and conversely. (b) 不可能 $𝑏$ 不在 $span (cols of A)$ 而 $𝐴 𝑥 = 𝑏$ be consistent. (c), least squares solutions: if $𝐴 𝑥 = 𝑏$ is not consistent, let $𝑉 = im 𝐴$ , $𝑏^{'} = {proj}_{𝑉 (𝑏)}$ . Then $𝐴 𝑥 = 𝑏^{'}$ has a solution because $𝑏^{'} \in 𝑉$ ; the solutions in that solution set are the least squares solution to this system.

The worksheet also records

𝐴 𝑥 \cdot 𝑦 = {(𝐴 𝑥)}^{𝑡} 𝑦 = 𝑥^{𝑡} 𝐴^{𝑡} 𝑦 = 𝑥 \cdot (𝐴^{𝑡} 𝑦) .

Theorem 9.15 : Image-kernel orthogonality

For an

𝑚 \times 𝑛

matrix

𝐴

ker (𝐴^{𝑡}) = {(im 𝐴)}^{⟂}

Write $𝐴 = (\begin{matrix} 𝑣_{1} & \dots & 𝑣_{𝑛} \end{matrix})$ , so $𝐴^{𝑡} = (\begin{matrix} 𝑣_{1}^{𝑡} \\ \dots \\ 𝑣_{𝑛}^{𝑡} \end{matrix})$ . If $𝑥 \in ker 𝐴^{𝑡}$ , then $𝑣_{𝑖} \cdot 𝑥 = 0$ for every $𝑖$ , so $𝑥 ⟂ im 𝐴$ . Conversely, if $𝑥 {\in (im 𝐴)}^{⟂}$ , then every $𝑣_{𝑖} \cdot 𝑥 = 0$ , so $𝐴^{𝑡} 𝑥 = 0$ .

A second proof uses the displayed transpose identity: $𝑦 \in ker 𝐴^{𝑡} \to 𝐴 𝑥 \cdot 𝑦 = 0$ for all $𝑥$ , hence $𝑦 {\in (im 𝐴)}^{⟂}$ ; conversely $𝐴 𝑥 \cdot 𝑦 = 0$ for all $𝑥$ implies $𝐴^{𝑡} 𝑦 = 0$ .

Consequently ${(ker 𝐴^{𝑡})}^{⟂} = im 𝐴$ , and $ker 𝐴 = {(im 𝐴^{𝑡})}^{⟂}$ . The rank computation is

rank (𝐴^{𝑡}) = 𝑛 - dim (ker 𝐴^{𝑡}) = dim (im 𝐴) = rank (𝐴) .

9.2 WS 19, p. 2: normal equations

The normal equation: $𝐴 𝑥 = 𝑏$ 的 least square 解当且仅当 $𝐴 𝑥 = proj (𝑏)$ is a solution of

𝐴^{𝑡} 𝐴 𝑥 = 𝐴^{𝑡} 𝑏 .

Theorem 9.16 : Kernel of $𝐴^{𝑡} 𝐴$

For an

𝑚 \times 𝑛

matrix

𝐴

ker 𝐴 = ker (𝐴^{𝑡} 𝐴)

If $𝑥 \in ker (𝐴^{𝑡} 𝐴)$ , then $𝐴^{𝑡} 𝐴 𝑥 = 0$ , so $𝐴 𝑥 \in ker 𝐴^{𝑡} = {(im 𝐴)}^{⟂}$ . Since $𝐴 𝑥 \in im 𝐴$ and $im 𝐴$ 与 ${(im 𝐴)}^{⟂}$ only meet at $0$ , so $𝐴 𝑥 = 0$ . Conversely $𝐴 𝑥 = 0$ directly implies $𝐴^{𝑡} 𝐴 𝑥 = 0$ .

For the rest proof of normal equation, the source diagram records $𝑏 - {proj}_{im 𝐴} (𝑏) {\in (im 𝐴)}^{⟂} = ker (𝐴^{𝑡})$ . Thus $𝐴 𝑥 - 𝑏 \in ker (𝐴^{𝑡})$ , whence

𝐴^{𝑡 (𝐴 𝑥 - 𝑏)} = 0, 𝐴^{𝑡} 𝐴 𝑥 = 𝐴^{𝑡} 𝑏 .

总结：对于 $𝑇_{𝐴} : ℝ^{𝑛} \to ℝ^{𝑚}$ , $𝑇_{𝐴^{𝑡}} : ℝ^{𝑚} \to ℝ^{𝑛}$ , 使得对于 $𝑥 \in ℝ^{𝑛}$ 及 $𝑦 \in ℝ^{𝑚}$ ， $(𝐴 𝑥) \cdot 𝑦 = 𝑥 \dot{𝐴^{𝑡} 𝑦}$ 是 transpose 本质的性质；它告诉我们 $𝐴$ 是 $ℝ^{𝑛}$ 到 $ℝ^{𝑚}$ 的 linear map.

10 Inner-product spaces

10.1 WS 20, p. 1: examples and generalized Gram–Schmidt

For the Inner product space $𝑉$ of functions on $[- 𝜋, 𝜋]$ , $< 𝑓, 𝑔 \geq \frac{1}{𝜋} \int_{- 𝜋}^{𝜋} 𝑓 (𝑡) 𝑔 (𝑡) d 𝑡$ :

$< 𝑓, 𝑔 \geq < 𝑔, 𝑓 >$ .
Linearity: $< 𝑎 𝑓_{1} + 𝑏 𝑓_{2}, 𝑔 \geq \frac{𝑎}{𝜋} \int_{- 𝜋}^{𝜋} 𝑓_{1} (𝑡) 𝑔 (𝑡) d 𝑡 + \frac{𝑏}{𝜋} \int_{- 𝜋}^{𝜋} 𝑓_{2} (𝑡) 𝑔 (𝑡) d 𝑡 = 𝑎 < 𝑓_{1}, 𝑔 > + 𝑏 < 𝑓_{2}, 𝑔 >$ .
Positive definite: $< 𝑓, 𝑓 \geq \frac{1}{𝜋} \int_{- 𝜋}^{𝜋} 𝑓^{2} (𝑡) d 𝑡$ , and $𝑓^{2} (𝑡) > 0 \to < 𝑓, 𝑓 ≫ 0$ .

(b) 同 (a)，inner product space $𝑉$ . (c) 不是 inner prod space: can diverge, $\infty$ 不在 $ℝ$ .

P4: find two different inner products in $ℝ^{2}$ : $< 𝑥, 𝑦 \geq 𝑥_{1} 𝑦_{1} + 𝑥_{2} 𝑦_{2}$ (dot product; here $𝑒_{1}, 𝑒_{2}$ are orthonormal), and different weight $< 𝑥, 𝑦 \geq 3 𝑥_{1} 𝑦_{1} + 2 𝑥_{2} 𝑦_{2}$ .

P5: Every finite dimensional inner product space has an orthonormal basis. Take

𝑢_{1} = \frac{𝑣_{1}}{norm (𝑣_{1})}, 𝑢_{2} = \frac{𝑣_{2} - (𝑣_{2} \cdot 𝑢_{1}) 𝑢_{1}}{‖ 𝑣_{2} - (𝑣_{2} \cdot 𝑢_{1}) 𝑢_{1} ‖},

and, in general,

𝑢_{𝑛} = \frac{𝑣_{𝑛} - \sum_{𝑗 = 1}^{𝑛 - 1} (𝑣_{𝑛} \cdot 𝑢_{𝑗}) 𝑢_{𝑗}}{‖ 𝑣_{𝑛} - \sum_{𝑗 = 1}^{𝑛 - 1} (𝑣_{𝑛} \cdot 𝑢_{𝑗}) 𝑢_{𝑗} ‖} .

proof: generalized Gram–Schmidt.

10.2 WS 20, p. 2: coordinates and matrix inner products

P8: 任选 orthonormal basis $𝑈$ for inner product $< \cdot, \cdot >$ . Then $(𝑉, < \cdot, \cdot >)$ 上

< 𝑥, 𝑦 \geq {[𝑥]}_{𝑈} \cdot {[𝑦]}_{𝑈} .

Indeed $𝑥 = \sum_{𝑖} 𝑎_{𝑖} 𝑢_{𝑖}$ , $𝑦 = \sum_{𝑗} 𝑏_{𝑗} 𝑢_{𝑗}$ , so

< 𝑥, 𝑦 \geq \sum_{𝑖} \sum_{𝑗} 𝑎_{𝑖} 𝑏_{𝑗} < 𝑢_{𝑖}, 𝑢_{𝑗} > = \sum_{𝑖} 𝑎_{𝑖} 𝑏_{𝑖} = {[𝑥]}_{𝑈} {\cdot [𝑦]}_{𝑈} .

P9: $\forall$ inner product $< \cdot, \cdot >$ on $ℝ^{𝑛}$ , $\exists$ an $𝑛 \times 𝑛$ symmetric matrix $𝐴$ such that

\forall 𝑥, 𝑦 \in ℝ^{𝑛}, < 𝑥, 𝑦 \geq 𝑥^{𝑡} 𝐴 𝑦 .

For the standard basis, the $(𝑖, 𝑗)$ th entry of $𝐴$ is $< 𝑒_{𝑖}, 𝑒_{𝑗} >$ . Hence

< 𝑥, 𝑦 \geq \sum_{𝑖} \sum_{𝑗} 𝑥_{𝑖} 𝑦_{𝑗} < 𝑒_{𝑖}, 𝑒_{𝑗} > = 𝑥^{𝑡} 𝐴 𝑦 .

Because the inner product is symmetric, so is $𝐴$ . Its diagonal entries are positive; $𝐴$ is 可逆 (full rank).

P10: All inner products on $ℝ^{2}$ have

𝐵_{𝐴 (𝑥, 𝑦)} = 𝑥^{𝑡} 𝐴 𝑦, 𝐴 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & 𝑐 \end{matrix}),

where (1) $𝐴$ is symmetric; (2) $𝑎 > 0$ and $det 𝐴 = 𝑎 𝑐 - 𝑏^{2} > 0$ . Indeed

(\begin{matrix} 𝑥 & 𝑦 \end{matrix}) (\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & 𝑐 \end{matrix}) (\begin{matrix} 𝑥 \\ 𝑦 \end{matrix}) = 𝑎 𝑥^{2} + 2 𝑏 𝑥 𝑦 + 𝑐 𝑦^{2} > 0 .

(这个条件等价：充分条件为 $𝑐 > 0$ 且 $𝑎 𝑐 - 𝑏^{2} > 0 = det 𝐴$ .) Linearity is guaranteed by matrix multiplication. 因而这三条为完整条件。

实际上的理解为：把 $ℝ^{𝑛}$ 的任何 inner product 都是：把一个 vector 做一个 linear trans 后再做一个 dot product.

11 Diagonalization and eigenspaces

11.1 WS 23, p. 1: four theorems

Theorem 11.17 : Eigenbasis and diagonal matrix

For a finite-dimensional vector space

𝑉

, a basis

𝛽

𝑉

is an eigenbasis for

𝑇 : 𝑉 \to 𝑉

iff

{[𝑇]}_{𝛽}

is diagonal.

Indeed,

{[𝑇]}_{𝛽} = (\begin{matrix} {[𝑇 (𝑏_{1})]}_{𝛽} & \dots & {[𝑇 (𝑏_{𝑛})]}_{𝛽} \end{matrix}) = (\begin{matrix} 𝜆_{1} 𝑏_{1} & \dots & 𝜆_{𝑛} 𝑏_{𝑛} \end{matrix}) .

Theorem 11.18 : Diagonalizable transformation

𝑇 : 𝑉 \to 𝑉

is diagonalizable iff

{[𝑇]}_{𝛽}

is similar to some diagonal matrix

𝐷

The definition notes: $𝑇$ diagonalizable iff 其 $𝐷 = 𝑛 \times 𝑛$ matrix of $𝑇$ is diagonal. 因而（所有 ${[𝑇]}_{𝛼}$ 都相似于 ${[𝑇]}_{𝛽}$ ）.

Theorem 11.19 : Diagonalizable matrix

𝐴 \in ℝ^{𝑛 \times 𝑛}

is diagonalizable iff

𝐴

is similar to some diagonal

𝐷

Theorem 11.20 : Eigenspace

𝐸_{𝜆} = ker (𝑇 - 𝜆 𝐼_{𝑛})

, so

dim (𝐸_{𝜆}) = dim (ker (𝑇 - 𝜆 𝐼_{𝑛})) = 𝑛 - rank (𝑇 - 𝜆 𝐼_{𝑛})

Proof: $(𝑇 - 𝜆 𝐼) (𝑣) = 0$ iff $𝑇 (𝑣) = 𝜆 𝐼 (𝑣) = 𝜆 𝑣$ , so the kernel is exactly the vectors changed only by $𝑇$ stretching by $𝜆$ , i.e. the eigenvectors with eigenvalue $𝜆$ together with zero. 没有被 $𝑇 - 𝜆 𝐼$ 改变的 vectors，即只被 $𝑇$ 拉伸 $𝜆$ 倍的 vectors.

(Thm 7.1.3) 如果 $𝐷 = (𝑣_{1}, \dots, 𝑣_{𝑛})$ 是 an eigenbasis of $𝑉$ for $𝑇$ , then

{[𝑇]}_{𝐷} = 𝑆_{𝐷 \to 𝜀}^{- 1} {[𝑇]}_{𝜀} 𝑆_{𝐷 \to 𝜀} = (\begin{matrix} 𝜆_{1} & \dots \\ \dots & 𝜆_{𝑛} \end{matrix}) .

Proof note: $𝑇_{𝐴 (𝑥)} = 𝐴 𝑥$ ( $𝐴 = {[𝑇]}_{𝜀}$ ). Then $𝑆_{𝐷 \to 𝜀} = (\begin{matrix} 𝑣_{1} & \dots & 𝑣_{𝑛} \end{matrix})$ and

{[𝑇]}_{𝜀} 𝑆_{𝐷 \to 𝜀} = (\begin{matrix} 𝐴 𝑣_{1} & \dots & 𝐴 𝑣_{𝑛} \end{matrix}) = (\begin{matrix} 𝜆_{1} 𝑣_{1} & \dots & 𝜆_{𝑛} 𝑣_{𝑛} \end{matrix}) .

Hence the change of basis yields the diagonal matrix.

12 Eigenvalues and eigenspaces

12.1 WS 24, p. 1: characteristic polynomial and multiplicities

Theorem 12.21 : Eigenvalues are roots

The eigenvalues of

𝑇

are precisely the roots of its characteristic polynomial.

简明 proof: $𝜆$ 是 $𝑇$ 的 eigenvalue iff $𝐸_{𝜆} \neq {0}$ iff $ker (𝑇 - 𝜆 𝐼) \neq {0}$ iff nullity $(𝑇 - 𝜆 𝐼) > 0$ iff $rank (𝑇 - 𝜆 𝐼) < 𝑛$ iff $𝑇 - 𝜆 𝐼$ 不可逆 iff $det (𝑇 - 𝜆 𝐼) = 0$ . Thus $𝜆$ is a root of $det (𝑇 - 𝜆 𝐼) = 0$ .

Theorem 12.22 : Distinct-eigenvalue eigenspaces

\forall 𝑥 \in 𝐸_{𝜆_{1}}

且

𝑥 \neq 0

, and

𝑦 \in 𝐸_{𝜆_{2}}

且

𝑦 \neq 0

, if

𝜆_{1} \neq 𝜆_{2}

, then

𝑥, 𝑦

are linearly independent.

因而 eigenbases of distinct eigenspaces have linearly independent union, and if $\sum_{𝑖} dim (𝐸_{𝜆_{𝑖}}) = dim 𝑉$ , that union is an eigenbasis of $𝑉$ for $𝑇$ .

Theorem 12.23 : Geometric and algebraic multiplicity

For eigenvalue

𝜆

gem (𝜆) \leq alm (𝜆)

Let $𝐴 \in ℝ^{𝑛 \times 𝑛}$ , let $𝜆$ be an eigenvalue and $gem (𝜆) = 𝑚$ . Let $(𝑣_{1}, \dots, 𝑣_{𝑚})$ be a basis of $𝐸_{𝜆}$ , put $𝑆 = (\begin{matrix} 𝑣_{1} & \dots & 𝑣_{𝑚} \end{matrix})$ , and define $𝐵 = 𝑆^{- 1} 𝐴 𝑆$ . Then for $1 \leq 𝑖 \leq 𝑚$ ,

𝐵 𝑒_{𝑖} = 𝑆^{- 1} 𝐴 𝑆 𝑒_{𝑖} = 𝑆^{- 1} 𝐴 𝑣_{𝑖} = 𝑆^{- 1} 𝜆 𝑣_{𝑖} = 𝜆 𝑒_{𝑖} .

Therefore

𝐵 = (\begin{matrix} 𝜆 𝐼_{𝑚} & 𝑃 \\ 0 & 𝑄 \end{matrix}),

𝑓_{𝐴 (𝑥)} = 𝑓_{𝐵 (𝑥)} = det (𝐵 - 𝑥 𝐼_{𝑛}) = {(𝑥 - 𝜆)}^{𝑚} 𝑓_{𝑄 (𝑥)},

and $alm (𝜆) \geq 𝑚$ .

Theorem 12.24 : Degree of the characteristic polynomial

For

𝑇 : 𝑉 \to 𝑉

𝜒_{𝑇 (𝑥)}

has degree

dim 𝑉

This is the corollary of the preceding two theorems: if $𝑇$ has $𝑛$ different eigenvalues, then $\sum_{𝑖} alm (𝜆_{𝑖}) \leq dim 𝑉$ . Proof: 见 P9.

13 Complex eigenvalues

13.1 WS 26, p. 1: complex roots and real $2 \times 2$ matrices

P5(b). Let $𝑧 = 𝑎 + 𝑏 𝑖$ and $𝑤 = 𝑐 + 𝑑 𝑖$ . Then

𝑧 + 𝑤 = (𝑎 + 𝑐) + (𝑏 + 𝑑) 𝑖,

z w=(a c-b d)+(b c+a d)i $,$

conjugate (𝑧) conjugate (𝑤) = (𝑎 - 𝑏 𝑖) (𝑐 - 𝑑 𝑖) = 𝑎 𝑐 - 𝑏 𝑑 - (𝑏 𝑐 + 𝑎 𝑑) 𝑖 = conjugate (𝑧 𝑤) .

(c) Fact: $𝜆$ is a root of $𝑓 (𝑥)$ iff $𝜆$ is a root when the coefficients of $𝑓 (𝑥)$ are real. If $𝜆^{𝑛} + 𝑎_{𝑛 - 1} 𝜆^{𝑛 - 1} + \dots + 𝑎_{0} = 0$ , then

\sum_{𝑘} 𝑎_{𝑘} 𝜆^{𝑘} = conjugate (\sum_{𝑘} 𝑎_{𝑘} 𝜆^{𝑘}) = 0 .

For

𝐴 = (\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix}) \in ℝ^{2 \times 2},

𝜒_{𝐴 (𝜆)} = {(𝑎 - 𝜆)}^{2} + 𝑏^{2}, 𝜒_{𝐴 (𝜆)} = 0 \to 𝜆 = 𝑎 \pm 𝑏 𝑖 .

(b) Factor $𝐴$ into a scalar matrix $𝑟 𝐼_{2}$ and a rotation matrix $𝑅_{𝜃}$ :

𝐴 = \sqrt{𝑎^{2} + 𝑏^{2}} (\begin{matrix} cos (𝜃) & - sin (𝜃) \\ sin (𝜃) & cos (𝜃) \end{matrix}),

where

cos (𝜃) = \frac{𝑎}{\sqrt{𝑎^{2} + 𝑏^{2}}}, sin (𝜃) = \frac{𝑏}{\sqrt{𝑎^{2} + 𝑏^{2}}}, 𝜃 = arctan (\frac{𝑏}{𝑎}) .

𝐷 = (\begin{matrix} 𝑎 + 𝑏 𝑖 & 0 \\ 0 & 𝑎 - 𝑏 𝑖 \end{matrix}) .

For $𝜆_{1} = 𝑎 + 𝑏 𝑖$ ,

det (𝐴 - 𝜆_{1} 𝐼_{2}) = det ((\begin{matrix} - 𝑏 𝑖 & - 𝑏 \\ 𝑏 & - 𝑏 𝑖 \end{matrix})),

so $(\begin{matrix} 𝑖 \\ 1 \end{matrix})$ is a basis for $𝐸_{𝜆_{1}}$ ; similarly $(\begin{matrix} - 𝑖 \\ 1 \end{matrix})$ is a basis for $𝐸_{𝜆_{2}}$ . Hence

𝐷 = 𝑆^{- 1} 𝐴 𝑆 = {(\begin{matrix} 𝑖 & - 𝑖 \\ 1 & 1 \end{matrix})}^{- 1} 𝐴 (\begin{matrix} 𝑖 & - 𝑖 \\ 1 & 1 \end{matrix}),

and

𝐴 = (\begin{matrix} 𝑖 & - 𝑖 \\ 1 & 1 \end{matrix}) (\begin{matrix} 𝑎 + 𝑏 𝑖 & 0 \\ 0 & 𝑎 - 𝑏 𝑖 \end{matrix}) {(\begin{matrix} 𝑖 & - 𝑖 \\ 1 & 1 \end{matrix})}^{- 1} .

P9: 任何有一对 complex eigenvalue $𝑎 \pm 𝑏 𝑖$ 的 $𝐴 \in ℝ^{2 \times 2}$ 都 similar to $(\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix})$ （一个 scaling matrix）. 因为 $𝐴$ diagonalizable: $𝐷 = (\begin{matrix} 𝑎 + 𝑏 𝑖 & 0 \\ 0 & 𝑎 - 𝑏 𝑖 \end{matrix})$ ，而 $𝐷$ 又 similar to $(\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix})$ by P8.

14 Spectral theorem

14.1 WS 27, p. 1: theorem and first two claims

Theorem 14.25 : Spectral theorem

𝐴

is orthogonally diagonalizable iff

𝐴

is symmetric:

\exists 𝑆 \in ℝ^{𝑛 \times 𝑛}

such that

𝑆^{- 1} = 𝑆^{𝑡}

and

𝑆^{𝑡} 𝐴 𝑆

is diagonal iff

𝐴

is symmetric.

Equivalently, $𝑆$ 的 cols 是 $ℝ^{𝑛}$ 的一个 orthonormal basis, 且为 $𝐴$ 的 eigenvectors. 普通 diagonalization: $𝐷 = 𝑆 𝐴 𝑆^{- 1}$ , $𝑆$ 的 cols 为 $𝑉$ 的一个 eigenbasis $(𝑣_{1}, \dots, 𝑣_{𝑛})$ . 而 orthogonal diagonalization: $𝐷 = 𝑆^{𝑡} 𝐴 𝑆$ , $𝑆$ 的 cols 不但是 eigenbasis，而且还是一个 orthonormal eigenbasis. 这意味着对于不同的 eigenvectors，它们都是 orthogonal 的（可以为 orthonormal），也就是说 $𝑉, 𝐸_{𝜆_{1}}, 𝐸_{𝜆_{2}}$ with $𝐸_{𝜆_{1}} ⟂ 𝐸_{𝜆_{2}}$ .

Claim 1: if $𝐴$ is orthogonally diagonalizable, then $𝐴$ is symmetric. If $𝑆^{𝑡} 𝐴 𝑆 = 𝐷$ and $𝑆^{- 1} = 𝑆^{𝑡}$ , then

𝐴 = 𝑆 𝐷 𝑆^{𝑡},

and

𝐴^{𝑡} = {(𝑆 𝐷 𝑆^{𝑡})}^{𝑡} = 𝑆 𝐷^{𝑡} 𝑆^{𝑡} = 𝑆 𝐷 𝑆^{𝑡} = 𝐴 .

Claim 2: if $𝐴 \in ℝ^{𝑛 \times 𝑛}$ is symmetric, then $𝐴$ is orthogonally diagonalizable over $ℝ$ . This claim is divided into three parts.

Claim 2 pt.(1): if $𝐴 \in ℝ^{𝑛 \times 𝑛}$ is symmetric, then $𝐴$ has real eigenvalues. By the fundamental theorem of Algebra, $𝜒_{𝑇 (𝑥)} = 0$ 有 $𝑛$ 个 complex roots 包含重复. Let $𝜆$ be any complex eigenvalue, so $\exists 𝑧 \in ℂ^{𝑛}$ , $𝐴 𝑧 = 𝜆 𝑧$ . Then $𝐴 𝑏$ (..) => ..{𝑧}=𝑏(..) => ..𝜆𝑏(..) => ..{𝑧}, and, after transpose, $𝑏$ (..) => ..{𝑧}𝑡𝐴=𝑏(..) => ..𝜆𝑏(..) => ..{𝑧}𝑡 because $𝐴$ is symmetric. Multiply $𝑧$ on the right:

𝑏

(..) => ..{𝑧}𝑡𝐴𝑧=𝜆(𝑏(..) => ..{𝑧}𝑡𝑧)=𝑏(..) => ..𝜆(𝑏(..) => ..{𝑧}𝑡𝑧).

Since $𝑏$ (..) => ..{𝑧}𝑡𝑧=∑𝑖|𝑧𝑖|2>0, $𝜆 = 𝑏$ (..) => ..𝜆; every complex eigenvalue is real.

Claim 2 pt.(2): consider $𝜆_{1}, 𝜆_{2}$ and nonzero $𝑣_{1} \in 𝐸_{𝜆_{1}}$ , $𝑣_{2} \in 𝐸_{𝜆_{2}}$ , with $𝜆_{1} \neq 𝜆_{2}$ . Then

𝜆_{1} 𝑣_{1}^{𝑡} 𝑣_{2} = {(𝐴 𝑣_{1})}^{𝑡} 𝑣_{2} = 𝑣_{1}^{𝑡} 𝐴^{𝑡} 𝑣_{2} = 𝑣_{1}^{𝑡 (𝐴 𝑣_{2})} = 𝜆_{2} 𝑣_{1}^{𝑡} 𝑣_{2} .

Thus $𝑣_{1} \cdot 𝑣_{2} = 0$ , so $𝑣_{1} ⟂ 𝑣_{2}$ .

14.2 WS 27, p. 2: induction proof

Claim 2 pt.(3): if $𝐴 \in ℝ^{𝑛 \times 𝑛}$ is symmetric, then $𝐴$ is orthogonally diagonalizable. Prove by induction.

Base case: a $1 \times 1$ matrix is diagonal and symmetric. Inductive step: if every symmetric $𝑛 \times 𝑛$ matrix is orthogonally diagonalizable, then every $(𝑛 + 1) \times (𝑛 + 1)$ matrix is too.

Let $𝐴 \in ℝ^{(𝑛 + 1) \times (𝑛 + 1)}$ be symmetric. Let $𝜆$ be an eigenvalue and $𝑢$ a corresponding unit eigenvector. Complete $𝑈 = (𝑢, 𝑢_{1}, \dots, 𝑢_{𝑛})$ to an orthonormal basis of $ℝ^{𝑛 + 1}$ . Put

𝑄 = (\begin{matrix} 𝑢 & 𝑢_{1} & \dots & 𝑢_{𝑛} \end{matrix}),

so $𝑄$ is orthogonal and $𝑄^{𝑡} = 𝑄^{- 1}$ . 注意 $𝑄$ 为 $𝑆_{𝑈 \to 𝜀}$ , 因而 $𝑄^{𝑡} = 𝑄^{- 1} = 𝑆_{𝜀 \to 𝑈}$ .

Then

𝑄^{𝑡} 𝐴 𝑄 = [𝜆, 0; 0, 𝐵]

for some $𝐵 \in ℝ^{𝑛 \times 𝑛}$ : $𝑄^{𝑡} 𝐴 𝑄$ is symmetric, and

𝑄^{𝑡} 𝐴 𝑄 𝑒_{1} = 𝑄^{𝑡 (𝐴 𝑢)} = 𝑄^{𝑡 (𝜆 𝑢)} = 𝜆 𝑆_{𝜀 \to 𝑈} 𝑢 = 𝜆 𝑒_{1} .

By the inductive hypothesis, $𝐵$ is orthogonally diagonalizable, $𝐵 = 𝑅 𝐷 𝑅^{𝑡}$ for an orthogonal $𝑅$ and diagonal $𝐷$ . Therefore

𝑄^{𝑡} 𝐴 𝑄 = (\begin{matrix} 1 & 0 \\ 0 & 𝑅 \end{matrix}) (\begin{matrix} 𝜆 & 0 \\ 0 & 𝐷 \end{matrix}) {(\begin{matrix} 1 & 0 \\ 0 & 𝑅 \end{matrix})}^{𝑡} .

The middle matrix is diagonal and $(\begin{matrix} 1 & 0 \\ 0 & 𝑅 \end{matrix})$ is orthogonal, since its transpose equals its inverse. Therefore we have constructed an orthogonal diagonalization of $𝐴$ .

14.3 Homework 1 — submitted work

Note

Visual transcription of the personal finished submission. Source: 217-Hw-1-finished.pdf; page references below are PDF pages.

14.3.1 Source p. 1 — Exercise 20

Find the values of $𝑘$ for which the system has infinitely many solutions, no solution, or one solution:

$(\begin{matrix} 1 & 1 & - 1 \\ 1 & 2 & 1 \\ 1 & 1 & 𝑘^{2} - 5 \end{matrix}) 𝑥 = (\begin{matrix} 2 \\ 3 \\ 𝑘 \end{matrix})$ .

The submitted row reduction is $(\begin{matrix} 1 & 1 & - 1 & 2 \\ 1 & 2 & 1 & 3 \\ 1 & 1 & 𝑘^{2} - 5 & 𝑘 \end{matrix}) \to (\begin{matrix} 1 & 1 & - 1 & 2 \\ 0 & 1 & 2 & 1 \\ 0 & 0 & 𝑘^{2} - 4 & 𝑘 - 2 \end{matrix}) \to (\begin{matrix} 1 & 0 & - 3 & 1 \\ 0 & 1 & 2 & 1 \\ 0 & 0 & 𝑘^{2} - 4 & 𝑘 - 2 \end{matrix})$ . For infinitely many solutions, the final row must be zero, so $𝑘^{2} - 4 = 0$ and $𝑘 - 2 = 0$ . Thus $𝑘 = 2$ .

14.3.2 Source p. 2 — Exercise 20; Exercise 32 begins

For no solution, $𝑘^{2} - 4 = 0$ while $𝑘 - 2 \neq 0$ , hence $𝑘 = - 2$ . For one solution, $𝑘 \neq - 2$ and $𝑘 \neq 2$ , namely $(- \infty, - 2) \cup (- 2, 2) \cup (2, \infty)$ .

Exercise 32 asks for a polynomial of degree at most two, $𝑓 (𝑡) = 𝑎 + 𝑏 𝑡 + 𝑐 𝑡^{2}$ , passing through $(1, 𝑝)$ , $(2, 𝑞)$ , and $(3, 𝑟)$ . The submitted equations are $𝑎 + 𝑏 + 𝑐 = 𝑝$ , $𝑎 + 2 𝑏 + 4 𝑐 = 𝑞$ , and $𝑎 + 3 𝑏 + 9 𝑐 = 𝑟$ .

14.3.3 Source p. 3 — Exercise 32

The augmented matrix is reduced as $(\begin{matrix} 1 & 1 & 1 & 𝑝 \\ 1 & 2 & 4 & 𝑞 \\ 1 & 3 & 9 & 𝑟 \end{matrix}) \to (\begin{matrix} 1 & 1 & 1 & 𝑝 \\ 0 & 1 & 3 & 𝑞 - 𝑝 \\ 0 & 2 & 8 & 𝑟 - 𝑝 \end{matrix})$ , giving $𝑎 = 3 𝑝 - 3 𝑞 + 𝑟$ , $𝑏 = - 5 \frac{𝑝}{2} + 4 𝑞 - 3 \frac{𝑟}{2}$ , and $𝑐 = \frac{𝑝}{2} - 𝑞 + \frac{𝑟}{2}$ . Therefore a polynomial exists for all choices of $𝑝$ , $𝑞$ , and $𝑟$ .

14.3.4 Source p. 4 — Exercise 34

Find a polynomial of degree at most two which passes through $(1, 1)$ and $(2, 0)$ and satisfies $\int_{1}^{2} 𝑓 (𝑡) d 𝑡 = - 1$ . The submitted system is $𝑎 + 𝑏 + 𝑐 = 1$ , $𝑎 + 2 𝑏 + 4 𝑐 = 0$ , and $𝑎 + 3 \frac{𝑏}{2} + 7 \frac{𝑐}{3} = - 1$ , with its augmented matrix set up for row reduction.

14.3.5 Source p. 5 — Exercise 34; Exercise 44

The reduction for Exercise 34 ends at $(\begin{matrix} 1 & 0 & 0 & 20 \\ 0 & 1 & 0 & - 28 \\ 0 & 0 & 1 & 9 \end{matrix})$ , so the submitted answer is $𝑓 (𝑡) = 20 - 28 𝑡 + 9 𝑡^{2}$ .

For Exercise 44, the line through $(1, 1, 1)$ and $(3, 5, 0)$ has direction $(\begin{matrix} 2 \\ 4 \\ - 1 \end{matrix})$ . The submitted vector equation is $(\begin{matrix} 𝑥 \\ 𝑦 \\ 𝑧 \end{matrix}) = (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}) + 𝑡 (\begin{matrix} 2 \\ 4 \\ - 1 \end{matrix})$ , or $𝑥 = 1 + 2 𝑡$ , $𝑦 = 1 + 4 𝑡$ , $𝑧 = 1 - 𝑡$ , for arbitrary $𝑡$ . Its equations are $2 𝑥 - 𝑦 = - 3$ and $𝑥 + 2 𝑧 = 3$ .

14.3.6 Source p. 6 — Exercise 12

The homogeneous augmented matrix recorded is $(\begin{matrix} 2 & 0 & - 3 & 0 & 7 & 7 & 0 \\ - 2 & 1 & 6 & 0 & - 6 & - 12 & 0 \\ 0 & 1 & - 3 & 0 & 1 & 5 & 0 \\ 0 & - 2 & 0 & 1 & 1 & 1 & 0 \\ 2 & 1 & - 3 & 0 & 8 & 7 & 0 \end{matrix})$ . The first displayed operation divides the first row by $2$ before continuing the row reduction.

14.3.7 Source p. 7 — Exercise 12

The submitted reduction clears the first column, then the second column. Its displayed intermediate rows include $(\begin{matrix} 1 & 0 & - \frac{3}{2} & 0 & \frac{7}{2} & \frac{7}{2} & 0 \\ 0 & 1 & - 3 & 0 & 1 & 5 & 0 \\ 0 & - 2 & 0 & 1 & 1 & 1 & 0 \\ 0 & 1 & 0 & 0 & 1 & 0 & 0 \end{matrix})$ , followed by clearing the remaining pivot columns.

14.3.8 Source p. 8 — Exercise 12

The final reduced matrix is $(\begin{matrix} 1 & 0 & 0 & 0 & \frac{7}{2} & 1 & 0 \\ 0 & 1 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & - \frac{5}{3} & 0 \\ 0 & 0 & 0 & 1 & 3 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 \end{matrix})$ . Setting $𝑥_{5} = 𝑡$ and $𝑥_{6} = 𝑟$ , the submission gives $(\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \\ 𝑥_{4} \\ 𝑥_{5} \\ 𝑥_{6} \end{matrix}) = (\begin{matrix} 7 \frac{𝑡}{2} + 𝑟 \\ - 𝑡 \\ 5 \frac{𝑟}{3} \\ - 3 𝑡 - 𝑟 \\ 𝑡 \\ 𝑟 \end{matrix}) = 𝑡 (\begin{matrix} \frac{7}{2} \\ - 1 \\ 0 \\ - 3 \\ 1 \\ 0 \end{matrix}) + 𝑟 (\begin{matrix} 1 \\ 0 \\ \frac{5}{3} \\ - 1 \\ 0 \\ 1 \end{matrix})$ .

14.3.9 Source p. 9 — Exercise 36

For a vector $𝑝$ perpendicular to $(\begin{matrix} 1 \\ 3 \\ - 1 \end{matrix})$ , let $𝑝 = (\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \end{matrix})$ . The condition is $𝑎 + 3 𝑏 - 𝑐 = 0$ . Setting $𝑏 = 𝑡$ and $𝑐 = 𝑟$ gives $𝑝 = (\begin{matrix} - 3 𝑡 + 𝑟 \\ 𝑡 \\ 𝑟 \end{matrix})$ , where $𝑡$ and $𝑟$ are arbitrary real numbers.

14.3.10 Source p. 10 — Exercise 44

For the traffic-flow diagram, the submitted equations are $300 + 𝑥 = 400 + 𝑦$ , $300 + 𝑤 = 320 + 𝑥$ , $150 + 120 = 𝑤 + 𝑧$ , and $𝑦 + 𝑧 + 100 = 250$ . They are rearranged as $𝑥 - 𝑦 = 100$ , $𝑤 - 𝑥 = 20$ , $𝑤 + 𝑧 = 270$ , and $𝑦 + 𝑧 = 150$ .

14.3.11 Source p. 11 — Exercise 44

The submitted row reduction reaches $(\begin{matrix} 1 & 0 & 0 & 1 & 270 \\ 0 & 1 & 0 & 1 & 250 \\ 0 & 0 & 1 & 1 & 150 \\ 0 & 0 & 0 & 0 & 0 \end{matrix})$ . The free variable is recorded as $𝑧 = 𝑡$ .

14.3.12 Source p. 12 — Exercise 44

The submitted solution is $(\begin{matrix} 𝑤 \\ 𝑥 \\ 𝑦 \\ 𝑧 \end{matrix}) = (\begin{matrix} 270 - 𝑡 \\ 250 - 𝑡 \\ 150 - 𝑡 \\ 𝑡 \end{matrix})$ . Nonnegativity gives $0 \leq 𝑡 \leq 150$ . The recorded flow ranges are $𝑤 \in [120, 270]$ , $𝑥 \in [100, 250]$ , $𝑦 \in [0, 150]$ , and $𝑧 \in [0, 150]$ .

14.3.13 Source p. 13 — Problem 1(a)–(c)

(a) True: “2 is even” and “3 is odd” are true, hence their disjunction is true.

(b) True: $217 = 7 \times 31$ , so the conclusion is true; an if–then statement is true regardless of the truth value of its hypothesis.

(c) False: the derivative claim is true, but the submitted work states that $tan (\frac{𝜋}{6}) = \sqrt{3}$ is false; thus the biconditional is false.

14.3.14 Source p. 14 — Problem 1(d)–(e); Problem 2(a)

(d) True: the premise “there are infinitely many even primes” is false, since the only even prime is $2$ , so the implication is true.

(e) True: each right triangle has two acute angles and every positive real number has a positive cube root; the implication is true.

For Problem 2(a), the table in the submission says that a proof proves nothing for a universal statement and that an existential statement is true; a counterexample proves a universal statement false and proves nothing for an existential statement.

14.3.15 Source p. 15 — Problem 2(b)–(d)

(b) True: every prime integer is either even or odd.

(c) False: $2$ is a prime which is even, so not every prime is odd; $3$ is a prime which is odd, so not every prime is even. Hence the asserted “or” is false.

(d) False: the submitted contradiction uses $𝑥 = 𝑛 - 1$ to refute the stated universal claim.

14.3.16 Source p. 16 — Problem 2(e)–(g)

(e) True: the submitted argument chooses the integer immediately above $𝑥$ , so there is an integer $𝑛 > 𝑥$ . [TODO(217-Hw-1-finished.pdf, p. 16): one handwritten bracket symbol in the displayed floor/ceiling argument is not visually unambiguous.]

(f) True: every square is a rectangle.

(g) False: $4$ has two square roots, $2$ and $- 2$ , which disproves the uniqueness statement.

14.3.17 Source p. 17 — Problem 3

The submitted negations are:

(a) 2 is not even and 3 is not odd.
(b) the Riemann hypothesis is true and 217 is prime.
(c) the derivative is not $2 𝑥$ or $tan (\frac{𝜋}{6}) \neq \sqrt{3}$ .
(d) there are infinitely many even primes, and 10 is not even or $10^{10}$ is not odd.
(e) every right triangle in $ℝ^{2}$ has two acute angles, and some real number has no positive cube root.
(f) $\forall 𝑛 \in ℕ$ there exists $𝑥 \in ℝ$ with $𝑥 \geq 𝑛$ .
(g) all squares are not rectangles.

14.3.18 Source p. 18 — Problem 4

The submitted converses and contrapositives are:

(a) Converse: “If something exists, it can think.” Contrapositive: “If something does not exist, it cannot think.”
(b) Converse: “If $𝑝^{2}$ is irrational, then $𝑝$ is irrational.” Contrapositive: “If $𝑝^{2}$ is not rational, then $𝑝$ is not rational.”
(c) Converse: “If $𝑛^{2} + 1$ is prime, then $𝑛$ is a natural number greater than 2 such that its Collatz sequence does not reach 1.” The contraposition is also written in terms of $𝑛^{2} + 1$ not being prime and the alternatives concerning $𝑛$ and its Collatz sequence.

14.3.19 Source p. 19 — Problem 5

(a1) The set is all odd natural numbers. (a2) The graph is the right half of the unit circle centred at the origin, including the boundary.

(b1) The submitted set notation is {(x,y,z) in RR^3 : x^2+y^2+z^2=1}. (b2) The submitted set notation is {sqrt(2)n : n in ZZ}.

(c) The submitted truth values are: $\sqrt{2} \in ℝ$ true; $\sqrt{2} \subset ℝ$ false; {sqrt(2)} in RR false; {sqrt(2)} subset RR true; $\emptyset \in ℝ$ false; $\emptyset \subset ℝ$ true; $\emptyset \in \emptyset$ false; and $\emptyset \subset \emptyset$ true.

14.3.20 Source p. 20 — Problem 6(a)

The submitted lists are 1/2 NN = {1/2,1,3/2,2,5/2,3,...}, 1/3 NN = {1/3,2/3,1,4/3,5/3,2,...}, and 3NN = {3,6,9,12,15,18,...}. It records 1/2 NN intersect 1/3 NN = NN, and writes the union beginning {1/3,1/2,2/3,1,4/3,3/2,...}. It also gives 1/2 NN without 1/3 NN = {1/2,3/2,5/2,7/2,9/2,11/2,...} and (3NN)^c = NN without 3NN = {1,2,4,5,7,8,10,...}.

14.3.21 Source p. 21 — Problem 6(b)

The claimed least $𝑛$ is $6$ . For $𝑥 \in \frac{1}{2} ℕ \cup \frac{1}{3} ℕ$ , the submission writes $𝑥 = \frac{3 𝑚 + 2 𝑝}{6}$ (with one of $𝑚, 𝑝$ allowed to be zero), proving inclusion in $\frac{1}{6} ℕ$ . It then rules out $𝑛 = 1, 3, 5$ because $\frac{1}{2}$ is not in $\frac{1}{𝑛} ℕ$ , and $𝑛 = 2, 4$ because $\frac{1}{3}$ is not in $\frac{1}{𝑛} ℕ$ .

14.3.22 Source p. 22 — Problem 7

Let $𝐹 (𝑥, 𝑡)$ mean “you can fool $𝑥$ at time $𝑡$ .” The submitted formalization of the recreational statement is $(\exists 𝑡 \forall 𝑥 𝐹 (𝑥, 𝑡)) \land (\exists 𝑥 \forall 𝑡 𝐹 (𝑥, 𝑡)) \land \neg (\forall 𝑡 \forall 𝑥 𝐹 (𝑥, 𝑡))$ .

14.3.23 Source p. 23 — Problem 7

Applying De Morgan’s law, the submitted negation is $(\forall 𝑡 \exists 𝑥 \neg 𝐹 (𝑥, 𝑡)) \lor (\forall 𝑥 \exists 𝑡 \neg 𝐹 (𝑥, 𝑡)) \lor (\forall 𝑡 \forall 𝑥 𝐹 (𝑥, 𝑡))$ .

The accompanying English reads: “for all time there are some people you cannot fool, or for some time you can fool no people in the world, or for all time you can fool all people. (You can at least achieve one of three things).”

14.4 Homework 2 — submitted work

Note

Visual transcription of the personal finished submission. Source: 217-Hw-2-finished.pdf; page references below are PDF pages.

14.4.1 Source p. 1 — Exercise 26

Let $𝐴$ be a $4 \times 3$ matrix and assume $𝐴 𝑥 = 𝑏$ has a unique solution. The submission records $rank (𝐴) = rank (𝐴 | 𝑏) = 3$ . For $𝐴 𝑥 = 𝑐$ , it begins the two RREF cases.

14.4.2 Source p. 2 — Exercise 26; Exercise 34 begins

If $[𝐴 : 𝑐]$ reduces to a form with three pivots, it has one solution. If it has a row $(\begin{matrix} 0 & 0 & 0 & 𝜀 \end{matrix})$ with $𝜀 \neq 0$ , it has no solution. By Theorem 1.3.1, infinitely many solutions cannot occur because there is no free variable.

Exercise 34 introduces the standard coordinate vectors $𝑒_{1}, 𝑒_{2}, 𝑒_{3}$ .

14.4.3 Source p. 3 — Exercise 34

For $𝐴 = (\begin{matrix} 𝑎 & 𝑏 & 𝑐 \\ 𝑑 & 𝑒 & 𝑓 \\ 𝑔 & ℎ & 𝑘 \end{matrix})$ , the submitted work writes

$𝐴 𝑒_{1} = (\begin{matrix} 𝑎 \\ 𝑑 \\ 𝑔 \end{matrix}) = 𝑣_{1}$ , $𝐴 𝑒_{2} = (\begin{matrix} 𝑏 \\ 𝑒 \\ ℎ \end{matrix}) = 𝑣_{2}$ , and $𝐴 𝑒_{3} = (\begin{matrix} 𝑐 \\ 𝑓 \\ 𝑘 \end{matrix}) = 𝑣_{3}$ .

For a matrix $𝐵$ with columns $𝑣_{1}, 𝑣_{2}, 𝑣_{3}$ , it likewise records $𝐵 𝑒_{𝑖} = 𝑣_{𝑖}$ .

14.4.4 Source p. 4 — Exercise 48(a)–(b)

(a) If $𝑥_{ℎ}$ solves $𝐴 𝑥 = 0$ and $𝑥_{1}$ solves $𝐴 𝑥 = 𝑏$ , then $𝐴 (𝑥_{1} + 𝑥_{ℎ}) = 𝐴 𝑥_{1} + 𝐴 𝑥_{ℎ} = 𝑏 + 0 = 𝑏$ .

(b) If $𝑥_{1}$ and $𝑥_{2}$ solve $𝐴 𝑥 = 𝑏$ , then $𝐴 (𝑥_{2} - 𝑥_{1}) = 𝐴 𝑥_{2} - 𝐴 𝑥_{1} = 𝑏 - 𝑏 = 0$ ; hence $𝑥_{2} - 𝑥_{1}$ is a homogeneous solution.

14.4.5 Source p. 5 — Exercise 48(c)

The drawing places the homogeneous solutions on a line through the origin and the particular solution $𝑥_{1}$ off that line. The submitted answer is the parallel affine line $𝑥_{1} + 𝑥_{ℎ}$ .

14.4.6 Source p. 6 — Exercise 48(c)

The geometric explanation concludes: all solutions form the line parallel to the homogeneous-solution line through the head of $𝑥_{1}$ , with its tail at $0$ .

14.4.7 Source p. 7 — Exercise 6

For $𝑇 : ℝ^{2} \to ℝ^{3}$ ,

$𝑇 (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \end{matrix}) = 𝑥_{1} (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) + 𝑥_{2} (\begin{matrix} 4 \\ 5 \\ 6 \end{matrix})$ .

The submitted standard matrix is $(\begin{matrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{matrix})$ , and the work verifies linearity by the definition cited on Worksheet 4.

14.4.8 Source p. 8 — Exercise 38; Exercise 44 begins

For Exercise 38,

$𝑇 (\begin{matrix} 2 \\ - 1 \end{matrix}) = 2 𝑣_{1} - 𝑣_{2}$ .

The submitted sketch shows this vector combination. Exercise 44 begins with $𝑇 (𝑥)$ equal to the cross product of $𝑣$ and $𝑥$ .

14.4.9 Source p. 9 — Exercise 44

With $𝑣 = (\begin{matrix} 𝑣_{1} \\ 𝑣_{2} \\ 𝑣_{3} \end{matrix})$ and $𝑥 = (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix})$ , the work expands

$𝑣 \times 𝑥 = (\begin{matrix} 𝑣_{2} 𝑥_{3} - 𝑣_{3} 𝑥_{2} \\ 𝑣_{3} 𝑥_{1} - 𝑣_{1} 𝑥_{3} \\ 𝑣_{1} 𝑥_{2} - 𝑣_{2} 𝑥_{1} \end{matrix}) = (\begin{matrix} 0 & - 𝑣_{3} & 𝑣_{2} \\ 𝑣_{3} & 0 & - 𝑣_{1} \\ - 𝑣_{2} & 𝑣_{1} & 0 \end{matrix}) 𝑥$ .

Thus the submitted matrix for $𝑇$ is $(\begin{matrix} 0 & - 𝑣_{3} & 𝑣_{2} \\ 𝑣_{3} & 0 & - 𝑣_{1} \\ - 𝑣_{2} & 𝑣_{1} & 0 \end{matrix})$ ; the work also verifies addition and scalar multiplication.

14.4.10 Source p. 10 — Exercise 46

For $𝐴 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ and $𝐵 = (\begin{matrix} 𝑝 & 𝑞 \\ 𝑟 & 𝑠 \end{matrix})$ , $𝑇 (𝑥) = 𝐵 (𝐴 𝑥)$ . The submitted column computation gives

$𝑇 (𝑒_{1}) = (\begin{matrix} 𝑎 𝑝 + 𝑐 𝑞 \\ 𝑎 𝑟 + 𝑐 𝑠 \end{matrix})$ and $𝑇 (𝑒_{2}) = (\begin{matrix} 𝑏 𝑝 + 𝑑 𝑞 \\ 𝑏 𝑟 + 𝑑 𝑠 \end{matrix})$ .

Hence the matrix is $(\begin{matrix} 𝑎 𝑝 + 𝑐 𝑞 & 𝑏 𝑝 + 𝑑 𝑞 \\ 𝑎 𝑟 + 𝑐 𝑠 & 𝑏 𝑟 + 𝑑 𝑠 \end{matrix})$ .

14.4.11 Source p. 11 — Part B, Problem 1(a)–(b)

(a) $𝑓 : [0, 4] \to [0, 18]$ , $𝑓 (𝑥) = 𝑥^{2} + 2$ , is injective but not surjective. A counterexample to surjectivity is $𝑦 = 0$ . For injectivity, $𝑎^{2} + 2 = 𝑏^{2} + 2$ implies $𝑎 = + - \lor - - 𝑏$ , and nonnegativity gives $𝑎 = 𝑏$ .

(b) $𝑔 : ℝ \to ℝ$ , $𝑔 (𝑥) = 2 𝑥 - 5$ , is bijective. Surjectivity uses $𝑥 = \frac{𝑦 + 5}{2}$ , and $2 𝑥_{1} - 5 = 2 𝑥_{2} - 5$ gives injectivity.

14.4.12 Source p. 12 — Part B, Problem 1(c)–(d)

(c) $ℎ : ℝ^{2} \to ℝ$ , $ℎ (𝑥, 𝑦) = 2 𝑥^{2} + 5 𝑦^{2}$ , is neither injective nor surjective: $ℎ (1, 2) = ℎ (- 1, - 2) = 22$ , and no negative number is in its image.

(d) $𝑞 : ℕ \to ℕ$ sends odd $𝑛$ to $𝑛$ and even $𝑛$ to $\frac{𝑛}{2}$ . It is surjective but not injective: $𝑞 (1) = 𝑞 (2) = 1$ ; for a target $𝑦$ , choose $𝑥 = 𝑦$ when $𝑦$ is odd and $𝑥 = 2 𝑦$ when $𝑦$ is even.

14.4.13 Source p. 13 — Part B, Problem 2(a)–(b)

The submitted truth values are (a) false, (b) true, (c) false, (d) false, (e) true.

For (a), take $𝑓 : [- 2, 2] \to [0, 4]$ , $𝑓 (𝑥) = 𝑥^{2}$ , $𝐴 = [- 2, - 1]$ , and $𝐵 = [1, 2]$ . Then A intersect B = emptyset but $4$ belongs to both images. The proof of (b) begins by contradiction.

14.4.14 Source p. 14 — Part B, Problem 2(b)–(c)

For (b), if $𝐴$ and $𝐵$ had a common $𝑎$ , then $𝑓 (𝑎)$ would lie in both $𝑓 [𝐴]$ and $𝑓 [𝐵]$ , contradicting their disjointness.

For (c), the same squaring map with $𝐴 = [0, 2]$ gives $𝑓^{-} 1 [𝑓 [𝐴]] = [- 2, 2] \neq 𝐴$ .

14.4.15 Source p. 15 — Part B, Problem 2(d)–(e)

For (d), using the same $𝑓$ and $𝐴 = [0, 2]$ , the submission gets $𝑓 [𝑋 ∖ 𝐴] = 𝑓 [- 2, 0] = [0, 4]$ , whereas $𝑌 ∖ 𝑓 [𝐴] = \emptyset$ .

For (e), it begins the two inclusions for a bijection $𝑓$ .

14.4.16 Source p. 16 — Part B, Problem 2(e)

If m in f[A intersect B], then $𝑚 = 𝑓 (𝑥)$ for some x in A intersect B, so m in f[A] intersect f[B]. Conversely, $𝑚 = 𝑓 (𝑥) = 𝑓 (𝑦)$ with $𝑥 \in 𝐴$ and $𝑦 \in 𝐵$ ; injectivity gives $𝑥 = 𝑦$ , thus m in f[A intersect B].

14.4.17 Source p. 17 — Part B, Problem 3(a)

Assume $𝑓 (𝑐 𝑥) = 𝑐 𝑓 (𝑥)$ . The submission first records $𝑓 (0) = 0$ and $𝑓 (- 𝑥) = - 𝑓 (𝑥)$ . If $𝑥 + 𝑦 = 0$ , additivity follows. Otherwise, writing $𝑐 = 𝑥 + 𝑦$ and applying the assumption to $\frac{𝑥}{𝑥 + 𝑦}$ and $\frac{𝑦}{𝑥 + 𝑦}$ gives

$(𝑥 + 𝑦) 𝑓 (𝑥 + 𝑦) = (𝑥 + 𝑦) (𝑓 (𝑥) + 𝑓 (𝑦))$ ,

then division yields $𝑓 (𝑥 + 𝑦) = 𝑓 (𝑥) + 𝑓 (𝑦)$ .

14.4.18 Source p. 18 — Part B, Problem 3(b)

The submitted example is $𝑓 (𝑥) = ‖ 𝑥 ‖$ . It records $𝑓 (𝑐 𝑥) = 𝑐 𝑓 (𝑥)$ , but says it is not additive: with $𝑥 = (\begin{matrix} 1 \\ 2 \end{matrix})$ and $𝑦 = (\begin{matrix} 2 \\ 2 \end{matrix})$ , the displayed norm values give $𝑓 (𝑥 + 𝑦) \neq 𝑓 (𝑥) + 𝑓 (𝑦)$ .

14.4.19 Source p. 19 — Part B, Problem 4(a)–(b)

For an additive $𝑓$ , $𝑓 (0) = 𝑓 (0) + 𝑓 (0)$ , hence $𝑓 (0) = 0$ ; also $𝑓 (- 𝑥) = - 𝑓 (𝑥)$ .

14.4.20 Source p. 20 — Part B, Problem 4(c)–(d) begins

The induction for $𝑓 (𝑛 𝑥) = 𝑛 𝑓 (𝑥)$ has base case $𝑛 = 1$ and the inductive step

$𝑓 ((𝑛 + 1) 𝑥) = 𝑓 (𝑛 𝑥 + 𝑥) = 𝑓 (𝑛 𝑥) + 𝑓 (𝑥) = (𝑛 + 1) 𝑓 (𝑥)$ .

Part (d), for negative integers, begins on this page.

14.4.21 Source p. 21 — Part B, Problem 4(d); recreational part (e)

The work completes the negative-integer case using $𝑓 (- 𝑥) = - 𝑓 (𝑥)$ . For the recreational rational case it concludes that $𝑓 (𝑞 𝑥) = 𝑞 𝑓 (𝑥)$ for rational $𝑞$ , using numerator/denominator multiplication and the preceding integer cases.

14.5 Homework 3 — submitted work

Note

Visual transcription of the personal finished submission. Source: 217-Hw-3-finished.pdf; page references below are PDF pages.

14.5.1 Source p. 1 — Exercise 20

Reflection about the $𝑥 - 𝑧$ plane sends $(\begin{matrix} 𝑣_{1} \\ 𝑣_{2} \\ 𝑣_{3} \end{matrix})$ to $(\begin{matrix} 𝑣_{1} \\ - 𝑣_{2} \\ 𝑣_{3} \end{matrix})$ . The submitted standard matrix is $(\begin{matrix} 1 & 0 & 0 \\ 0 & - 1 & 0 \\ 0 & 0 & 1 \end{matrix})$ , accompanied by a sketch of the reflected vector.

14.5.2 Source p. 2 — Exercise 38(a)–(c)

(a) Projection onto an arbitrary unit vector $𝑢 = (\begin{matrix} 𝑢_{1} \\ 𝑢_{2} \end{matrix})$ has matrix $(\begin{matrix} 𝑢_{1}^{2} & 𝑢_{1} 𝑢_{2} \\ 𝑢_{1} 𝑢_{2} & 𝑢_{2}^{2} \end{matrix})$ . Its determinant is $0$ , so it is not invertible.

(b) A reflection matrix is $(\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & - 𝑎 \end{matrix})$ with $𝑎^{2} + 𝑏^{2} = 1$ ; its determinant is $- 1$ , so it is invertible.

(c) A rotation matrix is $(\begin{matrix} 𝑎 & - 𝑏 \\ 𝑏 & 𝑎 \end{matrix})$ with $𝑎^{2} + 𝑏^{2} = 1$ ; its determinant is $𝑎^{2} + 𝑏^{2} = 1$ , so it is invertible.

14.5.3 Source p. 3 — Exercise 38(d); Exercise 18 begins

(d) The horizontal and vertical shear matrices are $(\begin{matrix} 1 & 𝑘 \\ 0 & 1 \end{matrix})$ and $(\begin{matrix} 1 & 0 \\ 𝑘 & 1 \end{matrix})$ . Each has determinant $1$ , so each is invertible.

For Exercise 18, let $𝐴 = (\begin{matrix} 2 & 3 \\ - 3 & 2 \end{matrix})$ and $𝐵 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ . Equating $𝐴 𝐵$ and $𝐵 𝐴$ yields $𝑏 = - 𝑐$ and $𝑎 = 𝑑$ , so $𝐵 = (\begin{matrix} 𝑎 & 𝑏 \\ - 𝑏 & 𝑎 \end{matrix})$ .

14.5.4 Source p. 4 — Exercise 34

For $𝐴 = (\begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix})$ , the submitted computations are

$𝐴^{2} = (\begin{matrix} 1 & 2 \\ 0 & 1 \end{matrix})$ , $𝐴^{3} = (\begin{matrix} 1 & 3 \\ 0 & 1 \end{matrix})$ , $𝐴^{4} = (\begin{matrix} 1 & 4 \\ 0 & 1 \end{matrix})$ , and $𝐴^{1001} = (\begin{matrix} 1 & 1001 \\ 0 & 1 \end{matrix})$ .

Thus $𝐴^{𝑘} = (\begin{matrix} 1 & 𝑘 \\ 0 & 1 \end{matrix})$ for positive $𝑘$ . Geometrically, repeated application is a horizontal shear by one unit each time.

14.5.5 Source p. 5 — Exercise 12

For $𝐴 = (\begin{matrix} 2 & 5 & 0 & 0 \\ 1 & 3 & 0 & 0 \\ 0 & 0 & 1 & 2 \\ 0 & 0 & 2 & 5 \end{matrix})$ , the submission applies the invertible-matrix test through row reduction of $[𝐴 : 𝐼_{4}]$ . The displayed operations first exchange the first two rows and then use pivots to clear the two $2 \times 2$ blocks.

14.5.6 Source p. 6 — Exercise 12; Exercise 34 begins

The final augmented matrix is $[𝐼_{4} : 𝐴^{-} 1]$ , with

$𝐴^{-} 1 = (\begin{matrix} 3 & - 5 & 0 & 0 \\ - 1 & 2 & 0 & 0 \\ 0 & 0 & 5 & - 2 \\ 0 & 0 & - 2 & 1 \end{matrix})$ .

By Theorem 2.4.5, $𝐴$ is invertible.

For a diagonal $𝐴 = (\begin{matrix} 𝑎 & 0 & 0 \\ 0 & 𝑏 & 0 \\ 0 & 0 & 𝑐 \end{matrix})$ , the submitted answer says $𝐴$ is invertible precisely when $𝑎, 𝑏, 𝑐 \neq 0$ , and then $𝐴^{-} 1 = (\begin{matrix} \frac{1}{𝑎} & 0 & 0 \\ 0 & \frac{1}{𝑏} & 0 \\ 0 & 0 & \frac{1}{𝑐} \end{matrix})$ . A diagonal matrix of arbitrary size is invertible iff every diagonal element is nonzero.

14.5.7 Source p. 7 — Part B, Problem 1(a)–(b)

The source defines trace, determinant, transpose, and symmetric matrix, then asks truth values for transpose claims. The submission marks (a) false and takes $𝐴 = (\begin{matrix} 1 & 2 \\ 2 & 1 \end{matrix})$ and $𝐵 = (\begin{matrix} 1 & 2 \\ 1 & 2 \end{matrix})$ . It computes $𝐴 𝐵 = (\begin{matrix} 3 & 6 \\ 3 & 6 \end{matrix})$ , $𝐵^{𝑇} = (\begin{matrix} 1 & 1 \\ 2 & 2 \end{matrix})$ , $𝐴^{𝑇} = (\begin{matrix} 1 & 2 \\ 2 & 1 \end{matrix})$ , and $𝐴^{𝑇} 𝐵^{𝑇} = (\begin{matrix} 5 & 5 \\ 4 & 4 \end{matrix})$ , so ${(𝐴 𝐵)}^{𝑇} \neq 𝐴^{𝑇} 𝐵^{𝑇}$ .

For (b), it marks false using the same matrices and records $𝐴 𝐵 = 𝐴^{𝑇} 𝐵^{𝑇} = (\begin{matrix} 5 & 4 \\ 4 & 5 \end{matrix})$ as a counterexample to the universal inequality.

14.5.8 Source p. 8 — Part B, Problem 1(c)

The statement ${(𝐴 𝐵)}^{𝑇} = 𝐵^{𝑇} 𝐴^{𝑇}$ is marked true. Let $𝐴$ be $𝑛 \times 𝑝$ and $𝐵$ be $𝑝 \times 𝑚$ . The work writes both matrices by entries and notes that $(𝐵^{𝑇} 𝐴^{𝑇})$ has the same $𝑚 \times 𝑛$ shape as ${(𝐴 𝐵)}^{𝑇}$ .

14.5.9 Source p. 9 — Part B, Problem 1(c)–(d)

The $𝑗, 𝑖$ entry of $𝐵^{𝑇} 𝐴^{𝑇}$ is computed as

$\sum_{𝑘 = 1}^{𝑝} 𝑏_{𝑘 𝑗} 𝑎_{𝑖 𝑘} = \sum_{𝑘 = 1}^{𝑝} 𝑎_{𝑖 𝑘} 𝑏_{𝑘 𝑗}$ ,

the $𝑗, 𝑖$ entry of ${(𝐴 𝐵)}^{𝑇}$ . Since $𝑖, 𝑗$ are arbitrary, ${(𝐴 𝐵)}^{𝑇} = 𝐵^{𝑇} 𝐴^{𝑇}$ .

For (d), the submission begins induction: for a symmetric $𝐴 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & 𝑎 \end{matrix})$ , $𝐴^{1} = 𝐴$ is symmetric, and if $𝐴^{𝑛} = (\begin{matrix} 𝑐 & 𝑑 \\ 𝑑 & 𝑐 \end{matrix})$ , then $𝐴^{𝑛 + 1} = 𝐴^{𝑛} 𝐴$ .

14.5.10 Source p. 10 — Part B, Problem 1(d)–(e)

The inductive multiplication is

$(\begin{matrix} 𝑐 & 𝑑 \\ 𝑑 & 𝑐 \end{matrix}) (\begin{matrix} 𝑎 & 𝑏 \\ 𝑏 & 𝑎 \end{matrix}) = (\begin{matrix} 𝑎 𝑐 + 𝑏 𝑑 & 𝑎 𝑑 + 𝑏 𝑐 \\ 𝑎 𝑑 + 𝑏 𝑐 & 𝑎 𝑐 + 𝑏 𝑑 \end{matrix})$ ,

which is symmetric. Hence $𝐴^{𝑛}$ is symmetric for all $𝑛 \in ℕ$ .

(e) is false: $𝐴 = (\begin{matrix} 1 & 3 \\ 2 & 7 \end{matrix})$ is not symmetric, while $𝐴^{2} = (\begin{matrix} 7 & 24 \\ 16 & 55 \end{matrix})$ is recorded as symmetric in the submitted counterexample.

14.5.11 Source p. 11 — Part B, Problem 2(a)–(c)

(a) True. A $3 \times 3$ matrix with a zero row has at most two leading 1s, so $rank (𝐴) \leq 2$ ; by Theorem 2.4.3 it is not invertible.

(b) False. The counterexample is $(\begin{matrix} 1 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{matrix})$ , whose RREF has rank $2 \neq 3$ , so it is not invertible.

(c) True. If $𝐴$ is an invertible $𝑛 \times 𝑛$ matrix, then $𝐴^{-} 1 𝐴 = 𝔸^{-} 1 = 𝐼_{𝑛}$ ; therefore $𝐴^{-} 1$ is invertible.

14.5.12 Source p. 12 — Part B, Problem 2(d)

(d) True. If $𝐴$ is invertible, then ${(𝐴^{-} 1)}^{𝑛}$ is an inverse for $𝐴^{𝑛}$ . By associativity, the submission writes the products until adjacent $𝐴^{-} 1 𝐴$ pairs become $𝐼_{𝑛}$ , so both products equal $𝐼_{𝑛}$ .

14.5.13 Source p. 13 — Part B, Problem 3(a)

If there is an $𝑛 \times 𝑚$ matrix $𝐵$ with $𝐵 𝐴 = 𝐼_{𝑛}$ , suppose $𝑥_{1}$ and $𝑥_{2}$ solve $𝐴 𝑥 = 0$ . Then $𝐴 (𝑥_{1} - 𝑥_{2}) = 0$ . Multiplying on the left by $𝐵$ gives $𝐵 𝐴 (𝑥_{1} - 𝑥_{2}) = 𝐵 0 = 0$ ; hence $𝑥_{1} - 𝑥_{2} = 0$ and $𝑥_{1} = 𝑥_{2}$ . Therefore $𝐴 𝑥 = 0$ has a unique solution.

14.5.14 Source p. 14 — Part B, Problem 4(a)

Statement (a) is marked true. Write $𝐴 = [𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑚}]$ and $𝐵 = [𝑤_{1}, 𝑤_{2}, \dots, 𝑤_{𝑘}]$ . By Theorem 2.3.2,

$𝐴 𝐵 = 𝐴 [𝑤_{1}, 𝑤_{2}, \dots, 𝑤_{𝑘}] = [𝐴 𝑤_{1}, 𝐴 𝑤_{2}, \dots, 𝐴 𝑤_{𝑘}]$ .

If $𝑤_{𝑖} = (\begin{matrix} 𝑤_{1 𝑖} \\ \dots \\ 𝑤_{𝑚 𝑖} \end{matrix})$ , then by Theorem 1.3.8,

$𝐴 𝑤_{𝑖} = 𝑤_{1 𝑖} 𝑣_{1} + 𝑤_{2 𝑖} 𝑣_{2} + \dots + 𝑤_{𝑚 𝑖} 𝑣_{𝑚}$ .

Thus every column of $𝐴 𝐵$ is a linear combination of the columns of $𝐴$ .

14.5.15 Source p. 15 — Part B, Problem 4(b)

Statement (b) is false. The counterexample uses $𝐴 = (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix})$ and $𝐵 = (\begin{matrix} 1 & 1 \\ 1 & 1 \end{matrix})$ , so $𝐴 𝐵 = (\begin{matrix} 1 & 1 \\ 0 & 0 \end{matrix})$ . Its first column $(\begin{matrix} 1 \\ 0 \end{matrix})$ is not a linear combination of the columns $(\begin{matrix} 1 \\ 1 \end{matrix})$ and $(\begin{matrix} 1 \\ 1 \end{matrix})$ of $𝐵$ : the submitted RREF shows that the corresponding system is inconsistent.

14.5.16 Source p. 16 — Part B, Problem 5(a)

For a linear $𝑓 : ℝ^{𝑑} \to ℝ^{𝑑}$ , the proof by induction has base $𝑛 = 1$ , where $𝑓^{1} (𝑥) = 𝑓 (𝑓^{0} (𝑥)) = 𝑓 (𝑥)$ is linear. Assuming $𝑓^{𝑛}$ is linear, write $𝑓 (𝑥) = 𝐴 𝑥$ . The key theorem gives a matrix $𝐵$ with $𝑓^{𝑛 (𝑥)} = 𝐵 𝑥$ ; hence

$𝑓^{𝑛 + 1} (𝑥) = 𝑓 (𝑓^{𝑛 (𝑥)}) = 𝑓 (𝐵 𝑥) = 𝐴 (𝐵 𝑥) = (𝐴 𝐵) 𝑥$ ,

so it is linear. Thus $𝑓^{𝑛}$ is linear for all $𝑛 \in ℕ$ .

14.5.17 Source p. 17 — Part B, Problem 5(a)

The submission restates the inductive conclusion: the base case and inductive step prove that if $𝑓$ is a linear transformation, then $𝑓^{𝑛}$ is a linear transformation for all $𝑛 \in ℕ$ .

14.5.18 Source p. 18 — Part B, Problem 5(b)

Define $𝑓 : ℝ^{2} \to ℝ^{2}$ by $𝑓 (𝑥) = - 𝑥$ if $‖ 𝑥 ‖ = 2$ , and $𝑓 (𝑥) = 𝑥$ otherwise. The submission says $𝑓$ is not linear: choose $𝑥_{1}, 𝑥_{2}$ with $‖ 𝑥_{1} ‖ = 2$ and $‖ 𝑥_{2} ‖ \neq 2$ , then $𝑓 (𝑥_{1} + 𝑥_{2}) = 𝑥_{1} + 𝑥_{2}$ while $𝑓 (𝑥_{1}) + 𝑓 (𝑥_{2}) = - 𝑥_{1} + 𝑥_{2}$ . But $𝑓^{2} (𝑥) = 𝑥$ for all $𝑥$ , so $𝑓^{2}$ is the identity and is linear.

14.5.19 Source p. 19 — Part B, Problem 5(c)

Let $𝑓 (𝑥) = 𝐴 𝑥$ . If $𝑓 (𝑥) = 0$ has a unique solution, then $𝐴 𝑥 = 0$ has a unique solution; by Theorem 1.3.4, $rank (𝐴) = 𝑑$ , hence by Theorem 2.4.3 $𝐴$ is invertible. Problem 3 has shown that $𝐴^{𝑛}$ is invertible for every $𝑛 \in ℕ$ . Since $𝑓^{𝑛 (𝑥)} = 𝐴^{𝑛} 𝑥$ , Theorem 1.3.4 gives a unique solution to $𝐴^{𝑛} 𝑥 = 0$ , proving the claim.

14.6 Homework 4 - submitted work

14.6.1 Exercise 28 - inverse of a linear transformation

For

$𝑇 (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \\ 𝑥_{4} \end{matrix}) = (\begin{matrix} 22 & 13 & 8 & 3 \\ - 16 & - 3 & - 2 & - 2 \\ 8 & 9 & 7 & 2 \\ 5 & 4 & 3 & 1 \end{matrix}) (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \\ 𝑥_{4} \end{matrix}),$

the standard matrix is

$𝐴 = (\begin{matrix} 22 & 13 & 8 & 3 \\ - 16 & - 3 & - 2 & - 2 \\ 8 & 9 & 7 & 2 \\ 5 & 4 & 3 & 1 \end{matrix}) .$

The submitted elimination of $(𝐴 | 𝐼_{4})$ ends at

$(\begin{matrix} 𝐼_{4} & | & (\begin{matrix} 1 & - 2 & 9 & - 25 \\ - 2 & 5 & - 22 & 60 \\ 4 & - 9 & 41 & - 112 \\ - 9 & 17 & - 80 & 222 \end{matrix}) \end{matrix}) .$

Thus

$𝐴^{- 1} = (\begin{matrix} 1 & - 2 & 9 & - 25 \\ - 2 & 5 & - 22 & 60 \\ 4 & - 9 & 41 & - 112 \\ - 9 & 17 & - 80 & 222 \end{matrix}),$

and $𝑇^{- 1} (𝑥) = 𝐴^{- 1} 𝑥$ for all $𝑥 \in ℝ^{4}$ .

14.6.2 Exercise 30 - invertibility

Let $𝐴 = (\begin{matrix} 0 & 1 & 𝑏 \\ - 1 & 0 & 𝑐 \\ - 𝑏 & - 𝑐 & 0 \end{matrix})$ . By Theorem 2.4.3, $𝐴$ is invertible exactly when $rref (𝐴) = 𝐼_{3}$ . Row reduction gives

$(\begin{matrix} 0 & 1 & 𝑏 \\ - 1 & 0 & 𝑐 \\ - 𝑏 & - 𝑐 & 0 \end{matrix}) \to (\begin{matrix} 1 & 0 & - 𝑐 \\ 0 & 1 & 𝑏 \\ 0 & 0 & 0 \end{matrix}) .$

Therefore $𝐴$ is not invertible, regardless of the values of $𝑏, 𝑐$ .

14.6.3 Exercise 42 - permutation matrices

By elementary transformations that change row order, any permutation matrix can be transformed into $𝐼_{𝑛}$ . Hence its rref is $𝐼_{𝑛}$ , so it is invertible by Theorem 2.4.3. If $𝐴$ is an arbitrary permutation matrix, then

$rref (𝐴 | 𝐼_{𝑛}) = (𝐼_{𝑛} | 𝐵)$

and $𝐵$ is the inverse of $𝐴$ . Since the calculation only changes row order, every row of $𝐵$ is chosen from $𝐼_{𝑛}$ without repetition. Thus $𝐵$ is again a permutation matrix.

14.6.4 Exercise 6 - kernel

For $𝐴 = (\begin{matrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{matrix})$ , the submitted row reduction is

$(\begin{matrix} 1 & 1 & 1 & | & 0 \\ 1 & 2 & 3 & | & 0 \end{matrix}) \to (\begin{matrix} 1 & 0 & - 1 & | & 0 \\ 0 & 1 & 2 & | & 0 \end{matrix}) .$

Hence

$(\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix}) = (\begin{matrix} 𝑡 \\ - 2 𝑡 \\ 𝑡 \end{matrix}) = 𝑡 (\begin{matrix} 1 \\ - 2 \\ 1 \end{matrix}), 𝑡 \in ℝ,$

so $ker (𝐴) = span ((\begin{matrix} 1 \\ - 2 \\ 1 \end{matrix}))$ .

14.6.5 Exercise 14 - image

For $𝐴 = (\begin{matrix} 1 & 2 & 3 \\ 1 & 2 & 3 \\ 1 & 2 & 3 \end{matrix})$ ,

$𝑇 (𝑥) = 𝐴 𝑥 = (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}) 𝑥_{1} + (\begin{matrix} 2 \\ 2 \\ 2 \end{matrix}) 𝑥_{2} + (\begin{matrix} 3 \\ 3 \\ 3 \end{matrix}) 𝑥_{3} = (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}) (𝑥_{1} + 2 𝑥_{2} + 3 𝑥_{3}) .$

Therefore $im (𝐴) = span ((\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}))$ .

14.7 Part B

14.7.1 Problem 1 - nilpotent transformations

Let $𝑇 : ℝ^{𝑛} \to ℝ^{𝑛}$ have standard matrix $𝐴$ .

14.7.1.1 (a)

We prove by induction on $𝑘$ that the standard matrix of $𝑇^{𝑘}$ is $𝐴^{𝑘}$ . For $𝑘 = 1$ , $𝑇 (𝑥) = 𝐴 𝑥 = 𝐴^{1} 𝑥$ . Assume $𝑇^{𝑛 (𝑥)} = 𝐴^{𝑛} 𝑥$ . Using the composition rule and associativity,

$𝑇^{𝑛 + 1} (𝑥) = 𝑇 (𝑇^{𝑛 (𝑥)}) = 𝑇 (𝐴^{𝑛} 𝑥) = 𝐴 (𝐴^{𝑛} 𝑥) = 𝐴^{𝑛 + 1} 𝑥 .$

Thus the statement holds for all $𝑘$ .

14.7.1.2 (b)

Assume that $𝑇$ is nilpotent. Then, for some positive integer $𝑘$ , $𝑇^{𝑘 (𝑥)} = 0$ for all $𝑥 \in ℝ^{𝑛}$ . By part (a), $𝐴^{𝑘} 𝑥 = 0$ for all $𝑥$ . Suppose, for contradiction, that $𝐴$ is invertible. Then $𝐴^{𝑘}$ is invertible, with inverse $(𝐴^{- 1} \cdot 𝐴^{- 1} \cdot \dots \cdot 𝐴^{- 1})$ ( $𝑘$ factors). Thus $rank (𝐴^{𝑘}) = 𝑛$ , its rref is $𝐼_{𝑛}$ , and the augmented system $(𝐴^{𝑘} | 0)$ cannot have a solution for every right-hand side as required. This contradicts $𝐴^{𝑘} 𝑥 = 0$ for all $𝑥$ . Therefore $𝐴$ is not invertible.

14.7.1.3 (c)

If $𝑘 = 1$ , then $𝑇 = 0$ , hence $𝐴 = 0$ and $𝐴 - 𝐼_{𝑛} = - 𝐼_{𝑛}$ is invertible. For $𝑘 \geq 2$ , set

$𝐵 = 𝐼_{𝑛} + 𝐴 + 𝐴^{2} + \dots + 𝐴^{𝑘 - 1} .$

Then

$(𝐼_{𝑛} - 𝐴) 𝐵 = 𝐼_{𝑛} - 𝐴^{𝑘} = 𝐼_{𝑛}$

and, similarly, $𝐵 (𝐼_{𝑛} - 𝐴) = 𝐼_{𝑛}$ . Thus $𝐼_{𝑛} - 𝐴$ is invertible, and so is $𝐴 - 𝐼_{𝑛} = - (𝐼_{𝑛} - 𝐴)$ .

14.7.2 Problem 2 - the vector space $ℱ︀ (𝑆, 𝑉)$

For $𝑓, 𝑔, ℎ \in ℱ︀ (𝑆, 𝑉)$ and $𝑥 \in 𝑆$ , pointwise addition gives the vector-space axioms:

(𝑓 + 𝑔) (𝑥) \in 𝑉, (𝑓 + 𝑔) (𝑥) + ℎ (𝑥) = 𝑓 (𝑥) + (𝑔 (𝑥) + ℎ (𝑥)), 𝑓 (𝑥) + 𝑔 (𝑥) = 𝑔 (𝑥) + 𝑓 (𝑥),

and the function $𝑝 (𝑥) = 0_{𝑉}$ is an additive identity. For each $𝑓$ , the function $𝑛 (𝑥) = - 𝑓 (𝑥)$ is its additive inverse. For scalars $𝛼, 𝛽$ ,

𝛼 (𝑓 + 𝑔) (𝑥) = 𝛼 𝑓 (𝑥) + 𝛼 𝑔 (𝑥), (𝛼 + 𝛽) 𝑓 (𝑥) = 𝛼 𝑓 (𝑥) + 𝛽 𝑓 (𝑥),

$𝛼 (𝛽 𝑓 (𝑥)) = (𝛼 𝛽) 𝑓 (𝑥), 1 𝑓 (𝑥) = 𝑓 (𝑥) .$

All expressions are defined in $𝑉$ , so $ℱ︀ (𝑆, 𝑉)$ is a vector space.

The element $0_{ℱ︀ (𝑆, 𝑉)}$ is the function $𝑥 \mapsto 0_{𝑉}$ , whereas $0_{𝑉}$ is an element of $𝑉$ ; they are different elements although the function value is always $0_{𝑉}$ .

$ℱ︀ (𝑉, 𝑆)$ is not necessarily a vector space. For example, take $𝑆 = {1, 2}$ with $1 + 2 = 3$ , which is not in $𝑆$ . The constant functions $𝑓 (𝑣) = 1$ and $𝑔 (𝑣) = 2$ lie in $ℱ︀ (𝑉, 𝑆)$ , but $𝑓 + 𝑔$ would have value $3$ , hence is not a function into $𝑆$ .

Finally,

$𝒫︀ \subset ℱ︀ (ℝ, ℝ), {𝒫︀}_{𝑛} \subset ℱ︀ (ℝ, ℝ), 𝐶^{\infty} \subset ℱ︀ (ℝ, ℝ),$

where $𝒫︀$ is the set of all real polynomial functions, ${𝒫︀}_{𝑛}$ those of degree at most $𝑛$ , and $𝐶^{\infty}$ the infinitely differentiable real functions.

14.7.3 Problem 3 - polynomial transformation

Let $𝑇 (𝑝) (𝑡) = 𝑝^{'} (𝑡) + 𝑝 (0)$ .

14.7.3.1 (a)

For $𝑝_{1}, 𝑝_{2} \in 𝒫︀$ ,

$𝑇 (𝑝_{1} + 𝑝_{2}) = {(𝑝_{1} + 𝑝_{2})}^{'} (𝑡) + (𝑝_{1} + 𝑝_{2}) (0) = 𝑇 (𝑝_{1}) + 𝑇 (𝑝_{2}),$

and for $𝑘 \in ℝ$ ,

$𝑇 (𝑘 𝑝) (𝑡) = 𝑘 𝑝^{'} (𝑡) + 𝑘 𝑝 (0) = 𝑘 𝑇 (𝑝) (𝑡) .$

Thus $𝑇$ is linear.

14.7.3.2 (b)

$𝑇_{𝑛} : {𝒫︀}_{𝑛} \to {𝒫︀}_{𝑛}$ is not surjective: $𝑝 (𝑡) = 𝑡^{𝑛}$ is in the target, but no element of ${𝒫︀}_{𝑛}$ maps to it because differentiation lowers the degree by one. It is not injective: for

$𝑝_{1} (𝑡) = 1 + 2 𝑡 + 𝑡^{2}, 𝑝_{2} (𝑡) = 2 + 𝑡 + 𝑡^{2},$

we have $𝑇_{𝑛 (𝑝_{1})} = 3 + 2 𝑡 = 𝑇_{𝑛 (𝑝_{2})}$ although $𝑝_{1} \neq 𝑝_{2}$ .

14.7.3.3 (c)

$𝑇$ is not injective by the same counterexample. It is surjective: if

$𝑝 (𝑡) = 𝑘 + 𝑎_{1} 𝑡 + 𝑎_{2} 𝑡^{2} + \dots + 𝑎_{𝑚} 𝑡^{𝑚},$

take

$𝑞 (𝑡) = 𝑘 𝑡 + \frac{1}{2} 𝑎_{1} 𝑡^{2} + \frac{1}{3} 𝑎_{2} 𝑡^{3} + \dots + \frac{1}{𝑚 + 1} 𝑎_{𝑚} 𝑡^{𝑚 + 1} .$

Then $𝑞^{'} (𝑡) + 𝑞 (0) = 𝑝 (𝑡)$ .

14.7.4 Problem 4 - left multiplication

For $𝐿_{𝐴} : ℝ^{𝑛 \times 𝑛} \to ℝ^{𝑛 \times 𝑛}$ , $𝐿_{𝐴 (𝐵)} = 𝐴 𝐵$ :

14.7.4.1 (a)

For $𝐵, 𝐶 \in ℝ^{𝑛 \times 𝑛}$ and $𝑘 \in ℝ$ ,

$𝐿_{𝐴 (𝐵 + 𝐶)} = 𝐴 (𝐵 + 𝐶) = 𝐴 𝐵 + 𝐴 𝐶 = 𝐿_{𝐴 (𝐵)} + 𝐿_{𝐴 (𝐶)},$

$𝐿_{𝐴 (𝑘 𝐵)} = 𝐴 (𝑘 𝐵) = 𝑘 𝐴 𝐵 = 𝑘 𝐿_{𝐴 (𝐵)} .$

So $𝐿_{𝐴}$ is linear.

14.7.4.2 (b)

If $𝐴$ is invertible and $𝐿_{𝐴 (𝐵_{1})} = 𝐿_{𝐴 (𝐵_{2})}$ , multiplication by $𝐴^{- 1}$ gives $𝐵_{1} = 𝐵_{2}$ , so $𝐿_{𝐴}$ is injective and hence invertible. Conversely, if $𝐿_{𝐴}$ is invertible, it is surjective. Thus some $𝐶$ satisfies $𝐿_{𝐴 (𝐶)} = 𝐼_{𝑛}$ , so $𝐴 𝐶 = 𝐼_{𝑛}$ and $𝐴$ is invertible.

14.7.4.3 (c)

If $𝐿_{𝐴} = 𝐿_{𝐵}$ , then evaluating both maps at $𝐼_{𝑛}$ gives $𝐴 = 𝐿_{𝐴 (𝐼_{𝑛})} = 𝐿_{𝐵 (𝐼_{𝑛})} = 𝐵$ . Hence $𝐴 \mapsto 𝐿_{𝐴}$ is injective.

14.7.4.4 (d)

The map $𝐿 : ℝ^{𝑛 \times 𝑛} \to ℱ︀$ is not surjective. The constant function $𝑓 (𝐶) = 𝐼_{𝑛}$ is in $ℱ︀$ , but if $𝐿 (𝐴) = 𝑓$ , then $𝐴 𝐶 = 𝐼_{𝑛}$ for all $𝐶$ . Taking $𝐶 = 0_{𝑛 \times 𝑛}$ is impossible.

14.7.5 Problem 5 - rotations and projection

For $𝑇 = {Rot}_{- 80 °} \circ {Proj}_{𝑦} \circ {Rot}_{35 °}$ :

14.7.5.1 (a)

${Rot}_{35 °}$ has image $ℝ^{2}$ . Projection onto the $𝑦$ -axis has image the $𝑦$ -axis, and rotation clockwise by $80 °$ sends this to a line $80 °$ clockwise from the $𝑦$ -axis. Thus the angle between $im (𝑇)$ and the $𝑥$ -axis is $90 ° - 80 ° = 10 °$ .

14.7.5.2 (b)

The final rotation does not affect the kernel. Projection kills exactly the $𝑥$ -axis, and undoing the initial $35 °$ counter-clockwise rotation gives a kernel line $35 °$ counter-clockwise from the $𝑥$ -axis.

14.7.5.3 (c)

For $𝑇_{𝜑, 𝜃} = {Rot}_{𝜑} \circ {Proj}_{𝑦} \circ {Rot}_{𝜃}$ , the image is a line $- 𝜑$ from the $𝑦$ -axis and the kernel is a line $𝜃$ from the $𝑥$ -axis. They agree when

$𝜑 = \frac{𝜋}{2} - 𝜃 + 𝑘 𝜋,$

equivalently $𝜃 - 𝜑 = \frac{𝜋}{2} + 𝑘 𝜋$ for $𝑘 \in ℤ$ .

14.8 Homework 5 - submitted work

14.8.1 Exercise 56 - linear independence

Rearrange the four given vectors as

𝑣_{1} = (\begin{matrix} 𝑒 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{matrix}), 𝑣_{2} = (\begin{matrix} 𝑘 \\ 𝑚 \\ 1 \\ 0 \\ 0 \\ 0 \end{matrix}), 𝑣_{3} = (\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \\ 𝑑 \\ 1 \\ 0 \end{matrix}), 𝑣_{4} = (\begin{matrix} 𝑓 \\ 𝑔 \\ ℎ \\ 𝑖 \\ 𝑗 \\ 1 \end{matrix}) .

Each $𝑣_{𝑖}$ has an entry where all preceding vectors have $0$ and it has a nonzero entry. Thus, in a relation $𝑐_{1} 𝑣_{1} + 𝑐_{2} 𝑣_{2} + 𝑐_{3} 𝑣_{3} + 𝑐_{4} 𝑣_{4} = 0$ , the sixth coordinate forces $𝑐_{4} = 0$ , and the same argument forces $𝑐_{3}, 𝑐_{2}, 𝑐_{1} = 0$ one by one. The vectors are linearly independent for all $𝑎, 𝑏, \dots, 𝑚 \in ℝ$ .

14.8.2 Exercise 33 - hyperplanes

For $𝑐_{1} 𝑥_{1} + \dots + 𝑐_{𝑛} 𝑥_{𝑛} = 0$ , let $𝐴 = (\begin{matrix} 𝑐_{1} & 𝑐_{2} & \dots & 𝑐_{𝑛} \end{matrix})$ . The hyperplane is $ker (𝑇_{𝐴})$ , and the image of $𝑇_{𝐴} : ℝ^{𝑛} \to ℝ$ is nonzero because some $𝑐_{𝑖} \neq 0$ . Hence its image has dimension $1$ , and rank-nullity gives

$dim (𝑉) = 𝑛 - 1 .$

Thus a hyperplane in $ℝ^{3}$ is a plane, and a hyperplane in $ℝ^{2}$ is a line.

14.8.3 Exercise 63 - equal-dimensional nested subspaces

Let $(𝑣_{1}, \dots, 𝑣_{𝑚})$ be a basis of $𝑉$ , with $𝑉 \subset 𝑊$ and $dim (𝑉) = dim (𝑊) = 𝑚$ . The vectors $𝑣_{1}, \dots, 𝑣_{𝑚}$ lie in $𝑊$ and are linearly independent. By Theorem 3.3.4 they form a basis of $𝑊$ . Every $𝑤 \in 𝑊$ is therefore a linear combination of the $𝑣_{𝑖}$ , so $𝑤 \in 𝑉$ . Thus $𝑊 \subset 𝑉$ , and $𝑉 = 𝑊$ .

14.8.4 Exercise 12 - arithmetic sequences

Let $𝑆$ be the set of all arithmetic sequences. It contains the zero sequence. If

$𝑚 = (𝑚_{0}, 𝑚_{0} + 𝑘, 𝑚_{0} + 2 𝑘, \dots), 𝑛 = (𝑛_{0}, 𝑛_{0} + 𝑙, 𝑛_{0} + 2 𝑙, \dots),$

then

$𝑚 + 𝑛 = (𝑚_{0} + 𝑛_{0}, 𝑚_{0} + 𝑛_{0} + (𝑘 + 𝑙), 𝑚_{0} + 𝑛_{0} + 2 (𝑘 + 𝑙), \dots) \in 𝑆 .$

For $𝑟 \in ℝ$ ,

$𝑟 (𝑚_{0}, 𝑚_{0} + 𝑘, 𝑚_{0} + 2 𝑘, \dots) = (𝑟 𝑚_{0}, 𝑟 𝑚_{0} + 𝑟 𝑘, 𝑟 𝑚_{0} + 2 𝑟 𝑘, \dots) \in 𝑆 .$

Hence $𝑆$ is a subspace.

14.8.5 Exercise 28 - commuting $2 \times 2$ matrices

Let $𝐴 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ and $𝐵 = (\begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix})$ . The equation $𝐴 𝐵 = 𝐵 𝐴$ gives

$(\begin{matrix} 𝑎 & 𝑎 + 𝑏 \\ 𝑐 & 𝑐 + 𝑑 \end{matrix}) = (\begin{matrix} 𝑎 + 𝑐 & 𝑏 + 𝑑 \\ 𝑐 & 𝑑 \end{matrix}),$

so $𝑐 = 0$ and $𝑎 = 𝑑$ . Therefore

$𝑆 = {(\begin{matrix} 𝑎 & 𝑏 \\ 0 & 𝑎 \end{matrix}) | 𝑎, 𝑏 \in ℝ} = {𝑎 (\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}) + 𝑏 (\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}) | 𝑎, 𝑏 \in ℝ} .$

The displayed matrices are linearly independent, so they are a basis and $dim (𝑆) = 2$ .

14.9 Part A - Problem 6

For each diagram, the submitted dependent triples are:

(a) $(𝑣_{1}, 𝑣_{2}, 𝑣_{3})$ , $(𝑣_{1}, 𝑣_{2}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{2}, 𝑣_{4})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{3}, 𝑣_{4}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{4}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{4}, 𝑣_{5})$ (10 sets). (b) $(𝑣_{1}, 𝑣_{2}, 𝑣_{3})$ , $(𝑣_{1}, 𝑣_{2}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{2}, 𝑣_{4})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{3}, 𝑣_{4}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{4}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{4}, 𝑣_{5})$ (8 sets). (c) $(𝑣_{1}, 𝑣_{2}, 𝑣_{3})$ , $(𝑣_{1}, 𝑣_{2}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{3}, 𝑣_{4}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{4}, 𝑣_{5})$ (6 sets). (d) $(𝑣_{1}, 𝑣_{2}, 𝑣_{3})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{1}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{4})$ , $(𝑣_{2}, 𝑣_{3}, 𝑣_{5})$ , $(𝑣_{2}, 𝑣_{4}, 𝑣_{5})$ .

14.10 Part B

14.10.1 Problem 1 - images of independent lists

14.10.1.1 (a)

False. Let $𝑇 : ℝ^{2} \to ℝ^{2}$ be $𝑇 (𝑥) = (\begin{matrix} 0 & 0 \\ 0 & 0 \end{matrix}) 𝑥$ . The vectors $(\begin{matrix} 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 1 \end{matrix})$ are linearly independent, but their images are both $0$ , so the image list is linearly dependent.

14.10.1.2 (b)

True. Assume $𝑌 = (𝑇 (𝑥_{1}), \dots, 𝑇 (𝑥_{𝑘}))$ is linearly independent and $𝑑_{1} 𝑥_{1} + \dots + 𝑑_{𝑘} 𝑥_{𝑘} = 0_{𝑉}$ . Then

$𝑑_{1} 𝑇 (𝑥_{1}) + \dots + 𝑑_{𝑘} 𝑇 (𝑥_{𝑘}) = 𝑇 (0_{𝑉}) = 0_{𝑊} .$

Independence of $𝑌$ gives $𝑑_{1} = \dots = 𝑑_{𝑘} = 0$ , so $𝑋$ is linearly independent.

14.10.2 Problem 2 - prescribed kernel and image

The rref required by

$ker (𝑇) = {𝑥 \in ℝ^{5} | 𝑥_{1} = 5 𝑥_{2}, 𝑥_{3} = 7 𝑥_{4}}$

$(\begin{matrix} 1 & - 5 & 0 & 0 & 0 \\ 0 & 0 & 1 & - 7 & 0 \\ 0 & 0 & 0 & 0 & 0 \end{matrix}) .$

The target image has basis $(\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix})$ . Choosing the first and second nonredundant columns accordingly gives

$𝐴 = (\begin{matrix} 1 & - 5 & 0 & 0 & 0 \\ 0 & 0 & 1 & - 7 & 0 \\ 1 & - 5 & 0 & 0 & 0 \end{matrix}),$

and $𝑇 (𝑥) = 𝐴 𝑥$ is one solution.

The transformation is not unique. For example, elementary transformations preserve the rref, and

$𝐴^{'} = (\begin{matrix} 5 & - 25 & 0 & 0 & 0 \\ 0 & 0 & 1 & - 7 & 0 \\ 5 & - 25 & 0 & 0 & 0 \end{matrix})$

has the same required kernel and image but is different from $𝐴$ .

14.10.3 Problem 3 - maps defined on a basis

14.10.3.1 (a)

Let $ℬ︀ = (𝑥_{1}, \dots, 𝑥_{𝑛})$ be a basis of $𝑋$ , and write $𝑣 = 𝑐_{1} 𝑥_{1} + \dots + 𝑐_{𝑛} 𝑥_{𝑛}$ . Define

$𝑇 (𝑣) = 𝑐_{1} 𝑇 (𝑥_{1}) + \dots + 𝑐_{𝑛} 𝑇 (𝑥_{𝑛}) = 𝑐_{1} 𝑦_{1} + \dots + 𝑐_{𝑛} 𝑦_{𝑛} .$

For $𝑣_{1} = \sum 𝑑_{𝑖} 𝑥_{𝑖}$ , $𝑣_{2} = \sum 𝑒_{𝑖} 𝑥_{𝑖}$ , this rule gives

$𝑇 (𝑣_{1} + 𝑣_{2}) = \sum (𝑑_{𝑖} + 𝑒_{𝑖}) 𝑇 (𝑥_{𝑖}) = 𝑇 (𝑣_{1}) + 𝑇 (𝑣_{2})$

and $𝑇 (𝑘 𝑣_{1}) = 𝑘 𝑇 (𝑣_{1})$ , so it is linear. If $𝑇^{'}$ has the same values on the basis, then $𝑇^{'} (𝑣) = \sum 𝑐_{𝑖} 𝑇^{'} (𝑥_{𝑖}) = \sum 𝑐_{𝑖} 𝑦_{𝑖} = 𝑇 (𝑣)$ , so it is unique.

14.10.3.2 (b)

Let $dim (𝑋) = 𝑛$ and let $(𝑢_{1}, \dots, 𝑢_{𝑘})$ be a basis of $𝑈$ . Extend it to a basis $(𝑢_{1}, \dots, 𝑢_{𝑘}, 𝑤_{𝑘 + 1}, \dots, 𝑤_{𝑛})$ of $𝑋$ . Since $dim (𝑉) = 𝑛 - 𝑘$ , choose a basis $(𝑣_{1}, \dots, 𝑣_{𝑛 - 𝑘})$ of $𝑉$ and define

$𝑇_{𝑈, 𝑉} (𝑢_{𝑖}) = 0, 𝑇_{𝑈, 𝑉} (𝑤_{𝑘 + 𝑗}) = 𝑣_{𝑗} .$

By part (a), this is a valid linear transformation. Its image is $𝑉$ and its kernel is $𝑈$ .

14.10.3.3 (c)

The map is not unique: the construction depends on an arbitrarily chosen basis of $𝑉$ . Choosing a different basis can give different images for the $𝑤_{𝑘 + 𝑗}$ while retaining the required kernel and image.

14.10.4 Problem 4 - ranks and nullities of a composition

14.10.4.1 (a)

True. Since $im (𝑆 \circ 𝑇) \subset im (𝑆)$ ,

$rank (𝑆 \circ 𝑇) = dim (im (𝑆 \circ 𝑇)) \leq rank (𝑆) .$

14.10.4.2 (b)

True. View the composition in two stages. First $𝑇 : 𝑈 \to 𝑉$ ; then $𝑆 : im (𝑇) \to 𝑊$ . Rank-nullity for the restricted second map gives

$rank (𝑆 \circ 𝑇) = rank (𝑇) - dim (ker (𝑆^{'})) \leq rank (𝑇) .$

14.10.4.3 (c)

True. Rank-nullity yields

$rank (𝑇) + nullity (𝑇) = rank (𝑆 \circ 𝑇) + nullity (𝑆 \circ 𝑇) .$

Part (b) then implies $nullity (𝑆 \circ 𝑇) \geq nullity (𝑇)$ .

14.10.4.4 (d)

False. If $𝑇$ is surjective, then $𝑆 (𝑉) = 0_{𝑊}$ , so $nullity (𝑆) = dim (𝑉)$ and $nullity (𝑆 \circ 𝑇) = dim (𝑈)$ . When $dim (𝑈) < dim (𝑉)$ , the claimed inequality fails.

14.10.5 Problem 5 - symmetric and skew-symmetric matrices

For $𝑇 (𝐴) = 𝐴 + 𝐴^{𝑇}$ :

14.10.5.1 (a)

$𝑇 (𝐴_{1} + 𝐴_{2}) = (𝐴_{1} + 𝐴_{2}) + {(𝐴_{1} + 𝐴_{2})}^{𝑇} = 𝑇 (𝐴_{1}) + 𝑇 (𝐴_{2})$ , and $𝑇 (𝑘 𝐴) = 𝑘 𝐴 + {(𝑘 𝐴)}^{𝑇} = 𝑘 𝑇 (𝐴)$ , so $𝑇$ is linear.

14.10.5.2 (b)

If $𝐴 \in ker (𝑇)$ , then $𝐴^{𝑇} = - 𝐴$ , so $𝐴 \in {Skew}_{𝑛}$ . Conversely, $𝐵^{𝑇} = - 𝐵$ implies $𝑇 (𝐵) = 0$ , so $ker (𝑇) = {Skew}_{𝑛}$ . Also $𝐶 = 𝑀 + 𝑀^{𝑇}$ satisfies $𝐶^{𝑇} = 𝐶$ , whence $im (𝑇) \subset {Sym}_{𝑛}$ . If $𝐷 \in {Sym}_{𝑛}$ , choose $𝑁 = \frac{1}{2} 𝐷$ ; then $𝑁 + 𝑁^{𝑇} = 𝐷$ . Therefore $im (𝑇) = {Sym}_{𝑛}$ .

14.10.5.3 (c)

Both are subspaces. Each contains the zero matrix. If $𝐴^{𝑇} = - 𝐴$ and $𝐵^{𝑇} = - 𝐵$ , then ${(𝐴 + 𝐵)}^{𝑇} = - (𝐴 + 𝐵)$ and ${(𝑘 𝐴)}^{𝑇} = - 𝑘 𝐴$ ; this proves the subspace conditions for ${Skew}_{𝑛}$ . Replacing $- 𝐴, - 𝐵$ by $𝐴, 𝐵$ gives the same argument for ${Sym}_{𝑛}$ .

14.10.5.4 (d)

There are $𝑛^{2}$ free entries in an arbitrary $𝑛 \times 𝑛$ matrix. In a symmetric matrix, entries below the diagonal mirror entries above it, while the diagonal entries are free. Thus

$dim ({Sym}_{𝑛}) = \frac{𝑛^{2} - 𝑛}{2} + 𝑛 = \frac{𝑛^{2} + 𝑛}{2} .$

For a skew-symmetric matrix, the lower entries are the negations of the upper entries and every diagonal entry is zero. Thus

$dim ({Skew}_{𝑛}) = \frac{𝑛^{2} - 𝑛}{2} .$

14.11 Homework 6 - submitted work

14.11.1 Exercise 50 - hexagonal coordinates

For the basis $ℬ︀ = (𝑣, 𝑤)$ in the hexagonal tiling,

${[\vec{OP}]}_{ℬ︀} = {[2 𝑣 + 𝑤]}_{ℬ︀} = (\begin{matrix} 2 \\ 1 \end{matrix}),$

${[\vec{OQ}]}_{ℬ︀} = {[𝑣 + 2 𝑤]}_{ℬ︀} = (\begin{matrix} 1 \\ 2 \end{matrix}) .$

The point with ${[\vec{OR}]}_{ℬ︀} = (\begin{matrix} 3 \\ 2 \end{matrix})$ is the center of a tile. Further,

$(\begin{matrix} 17 \\ 13 \end{matrix}) = 8 (\begin{matrix} 2 \\ 1 \end{matrix}) + (\begin{matrix} 1 \\ 2 \end{matrix}) + (\begin{matrix} 0 \\ 3 \end{matrix}) .$

Moving by the first two summands means moving in parallel to $\vec{OQ}$ and $\vec{OP}$ by whole tile lengths, which does not change whether a point is a vertex or center. Since $(\begin{matrix} 0 \\ 3 \end{matrix})$ is a vertex, $(\begin{matrix} 17 \\ 13 \end{matrix})$ is a vertex.

14.11.2 Exercise 70 - upper triangular coordinate matrix

There is no basis $ℬ︀ = (𝑣_{1}, 𝑣_{2})$ of $ℝ^{2}$ whose $ℬ︀$ -matrix for

$𝑇 (𝑥) = (\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix}) 𝑥$

is upper triangular. Suppose, for a contradiction, that

${[𝑇]}_{ℬ︀} = (\begin{matrix} 𝑎 & 𝑏 \\ 0 & 𝑐 \end{matrix}) .$

Then the generalized key theorem gives

$(\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix}) 𝑣_{1} = 𝑎 𝑣_{1} .$

Writing $𝑣_{1} = (\begin{matrix} 𝑥 \\ 𝑦 \end{matrix})$ yields

$- 𝑦 = 𝑎 𝑥, 𝑥 = 𝑎 𝑦, (𝑎 - 1) 𝑥 = (- 𝑎 - 1) 𝑦 .$

Thus $𝑥 = 𝑦 = 0$ , since $𝑎 - 1$ and $- 𝑎 - 1$ cannot both vanish. This is impossible because $0$ cannot be a basis vector.

14.11.3 Exercise 58 - the solutions of $𝑓^{″} = - 𝑓$

14.11.3.1 (a)

For $𝑔 \in 𝑉$ , $𝑔^{″} = - 𝑔$ . Let

$𝑓 (𝑥) = {𝑔 (𝑥)}^{2} + 𝑔^{'} {(𝑥)}^{2} .$

Then

$𝑓^{'} (𝑥) = 2 𝑔 (𝑥) 𝑔^{'} (𝑥) + 2 𝑔^{'} (𝑥) 𝑔^{″} (𝑥) = 2 𝑔 (𝑥) 𝑔^{'} (𝑥) - 2 𝑔 (𝑥) 𝑔^{'} (𝑥) = 0,$

so $𝑓$ is constant.

14.11.3.2 (b)

If $𝑔 (0) = 𝑔^{'} (0) = 0$ , the constant from part (a) is $𝑘 = {𝑔 (0)}^{2} + 𝑔^{'} {(0)}^{2} = 0$ . Therefore ${𝑔 (𝑥)}^{2} + 𝑔^{'} {(𝑥)}^{2} = 0$ for all $𝑥$ , and $𝑔 (𝑥) = 𝑔^{'} (𝑥) = 0$ for all $𝑥$ .

14.11.3.3 (c)

$𝑉$ is a vector space; moreover, ${(sin 𝑥)}^{″} = - sin 𝑥$ and ${(cos 𝑥)}^{″} = - cos 𝑥$ . Hence for $𝑓 \in 𝑉$ ,

$𝑔 (𝑥) = 𝑓 (𝑥) - 𝑓 (0) cos 𝑥 - 𝑓^{'} (0) sin 𝑥$

is in $𝑉$ . We have $𝑔 (0) = 0$ and

$𝑔^{'} (𝑥) = 𝑓^{'} (𝑥) + 𝑓 (0) sin 𝑥 - 𝑓^{'} (0) cos 𝑥, 𝑔^{'} (0) = 0 .$

Part (b) gives $𝑔 = 0$ , so

$𝑓 (𝑥) = 𝑓 (0) cos 𝑥 + 𝑓^{'} (0) sin 𝑥 .$

Thus $(cos 𝑥, sin 𝑥)$ spans $𝑉$ .

14.11.4 Exercise 46 - multiplication by $𝑡 - 1$

For $𝑇 (𝑓 (𝑡)) = (𝑡 - 1) 𝑓 (𝑡)$ ,

$𝑇 (𝑓 + 𝑔) = (𝑡 - 1) (𝑓 + 𝑔) = 𝑇 (𝑓) + 𝑇 (𝑔),$

$𝑇 (𝑘 𝑓) = (𝑡 - 1) 𝑘 𝑓 = 𝑘 𝑇 (𝑓) .$

Thus $𝑇$ is linear. It is not an isomorphism because it is not surjective: a nonzero constant target polynomial cannot be $(𝑡 - 1) 𝑓 (𝑡)$ for a polynomial $𝑓$ .

14.11.5 Exercise 68 - isomorphism condition

For $𝑀 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ ,

$𝑇 (𝑀) = (\begin{matrix} 5 𝑎 & 𝑏 \\ 5 𝑐 & 𝑑 \end{matrix}) - (\begin{matrix} 2 𝑎 & 2 𝑏 \\ 𝑘 𝑐 & 𝑘 𝑑 \end{matrix}) = (\begin{matrix} 3 𝑎 & - 𝑏 \\ (5 - 𝑘) 𝑐 & (1 - 𝑘) 𝑑 \end{matrix}) .$

The source and target have equal dimension, so $𝑇$ is an isomorphism exactly when it is injective, equivalently when $ker (𝑇) = {0}$ . This holds for $𝑘 \neq 1, 5$ . For $𝑘 = 1$ or $𝑘 = 5$ , the kernel has dimension $1$ , so $𝑇$ is not an isomorphism.

14.12 Part B

14.12.1 Problem 1 - coefficient map

Let $𝑇 : ℝ^{𝑛} \to 𝑉$ be defined by

$𝑇 ((\begin{matrix} 𝑐_{1} \\ ⋮ \\ 𝑐_{𝑛} \end{matrix})) = 𝑐_{1} 𝑣_{1} + \dots + 𝑐_{𝑛} 𝑣_{𝑛} .$

14.12.1.1 (a)

For coefficient columns $𝑎 = (𝑎_{𝑖})$ and $𝑐 = (𝑐_{𝑖})$ ,

$𝑇 (𝑎 + 𝑐) = \sum (𝑎_{𝑖} + 𝑐_{𝑖}) 𝑣_{𝑖} = 𝑇 (𝑎) + 𝑇 (𝑐),$

and $𝑇 (𝑘 𝑎) = \sum 𝑘 𝑎_{𝑖} 𝑣_{𝑖} = 𝑘 𝑇 (𝑎)$ . Hence $𝑇$ is linear.

14.12.1.2 (b)

If $𝑇$ is injective and $𝑐_{1} 𝑣_{1} + \dots + 𝑐_{𝑛} 𝑣_{𝑛} = 0$ , then

$𝑇 ((\begin{matrix} 𝑐_{1} \\ ⋮ \\ 𝑐_{𝑛} \end{matrix})) = 𝑇 ((\begin{matrix} 0 \\ ⋮ \\ 0 \end{matrix})),$

so $𝑐_{𝑖} = 0$ for every $𝑖$ . The list is linearly independent. Conversely, if $(𝑣_{1}, \dots, 𝑣_{𝑛})$ is linearly independent and $𝑇 (𝑎) = 𝑇 (𝑏)$ , then

$(𝑎_{1} - 𝑏_{1}) 𝑣_{1} + \dots + (𝑎_{𝑛} - 𝑏_{𝑛}) 𝑣_{𝑛} = 0 .$

Thus $𝑎_{𝑖} = 𝑏_{𝑖}$ for all $𝑖$ , and $𝑇$ is injective.

14.12.1.3 (c)

If $𝑇$ is surjective, every $𝑣 \in 𝑉$ equals $𝑇 ((\begin{matrix} 𝑎_{1} \\ ⋮ \\ 𝑎_{𝑛} \end{matrix})) = 𝑎_{1} 𝑣_{1} + \dots + 𝑎_{𝑛} 𝑣_{𝑛}$ , so the list spans $𝑉$ . Conversely, if it spans $𝑉$ , each $𝑣 \in 𝑉$ has this form and is in the image of $𝑇$ . Therefore $𝑇$ is surjective.

14.12.1.4 (d)

An isomorphism is injective and surjective, so by (b) and (c) its list is linearly independent and spans $𝑉$ : it is an ordered basis. Conversely, an ordered basis gives both properties, so the linear map $𝑇$ is a bijection and hence an isomorphism.

14.12.2 Problem 2 - coordinate matrices for symmetrization

Let $𝑇 (𝐴) = \frac{1}{2} (𝐴 + 𝐴^{𝑇})$ for $𝐴 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ .

14.12.2.1 (a)

$𝑇 (𝐴) = (\begin{matrix} 𝑎 & \frac{𝑏 + 𝑐}{2} \\ \frac{𝑏 + 𝑐}{2} & 𝑑 \end{matrix}) .$

With ${[𝐴]}_{ℰ︀} = (\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \\ 𝑑 \end{matrix})$ , the $ℰ︀$ -matrix is

${[𝑇]}_{ℰ︀} = (\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & \frac{1}{2} & \frac{1}{2} & 0 \\ 0 & \frac{1}{2} & \frac{1}{2} & 0 \\ 0 & 0 & 0 & 1 \end{matrix}) .$

14.12.2.2 (b)

For $𝒞︀ = ((\begin{matrix} 0 & 1 \\ 1 & 0 \end{matrix}), (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}), (\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix}), (\begin{matrix} 0 & 1 \\ - 1 & 0 \end{matrix}))$ ,

${[𝑇]}_{𝒞︀} = (\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 \end{matrix}) .$

14.12.2.3 (c)

Solving ${[𝑇]}_{ℰ︀} (\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \\ 𝑑 \end{matrix}) = 0$ gives

$ker ({[𝑇]}_{ℰ︀}) = {𝑟 (\begin{matrix} 0 \\ - 1 \\ 1 \\ 0 \end{matrix}) | 𝑟 \in ℝ} .$

14.12.2.4 (d)

The corresponding subspace of $ℝ^{2 \times 2}$ is

${𝑟 (\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix}) | 𝑟 \in ℝ},$

with basis $(\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix})$ .

14.12.2.5 (e)

Solving ${[𝑇]}_{𝒞︀} (\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \\ 𝑑 \end{matrix}) = 0$ gives

$ker ({[𝑇]}_{𝒞︀}) = {𝑟 (\begin{matrix} 0 \\ 0 \\ 0 \\ 1 \end{matrix}) | 𝑟 \in ℝ} .$

14.12.2.6 (f) and (g)

The coordinate isomorphism sends

$(\begin{matrix} 𝑥 \\ 𝑦 \\ 𝑧 \\ 𝑤 \end{matrix}) \mapsto (\begin{matrix} 𝑦 & 𝑥 + 𝑤 \\ 𝑥 - 𝑤 & 𝑧 \end{matrix}) .$

Hence the image of the $𝒞︀$ -coordinate kernel is

${𝑟 (\begin{matrix} 0 & 1 \\ - 1 & 0 \end{matrix}) | 𝑟 \in ℝ},$

the same subspace as in part (d). Both are $ker (𝑇)$ .

14.12.2.7 (h)

Using $𝒞︀$ -coordinates, the image is spanned by the first three coordinate vectors. A basis is therefore

$((\begin{matrix} 0 & 1 \\ 1 & 0 \end{matrix}), (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}), (\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix})) .$

14.12.3 Problem 3 - a trigonometric vector space

Let

$𝑓_{1} = 1, 𝑓_{2} = sin (2 𝑥), 𝑓_{3} = cos (2 𝑥), 𝑓_{4} = {sin}^{2} (𝑥), 𝑓_{5} = {cos}^{2} (𝑥), 𝑓_{6} = sin 𝑥 cos 𝑥,$

and $𝑉 = Span (𝑓_{1}, \dots, 𝑓_{6})$ , $ℬ︀ = (𝑓_{1}, 𝑓_{2}, 𝑓_{4})$ .

14.12.3.1 (a)

For $𝑓 = 𝑎_{1} + 𝑎_{2} sin (2 𝑥) + 𝑎_{3} cos (2 𝑥) + 𝑎_{4} {sin}^{2} (𝑥) + 𝑎_{5} {cos}^{2} (𝑥) + 𝑎_{6} sin 𝑥 cos 𝑥$ , the identities $cos (2 𝑥) = 1 - 2 {sin}^{2} (𝑥)$ , ${cos}^{2} (𝑥) = 1 - {sin}^{2} (𝑥)$ , and $sin 𝑥 cos 𝑥 = \frac{1}{2} sin (2 𝑥)$ give

$𝑓 = (𝑎_{1} + 𝑎_{3} + 𝑎_{5}) + (𝑎_{2} + \frac{1}{2} 𝑎_{6}) sin (2 𝑥) + (𝑎_{4} - 2 𝑎_{3} - 𝑎_{5}) {sin}^{2} (𝑥) .$

So $(𝑓_{1}, 𝑓_{2}, 𝑓_{4})$ spans $𝑉$ . If $𝑏_{1} + 𝑏_{2} sin (2 𝑥) + 𝑏_{4} {sin}^{2} (𝑥) = 0$ , evaluate at $𝑥 = 𝜋$ , $𝑥 = \frac{𝜋}{2}$ , and $𝑥 = \frac{𝜋}{4}$ to get $𝑏_{1} = 𝑏_{4} = 𝑏_{2} = 0$ . Thus $ℬ︀$ is an ordered basis.

14.12.3.2 (b)

{[𝑓_{1}]}_{ℬ︀} = (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}), {[𝑓_{2}]}_{ℬ︀} = (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}), {[𝑓_{3}]}_{ℬ︀} = (\begin{matrix} 1 \\ 0 \\ - 2 \end{matrix}),

{[𝑓_{4}]}_{ℬ︀} = (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}), {[𝑓_{5}]}_{ℬ︀} = (\begin{matrix} 1 \\ 0 \\ - 1 \end{matrix}), {[𝑓_{6}]}_{ℬ︀} = (\begin{matrix} 0 \\ \frac{1}{2} \\ 0 \end{matrix}) .

14.12.3.3 (c)

For $𝑓 = 𝑎_{1} + 𝑎_{2} sin (2 𝑥) + 𝑎_{3} {sin}^{2} (𝑥)$ ,

$𝑓^{'} = 2 𝑎_{2} cos (2 𝑥) + 𝑎_{3} sin (2 𝑥) = 2 𝑎_{2} + 𝑎_{3} sin (2 𝑥) - 4 𝑎_{2} {sin}^{2} (𝑥) \in 𝑉 .$

Thus $𝑉$ is closed under differentiation.

14.12.3.4 (d)

$𝑇 (𝑓) = 𝑓^{'} + 2 𝑓$ has

${[𝑇 (𝑓)]}_{ℬ︀} = (\begin{matrix} 2 𝑎_{1} + 2 𝑎_{2} \\ 2 𝑎_{2} + 𝑎_{3} \\ - 4 𝑎_{2} + 2 𝑎_{3} \end{matrix}),$

${[𝑇]}_{ℬ︀} = (\begin{matrix} 2 & 2 & 0 \\ 0 & 2 & 1 \\ 0 & - 4 & 2 \end{matrix}) .$

14.12.3.5 (e)

Row reduction of $({[𝑇]}_{ℬ︀} | 𝐼_{3})$ gives

${[𝑇]}_{ℬ︀}^{- 1} = (\begin{matrix} \frac{1}{2} & - \frac{1}{4} & \frac{1}{8} \\ 0 & \frac{1}{4} & - \frac{1}{8} \\ 0 & \frac{1}{2} & \frac{1}{4} \end{matrix}) .$

Therefore

$𝑇^{- 1} (𝑎_{1} + 𝑎_{2} sin (2 𝑥) + 𝑎_{3} {sin}^{2} (𝑥)) = (\frac{1}{2} 𝑎_{1} - \frac{1}{4} 𝑎_{2} + \frac{1}{8} 𝑎_{3}) + (\frac{1}{4} 𝑎_{2} - \frac{1}{8} 𝑎_{3}) sin (2 𝑥) + (\frac{1}{2} 𝑎_{2} + \frac{1}{4} 𝑎_{3}) {sin}^{2} (𝑥) .$

14.12.3.6 (f)

The coordinate vector of $4 + 8 {sin}^{2} (𝑥)$ is $(\begin{matrix} 4 \\ 0 \\ 8 \end{matrix})$ . Applying the inverse matrix gives $(\begin{matrix} 3 \\ - 1 \\ 2 \end{matrix})$ , so

$𝑓 (𝑥) = 3 - sin (2 𝑥) + 2 {sin}^{2} (𝑥) .$

14.12.4 Problem 4 - products of vector spaces

14.12.4.1 (a)

If $0_{𝑋}$ and $0_{𝑌}$ are the zero vectors of $𝑋$ and $𝑌$ , the zero vector of $𝑋 \times 𝑌$ is $(0_{𝑋}, 0_{𝑌})$ .

14.12.4.2 (b)

Let $(𝑥_{1}, \dots, 𝑥_{𝑚})$ be a basis of $𝑋$ and $(𝑦_{1}, \dots, 𝑦_{𝑛})$ a basis of $𝑌$ . For $(𝑎, 𝑏) \in 𝑋 \times 𝑌$ , write

$𝑎 = \sum 𝑎_{𝑖} 𝑥_{𝑖}, 𝑏 = \sum 𝑏_{𝑗} 𝑦_{𝑗} .$

Then

$(𝑎, 𝑏) = \sum 𝑎_{𝑖 (𝑥_{𝑖}, 0_{𝑌})} + \sum 𝑏_{𝑗 (0_{𝑋}, 𝑦_{𝑗})},$

so the listed vectors span. If their linear combination is $(0_{𝑋}, 0_{𝑌})$ , then $\sum 𝑎_{𝑖} 𝑥_{𝑖} = 0_{𝑋}$ and $\sum 𝑏_{𝑗} 𝑦_{𝑗} = 0_{𝑌}$ . Independence of the two bases forces all coefficients to vanish. Therefore

$((𝑥_{1}, 0_{𝑌}), \dots, (𝑥_{𝑚}, 0_{𝑌}), (0_{𝑋}, 𝑦_{1}), \dots, (0_{𝑋}, 𝑦_{𝑛}))$

is a basis of $𝑋 \times 𝑌$ .

14.12.4.3 (c)

The basis in (b) has $𝑚 + 𝑛$ vectors, so

$dim (𝑋 \times 𝑌) = dim (𝑋) + dim (𝑌) .$

14.12.5 Problem 5 - sum and intersection

Let $𝑇 : 𝑋 \times 𝑌 \to 𝑋 + 𝑌$ be $𝑇 (𝑥, 𝑦) = 𝑥 + 𝑦$ .

14.12.5.1 (a)

$𝑇 ((𝑥_{1}, 𝑦_{1}) + (𝑥_{2}, 𝑦_{2})) = 𝑥_{1} + 𝑥_{2} + 𝑦_{1} + 𝑦_{2} = 𝑇 (𝑥_{1}, 𝑦_{1}) + 𝑇 (𝑥_{2}, 𝑦_{2})$ , and $𝑇 (𝑘 (𝑥, 𝑦)) = 𝑘 𝑇 (𝑥, 𝑦)$ , so $𝑇$ is linear. Each $𝑧 \in 𝑋 + 𝑌$ has the form $𝑧 = 𝑥 + 𝑦 = 𝑇 (𝑥, 𝑦)$ , so $𝑇$ is surjective.

14.12.5.2 (b)

If $𝑥 + 𝑦 = 0$ , then $𝑥 = - 𝑦$ , so it belongs to both $𝑋$ and $𝑌$ . Hence

$ker (𝑇) = {(𝑎, - 𝑎) | 𝑎 \in 𝑋 \cap 𝑌} .$

The map $𝑇_{1} : ker (𝑇) \to 𝑋 \cap 𝑌$ , $(𝑎, - 𝑎) \mapsto 𝑎$ , is linear, injective, and surjective; hence it is an isomorphism.

14.12.5.3 (c)

Since $ker (𝑇)$ is isomorphic to $𝑋 \cap 𝑌$ and $𝑇$ is surjective, rank-nullity with part 4(c) gives

$dim (𝑋 + 𝑌) + dim (𝑋 \cap 𝑌) = dim (𝑋) + dim (𝑌) .$

14.12.5.4 (d)

For three-dimensional subspaces of $ℝ^{5}$ , $𝑋 \cap 𝑌 = {0}$ would give $dim (𝑋 + 𝑌) = 6$ , contradicting $dim (𝑋 + 𝑌) \leq 5$ . Thus it is impossible. In $ℝ^{6}$ it is possible; for example,

$𝑋 = {(\begin{matrix} 𝑎 \\ 𝑏 \\ 𝑐 \\ 0 \\ 0 \\ 0 \end{matrix}) | 𝑎, 𝑏, 𝑐 \in ℝ}, 𝑌 = {(\begin{matrix} 0 \\ 0 \\ 0 \\ 𝑥 \\ 𝑦 \\ 𝑧 \end{matrix}) | 𝑥, 𝑦, 𝑧 \in ℝ} .$

Then $𝑋 + 𝑌 = ℝ^{6}$ and $𝑋 \cap 𝑌 = {0}$ .

14.13 Homework 7 — submitted work

15 Part A

15.1 4.3 Exercise 14

For $𝑇 (𝑀) = (\begin{matrix} 1 & 1 \\ 2 & 2 \end{matrix}) 𝑀$ relative to $ℬ︀ = ((\begin{matrix} 1 & 0 \\ - 1 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & - 1 \end{matrix}), (\begin{matrix} 1 & 0 \\ 2 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & 2 \end{matrix}))$ , the submission forms ${[𝑇]}_{ℬ︀}$ from ${[𝑇 (𝑏_{𝑖})]}_{ℬ︀}$ . It finds

$𝑇 (𝑏_{1}) = 𝑇 (𝑏_{2}) = (\begin{matrix} 0 & 0 \\ 0 & 0 \end{matrix}), 𝑇 (𝑏_{3}) = (\begin{matrix} 3 & 0 \\ 6 & 0 \end{matrix}), 𝑇 (𝑏_{4}) = (\begin{matrix} 0 & 3 \\ 0 & 6 \end{matrix}),$

${[𝑇]}_{ℬ︀} = (\begin{matrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 3 & 0 \\ 0 & 0 & 0 & 3 \end{matrix}) \mapsto (\begin{matrix} 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix}) .$

Hence

${[ker 𝑇]}_{𝑀} = span ((\begin{matrix} 1 \\ 0 \\ 0 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 0 \\ 0 \end{matrix})),$

${[im 𝑇]}_{𝑀} = span ((\begin{matrix} 0 \\ 0 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 0 \\ 0 \\ 1 \end{matrix})) .$

A basis for the kernel is $((\begin{matrix} 1 & 0 \\ - 1 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & - 1 \end{matrix}))$ ; a basis for the image is $((\begin{matrix} 1 & 0 \\ 2 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & 2 \end{matrix}))$ ; and $rank (𝑇) = 2$ .

15.2 4.3 Exercise 28

For $𝑇 (𝑓 (𝑡)) = 𝑓 (2 𝑡 - 1)$ and $ℬ︀ = (1, 𝑡 - 1, {(𝑡 - 1)}^{2})$ ,

${[𝑇]}_{ℬ︀} = ({[1]}_{ℬ︀} {[2 𝑡 - 2]}_{ℬ︀} {[4 {(𝑡 - 1)}^{2}]}_{ℬ︀}) = (\begin{matrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 4 \end{matrix}) .$

This has full rank, so $𝑇$ is invertible, is an isomorphism, and $rank (𝑇) = 3$ .

15.3 4.3 Exercise 60

In $2 𝑥_{1} + 𝑥_{2} - 2 𝑥_{3} = 0$ let

$𝒜︀ = ((\begin{matrix} 1 \\ 2 \\ 2 \end{matrix}), (\begin{matrix} 2 \\ - 2 \\ 1 \end{matrix})), ℬ︀ = ((\begin{matrix} 1 \\ 2 \\ 2 \end{matrix}), (\begin{matrix} 3 \\ 0 \\ 3 \end{matrix})) .$

The submitted change-of-basis matrices are

$𝑆_{ℬ︀ \to 𝒜︀} = (\begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix}), 𝑆_{𝒜︀ \to ℬ︀} = (\begin{matrix} 1 & - 1 \\ 0 & 1 \end{matrix}),$

and $[\vec{𝑎_{1}} \vec{𝑎_{2}}] = 𝑆_{ℬ︀ \to 𝒜︀} [\vec{𝑏_{1}} \vec{𝑏_{2}}]$ .

15.4 5.1 Exercise 6

For $𝑢 = (\begin{matrix} 1 \\ - 1 \\ 2 \\ - 2 \end{matrix})$ and $𝑣 = (\begin{matrix} 2 \\ 3 \\ 4 \\ 5 \end{matrix})$ ,

$cos 𝜃 = \frac{𝑢 \cdot 𝑣}{‖ 𝑢 ‖ ‖ 𝑣 ‖} = \frac{- 3}{\sqrt{540}} = \frac{- 1}{2 \sqrt{15}} .$

Thus $𝜃 = arccos (- \frac{\sqrt{15}}{30}) \approx 1.7$ rad (approximately 97.4 degrees).

15.5 5.1 Exercise 17

For $𝑊 = span ((\begin{matrix} 1 \\ 2 \\ 3 \\ 4 \end{matrix}), (\begin{matrix} 5 \\ 6 \\ 7 \\ 8 \end{matrix}))$ , the equations for $𝑥 = (\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \\ 𝑥_{4} \end{matrix}) \in 𝑊^{⟂}$ reduce as

$(\begin{matrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{matrix}) \mapsto (\begin{matrix} 1 & 0 & - 1 & - 2 \\ 0 & 1 & 2 & 3 \end{matrix}) .$

Thus $𝑥_{1} = 𝑥_{3} + 2 𝑥_{4}$ , $𝑥_{2} = - 2 𝑥_{3} - 3 𝑥_{4}$ , and

$𝑊^{⟂} = {𝑟 (\begin{matrix} 1 \\ - 2 \\ 1 \\ 0 \end{matrix}) + 𝑠 (\begin{matrix} 2 \\ - 3 \\ 0 \\ 1 \end{matrix}) | 𝑟, 𝑠 \in ℝ} .$

15.6 5.1 Exercise 26

With $𝑢_{1} = \frac{1}{7} (\begin{matrix} 2 \\ 3 \\ 6 \end{matrix})$ and $𝑢_{2} = \frac{1}{7} (\begin{matrix} 3 \\ - 6 \\ 2 \end{matrix})$ ,

${proj}_{𝑊 (𝑥)} = (𝑢_{1} \cdot 𝑥) 𝑢_{1} + (𝑢_{2} \cdot 𝑥) 𝑢_{2}$

$= (2 + 3 + 6) (\begin{matrix} 2 \\ 3 \\ 6 \end{matrix}) + (3 - 6 + 2) (\begin{matrix} 3 \\ - 6 \\ 2 \end{matrix}) = (\begin{matrix} 19 \\ 35 \\ 64 \end{matrix}) .$

16 Part B

16.1 Problem 1

For ordered bases $𝐴, 𝐵, 𝐶$ , writing $𝐶 = (𝑐_{1}, 𝑐_{2}, \dots, 𝑐_{𝑛})$ and taking $𝑓 \in 𝑊$ ,

${[𝑓]}_{𝐴} = 𝑆_{𝐶 \to 𝐴} {[𝑓]}_{𝐶} = 𝑆_{𝐵 \to 𝐴} {[𝑓]}_{𝐵}, {[𝑓]}_{𝐵} = 𝑆_{𝐶 \to 𝐵} {[𝑓]}_{𝐶} .$

$𝑆_{𝐶 \to 𝐴} {[𝑓]}_{𝐶} = (𝑆_{𝐵 \to 𝐴} 𝑆_{𝐶 \to 𝐵}) {[𝑓]}_{𝐶} .$

Taking ${[𝑓]}_{𝐶} = 𝑒_{𝑖}$ makes corresponding columns equal; therefore

$𝑆_{𝐶 \to 𝐴} = 𝑆_{𝐵 \to 𝐴} 𝑆_{𝐶 \to 𝐵} .$

Consequently,

$𝑆_{𝐶 \to 𝐴} 𝑆_{𝐵 \to 𝐶} 𝑆_{𝐴 \to 𝐵} = 𝑆_{𝐵 \to 𝐴} 𝑆_{𝐶 \to 𝐵} 𝑆_{𝐵 \to 𝐶} 𝑆_{𝐴 \to 𝐵} = 𝑆_{𝐵 \to 𝐴} 𝐼_{𝑛} 𝑆_{𝐴 \to 𝐵} = 𝐼_{𝑛} .$

16.2 Problem 2

For $𝑓_{1} = sin (2 𝑥)$ , $𝑓_{2} = cos (2 𝑥)$ , $𝑓_{3} = 𝑒^{3 𝑥}$ and $ℬ︀ = (𝑓_{1}, 𝑓_{2}, 𝑓_{3})$ ,

${[𝐷 (𝑓_{1})]}_{ℬ︀} = (\begin{matrix} 0 \\ 2 \\ 0 \end{matrix}), {[𝐷 (𝑓_{2})]}_{ℬ︀} = (\begin{matrix} - 2 \\ 0 \\ 0 \end{matrix}), {[𝐷 (𝑓_{3})]}_{ℬ︀} = (\begin{matrix} 0 \\ 0 \\ 3 \end{matrix}),$

so ${[𝐷]}_{ℬ︀} = (\begin{matrix} 0 & - 2 & 0 \\ 2 & 0 & 0 \\ 0 & 0 & 3 \end{matrix})$ .

The submitted interpretation is that it rotates every vector in $ℝ^{2}$ counterclockwise by $\frac{𝜋}{2}$ , stretches the $𝑥, 𝑦$ coordinates to twice their length, and the $𝑧$ coordinate to three times its length.

16.3 Problem 3

Let $𝐵 = {[𝑇]}_{ℬ︀}$ and $𝐶 = {[𝑇]}_{𝒞︀}$ . By the change-of-basis theorem, $𝐶 = 𝑆^{- 1} 𝐵 𝑆$ , where $𝑆 = 𝑆_{𝐶 \to 𝐵}$ . The submission proves by induction that

$𝐵^{𝑘} = 𝑆^{- 1} 𝐶^{𝑘} 𝑆$

for each integer $𝑘 \geq 1$ . The inductive step is

$𝐵^{𝑘 + 1} = 𝐵^{𝑘} 𝐵 = (𝑆^{- 1} 𝐶^{𝑘} 𝑆) (𝑆^{- 1} 𝐶 𝑆) = 𝑆^{- 1} 𝐶^{𝑘 + 1} 𝑆 .$

Thus $𝐵^{𝑘}$ and $𝐶^{𝑘}$ are similar.

For the false kernel claim, it takes

$𝐴 = (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{matrix}),$

$ℰ︀ = ((\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix})),$

$ℬ︀ = ((\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix})) .$

It records

${[𝐴]}_{ℰ︀} = 𝐴, {[𝐴]}_{ℬ︀} = (\begin{matrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 1 & 1 \end{matrix}),$

but

$ker ({[𝐴]}_{ℰ︀}) = {𝑟 (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) | 𝑟 \in ℝ},$

$ker ({[𝐴]}_{ℬ︀}) = {𝑟 (\begin{matrix} 0 \\ 1 \\ - 1 \end{matrix}) | 𝑟 \in ℝ} .$

For equal nullities, rank-nullity gives

$dim (ker 𝐵) = 𝑛 - rank (𝐵), dim (ker 𝐶) = 𝑛 - rank (𝐶) .$

The page proves $rank (𝑀 𝑁) \leq rank (𝑁)$ from $ker 𝑁 \subseteq ker (𝑀 𝑁)$ , the analogous bound for $𝑀$ by transposition, and equality $rank (𝑀 𝑁) = rank (𝑁)$ when $𝑀$ is invertible. Since $𝐵 = 𝑆^{- 1} 𝐶 𝑆$ , $rank (𝐵) = rank (𝐶)$ , so the kernel dimensions agree.

16.4 Problem 4

For $𝑇 : 𝑈 \to 𝑊$ , bases $ℬ︀ = (𝑢_{1}, \dots, 𝑢_{𝑘})$ and $𝒞︀ = (𝑤_{1}, \dots, 𝑤_{𝑑})$ , define

$𝑇^{'} ({[𝑢]}_{ℬ︀}) = {[𝑤]}_{𝒞︀} whenever 𝑇 (𝑢) = 𝑤 .$

The source diagram verifies

$𝑇^{'} \circ 𝐿_{ℬ︀} (𝑢) = 𝑇^{'} ({[𝑢]}_{ℬ︀}) = {[𝑤]}_{𝒞︀} = 𝐿_{𝒞︀} \circ 𝑇 (𝑢),$

hence $𝑇^{'} \circ 𝐿_{ℬ︀} = 𝐿_{𝒞︀} \circ 𝑇$ . Thus

${[𝑇 (𝑢)]}_{𝒞︀} = {[𝑇]}_{ℬ︀, 𝒞︀} {[𝑢]}_{ℬ︀} .$

For $𝑢 = 𝑎_{1} 𝑢_{1} + \dots + 𝑎_{𝑘} 𝑢_{𝑘}$ ,

${[𝑇 (𝑢)]}_{𝒞︀} = 𝑎_{1} {[𝑇 (𝑢_{1})]}_{𝒞︀} + \dots + 𝑎_{{𝑘 [𝑇 (𝑢_{𝑘})]}_{𝒞︀}},$

and therefore

${[𝑇]}_{ℬ︀, 𝒞︀} = ({[𝑇 (𝑢_{1})]}_{𝒞︀} {[𝑇 (𝑢_{2})]}_{𝒞︀} \dots {[𝑇 (𝑢_{𝑘})]}_{𝒞︀}) .$

16.5 Problem 5

For $𝑓_{1} = sin 𝑥$ , $𝑓_{2} = cos 𝑥$ , $𝑓_{3} = 𝑒^{𝑥}$ :

$𝑇 (𝑓_{1}) = 𝑥 - \frac{𝑥^{3}}{6}, 𝑇 (𝑓_{2}) = 1 - \frac{𝑥^{2}}{2},$

$𝑇 (𝑓_{3}) = 1 + 𝑥 + \frac{𝑥^{2}}{2} + \frac{𝑥^{3}}{6} .$

The source chooses $𝒞︀ = (1, 𝑥, \frac{𝑥^{2}}{2}, \frac{𝑥^{3}}{6})$ . For $ℬ︀ = (𝑓_{1} + 𝑓_{2}, 𝑓_{1} - 𝑓_{2}, 𝑓_{3} + 𝑓_{1})$ it computes

${[𝑇 (𝑓_{1} + 𝑓_{2})]}_{𝒞︀} = (\begin{matrix} 1 \\ 1 \\ - 1 \\ - 1 \end{matrix}),$

${[𝑇 (𝑓_{1} - 𝑓_{2})]}_{𝒞︀} = (\begin{matrix} - 1 \\ 1 \\ 1 \\ - 1 \end{matrix}),$

${[𝑇 (𝑓_{3} + 𝑓_{1})]}_{𝒞︀} = (\begin{matrix} 1 \\ 2 \\ 1 \\ 0 \end{matrix}),$

${[𝑇]}_{ℬ︀, 𝒞︀} = (\begin{matrix} 1 & - 1 & 1 \\ 1 & 1 & 2 \\ - 1 & 1 & 1 \\ - 1 & - 1 & 0 \end{matrix}) .$

16.6 Problem 6

Let $𝐴 = (\begin{matrix} - 6 & - 30 \\ - 30 & 19 \end{matrix})$ and $𝑉 = span ((\begin{matrix} 3 \\ 2 \end{matrix}))$ . For $𝑣 = 𝑎 (\begin{matrix} 3 \\ 2 \end{matrix})$ ,

$𝐴 𝑣 = 𝑎 (\begin{matrix} - 78 \\ - 52 \end{matrix}) = - 26 𝑎 (\begin{matrix} 3 \\ 2 \end{matrix}) \in 𝑉 .$

$𝑉^{⟂} = {𝑟 (\begin{matrix} - 2 \\ 3 \end{matrix}) | 𝑟 \in ℝ}$ , and

$𝐴 (\begin{matrix} - 2 𝑟 \\ 3 𝑟 \end{matrix}) = 𝑟 (\begin{matrix} - 78 \\ 117 \end{matrix}) = - 39 𝑟 (\begin{matrix} - 2 \\ 3 \end{matrix}) \in 𝑉^{⟂} .$

With $ℬ︀ = ((\begin{matrix} - 2 \\ 3 \end{matrix}), (\begin{matrix} 3 \\ 2 \end{matrix}))$ ,

${[𝑇]}_{ℬ︀} = (\begin{matrix} 26 & 0 \\ 0 & - 39 \end{matrix}), {[𝑇^{10}]}_{ℬ︀} = (\begin{matrix} 26^{10} & 0 \\ 0 & 39^{10} \end{matrix}) .$

The source uses

$𝑆_{ℬ︀ \to ℰ︀} = (\begin{matrix} - 2 & 3 \\ 3 & 2 \end{matrix}), 𝑆_{ℬ︀ \to ℰ︀}^{- 1} = (\begin{matrix} \frac{2}{13} & - \frac{3}{13} \\ - \frac{3}{13} & - \frac{2}{13} \end{matrix})$

to obtain

${[𝑇^{10}]}_{ℰ︀} = (\begin{matrix} \frac{4 \cdot 26^{10} + 9 \cdot 39^{10}}{13} & \frac{- 6 \cdot 26^{10} + 6 \cdot 39^{10}}{13} \\ \frac{- 6 \cdot 26^{10} + 6 \cdot 39^{10}}{13} & \frac{9 \cdot 26^{10} + 4 \cdot 39^{10}}{13} \end{matrix}) .$

16.7 Homework 8 — submitted work

17 Part A

The assigned book exercises are 5.1: 45; 5.2: 14, 26; 5.3: 36; and 5.4: 26, 32.

17.1 5.1 Exercise 45

For $𝐴 = (\begin{matrix} 3 & 5 & 11 \\ 5 & 9 & 20 \\ 11 & 20 & 49 \end{matrix})$ and $𝑉 = span (\vec{𝑣_{2}}, \vec{𝑣_{3}})$ , put ${proj}_{𝑉 (\vec{𝑣_{1}})} = 𝑐_{2} \vec{𝑣_{2}} + 𝑐_{3} \vec{𝑣_{3}}$ . Orthogonality gives

$9 𝑐_{2} + 20 𝑐_{3} = 5, 20 𝑐_{2} + 49 𝑐_{3} = 11,$

so $𝑐_{2} = \frac{25}{41}$ , $𝑐_{3} = - \frac{1}{41}$ , and

${proj}_{𝑉 (\vec{𝑣_{1}})} = \frac{25}{41} \vec{𝑣_{2}} - \frac{1}{41} \vec{𝑣_{3}} .$

17.2 5.2 Exercise 14

For $(\begin{matrix} 1 \\ 7 \\ 1 \\ 7 \end{matrix}), (\begin{matrix} 0 \\ 7 \\ 2 \\ 7 \end{matrix}), (\begin{matrix} 1 \\ 8 \\ 1 \\ 6 \end{matrix})$ , the submitted Gram–Schmidt calculation gives

$𝑢_{1} = \frac{1}{10} (\begin{matrix} 1 \\ 7 \\ 1 \\ 7 \end{matrix}), 𝑢_{2} = \frac{1}{\sqrt{2}} (\begin{matrix} - 1 \\ 0 \\ 1 \\ 0 \end{matrix}), 𝑢_{3} = \frac{1}{\sqrt{2}} (\begin{matrix} 0 \\ 1 \\ 0 \\ - 1 \end{matrix}) .$

It states that $(𝑢_{1}, 𝑢_{2}, 𝑢_{3})$ is the orthonormal basis.

17.3 5.2 Exercise 26

For $𝑚_{1} = (\begin{matrix} 2 \\ 3 \\ 0 \\ 6 \end{matrix})$ and $𝑚_{2} = (\begin{matrix} 4 \\ 4 \\ 2 \\ 13 \end{matrix})$ ,

$𝑢_{1} = \frac{1}{7} (\begin{matrix} 2 \\ 3 \\ 0 \\ 6 \end{matrix}), 𝑢_{2} = \frac{1}{3} (\begin{matrix} 0 \\ - 2 \\ 2 \\ 1 \end{matrix}),$

and

$𝑄 = (\begin{matrix} \frac{2}{7} & 0 \\ \frac{3}{7} & - \frac{2}{3} \\ 0 & \frac{2}{3} \\ \frac{6}{7} & \frac{1}{3} \end{matrix}), 𝑅 = (\begin{matrix} 7 & 14 \\ 0 & 3 \end{matrix}) .$

17.4 5.3 Exercise 36

For $(\begin{matrix} \frac{2}{3} & \frac{1}{\sqrt{2}} & 𝑎 \\ \frac{2}{3} & - \frac{1}{\sqrt{2}} & 𝑏 \\ \frac{1}{3} & 0 & 𝑐 \end{matrix})$ , the roles must be orthonormal. The page solves

$\frac{4}{9} - \frac{1}{2} + 𝑎 𝑏 = 0, \frac{2}{9} + 𝑎 𝑐 = 0, \frac{2}{9} + 𝑏 𝑐 = 0,$

giving $𝑎 = 𝑏 = \frac{\sqrt{2}}{6}$ and $𝑐 = - 2 \frac{\sqrt{2}}{3}$ .

17.5 5.4 Exercise 26

For $𝐴 = (\begin{matrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{matrix})$ , $𝑏 = (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix})$ , the normal equation is

$(\begin{matrix} 66 & 78 & 90 \\ 78 & 93 & 108 \\ 90 & 108 & 126 \end{matrix}) 𝑥 = (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) .$

Its reduced echelon form is

$(\begin{matrix} 1 & 0 & - 1 & \frac{7}{6} \\ 0 & 1 & 2 & 1 \\ 0 & 0 & 0 & 0 \end{matrix}),$

$𝑥^{*} = (\begin{matrix} - \frac{7}{6} + 𝑡 \\ 1 - 2 𝑡 \\ 𝑡 \end{matrix}) = (\begin{matrix} - \frac{7}{6} \\ 1 \\ 0 \end{matrix}) + 𝑡 (\begin{matrix} 1 \\ - 2 \\ 1 \end{matrix}) .$

17.6 5.4 Exercise 32

For $(0, 27), (1, 0), (2, 0), (3, 0)$ , let $𝑓 (𝑥) = 𝑐_{0} + 𝑐_{1} 𝑥 + 𝑐_{2} 𝑥^{2}$ , with

$𝐴 = (\begin{matrix} 1 & 0 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \end{matrix}), 𝑏 = (\begin{matrix} 27 \\ 0 \\ 0 \\ 0 \end{matrix}) .$

The source gives

$(\begin{matrix} 4 & 6 & 14 \\ 6 & 14 & 36 \\ 14 & 36 & 98 \end{matrix}) 𝑥 = (\begin{matrix} 27 \\ 0 \\ 0 \end{matrix}),$

$𝑥^{*} = (\begin{matrix} \frac{513}{20} \\ - \frac{567}{20} \\ \frac{21}{4} \end{matrix}),$

and

$𝑓^{*} (𝑥) = \frac{513}{20} - \frac{567}{20} 𝑥 + \frac{21}{4} 𝑥^{2} = 25.65 - 28.35 𝑥 + 6.25 𝑥^{2} .$

18 Part B

18.1 Problem 1

For $𝜋 (𝑣) = \sum_{𝑖 = 1}^{𝑑} \frac{𝑣 \cdot 𝑣_{𝑖}}{𝑣_{𝑖} \cdot 𝑣_{𝑖}} 𝑣_{𝑖}$ :

18.1.1 (a)

If $𝑣_{𝑖} \cdot 𝑣_{𝑗} = 0$ for $𝑖 \neq 𝑗$ , then $(\frac{𝑣_{1}}{‖ 𝑣_{1} ‖}, \dots, \frac{𝑣_{𝑑}}{‖ 𝑣_{𝑑} ‖})$ is an orthonormal basis. Writing $𝑢_{𝑖} = \frac{𝑣_{𝑖}}{‖ 𝑣_{𝑖} ‖}$ ,

$𝜋 (𝑣) = \sum_{𝑖 = 1}^{𝑑} \frac{𝑣 \cdot 𝑣_{𝑖}}{{‖ 𝑣_{𝑖} ‖}^{2}} 𝑣_{𝑖} = \sum_{𝑖 = 1}^{𝑑 (𝑣 \cdot 𝑢_{𝑖})} 𝑢_{𝑖},$

so $𝜋$ is the orthogonal projection onto $𝑊$ .

18.1.2 (b)

If the basis is not perpendicular, some $𝑣_{𝑖} \cdot 𝑣_{𝑗} = 𝑎 \neq 0$ . While ${proj}_{𝑊 (𝑣_{𝑖})} = 𝑣_{𝑖}$ ,

$𝜋 (𝑣_{𝑖}) = 𝑣_{𝑖} + \dots + \frac{𝑎}{{‖ 𝑣_{𝑗} ‖}^{2}} 𝑣_{𝑗} + \dots \neq 𝑣_{𝑖}$

by linear independence. Thus $𝜋$ is not ${proj}_{𝑊}$ .

18.2 Problem 2

For the set $𝑂_{𝑛}$ of orthogonal matrices:

(a) False. $𝐼_{𝑛} \in 𝑂_{𝑛}$ , but $𝐼_{𝑛} + 𝐼_{𝑛}$ is not orthogonal; the page gives ${(𝐼_{𝑛} + 𝐼_{𝑛})}^{𝑇 (𝐼_{𝑛} + 𝐼_{𝑛})} = (\begin{matrix} 2 & 0 \\ 0 & 2 \end{matrix})$ , whereas its inverse is $(\begin{matrix} \frac{1}{2} & 0 \\ 0 & \frac{1}{2} \end{matrix})$ .
(b) True. The composition of the orthogonal maps represented by $𝐴, 𝐵$ has standard matrix $𝐴 𝐵$ , so $𝐴 𝐵 \in 𝑂_{𝑛}$ .
(c) True by the same composition argument for $𝐴^{2}$ .
(d) True. If $𝐴^{2}$ were orthogonal but $𝐴$ were not, with $‖ 𝑇_{𝐴 (𝑥)} ‖ = 𝑎 ‖ 𝑥 ‖$ and $𝑎 \neq 1$ , then $‖ 𝑇_{𝐴} \circ 𝑇_{𝐴 (𝑥)} ‖ = 𝑎^{2} ‖ 𝑥 ‖ \neq ‖ 𝑥 ‖$ , a contradiction.
(e) True. $𝐴 \in 𝑂_{𝑛}$ gives $𝐴^{𝑇} = 𝐴^{- 1}$ ; $𝐴^{2} = 𝐼_{𝑛}$ gives $𝐴 = 𝐴^{- 1} = 𝐴^{𝑇}$ , so $𝐴$ is symmetric.

18.3 Problem 3

For an orthonormal basis $ℬ︀$ ,

$𝑣 = 𝑣_{1} 𝑏_{1} + \dots + 𝑣_{𝑟} 𝑏_{𝑟}, 𝑤 = 𝑤_{1} 𝑏_{1} + \dots + 𝑤_{𝑟} 𝑏_{𝑟},$

and the calculation on the page is

$𝑣 \cdot 𝑤 = \sum_{𝑖 = 1}^{𝑟} 𝑣_{𝑖} 𝑤_{𝑖} = {[𝑣]}_{ℬ︀} \cdot {[𝑤]}_{ℬ︀} .$

For two orthonormal bases $ℬ︀, 𝒞︀$ , $𝑆_{𝒞︀ \to ℬ︀}$ has $(𝑖, 𝑗)$ entry $𝑐_{𝑗} \cdot 𝑏_{𝑖}$ and $𝑆_{ℬ︀ \to 𝒞︀}$ has transpose entry $𝑏_{𝑖} \cdot 𝑐_{𝑗}$ . Hence

$𝑆_{𝒞︀ \to ℬ︀} = 𝑆_{ℬ︀ \to 𝒞︀}^{𝑇} = 𝑆_{ℬ︀ \to 𝒞︀}^{- 1},$

so $𝑆_{ℬ︀ \to 𝒞︀}$ is orthogonal.

18.4 Problem 4

The submitted answers are:

(a) True. ${(ker 𝐴)}^{⟂} = im 𝐴^{𝑇}$ , from $ker (𝐴^{𝑇}) = {(im 𝐴)}^{⟂}$ and double orthogonal complement.
(b) True. $ker 𝐴 = ker (𝐴^{𝑇} 𝐴)$ , then rank-nullity gives $rank 𝐴 = rank (𝐴^{𝑇} 𝐴)$ .
(c) True. $ker (𝐴^{𝑇}) = {(im 𝐴)}^{⟂}$ and rank-nullity yield $rank 𝐴 = rank (𝐴^{𝑇})$ .
(d) True. With (b), (c), and ${(𝐴^{𝑇})}^{𝑇} = 𝐴$ , $rank (𝐴^{𝑇} 𝐴) = rank (𝐴 𝐴^{𝑇})$ .
(e) False. $dim (ker 𝐴) = 𝑚 - rank 𝐴$ but $dim (ker 𝐴 𝐴^{𝑇}) = 𝑛 - rank 𝐴$ ; if $𝑛 \neq 𝑚$ they cannot be equal.

18.5 Problem 5

18.5.1 (a)

For $𝑋 = {𝑥_{1}, \dots, 𝑥_{𝑟}}$ , $𝑌 = {𝑦_{1}, \dots, 𝑦_{𝑠}}$ and $𝑋 ⟂ 𝑌$ ,

$𝑥 = \sum 𝑎_{𝑖} 𝑥_{𝑖}, 𝑦 = \sum 𝑏_{𝑗} 𝑦_{𝑗}$

implies $𝑥 \cdot 𝑦 = 0$ , because every $𝑥_{𝑖} \cdot 𝑦_{𝑗} = 0$ . Thus $span 𝑋 ⟂ span 𝑌$ .

18.5.2 (b)

For a relation

$\sum 𝑎_{𝑖} 𝑥_{𝑖} + \sum 𝑏_{𝑗} 𝑦_{𝑗} = 0,$

if coefficients on both sides are nonzero then the equal nonzero vectors would belong to $span 𝑋 \cap span 𝑌$ . But (a) and WS 16 give this intersection as $0$ . Hence all coefficients vanish and $𝑋 \cup 𝑌$ is linearly independent.

18.5.3 (c)

A pairwise orthogonal set with fewer than $𝑛 + 1$ members is not maximal: append $0$ , use Gram–Schmidt, and add a new unit vector. A set with more than $𝑛 + 1$ members would make $𝑌 \cup {0}$ linearly independent in $ℝ^{𝑛}$ , which is impossible. Therefore a maximal pairwise orthogonal set has exactly $𝑛 + 1$ elements.

18.6 Problem 6

18.6.1 (a)

The source applies Gram–Schmidt to full-rank $𝐴$ to obtain an orthonormal basis, reverses the chosen basis, and uses a QR factorization. It then sets

It takes the orthonormal basis matrix and the transpose of the upper factor, obtaining the stated QL factorization. The resulting lower factor is triangular with positive diagonal.

18.6.2 (b)

The finished submission stops after the construction for (a); no visible proof for part (b) appears in the source PDF.

18.7 Homework 9 — submitted work

19 Part A

The assigned exercises are 5.4: 27, 31; and 5.5: 15, 23, 32(a–d).

19.1 5.4 Exercise 27

The work observes that $𝑆^{⟂}$ means the least-squares error is orthogonal to $𝑆$ . Thus, for the displayed vector, the least-squares solution is unchanged:

$𝑥^{=} (\begin{matrix} 7 \\ 11 \end{matrix}) .$

19.2 5.4 Exercise 31

For the three points $(0, 3), (1, 3), (1, 6)$ , the line of best fit uses

$𝐴 = (\begin{matrix} 1 & 0 \\ 1 & 1 \\ 1 & 1 \end{matrix}), 𝑏 = (\begin{matrix} 3 \\ 3 \\ 6 \end{matrix}) .$

The normal equations recorded in the submission are

$(\begin{matrix} 3 & 2 \\ 2 & 2 \end{matrix}) (\begin{matrix} 𝑐_{0} \\ 𝑐_{1} \end{matrix}) = (\begin{matrix} 12 \\ 9 \end{matrix}),$

and solving them gives $𝑐_{0} = 3$ and $𝑐_{1} = \frac{3}{2}$ . Hence the submitted line is

$𝑓 (𝑥) = 3 + 3 \frac{𝑥}{2} .$

19.3 5.5 Exercise 15

For the bilinear expression in the exercise, symmetry requires $𝑏 = 𝑐$ . The submitted positive-definiteness test yields the additional condition

$𝑑 > 𝑏^{2} .$

19.4 5.5 Exercise 23

With

$∠ (𝑓, 𝑔) = \frac{1}{2} (𝑓 (0) 𝑔 (0) + 𝑓 (1) 𝑔 (1)),$

the answer verifies the inner-product properties and gives the orthonormal basis

$1, 2 𝑥 - 1 .$

19.5 5.5 Exercise 32

For the weighted integral inner product

$∠ (𝑓, 𝑔) = \frac{1}{2} \int_{- 1}^{1} 𝑓 (𝑡) 𝑔 (𝑡) d 𝑡,$

the computation in the submitted pages records

when $𝑛 + 𝑚$ is even, and

$∠ (𝑡^{𝑛}, 𝑡^{𝑚}) = \frac{1}{𝑛 + 𝑚 + 1}$

when $𝑛 + 𝑚$ is odd. Also $‖ 𝑡^{𝑛} ‖ = \sqrt{\frac{1}{2 𝑛 + 1}}$ . Applying Gram–Schmidt, it writes

$𝑔_{0} = 1, 𝑔_{1} = \sqrt{3} 𝑡, 𝑔_{2} = \sqrt{\frac{5}{2}} (3 𝑡^{2} - 1),$

$𝑔_{3} = \frac{1}{\sqrt{\frac{4}{175}}} (𝑡^{3} - \frac{3}{5} 𝑡) .$

The associated polynomial sequence is recorded as

$1, 𝑡, \frac{3 𝑡^{2} - 1}{2}, \frac{5 𝑡^{3} - 3 𝑡}{2} .$

20 Part B

20.1 Problem 1

The submitted solution finds the plane of best fit for the three displayed points by writing its normal-equation system. Its computed coefficient vector is

$(\begin{matrix} \frac{7}{3} \\ 1 \\ - \frac{8}{3} \end{matrix}),$

which is also plotted on the graph in the original submission.

20.2 Problem 2

20.2.1 (a)

For (i), the proposed expression is not an inner product: the work uses

$𝑓 (𝑥) = 𝑥^{2} - 4 𝑥 + 3,$

which is nonzero while $𝑓 (1) = 𝑓 (3) = 0$ . For (ii), it verifies the nonnegativity and definiteness of the sum-of-squares evaluation expression, and concludes that it is an inner product.

20.2.2 (b)

For (i), the weighted integral with weight $𝑥$ fails positive definiteness; the submission gives $𝑓 (𝑥) = sin (𝑥)$ as the counterexample. For (ii), the weight $𝑥^{2}$ gives the required symmetric, bilinear, positive-definite inner product.

20.3 Problem 3

Let

$∠ (𝑓, 𝑔) = \int_{- \frac{𝜋}{2}}^{\frac{𝜋}{2}} 𝑓 (𝑥) 𝑔 (𝑥) {sin}^{2} (𝑥) d 𝑥 .$

20.3.1 (a)

The work finds $∠ (1, 𝑥) = 0$ and $‖ 1 ‖ = \sqrt{\frac{𝜋}{2}}$ , then evaluates the remaining displayed polynomial inner products to prepare Gram–Schmidt.

20.3.2 (b)

The submitted orthonormalized functions are recorded approximately as

$𝑢_{1} = \frac{2}{\sqrt{𝜋}}, 𝑢_{2} = 0.974 𝑥, 𝑢_{3} = \frac{𝑥^{2} - 4.435}{15.8376} .$

20.3.3 (c)

For $𝑓 (𝑥) = 𝑒^{𝑥}$ , the work records

${proj}_{𝑊 (𝑓)} = 1.7521 𝑢_{1} + 0.2492 𝑢_{2} - 8.1697 𝑢_{3}$

and, after expansion,

${proj}_{𝑊 (𝑓)} \approx 3.3472 + 0.0236 𝑥 - 0.5142 𝑥^{2} .$

20.4 Homework 10 — submitted work

21 Part A

The assigned exercises are 6.1: 20, 54; 6.2: 42, 50; 6.3: 14; and 7.1: 12, 18, 42.

21.1 6.1 Exercise 20

By the row operations shown in the submission, the determinant of the given $𝑘 \times 𝑘$ matrix reduces to

$det (𝐴) = 1$

for every $𝑘$ .

21.2 6.1 Exercise 54

The answer uses positivity to conclude that the determinant of the displayed $5 \times 5$ matrix is positive, and records

$det (𝐴) = 1000 .$

21.3 6.2 Exercise 42

For a QR factorization $𝐴 = 𝑄 𝑅$ , the submission writes

$det (𝐴^{𝑇} 𝐴) = det (𝑅^{𝑇} 𝑄^{𝑇} 𝑄 𝑅) = {det (𝑅)}^{2},$

so the determinant is positive.

21.4 6.2 Exercise 50

For the matrix whose $(𝑖, 𝑗)$ entry is $min (𝑖, 𝑗)$ , repeated determinant reduction gives

$det (𝐴) = 1 .$

21.5 6.3 Exercise 14

The parallelepiped volume is found from the determinant. Since the displayed vectors are linearly dependent, the result is

$volume = 0 .$

21.6 7.1 Exercise 12

For

$𝐴 = (\begin{matrix} 2 & 0 \\ 3 & 4 \end{matrix}),$

the characteristic equation gives eigenvalues $2$ and $4$ . The corresponding eigenvector directions in the work are

$𝜆 = 2 : (\begin{matrix} - \frac{2}{3} \\ 1 \end{matrix}), 𝜆 = 4 : (\begin{matrix} 0 \\ 1 \end{matrix}) .$

Thus it diagonalizes to $𝐷 = (\begin{matrix} 2 & 0 \\ 0 & 4 \end{matrix})$ .

21.7 7.1 Exercise 18

For reflection in a plane, the plane is the $1$ -eigenspace and has dimension two; its normal direction is the $- 1$ -eigenspace. With an eigenbasis, the submitted diagonal form is

$(\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & - 1 \end{matrix}) .$

21.8 7.1 Exercise 42

The matrices in $𝑉$ are written as

$(\begin{matrix} 𝑎 & 𝑏 & 0 \\ 0 & 𝑐 & 0 \\ 0 & 𝑑 & 𝑒 \end{matrix}) .$

The five matrix units in positions $(1, 1), (1, 2), (2, 2), (3, 2), (3, 3)$ form the submitted basis, so

$dim (𝑉) = 5 .$

22 Part B

22.1 Problem 1

The proof shows that an alternating bilinear form $𝐹$ is antisymmetric:

$𝐹 (𝑢 + 𝑣, 𝑢 + 𝑣) = 0 = 𝐹 (𝑢, 𝑣) + 𝐹 (𝑣, 𝑢) .$

Conversely, antisymmetry gives $𝐹 (𝑢, 𝑢) = 0$ . Since $𝐹 (𝑒_{1}, 𝑒_{2}) = 1$ , bilinearity then identifies the form with the determinant:

$𝐹 ((\begin{matrix} 𝑎 \\ 𝑏 \end{matrix}), (\begin{matrix} 𝑐 \\ 𝑑 \end{matrix})) = 𝑎 𝑑 - 𝑏 𝑐 .$

22.2 Problem 2

Let $𝑀 = (\begin{matrix} 𝑎 & 𝑏 \\ 𝑐 & 𝑑 \end{matrix})$ and let $𝑇 (𝐴) = 𝐴 𝑀$ . The submission verifies linearity. In the ordered basis

$ℰ︀ = (𝐸_{11}, 𝐸_{12}, 𝐸_{21}, 𝐸_{22}),$

it computes

${[𝑇]}_{ℰ︀} = (\begin{matrix} 𝑎 & 𝑐 & 0 & 0 \\ 𝑏 & 𝑑 & 0 & 0 \\ 0 & 0 & 𝑎 & 𝑐 \\ 0 & 0 & 𝑏 & 𝑑 \end{matrix}) .$

The determinant calculation is

$det ({[𝑇]}_{ℰ︀}) = {(𝑎 𝑑 - 𝑏 𝑐)}^{2},$

which agrees with the determinant in any other basis. Finally, for

$𝑀 = (\begin{matrix} 2 & 1 \\ 0 & 2 \end{matrix}),$

the only eigenvalue is $2$ , with an eigenspace of dimension $2 < 4$ for the induced map; therefore the submitted conclusion is that $𝑇$ is not diagonalizable.

22.3 Problem 3

The construction defines $𝑧$ by the determinant/cross-product functional in $ℝ^{4}$ . For the standard choice $𝑢 = 𝑒_{1}$ , $𝑣 = 𝑒_{2}$ , $𝑤 = 𝑒_{3}$ , the work finds

$𝑧 = - 𝑒_{4} .$

It proves that $𝑧 = 0$ exactly when $𝑢, 𝑣, 𝑤$ are linearly dependent, that $𝑧$ is orthogonal to each of $𝑢, 𝑣, 𝑤$ , and that

$det (𝑧, 𝑢, 𝑣, 𝑤) = {‖ 𝑧 ‖}^{2} .$

22.4 Problem 4

The response uses the characteristic polynomial to find eigenvalues, and uses similarity to preserve the polynomial. It then applies the displayed eigenvector criterion to decide diagonalizability.

22.5 Problem 5

For a $2 \times 2$ matrix satisfying $𝐴^{2} = 𝐼$ , the submission separates the $+ 1$ and $- 1$ eigenvector cases and obtains a diagonal form. The final argument extends this to every dimension by decomposing an arbitrary vector as

$𝑣 = \frac{1}{2} (𝑣 + 𝐴 𝑣) + \frac{1}{2} (𝑣 - 𝐴 𝑣),$

where the two summands lie in the $1$ - and $- 1$ -eigenspaces respectively. Thus $𝐴$ is diagonalizable.

23 Review on Basic Concepts

23.1 Subspace and direct sum

Definition 23.15 : Subspace

vector space 的 subset $𝑈 \subset 𝑉$ 为一个 subspace，if 它满足条件：

包含 0
对 addition 和 scalar multiplication 闭合

两个 subset 的和就是各取一个元素相加的所有情况.
很显然我们知道：

Proposition 23.1

两个 subspace $𝑈_{1}, 𝑈_{2}$ 的 sum $𝑈_{1} + 𝑈_{2}$ 也是一个 subspace, 并且

dim (𝑈_{1} + 𝑈_{2}) \leq dim (𝑈_{1}) + dim (𝑈_{2})

且 $𝑈_{1} + 𝑈_{2}$ 是同时包含 $𝑈_{1}$ 和 $𝑈_{2}$ 的 $𝑉$ 的最小 subspace.

显然可以随便和。同一个 $𝑈$ 自己和自己的和就是自己。所以 subspace sum 这个概念比较大，没什么用。我们需要用 direct sum 来作为一个小一点但是更有用的概念，表达出一种垂直的 subspace 的直观.

Definition 23.16 : Direct sum

如果

𝑈_{1} + 𝑈_{2} + \dots + 𝑈_{𝑚}

中的任意元素

𝑣

，都存在唯一的

𝑣_{𝑘} \in 𝑈_{𝑘}

for each

𝑘

使得

𝑣 = \sum_{𝑘} 𝑣_{𝑘}

，就称

𝑈_{1} + \dots + 𝑈_{𝑚} = \oplus_{𝑖 = 1}^{𝑚} 𝑈_{𝑖}

为一个 direct sum.

我们显然发现：

Proposition 23.2

dim (\oplus_{𝑖 = 1}^{𝑚} 𝑈_{𝑖}) = \sum_{𝑖 = 1}^{𝑚} dim (𝑈_{𝑖})

我们发现，其实可以 direct sum 的 subspaces 是 “垂直的”，意思是:

Theorem 23.26

𝑈_{1} + 𝑈_{2} + \dots + 𝑈_{𝑚}

是一个 direct sum (这几个空间“垂直”) iff 任取

𝑢_{1}, 𝑢_{2}, \dots, 𝑢_{𝑚}

分别来自

𝑈_{1}, 𝑈_{2}, \dots, 𝑈_{𝑚}

，它们都 lin. ind.

并且：

Theorem 23.27

𝑈_{1} + 𝑈_{2}

为一个 direct sum iff

𝑈_{1} \cap 𝑈_{2} = {0}

24 Linear functional and Duality

Definition 24.17 : Linear functional

25 Eigenvalues and Operators

26 Operators on complex VS

27 Multilinear Algebra

28 tensor product and matrix multiplication

这里我们放弃陈述两个 over 同一 field 的 vector spaces 的 tensor product 的 algebraic definition，直接看应用的。(完整的 definition 是两个 over 同一个 ring 的 modules $𝐴, 𝐵$ ，取它们 free abelian group generated by $𝐴 \times 𝐵$ ，再 quotient 掉一个用来形成 bilinearity 的 subgroup，就是它们的 tensor product。当这两个东西是 vector spaces 时，它们并且是 isomorphic to 其对应的 bilinear functional vector space的。)

Definition 28.18 : (not rigorous) Tensor product

令 $𝑉, 𝑊$ be vector spaces over $𝔽$ ，我们定义一个 vector 与 vector 之间的 tensor product operation：

$\otimes : (𝑣, 𝑤) \to 𝑣 \otimes 𝑤$

s.t. 对于 $𝑉, 𝑊$ 的 basis $𝑒_{1}, \dots, 𝑒_{𝑛}$ 和 $𝜀_{1}, \dots, 𝜀_{𝑚}$ ，我们给每个 $(𝑒_{𝑖}, 𝜀_{𝑗})$ 都赋予一个不同的 image $𝑒_{𝑖} \otimes 𝜀_{𝑗}$ ，其 over $𝔽$ 具有 bilinear 性。

对于

$𝑉 \otimes 𝑊 ≔ span ({𝑒_{𝑖} \otimes 𝑒_{𝑗} : 1 \leq 𝑖 \leq 𝑛, 1 \leq 𝑗 \leq 𝑚})$

$𝑉 \otimes 𝑊$ 也是一个 over $𝔽$ 的 vector space，且 $dim (𝑉 \otimes 𝑊) = 𝑚 𝑛$ 。

Proposition 28.3 : representing tensor product as matrix

如果我们有两个矩阵：

$𝐴 = (\begin{matrix} 𝑎_{11} & 𝑎_{12} \\ 𝑎_{21} & 𝑎_{22} \end{matrix}), 𝐵 = (\begin{matrix} 𝑏_{11} & 𝑏_{12} \\ 𝑏_{21} & 𝑏_{22} \end{matrix})$

可以把它们的 tensor product $𝐴 \otimes 𝐵$ 表示为：

$𝐴 \otimes 𝐵 = (\begin{matrix} 𝑎_{11} 𝐵 & 𝑎_{12} 𝐵 \\ 𝑎_{21} 𝐵 & 𝑎_{22} 𝐵 \end{matrix}) = (\begin{matrix} 𝑎_{11} 𝑏_{11} & 𝑎_{11} 𝑏_{12} & 𝑎_{12} 𝑏_{11} & 𝑎_{12} 𝑏_{12} \\ 𝑎_{11} 𝑏_{21} & 𝑎_{11} 𝑏_{22} & 𝑎_{12} 𝑏_{21} & 𝑎_{12} 𝑏_{22} \\ 𝑎_{21} 𝑏_{11} & 𝑎_{21} 𝑏_{12} & 𝑎_{22} 𝑏_{21} & 𝑎_{22} 𝑏_{22} \\ 𝑎_{21} 𝑏_{21} & 𝑎_{21} 𝑏_{22} & 𝑎_{22} 𝑏_{21} & 𝑎_{22} 𝑏_{22} \end{matrix})$

can verify：这个表示是符合 tensor product 的 bilinearity 的。

(我们可以把 $𝐴$ ， $𝐵$ 分别看作 $𝑚 \times 𝑛$ ， $𝑝 \times 𝑞$ dim 的向量，这个 $𝐴 \otimes 𝐵$ 是 $𝑚 \times 𝑛 \times 𝑝 \times 𝑞$ dim 的向量。)

Definition 28.19 : outer product of two vectors

对于 $𝑣 \in 𝔽^{𝑛}$ ， $𝑤 \in 𝔽^{𝑚}$ ，我们定义它们的 outer product $𝑣 \otimes 𝑤$ 为：

$𝑣 𝑤 ≔ 𝑣 \otimes 𝑤^{𝑇} = (\begin{matrix} 𝑣_{1} \\ \dots \\ 𝑣_{𝑛} \end{matrix}) \otimes (\begin{matrix} 𝑤_{1} & \dots & 𝑤_{𝑚} \end{matrix}) = (\begin{matrix} 𝑣_{1} 𝑤^{𝑇} \\ \dots \\ 𝑣_{𝑛} 𝑤^{𝑇} \end{matrix}) = (\begin{matrix} 𝑤_{1} 𝑣 & \dots & 𝑤_{𝑚} \end{matrix})$

28.1 matrix product through outer product

Theorem 28.28 : representing matrix multiplication by outer products

对于 $𝑚 \times 𝑛$ 的矩阵 $𝐴$ 和 $𝑛 \times 𝑘$ 的矩阵 $𝐵$ ，we have：

$𝐴 𝐵 = \sum_{𝑖 = 1}^{𝑛} 𝐴_{* 𝑖} \otimes 𝐵_{𝑖 *}$

Proof

In md. □

29 orthogonal vectors and matrices

Definition 29.20 : adjoint(hermitian conjugate)

^*: $𝐴 = (\begin{matrix} 𝑎_{11} & 𝑎_{12} \\ 𝑎_{21} & 𝑎_{22} \\ 𝑎_{31} & 𝑎_{32} \end{matrix}) \to 𝐴^{*} = (\begin{matrix} | 𝑎_{11} | & | 𝑎_{21} | & | 𝑎_{31} | \\ | 𝑎_{12} | & | 𝑎_{22} | & | 𝑎_{32} | \end{matrix})$

(where each overline means complex conjuate.)

Definition 29.21 : standard inner product and norm on $ℂ^{𝑚}$

The standard inner product:

$< 𝑥, 𝑦 > ≔ 𝑥^{*} 𝑦 = \sum_{𝑖 = 1}^{𝑚} | 𝑥_{𝑖} | 𝑦_{𝑖}$

The standard norm:

$‖ 𝑥 ‖ ≔ \sqrt{𝑥^{*} 𝑥}$

Definition 29.22 : orthogonal, orthonomal vectors

Say $𝑥, 𝑦 \in ℂ^{𝑚}$ 是 orthogonal vectors，if $𝑥^{*} 𝑦 = 0$ 。

Say $𝑆 \subset ℂ^{𝑚}$ 是 orthogonal 的，如果其中的 vectors 相互 orthogonal。

Say $𝑆 \subset ℂ^{𝑚}$ 是 orthonomal 的，如果其中的 vectors 相互 orthogonal，并且每个 vector 的 norm 都是 1。

Theorem 29.29 : orthogonal $\Rightarrow$ lin.ind

orthogonal 的 vectors 一定 linearly independent。

Proof

trivial. □

Corollary 29.1

orthogonal 的

dim (𝑉)

个 vectors 一定是

𝑉

的一个 basis。

29.1 decomposing vector by an orthonormal set

Theorem 29.30 : decomposing vector by an orthonormal set

给定 $ℂ^{𝑚}$ 中的一个 orthonormal set ${𝑞_{1}, \dots, 𝑞_{𝑛}}$ (by inner product $< \cdot >$ ，这里以 standard complex inner product 为例)，对于一个 arbitrary vector $𝑣$ ，我们 define:

$𝑟 ≔ 𝑣 - \sum_{𝑖 = 1}^{𝑛} < 𝑞_{𝑖}, 𝑣 > 𝑞_{𝑖}$

Claim：这个 $𝑟$ is orthogonal to ${𝑞_{1}, \dots, 𝑞_{𝑛}}$ , 即我们把这个 $𝑣$ 分解成了在这个 orthonormal set 上的投影与一个和它们都正交的 vector。

Proof

注意：由于 $𝑞_{𝑖}$ 都是 unit vectors， $< 𝑞_{𝑖}, 𝑣 > 𝑞_{𝑖} = \frac{𝑣}{‖ 𝑣 ‖} cos 𝛼 = {proj}_{𝑞_{𝑖}} (𝑣)$ is the projection of $𝑣$ onto the direction of $𝑞_{𝑖}$ .

我们在两边取和 $𝑞_{𝑖}$ 的 inner product，for each $𝑖$ 。由 linearity 可拆开，由 orthgonality 可得到：

$< 𝑞_{𝑖}, 𝑟 > = < 𝑞_{𝑖}, 𝑣 > - < 𝑞_{𝑖}, 𝑣 > < 𝑞_{𝑖}, 𝑞_{𝑖} >$

并且由于 $𝑞_{𝑖}$ 是 unit vector，得到 $< 𝑞_{𝑖}, 𝑞_{𝑖} > = 1$ ，从而右边为 0。 □

对于 unit vector $𝑤$ ，我们刚才已经展示了一个 arbitrary vector $𝑣$ 在它上面的 projection 是：

${proj}_{𝑤 (𝑣)} = < 𝑤, 𝑣 > 𝑤$

现在我们引入另一个形式的 projection 表达：projection matrix

Theorem 29.31 : projection matrix

对于任意的 unit vector $𝑤$ ，we have

${proj}_{𝑤 (𝑣)} = (𝑤 \otimes 𝑤^{*}) 𝑣$

其中 $𝑤 \otimes 𝑤^{*}$ is called the projection matrix onto $𝑤$ .

Proof

In md.

Notice that this matrix is rank 1. □

Definition 29.23 : Unitary matrix

一个 square matrix

𝑄 \in ℂ^{𝑚 \times 𝑚}

被称为 unitrary 的，if

𝑄^{*} = 𝑄^{- 1}

。

Theorem 29.32 : unitrary matrix 的充要条件

𝑄 \in ℂ^{𝑚 \times 𝑚}

is unitrary

\Leftrightarrow

its columns are orthonormal

\Leftrightarrow

its rows are orthonormal

Proof

显然，因为 unitrary

\Leftrightarrow 𝑄 𝑄^{*} = 𝑄^{*} 𝑄 = 𝐼 \Leftrightarrow ⟨ 𝑞_{𝑖}, 𝑞_{𝑗} ⟩ = 𝛿_{𝑖 𝑗}

□

Theorem 29.33 : unitrary transfromation preserves inner product and length

如果 $𝑄 \in ℂ^{𝑚 \times 𝑚}$ is unitrary，那么对于任意的 $𝑥, 𝑦 \in ℂ^{𝑚}$ ，都有：

${(𝑄 𝑥)}^{*} (𝑄 𝑦) = 𝑥^{*} 𝑦$

并且自然得到 $‖ 𝑄 𝑥 ‖ = ‖ 𝑥 ‖$

Proof

Follows from:

{(𝐴 𝐵)}^{*} = 𝐵^{*} 𝐴^{*}

{(𝑄 𝑥)}^{*} (𝑄 𝑦) = 𝑥^{*} 𝑄^{*} 𝑄 𝑦 = 𝑥^{*} 𝑦

□

30 norms

Definition 30.24 : norm

一个 norm on a vector space $𝑉$ 是一个满足：

nonnegativity (0 iff $𝑥 = 0$ )
trianglar ineq
homogenity

的 function $‖ \cdot ‖ : 𝑉 \to ℝ$

30.1 norms on $ℂ^{𝑚}$

Example 30.1

以下为 $ℂ^{𝑚}$ 上的典型 norms: (absolute value 表示 length, 即 $\sqrt{𝑥^{*} 𝑥}$ )

Lp-norm: $𝑝$ 越大，the largest length dimension 占 norm 的比重就越大

${‖ 𝑥 ‖}_{𝑝} = {(\sum_{𝑖 = 1}^{𝑚} {| 𝑥_{𝑖} |}^{𝑝})}^{\frac{1}{𝑝}}$

$𝐿_{\infty}$ -norm:最长维度.

${‖ 𝑥 ‖}_{\infty} = {max}_{𝑖} | 𝑥_{𝑖} |$

weighted norm: 给定一个 norm ${‖ \cdot ‖}_{𝑘}$ ，这是 weighted version of this norm. 其中 $𝑊$ 是一个 diagonal matrix, diag 上的是 weights.

${‖ 𝑥 ‖}_{𝑊, 𝑘} = {‖ 𝑊 𝑥 ‖}_{𝑘}$

TODO (source 03-norms.tex, line 28): selected TeX refers to 01-fundamentals.assets/Screenshot 2025-01-29 at 22.41.10.png; the asset is not among the selected chapter sources.

TODO (source 03-norms.tex, line 29): selected TeX refers to 01-fundamentals.assets/Screenshot 2025-01-29 at 22.41.52.png; the asset is not among the selected chapter sources.

30.2 operator norms on matrix spaces

We know: 所有的 $𝑚 \times 𝑛$ matrix, every entry in $𝔽$ 也是一个 vector space of $dim 𝑛 𝑚$ over $𝔽$ 。

所以我们当然也可以给 matrix 赋范。

matrix 代表一个 linear transformation，所以 norm 的意义实际上是它 stretch vector 的程度的一种评估。

Example 30.2

Definition 30.25 : Operator norm

${‖ 𝐴 ‖}_{𝑚, 𝑛} = {sup}_{𝑧 \in ℂ^{𝑚}, {‖ 𝑧 ‖}_{𝑛} = 1} {‖ 𝐴 𝑥 ‖}_{𝑚}$

induced by vector norm. 表示它stretch vector 的最大程度。其中, source 和 image vector 分别用 norm n,m 来判定。

如果 source 和 image vector 的 norm 是一样的，比如都使用某个 $𝐿_{𝑝}$ norm，那么我们可以用单个符号表示(induced by $𝑝$ -norm):

${‖ 𝐴 ‖}_{𝑝} = {sup}_{𝑧 \in ℂ^{𝑚}, {‖ 𝑧 ‖}_{𝑝} = 1} {‖ 𝐴 𝑥 ‖}_{𝑝}$

(这更加常用，因为通常我们会对 source 和 image vector 的大小使用相同的评估)

Proposition 30.4 : diagonal matrix 的 norm: reduced to max diag element

如果 $𝐷$ 是一个 diagonal matrix，那么不论取什么 $𝑝$ -norm，我们都有：

${‖ 𝐷 ‖}_{𝑝} = {max}_{1 \leq 𝑖 \leq 𝑚} | 𝑑_{𝑖} |$

其中 $𝑑_{𝑖}$ 为对角线上的元素。

Proof

很直观。我们要把一个以 ${‖ \cdot ‖}_{𝑝}$ 为衡量的 unit ball 上的哪个 vector 被拉伸的程度最大，而 diagonal matrix 把每个坐标 $𝑖$ 上的点固定放大 $𝑑_{𝑖}$ 倍，

因而选择绝对值最大的 $𝑑_{𝑘}$ ，拉伸最大的 vector 一定是 $[0 \dots 1 \dots 0]$ where only the $𝑘$ -th coordinate is $1$ ，因为这个 ball 上所有的 vectors 原本的 norm 都是一样的，而这个 vector 完整地吃到了最大的拉伸程度，其他 vectors 都或多或少吃到了其他 $𝑑_{𝑖}$ 的拉伸效果。 □

Proposition 30.5 : 1-norm: reduced to max column sum

${‖ 𝐴 ‖}_{1} = {max}_{1 \leq 𝑗 \leq 𝑛} {‖ 𝐴_{* 𝑗} ‖}_{1}$

matrix 的 1-norm 实则就是 1-norm 最大列的 1-norm.

Proof

因为

${‖ 𝐴 𝑥 ‖}_{1} = {‖ \sum_{𝑖} 𝑥_{𝑗} 𝑎_{𝑗} ‖}_{1} \leq \sum_{𝑗} | 𝑥_{𝑗} | {| 𝑎_{𝑗} |}_{1}$

并且 $\sum_{𝑗} | 𝑥_{𝑗} | = 1$ ，因而这个和 $\leq {max}_{𝑗} {‖ 𝑎_{𝑗} ‖}_{1}$ 。

并且我们发现，这个值是可以取到的: suppose ${‖ 𝑎_{𝑘} ‖}_{1}$ 最大，那么取 $𝑒_{𝑘}$ 就可以了。

直观而言，由于 1-norm 的单位球和它的 image 都是一个多面体，它取到最大的点一定是某个顶点。以这里的 $ℝ^{2}$ 为例，一定是 $𝑒_{1}$ , $𝑒_{2}$ 中的一个。 □

Proposition 30.6 : $\infty$ -norm: reduced to max row sum

${‖ 𝐴 ‖}_{1} = {max}_{1 \leq 𝑖 \leq 𝑚} {‖ 𝐴_{𝑖 *} ‖}_{1}$

matrix 的 $\infty$ -norm 实则就是 $1$ -norm 最大行的 $1$ -norm.

Proof

直观上，image 的 sup norm 只取最大的那一个 entry，因而一定是取矩阵总(absolute)长度最大的一列, 因为每一列都只贡献 image vector 中的一个 entry。

并且，我们注意到，source vector (on单位球) 包括了所有的最大 entry 为 $1$ 的 vectors，这些 vectors 的 sup norm 都是一样的。而要使得 image vector 的 entries 尽可能大，我们一定会取所有 entries 都为 1 的 vector 作为 input.

TODO (source 03-norms.tex, line 89): selected TeX refers to 01-fundamentals.assets/image-20250130003611232.png; the asset is not among the selected chapter sources.

Note: sup norm 的单位球和它的 image 也都是一个多面体。 □

30.3 Caychy-Swartz and Frobeniu norm

Theorem 30.34 : Hölder inequility and Cachy-Swartz

Let $𝑥, 𝑦 \in ℂ^{𝑚}$ , let $𝑝 \geq 1, 𝑞 \leq \infty$ s.t.

$\frac{1}{𝑝} + \frac{1}{𝑞} = 1$

Holder ineq:

$| 𝑥^{*} 𝑦 | \leq {‖ 𝑥 ‖}_{𝑝} {‖ 𝑦 ‖}_{𝑞}$

Cauchy-Schwarz ineq(special case of Hölder ineq when $𝑝 = 𝑞 = 2$ ):

$| 𝑥^{*} 𝑦 | \leq {‖ 𝑥 ‖}_{2} {‖ 𝑦 ‖}_{2}$

Proof

of Cauchy-Swartz:

By homogenity of inner product and norm, it suffices to prove for unit vector $𝑢, 𝑣$ .

${(𝑢 - 𝑣)}^{2} = {‖ 𝑢 ‖}^{2} - 2 𝑢^{*} 𝑣 + {‖ 𝑣 ‖}^{2}$

因而

$𝑢^{*} 𝑣 \leq \frac{{‖ 𝑢 ‖}^{2} + {‖ 𝑣 ‖}^{2}}{2} = 1 = ‖ 𝑢 ‖ ‖ 𝑣 ‖$

等号成立 iff $𝑢 = 𝑣$ . □

Example 30.3

Applying Cauchy-Swartz 可以发现: row vector 的 matrix 2-norm 等于它 (adjointed) 作为 vector 的 vector 2-norm.

这是因为 consider $𝑎 ≔ 𝐴^{*}$ , 则 $‖ 𝐴 𝑥 ‖ = | 𝑎^{*} 𝑥 | \leq {‖ 𝑎 ‖}_{2} {‖ 𝑥 ‖}_{2}$ ，因而总有 $\frac{‖ 𝐴 𝑥 ‖}{{‖ 𝑥 ‖}_{2}} \leq {‖ 𝑎 ‖}_{2}$ 。并且这个等号可以取到, by taking $𝑥 ≔ 𝑎$ .

Example 30.4

任取两个 vectors $𝑢, 𝑣$ ，它们 outer product 成的 rank-one matrix，其 operator 2-norm 小于等于它们自身的 2-norm 的乘积。

${‖ 𝐴 𝑥 ‖}_{2} = {‖ 𝑢 𝑣^{*} 𝑥 ‖}_{2} = {‖ 𝑢 ‖}_{2} | 𝑣^{*} 𝑥 | \leq {‖ 𝑢 ‖}_{2} {‖ 𝑣 ‖}_{2} {‖ 𝑥 ‖}_{2}$

这是因为: $𝑢 𝑣^{*}$ 这一 outer product 乘以一个向量，即每行都是 $𝑣^{*}$ 的一个倍数 ( $𝑢_{𝑖}$ 倍) 的矩阵乘以这个向量。因而，每行得到的都是 $𝑢_{𝑖}$ 乘上 $𝑣^{*} 𝑥$ 这个 inner product，最后得到的就是

$𝐴 𝑥 = (𝑣^{*} 𝑥) 𝑢$

即 $𝑢$ 的一个倍数，这个倍数等于 $𝑣^{*} 𝑥$ 。

Example 30.5

Theorem 30.35

{‖ 𝐴 𝐵 ‖}_{𝑙, 𝑛} \leq {‖ 𝐴 ‖}_{𝑙, 𝑚} {‖ 𝐵 ‖}_{𝑚, 𝑛}

(并且通常取不到等号.)

Proof

不证明了. Playing with definition 加上 Cauchy-Swartz. □

Definition 30.26 : Frobenious norm

${‖ 𝐴 ‖}_{𝐹} ≔ (\sum_{𝑚} \sum_{𝑛} {| 𝑎_{𝑖 𝑗} |}^{2})$

等于把这个 matrix 展开为 $𝑚 \times 𝑛$ 的 vector 的 vector 2-norm.

Theorem 30.36 : equivalent form of Frobenius norm

{‖ 𝐴 ‖}_{𝐹} = \sqrt{tr (𝐴^{*} 𝐴)} = \sqrt{tr (𝐴 𝐴^{*})}

Proof

trivial.

𝐴^{*} 𝐴

𝐴 𝐴^{*}

的 trace 上每个元素，都是

𝐴

的一行与自己的 dot product，即这一行作为 row vector 的 2-norm 的平方; □

Proposition 30.7

{‖ 𝐴 𝐵 ‖}_{𝐹}^{2} \leq {‖ 𝐴 ‖}_{𝐹}^{2} {‖ 𝐵 ‖}_{𝐹}^{2}

Proof

因为 $𝐴 𝐵$ 的每个 entry $𝑐_{𝑖 𝑗}$ 作为 $𝐴_{𝑖}$ 和 $𝐵_{𝑗}$ 的 inner product, by Cauchy-Swartz, have

$| 𝑐_{𝑖 𝑗} | \leq {‖ 𝐴_{𝑖} ‖}_{2} {‖ 𝐵_{𝑖} ‖}_{2}$

因而：

${‖ 𝐴 𝐵 ‖}_{𝐹} \leq \sum_{𝑛} \sum_{𝑚} ({‖ 𝐴_{𝑖} ‖}_{2} {‖ 𝐵_{𝑗} ‖}_{2})$

$= (\sum_{𝑛} {‖ 𝐴_{𝑖} ‖}_{2}) (\sum_{𝑚} {‖ 𝐵_{𝑗} ‖}_{2})$

$= {‖ 𝐴 ‖}_{𝐹} {‖ 𝐵 ‖}_{𝐹}$

(虽然这看起来很不对, 但容易验证, 这上下两个 sum 是相等的. ) □

Theorem 30.37 : unitrary matrix preserves 2-norm 和 Frobenius norm

Let $𝑄$ be unitrary, then

${‖ 𝑄 𝐴 ‖}_{2} = {‖ 𝐴 ‖}_{2}, {‖ 𝑄 𝐴 ‖}_{𝐹} = {‖ 𝐴 ‖}_{𝐹}$

Proof

因为 ${‖ 𝑄 𝑥 ‖}_{2} = {‖ 𝑥 ‖}_{2}$ for each $𝑥$ .

Frobenius norm:

$tr ({(𝑈 𝐴)}^{*} (𝑈 𝐴)) = tr (𝐴^{*} 𝑈^{*} 𝑈 𝐴) = tr (𝐴^{*} 𝐴)$ □

31 SVD

SVD 的 motivation：一个 linear transformation 可以通过 unit sphere 的 image 来唯一确定。并且，这个 unit sphere 的 image 一定是一个 hyperellipse (高维椭圆)。

Definition 31.27 : principal semiaxes, Singular value

对于一个 linear transformation $𝑇 : ℝ^{𝑛} \to ℝ^{𝑚}$ ，我们 denote the unit sphere in $ℝ^{𝑛}$ as $𝑆$ ，把 $𝑇 (𝑆)$ 这一 hyperellipse 中相互 orthogonal 的各轴上的 vectors 表示为 ${𝜎_{1} 𝑢_{1}, \dots, 𝜎_{𝑛} 𝑢_{𝑛}}$ 。其中 $𝜎_{𝑖}$ decsending， $𝑢_{1}, \dots, 𝑢_{𝑛}$ 为 unit vectors。

我们称 $𝑢_{1}, \dots, 𝑢_{𝑛}$ 为 left singular vectors， $𝜎_{1}, \dots, 𝜎_{𝑛}$ 为 singular values，而 ${𝑣_{1}, \dots, 𝑣_{𝑚}}$ 作为

31.1 reduced SVD

32 QR factorization

32.1 projector

Definition 32.28

我们称一个 operator $𝑃 : ℂ^{𝑛} \to ℂ^{𝑛}$ 为一个 projector, if

$𝑃^{2} = 𝑃$

Note: 不要求是 linear 的. For linear case, 这是一个idempotent linear map.

(而我们将主要关注于 linear orthogonal projector.)

以下，我们都只考虑 projector linear 的情况. nonlinear 的情况是类似的.

Lemma 32.1

我们发现，一个 projector

𝑃

沿着

𝑆_{1} ≔ ker (𝑃)

把空间投影到

𝑆_{2} ≔ im (𝑃)

上.

Lemma 32.2

一个 projector 的 eigenvalue 只有可能是 $0$ 或者 $1$ . 它的 SVD 同时也是 eigenvalue decomposition:

$𝑃 = 𝑄 Σ 𝑄^{*}$

其中 $Σ$ 是一个前面全 1, 后面全 0 的对角矩阵.

Lemma 32.3 : complement projector

如果 $𝑃$ 是一个 projector，那么 $𝐼 - 𝑃$ 也是一个 projector.

我们称 $𝐼 - 𝑃$ 为 $𝑃$ 的 complementary projector.

并且我们有: complementary projector 的 ker 是原 projector 的 im, im 是原 projector 的 ker.

Definition 32.29 : Orthogonal projector

我们称一个 projector 是 orthogonal projector，如果

ker (𝑃) ⟂ im (𝑃)

Theorem 32.38

一个 projector

𝑃

是 orthogonal projector

\Leftrightarrow 𝑃 = 𝑃^{*}

，即

𝑃

是 Hermitian 的.

Theorem 32.39

一个 projector

𝑃

是 orthogonal projector，则它的 complementary projector $𝐼 - 𝑃$ 也是 orthogonal projector.

Corollary 32.2

orthogonal projector

𝑃

的 complementary

𝐼 - 𝑃

把 vectors 投影到

{im (𝑃)}^{⟂}

上.

32.2 classical Gram-Schmidt orthogonalization

classical Gram-Schmidt 是计算 reduced QR 分解的算法.

TODO (source 05-qr-factorization.tex, lines 77–82): selected TeX includes the figure assets/Screenshot 2025-04-17 at 11.44.46.png, captioned reduced QR and labelled fig:reduced QR; the asset is not among the selected chapter sources.

32.2.1 idea of triangular orthogonalization

classical Gram-Schmidt orthogonalization 的 idea 是: 我们逐列地将 $𝐴$ 的 columns 转变为相互 orthogonal 的新列.

具体: 我们每次都把 $𝑎_{𝑗}$ 减去 $𝑎_{1}, \dots, 𝑎_{𝑗 - 1}$ 的 span 包含的成分，从而制作成和 $𝑎_{1}, \dots, 𝑎_{𝑗 - 1}$ 的 span 正交的新列 $𝑞_{𝑗}$ :

$𝑞_{𝑗} ≔ normalized (𝑎_{𝑗} - {proj}_{⟨ 𝑎_{1}, \dots, 𝑎_{𝑗 - 1} ⟩} 𝑎_{𝑗})$

$= normalized (𝑎_{𝑗} - {proj}_{⟨ 𝑞_{1}, \dots, 𝑞_{𝑗 - 1} ⟩} 𝑎_{𝑗})$

展开这个定义:

$𝑣_{𝑗} ≔ 𝑎_{𝑗} - (𝑞_{1}^{*} 𝑎_{𝑗}) 𝑞_{1} - (𝑞_{2}^{*} 𝑎_{𝑗}) 𝑞_{2} - \dots - (𝑞_{𝑗 - 1}^{*} 𝑎_{𝑗}) 𝑞_{𝑗 - 1}$

$𝑞_{𝑗} ≔ \frac{𝑣_{𝑗}}{| 𝑣_{𝑗} |}$

这个过程可以通过定义:

$𝑟_{𝑖 𝑗} = 𝑞_{𝑖}^{*} 𝑎_{𝑗} (𝑖 \neq 𝑗), | 𝑟_{𝑗 𝑗} | = {‖ 𝑎_{𝑗} - \sum_{𝑖 = 1}^{𝑗 - 1} 𝑟_{𝑖 𝑗} 𝑞_{𝑖} ‖}_{2}$

(Note that the sign of $𝑟_{𝑗 𝑗}$ is not determined. Arbitrarily, we may choose $𝑟_{𝑗 𝑗} > 0$ , in which case we shall finish with a factorization $𝐴 = \hat{𝑄} \hat{𝑅}$ in which $\hat{𝑅}$ has positive entries along the diagonal.)

从而这个过程写作:

$𝑣_{𝑗} ≔ 𝑎_{𝑗} - \sum_{𝑖 = 1}^{𝑗 - 1} 𝑟_{𝑖 𝑗} 𝑞_{𝑖}$

$𝑞_{𝑗} ≔ \frac{𝑣_{𝑗}}{𝑟_{𝑗 𝑗}}$

我们发现:

$𝑞_{1} = \frac{𝑎_{1}}{𝑟_{11}}$

$𝑞_{2} = \frac{𝑎_{2} - 𝑟_{12} 𝑞_{1}}{𝑟_{22}}$

$𝑞_{3} = \frac{𝑎_{3} - 𝑟_{13} 𝑞_{1} - 𝑟_{23} 𝑞_{2}}{𝑟_{33}}$

$⋮$

$𝑞_{𝑛} = \frac{𝑎_{𝑛} - \sum_{𝑖 = 1}^{𝑛 - 1} 𝑟_{𝑖 𝑛} 𝑞_{𝑖}}{𝑟_{𝑛 𝑛}}$

这个过程使得:

$𝑎_{𝑗} = \sum_{𝑖 = 1}^{𝑗} 𝑟_{𝑖 𝑗} 𝑞_{𝑖}$

从而:

$𝐴 = \hat{𝑄} \hat{𝑅}$

32.2.2 algorithm

Classical Gram-Schmidt (unstable)

FOR j = 1 TO n
    v_j ← a_j
    FOR i = 1 TO j-1
        r_ij ← q_i* a_j
        v_j ← v_j - r_ij q_i
    ENDFOR
    r_jj ← ||v_j||_2
    q_j ← v_j / r_jj
ENDFOR

32.3 modified Gram-Shimitdt (triangular orthogonalization)

32.4 Household Triangularization

33 Discrete Fourier transform and FFT algorithm

34 conditioning and stability

Source attribution in the selected TeX chapter title: doi:10.1137/1.9780898719574.ch3.

接下来 chapter 中我们将讨论 numerical analysis 中的两个 fundamental issues: Conditioning 和 Stability. Conditioning 指的是 perturbation behavior of a mathematical problem; 而 Stability 指的是解决这一问题的 algorithm 的 perturbation behavior.

Definition 34.30 : problem, problem instance

我们把一个 problem 看作是一个 function, 把 normed VS $𝑋$ of data map to normed VS $𝑌$ of solutions. 即：

$𝑓 : 𝑋 \to 𝑌$

其中，这个 problem $𝑓$ together with a data point $𝑥$ 被称为一个 problem instance. (比如:输入是 $𝑥$ ，问题是求 $𝑥$ 的平方根， $𝑓 : 𝑥 \to \sqrt{𝑥}$ ，那么 $𝑓$ together with input $𝑥 = 3$ 就是一个 problem instance.)

Conditioning 研究的就是一个 problem instance $(𝑓, 𝑥)$ , 其附近 solutions 的变动行为。

一个 well-conditioned problem instance 就是指， $𝑥$ 附近的 small perturbations 只 lead to small changes; 而 ill-conditioned problem instance 就是指， $𝑥$ 附近的 small perturbations 可能引起 big changes.

34.1 absolute/relative condition number of a problem

Definition 34.31 : absolute condition number

$𝛿 𝑥$ 表示 $𝑥$ 附近的一个 small perturbation，并用

$𝛿 𝑓 = 𝑓 (𝑥 + 𝛿 𝑥) - 𝑓 (𝑥)$

来表示 $𝑓$ 随之产生的变化。

定义 absolute condition number 为：

$\hat{𝜅} (𝑥) ≔ {lim}_{𝛿 \to 0} {sup}_{‖ 𝛿 𝑥 ‖ \leq 𝛿} \frac{{‖ 𝛿 𝑓 ‖}_{𝑦}}{{‖ 𝛿 𝑥 ‖}_{𝑥}}$

Remark

即， $𝑥$ 附近比较小的区域上， $𝑓$ 的值随 $𝑥$ 的变化的变化量。

我们发现，这和导数的定义很相近，就是加了一个 norm。当然，并不完全是。因为这里取的是 $sup$ 。导数未必存在，但是这个 $sup$ 总是存在的(如果 count into infty 的话). absolute condition number 是一种上极限概念，表示 $𝑥$ 周围的 $𝑓$ 变动幅度最大的值; 而它也不完全是导数值 (recall: 导数是 best linear approximation of this function near this point, 和原 function 的形状是一样的, 这里 take 的是导数的某个 matrix operator norm, determined by 我们对 $𝑦, 𝑥$ 使用的 norm)

具体：我们发现，当 $𝛿$ 足够小的时候，we have:

$𝛿 𝑓 \approx 𝐽 (𝑥) 𝛿 𝑥$

(这里使用 approx，但是可以严格证明.) 从而，如果 $𝑓$ differentiable,

$\hat{𝜅} (𝑥) ≔ {lim}_{𝛿 \to 0} {sup}_{‖ 𝛿 𝑥 ‖ \leq 𝛿} \frac{{‖ 𝐽 (𝑥) 𝛿 𝑥 ‖}_{𝑦}}{{‖ 𝛿 𝑥 ‖}_{𝑥}} = {‖ 𝐽 (𝑥) ‖}_{𝑥 \to 𝑦}$

简写：

$\hat{𝜅} (𝑥) = ‖ 𝐽 (𝑥) ‖$

这里还有另外一个 condition number:

Definition 34.32 : relative condition number

$𝜅 (𝑥) ≔ {lim}_{𝛿 \to 0} {sup}_{‖ 𝛿 𝑥 ‖ \leq 𝛿} \frac{\frac{‖ 𝛿 𝑓 ‖}{‖ 𝑓 (𝑥) ‖}}{\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}}$

(recall: $𝛿 𝑓$ 并不是 $𝑓$ 的倍数而是:

$𝛿 𝑓 = 𝑓 (𝑥 + 𝛿 𝑥) - 𝑓 (𝑥)$

即 $𝑥$ perturbated 后的函数值和原先的函数值的差.)

这里对于 absolute/relative condition number 有一种不严谨的记法: 我们把 $𝛿 𝑥, 𝛿 𝑓$ 看作 infinitesimal (当然,严格的分析里并不存在) 那么可以简写为:

$\hat{𝜅} = {sup}_{𝛿 𝑥} \frac{‖ 𝛿 𝑓 ‖}{‖ 𝛿 𝑥 ‖}, 𝜅 = {sup}_{𝛿 𝑥} (\frac{\frac{‖ 𝛿 𝑓 ‖}{‖ 𝑓 (𝑥) ‖}}{\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}})$

Example 34.6

问题 1. $𝑓 : 𝑥 \to \frac{𝑥}{2}$ , 即把一个数取半. 那么对于任意 $𝑥$ 都有:

$𝜅 (𝑥) = \frac{‖ 𝐽 (𝑥) ‖}{\frac{‖ 𝑓 (𝑥) ‖}{‖ 𝑥 ‖}} = \frac{\frac{1}{2}}{\frac{\frac{𝑥}{2}}{𝑥}} = 1$

well-conditioned.

问题 2. $𝑓 : 𝑥 \to \sqrt{𝑥}$ , 即取一个数的 sqrt, $𝑥 > 0$ , 有:

$𝜅 (𝑥) = \frac{‖ 𝐽 (𝑥) ‖}{\frac{‖ 𝑓 (𝑥) ‖}{‖ 𝑥 ‖}} = \frac{\frac{1}{2 \sqrt{𝑥}}}{\frac{\sqrt{𝑥}}{𝑥}} = \frac{1}{2}$

well-conditioned.

TODO (source 07-conditioning-and-stability.tex, lines 70–73): selected TeX includes assets/condition1.png; the asset is not among the selected chapter sources.

34.1.1 examples: 两数相减

Example 34.7

问题 3: $𝑓 : (𝑥_{1}, 𝑥_{2}) \to 𝑥_{1} - 𝑥_{2}$ 两数字相减.

$𝐽 (𝑥) = [𝜕 \frac{𝑓}{𝜕} 𝑥_{1}, 𝜕 \frac{𝑓}{𝜕} 𝑥_{2}] = [1, - 1]$

For simplicity, 取 $\infty$ -norm, 得到 $‖ 𝐽 (𝑥) ‖ = 2$ , 于是

$𝜅 (𝑥) = \frac{2}{\frac{| 𝑥_{1} - 𝑥_{2} |}{max} {| 𝑥_{1} |, | 𝑥_{2} |}}$

如果 $| 𝑥_{1} - 𝑥_{2} |$ large 时， $𝜅$ 就会变的很大。因而 this problem is ill-conditioned when $𝑥_{1} \approx 𝑥_{2}$ . 这符合 “cancellation error”: 相近的两个数相减会损失有效数字, 放大误差.

For example:

$𝑎 = 123456.789012, 𝑏 = 123456.789011$

它们的差:

$𝑎 - 𝑏 = 0.000001$

如果浮点数只能保留 7 位有效数字 (单精度), 那么 $𝑎$ 被存为 123456.8, $𝑏$ 被存为 123456.8, 相减后结果是 $0.0$ , 完全错误. 这就是 cancellation error：由于精度丢失导致的小差值计算结果失真.

34.1.2 example: polynomial 求根

Example 34.8

问题 4: polynomial 求根. $𝑓 : ℂ^{𝑛} \to ℂ^{𝑛}$ , 把 $𝑛$ 个系数 maps to $𝑛$ 个 roots.

我们考虑

$𝑝 (𝑥) = 𝑎_{0} + 𝑎_{1} 𝑥 + 𝑎_{2} 𝑥^{2} + \dots + 𝑎_{𝑛} 𝑥^{𝑛} = \sum_{𝑘 = 0}^{𝑛} 𝑎_{𝑘} 𝑥^{𝑘}$

如果 coefficient $𝑎_{𝑖}$ 被 perturbed by an infinitesimal quantity $𝛿 𝑎_{𝑖}$ , 那么 the perturbation of root $𝑥_{𝑗}$ 是多少? 答案是:

$𝛿 𝑥_{𝑗} = - (𝛿 𝑎_{𝑖}) \frac{𝑥_{𝑗}^{𝑖}}{𝑝^{'}} (𝑥_{𝑗})$

从而对于这个问题:

$𝜅_{𝑗 (𝑎_{𝑖})} = \frac{\frac{| 𝛿 𝑥_{𝑗} |}{| 𝑥_{𝑗} |}}{\frac{| 𝛿 𝑎_{𝑖} |}{| 𝑎_{𝑖} |}} = \frac{| 𝑎_{𝑖} 𝑥_{𝑗}^{𝑖 - 1} |}{| 𝑝^{'} (𝑥_{𝑗}) |}$

证明 selected-TeX label perturbation of a root given perturbation of a coeff: (非 rigorous)

perturbed polynomial 即:

$\tilde{𝑝} (𝑥) = 𝑝 (𝑥) + 𝛿 𝑎_{𝑖} 𝑥^{𝑖}$

我们要求的 perturbation $𝛿 𝑥_{𝑗}$ , 无法直接得到等式关系. 但是我们知道新的 root 是: $𝑥_{𝑗} + 𝛿 𝑥_{𝑗}$ .

即:

$\tilde{𝑝} (𝑥_{𝑗} + 𝛿 𝑥_{𝑗}) = 0$

从而:

$𝑝 (𝑥_{𝑗} + 𝛿 𝑥_{𝑗}) + 𝛿 𝑎_{𝑖} {(𝑥_{𝑗} + 𝛿 𝑥_{𝑗})}^{𝑖} = 0$

Using Taylor expansions:

$𝑝 (𝑥_{𝑗} + 𝛿 𝑥_{𝑗}) \approx 𝑝 (𝑥_{𝑗}) + 𝑝^{'} (𝑥_{𝑗}) 𝛿 𝑥_{𝑗}$

其中 $𝑝 (𝑥_{𝑗}) = 0$ . 并且， ${(𝑥_{𝑗} + 𝛿 𝑥_{𝑗})}^{𝑖} \approx 𝑥_{𝑗}^{𝑖}$ ，因为在乘方的作用下这个 perturbation 作用可以忽略 (作为高阶无穷小). 从而得到

$𝑝^{'} (𝑥_{𝑗}) 𝛿 𝑥_{𝑗} + 𝛿 𝑎_{𝑖} 𝑥_{𝑗}^{𝑖} = 0$

从而得到 $𝛿 𝑥_{𝑗} = - \frac{𝛿 𝑎_{𝑖} 𝑥_{𝑗}^{𝑖}}{𝑝^{'}} (𝑥_{𝑗})$

Polynomial rootfinding 是 ill-conditioned, 即便不涉及 multiple roots 问题. 比如经典的 “Wilkinson polynomial”:

$𝑝 (𝑥) = \prod_{𝑖 = 1}^{20} (𝑥 - 𝑖) = 𝑎_{0} + 𝑎_{1} 𝑥 + \dots + 𝑎_{19} 𝑥^{19} + 𝑥^{20}$

它的 most sensitive root 是 $𝑥 = 15$ , 并且对于这个 root, 最 sensitive 的 coefficient to change 是 $𝑎_{15} \approx 1.67 \times 10^{9}$ , 这个 root 和这个 coeff 之间的 condition number 为:

$𝜅 \approx \frac{1.67 \times 10^{9} \cdot 15^{14}}{5! 14!} \approx 5.1 \times 10^{13}$

TODO (source 07-conditioning-and-stability.tex, lines 138–143): selected TeX includes assets/Screenshot 2025-04-15 at 00.21.49.png, captioned Wilkinson's example 中 roots 的 perturbation, by $tilde(a)_k = a_k(1 + 10^(-10) r_k)$ and labelled fig:wilkinson-root-perturbation; the asset is not among the selected chapter sources.

34.1.3 example: matrix 乘 vector

这个 example 分为三部分:

Fixing $𝐴$ , $𝑥 \to 𝑏$
Fixing $𝐴$ , inverse problem: $𝑏 \to 𝑥$
fixing $𝑏$ , $𝐴 \to 𝑥$

Example 34.9

Matrix-vector multiplication: $𝐴 𝑥 = 𝑏$ (fixing $𝐴$ )

$𝜅_{fwd (𝑥)} = {sup}_{𝛿 𝑥} (\frac{\frac{‖ 𝐴 (𝑥 + 𝛿 𝑥) - 𝐴 𝑥 ‖}{‖ 𝐴 𝑥 ‖}}{\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}}) = {sup}_{𝛿 𝑥} \frac{\frac{‖ 𝐴 𝛿 𝑥 ‖}{‖ 𝛿 𝑥 ‖}}{\frac{‖ 𝐴 𝑥 ‖}{‖ 𝑥 ‖}}$

即:

$𝜅_{fwd (𝑥)} = ‖ 𝐴 ‖ \frac{‖ 𝑥 ‖}{‖ 𝐴 𝑥 ‖}$

Note: 对于任意非零 $𝑥$ , 都有

$‖ 𝐴 𝑥 ‖ \geq \frac{1}{‖ 𝐴^{†} ‖} \cdot ‖ 𝑥 ‖ \to \frac{‖ 𝑥 ‖}{‖ 𝐴 𝑥 ‖} \leq ‖ 𝐴^{†} ‖$

所以

$𝜅_{fwd (𝑥)} = ‖ 𝐴 ‖ \cdot \frac{‖ 𝑥 ‖}{‖ 𝐴 𝑥 ‖} \leq ‖ 𝐴 ‖ ‖ 𝐴^{†} ‖$

(2) Inverse problem: given $𝑏$ 求 $𝑥$ , 即 $𝑥 = 𝐴^{- 1} 𝑏$ , 也有同样的 condition number bound (这显然，因为对称):

$𝜅_{inverse (𝑏)} = ‖ 𝐴^{†} ‖ \frac{‖ 𝑏 ‖}{‖ 𝑥 ‖} \leq ‖ 𝐴 ‖ ‖ 𝐴^{†} ‖$

因而我们把 $‖ 𝐴 ‖ ‖ 𝐴^{- 1} ‖$ 称为 一个 matrix 的 condition number.

Definition 34.33 : Condition number of a matrix

我们定义 condition number of a matrix:

$𝜅 (𝐴) ≔ ‖ 𝐴 ‖ ‖ 𝐴^{†} ‖$

Notice: 如果 $‖ \cdot ‖ ≔ {‖ \cdot ‖}_{2}$ , 那么 for $𝐴 \in ℂ^{𝑚 \times 𝑛}$ , we have:

${‖ 𝐴 ‖}_{2} = 𝜎_{1}, {‖ 𝐴^{†} ‖}_{2} = \frac{1}{𝜎_{𝑟}}$

where $𝜎_{𝑟}$ 是最小的非零的 singular value.

对于 $‖ \cdot ‖ ≔ {‖ \cdot ‖}_{2}$ ,

$𝜅 (𝐴) = \frac{𝜎_{1}}{𝜎_{𝑟}}$

即 image of the unit sphere 作为 hyperrellipse 的 eccentricity.

Remark

我们已经知道了，对于 $𝐴 𝑥 = 𝑏$ ( $𝐴 \in ℂ^{𝑚 \times 𝑛}$ ): 不论是 forward problem $𝑥 \to 𝑏$ 还是 inverse problem $𝑏 \to 𝑥$ ，都有

$𝜅 \leq 𝜅 (𝐴)$

那么等号在什么时候取到呢? 我们以 forward problem 为例: 取 $𝐴$ 的 SVD: $𝐴 = 𝑈 Σ 𝑉^{*}$ , where

$𝑈 \in ℂ^{𝑚 \times 𝑚}, 𝑉 \in ℂ^{𝑛 \times 𝑛}$ unitary，
$Σ = diag (𝜎_{1}, \dots, 𝜎_{𝑟}, 0, \dots, 0) \in ℝ^{𝑚 \times 𝑛}, 𝑟 = rank (𝐴)$
$𝜎_{1} \geq 𝜎_{2} \geq \dots \geq 𝜎_{𝑟} > 0$

我们令 $𝑥 = 𝑉 𝑧$ ，因为 $𝑉$ unitary 不改变范数，

$𝐴 𝑥 = 𝑈 Σ 𝑉^{*} 𝑉 𝑧 = 𝑈 Σ 𝑧 \Rightarrow ‖ 𝐴 𝑥 ‖ = ‖ Σ 𝑧 ‖, ‖ 𝑥 ‖ = ‖ 𝑧 ‖$

所以

$\frac{‖ 𝑥 ‖}{‖ 𝐴 𝑥 ‖} = \frac{‖ 𝑧 ‖}{‖ Σ 𝑧 ‖} = \frac{{(\sum_{𝑖 = 1}^{𝑛} {| 𝑧_{𝑖} |}^{2})}^{\frac{1}{2}}}{{(\sum_{𝑖 = 1}^{𝑟} 𝜎_{𝑖}^{2} {| 𝑧_{𝑖} |}^{2})}^{\frac{1}{2}}}$

这个最大值就是:

${max}_{𝑧 \neq 0} \frac{‖ 𝑧 ‖}{‖ Σ 𝑧 ‖} = {max}_{𝑧 \neq 0} \frac{1}{{(\sum_{𝑖 = 1}^{𝑟} 𝜎_{𝑖}^{2} \cdot (\frac{{| 𝑧_{𝑖} |}^{2}}{{‖ 𝑧 ‖}^{2}}))}^{\frac{1}{2}}}$

这在 $| 𝑧_{𝑖} | = 1$ 只有一个非零坐标, 且对应于最小非零奇异值 $𝜎_{𝑟}$ 时最大. 所以最大值是 $\frac{1}{𝜎_{𝑟}} = ‖ 𝐴^{†} ‖$ 。因此：

$\frac{‖ 𝑥 ‖}{‖ 𝐴 𝑥 ‖} = ‖ 𝐴^{†} ‖ \Leftrightarrow 𝑥 \in span (𝑣_{𝑟})$

即，当 $𝑥$ 是 $𝐴$ 的 minimal nonzero singular value $𝜎_{𝑟}$ 的 right singular vector 时, 取到

$𝜅_{fwd (𝑥)} = 𝜅 (𝐴)$

同理，对于 inverse problem，令 $𝑏 = 𝑈 𝑦$ , 有:

$𝐴^{†} 𝑏 = 𝑉 Σ^{†} 𝑈^{*} 𝑈 𝑦 = 𝑉 Σ^{†} 𝑦 \Rightarrow ‖ 𝐴^{†} 𝑏 ‖ = ‖ Σ^{†} 𝑦 ‖, ‖ 𝑏 ‖ = ‖ 𝑦 ‖$

其中 $Σ^{†} = diag (\frac{1}{𝜎_{1}}, \dots, \frac{1}{𝜎_{𝑟}}, 0, \dots, 0)$ . 所以：

$\frac{‖ 𝑏 ‖}{‖ 𝐴^{†} 𝑏 ‖} = \frac{‖ 𝑦 ‖}{‖ Σ^{†} 𝑦 ‖} = {(\sum_{𝑖 = 1}^{𝑚} \frac{{| 𝑦_{𝑖} |}^{2}}{\sum_{𝑖 = 1}^{𝑟}} \frac{1}{𝜎_{𝑖}^{2}} {| 𝑦_{𝑖} |}^{2})}^{\frac{1}{2}}$

最大值发生在 $𝑦 = 𝑒_{1}$ 对应于最大奇异值 $𝜎_{1}$ , 此时

$‖ 𝐴^{†} 𝑏 ‖ = \frac{1}{𝜎_{1}} ‖ 𝑏 ‖ \Rightarrow \frac{‖ 𝑏 ‖}{‖ 𝐴^{†} 𝑏 ‖} = 𝜎_{1} = ‖ 𝐴 ‖$

因此

$𝜅_{inv (𝑏)} = ‖ 𝐴^{†} ‖ \cdot ‖ 𝐴 ‖ \Leftrightarrow 𝑏 \in span (𝑢_{1})$

即，当 $𝑏$ 是 $𝐴$ 的 maximal singular value $𝜎_{1}$ 的 left singular vector 时, 取到

$𝜅_{inverse (𝑏)} = 𝜅 (𝐴)$

至此，我们可以总结这个 theorem (a general version of Theorem 12.2 in textbook Ch12):

Theorem 34.40 : conditioning of matrix times vector

For problem $𝐴 𝑥 = 𝑏$ fixing $𝐴$ , 不论是 $𝑥 \to 𝑏$ 的 forward problem 还是 $𝑏 \to 𝑥$ 的 inverse problem，都有:

$𝜅 \leq 𝜅 (𝐴) ≔ ‖ 𝐴 ‖ ‖ 𝐴^{†} ‖$

并且，对于 forward problem，等号当且仅当 $𝑥$ 是 $𝐴$ 的 minimal nonzero singular value $𝜎_{𝑟}$ 的 right singular vector 时取到; 对于 inverse problem，等号当且仅当 $𝑏$ 是 $𝐴$ 的 maximal singular value $𝜎_{1}$ 的 left singular vector 时取到.

现在我们来考虑: Fixing $𝑏$ , 求 $𝐴 \to 𝑥$ 的问题.

我们有:

$(𝐴 + 𝛿 𝐴) (𝑥 + 𝛿 𝑥) = 𝑏$

我们知道 $𝐴 𝑥 = 𝑏$ , 并且可以 drop the doubly infinitesimal term $(𝛿 𝐴) (𝛿 𝑥)$ , 从而得到 $(𝛿 𝐴) 𝑥 + 𝐴 (𝛿 𝑥) = 0$ 即

$𝛿 𝑥 = - 𝐴^{† (𝛿 𝐴)} 𝑥$

By matrix norm 小于等于拆分后 norms 的乘积的定理，我们于是有:

$‖ 𝛿 𝑥 ‖ \leq ‖ 𝐴^{†} ‖ ‖ 𝛿 𝐴 ‖ ‖ 𝑥 ‖$

即

$\frac{\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}}{\frac{‖ 𝛿 𝐴 ‖}{‖ 𝐴 ‖}} \leq ‖ 𝐴^{†} ‖ ‖ 𝐴 ‖ = 𝜅 (𝐴)$

于是我们得到

$𝜅_{𝐴 \to 𝑥} \leq 𝜅 (𝐴)$

神奇地发现，它也被 $𝜅 (𝐴)$ bound.

并且, equality in this bound will hold whenever $𝛿 𝐴$ is such that

$‖ 𝐴^{† (𝛿 𝐴)} 𝑥 ‖ = ‖ 𝐴^{†} ‖ ‖ 𝛿 𝐴 ‖ ‖ 𝑥 ‖$

而，我们可以发现对于任意 $𝐴, 𝑏$ , 这个 $𝛿 𝐴$ 一定存在，即等号一定可以取到. 这是因为 operator norm 与其 dual norm 的等价性:

$𝐿 : 𝛿 𝐴 \to 𝐴^{† (𝛿 𝐴)} 𝑥$

是一个从 $ℂ^{𝑚 \times 𝑛} \to ℂ^{𝑛}$ 的线性算子，它的 operator norm 是:

$‖ 𝐿 ‖ = {sup}_{𝛿 𝐴 \neq 0} \frac{‖ 𝐴^{† (𝛿 𝐴)} 𝑥 ‖}{‖ 𝛿 𝐴 ‖}$

我们可以证明这个 supremum 可以达到. 选择

$𝛿 𝐴 = 𝑢 𝑣^{*}$

其中 $𝑢 \in ℂ^{𝑚}$ 是使得 $‖ 𝐴^{†} 𝑢 ‖ = ‖ 𝐴^{†} ‖$ 的单位向量， $𝑣 = \frac{𝑥}{‖ 𝑥 ‖}$ 是单位方向向量. 于是:

$(𝛿 𝐴) 𝑥 = (𝑢 𝑣^{*}) 𝑥 = 𝑢 \cdot (𝑣^{*} 𝑥) = 𝑢 \cdot ‖ 𝑥 ‖ \Rightarrow 𝐴^{† (𝛿 𝐴)} 𝑥 = ‖ 𝑥 ‖ \cdot 𝐴^{†} 𝑢 \Rightarrow ‖ 𝐴^{† (𝛿 𝐴)} 𝑥 ‖ = ‖ 𝑥 ‖ \cdot ‖ 𝐴^{†} 𝑢 ‖ = ‖ 𝑥 ‖ \cdot ‖ 𝐴^{†} ‖$

从而我们可以得到这个结论:

Theorem 34.41 : conditioning of matrix times vector: given $𝑏$ , problem $𝐴 \to 𝑥$

对于 $𝐴 𝑥 = 𝑏$ fixing $𝑏$ , 考虑 problem $𝐴 \to 𝑥$ , 这一问题一定有 condition number:

$𝜅 = 𝜅 (𝐴)$

34.2 float number and machine epsilon

34.2.1 float number system

我们知道计算机处理的是离散的数值. 即，一个 computer 的 number system 并非 $ℝ$ 而是 $ℝ$ 的一个 discrete (and finite, 但是 ideally 可以看作 infinite) subset $𝐹$ , 称之为 float number system.

这个 $𝐹$ 由这两个参数决定决定:

base integer $𝛽$
precision integer $𝑡$

(通常 $𝛽 = 2$ 即二进制，而 $𝑡 = 24, 53$ for IEEE single/double precision.)

precision 决定了这个系统的对数字表示的相对精度 (即即将定义的 machine epsilon); biased exponent 决定了这个系统能够表示的数的范围的上下限.

从而,

$𝐹 = {0} \cup {\pm (\frac{𝑚}{𝛽^{𝑡}}) 𝛽^{𝑒} : 𝑚 \in [1, 𝛽^{𝑡} - 1] int, 𝑒 int}$

这里的 $\pm \frac{𝑚}{𝛽^{𝑡}}$ 称为 mantissa of $𝑥$ ; $𝑒$ 称为 exponent.

现实中， $𝑒$ 也有范围，取决于计算机位数和架构. 比如说 ieee 双精度 float: 这里 $𝐸$ 的范围是 $0 \sim 2047$ , 因而 $𝑒 = 𝐸 - 1023$ 的范围是 $- 1023 \sim 1024$ .

TODO (source 07-conditioning-and-stability.tex, lines 297–302): selected TeX includes assets/Screenshot 2025-04-15 at 10.56.30.png, captioned IEEE and labelled fig:ieee-double-precision; the asset is not among the selected chapter sources.

IEEE double precision:

$𝑥 = {(- 1)}_{2}^{sign (1 . 𝑏_{51} 𝑏_{50} \dots 𝑏_{0})} \times 2^{𝐸 - 1023}$

因而更加现实的 system $𝐹$ 和我们这里的理论 model $𝐹$ 有这些差别:

还要包括一个额外的参数: exponent offset $𝑠$ , 控制 $𝑒$ bounded by some $𝑒_{min}$ 和 $𝑒_{max}$ .
现实的 ieee standard 和我们的 ideal 模型 $𝐹$ 不同的点, 不仅是 $𝑒$ bounded 具有 $𝑒_{min}$ 和 $𝑒_{max}$ , 还有: 它的每个数其实是 $\pm (1 + \frac{𝑚}{𝛽^{𝑡}}) 𝛽^{𝑒}$ 而不是 $\pm (\frac{𝑚}{𝛽^{𝑡}}) 𝛽^{𝑒}$ . 前面的 $1$ 称为 leading bit. 这是因为在规格化二进制浮点数系统中, 所有非零数的尾数都可以唯一表示成以 $1 .$ 开头的形式. 因为这个 $1 .$ 总是存在, 可以省略它来节省空间.
考虑更多的 symbols, 例如:

Symbol	Meaning
$+ 0$	Postitive underflow; between $0$ and the smallest positive representable float
$- 0$	Negative underflow
$+ \infty$	Positive overflow; bigger than biggest representable float. E.g., $\frac{1}{0} = \frac{1}{+ 0}$
$- \infty$	Negative overflow
NaN	Not-a-Number, e.g., $\frac{0}{0}$ .

Note: $0$ 也是一个 symbol. 并且，现实的 system 里，还要区分正负方向上的 underflow 得到的 $0$ .

Definition 34.34 : machine epsilon

对于一个 discrete number system $𝐹$ with precision $𝑡$ 和 base $𝛽$ ，我们定义:

$𝜀_{machine} ≔ \frac{1}{2} 𝛽^{1 - 𝑡}$

为什么要这样定义: 因为这两点:

Proposition 34.8

对于任意的 $𝑥 \in ℝ$ that is within machine 的表示范围，都存在一个 $𝜀$ s.t. $| 𝜀 | < 𝜀_{machine}$ 使得

$fl (𝑥) = 𝑥 (1 + 𝜀)$

这一点是显然的. 任意的大小不能过大的实数，都可以在 machine epsilon 的误差内被 float number 表示.

更加好的是:

Theorem 34.42 : Fundamental Axion of Floating Point Arithmetic

对于一个 discrete number system $𝐹$ , 对于任意的 $𝑥, 𝑦 \in 𝐹$ , 都存在一个 error $𝜀$ s.t.

$| 𝜀 | \leq 𝜀_{machine}$

such that:

$𝑥 ⋆_{fl} 𝑦 = (𝑥 ⋆_{ℝ} 𝑦) (1 + 𝜀)$

for 任意的 $⋆ ≔ +, -, \times, \div$ .

(Exclusion: relative error 并不包括 $𝑥 - 𝑥$ 时出现的 cancellation error, 以及其他的 overflow, underflow! 这些是symbolic hacks, 例如 perturbing $0$ to $0.1$ gives relative error $\frac{0.1 - 0}{0} = + \infty$ ; 并且需要注意的是, relative errors are only useful when small, well below $100 %$ .)

即：任意基本运算的相对于自身的误差，都被 bound 在 $𝜀_{machine}$ 之内.

为什么是相对误差而不是绝对误差? 因为我们能表示的有效数字位数是固定的. 越大的数，其小数点后的有效数字就越小. 从而，绝对误差就越大. 但是相对误差仅和 $𝑡$ 和 base $𝛽$ 有关.

(Note: On a computer in which intermediate quantities are truncated rather than rounded, Fundamental Axion of Floating Point Arithmetic hold with $𝜀_{machine}$ replaced by $2 𝜀_{machine}$ .)

Remark

之所以 Float number 的讨论是重要的，因为它是计算机用来近似表示一个实数的方法，而所有的数值计算都要经由此为媒介. 需要注意的是:ultimate access to numbers is via +,−,⋆,÷, 复杂度和误差最终由基本运算衡量.

对于 losing 1 bit 的 round off error 等问题，我们并不在意; 但是，当大量计算 iteratively 堆叠时，一些 first algorithms for many problems might lose half the bits. 这是一个很大的数量: 例如 Classical Gram Schmidt 会失去 half the digits, 相比 modified Gram-Schmidt 而言. 例如, accurate bits 从 52 变为 26.

因而，基于 float number (via machine epsilon) 对一个 algorithm 的 stability 进行分析是重要的. 接下来我们将讲解 stability of an algorithm 这个概念.

34.3 stability

Review: 一个 Problem (in our def) 是一个 function $𝑓 : 𝑋 \to 𝑌$ , $𝑋, 𝑌$ 都是 NVS.

而我们现在定义什么是一个 algorithm:

34.3.1 def: algorithm, stability, accuracy

Definition 34.35 : algorithm

一个 algorithm for a problem

𝑓 : 𝑋 \to 𝑌

是另一个函数

\tilde{𝑓} : 𝑋 \to 𝑌

定义上就是这么简单.

注意: 我们这里的 algorithm 是一个比较 restricted 的定义. Specially, 它并不考虑 randomized algorithms.

Example 34.10

Randomized rounding:

把 $7.3$ round to: 8, with a prob of $0.3$ ; $7$ , with a prob of $0.7$ .

我们把 round 得到的结果标记为 $𝑋$ , 那么它则是一个 random variable. 并且，它是一个 unbiased random variable，即:

$𝔼 [𝑋] = 7.3$

这个 rounding 是一个 algorithm, 但是不包含在我们这里的定义里. 因为原问题是 $ℝ \to ℕ$ 的, 而这个问题则是 maps to random variables (我们知道一个 random variable 是一个函数, 这是一个 function space) 的. 因而它并不是我们定义的算法.

因而我们的定义其实是 restricted 的. 我们这里只考虑 determinstic 的 algorithm.

Definition 34.36 : accuracy of an algorithm

给定一个 problem $𝑓$ 和一个对应的 algorithm $\tilde{𝑓}$ , 我们定义 $\tilde{𝑓}$ 的 relative error 为

$\frac{‖ \tilde{𝑓} (𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖}$

即: algorithm 给出的答案和正确答案的相对 difference.

如果对于每个 $𝑥 \in 𝑋$ 都有:

$\frac{‖ \tilde{𝑓} (𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖} = 𝑂 (𝜀_{machine})$

即 relative error is on the order of machine epsilon, 那么我们称这个 algorithm 是 accurate 的.

Problem: 对于一个 well-conditioned 的问题，我们自然地想要一个足够 accurate 的 algorithm; 但是对于 ill-conditioned 的问题，要求给出一个足够 accurate 的 algorithm 是很困难的事情，因为 perturbations on ill-conditioned inputs 使得它给出准确结果的难度很大.

因而，generally, 我们应该放低要求.

Definition 34.37 : stability of an algorithm

给定一个 problem $𝑓$ 和一个对应的 algorithm $\tilde{𝑓}$ , 如果对于每个 $𝑥 \in 𝑋$ 都存在一个 $\tilde{𝑥} \in 𝑋$ , 其满足

$\frac{‖ \tilde{𝑥} - 𝑥 ‖}{‖ 𝑥 ‖} = 𝑂 (𝜀_{machine})$

能够使得

$\frac{‖ \tilde{𝑓} (𝑥) - 𝑓 (\tilde{𝑥}) ‖}{‖ 𝑓 (\tilde{𝑥}) ‖} = 𝑂 (𝜀_{machine})$

那么则称，这个 algorithm $\tilde{𝑓}$ 是 statble 的.

Remark

显然，stable 是比 accurate 稍微宽松的要求. 它的要求是: 这个 algorithm gives nearly the right answer to nearly the right question. 比起 accurate, 它放宽在于: 不需要 solve exactly the same question, 只需要 solve 一个很相近的 question 就可以了.

为什么要这样要求? 因为一个计算机很大的问题是: 我们的 input 和理论上的 input 还是不一样的. 比如，我们要输入一个 $𝑥 = \frac{2}{3}$ , 这是一个无限循环的小数，而计算机只能输入有限位数的小数来近似. 因而关于它的计算问题，从输入起就有了误差. 这个误差可能很小，但是一旦遇到 ill-conditioned 的问题，在 condition 比较差的地方 (比如说某个问题在 $0$ 处, $𝜅 (𝑥) \to \infty$ )，那么这个 input 的小误差很可能导致很大的 Output 不同.

从而，我们需要对 input 放宽，尽可能去容忍类似于 round up 这样的问题.

这个时候我有一个问题: stable 这个概念，相比于 accurate，是为了迁就 ill-conditioned 的问题，只要保证我们的算法给出的结果一定是原问题周围某个相似的问题的数值解就可以了. 但是如果这个 problem 是 ill-conditioned，那么原问题的解可能和它周围的其他问题的解差别非常大. 一个 stable 但不 accuratae 的 algorithm，意思就是: 在某些 ill-conditioned 的 input 上，给出的解和我们想要的原问题的解差别非常大，那这个算法还能成为好吗?

这就是数值分析的核心哲学问题之一. stability 已经是一个足够的条件，因为你并不能要求算法给你“一个问题本身都无法承诺”的东西. Stable algorithm 是对现实的诚实反应，承认输入的不可避免误差，并保证: 你得到的结果是“某个微小扰动问题”的真实答案. 它告诉你: 在你所能拥有的误差范围内, 这就是最合理的答案了. 稳定性 = 不人为放大错误.

之所以我们只要求算法的稳定性，就是因为: 如果非要要求准确性，那么很多优秀的 algorithm 可能会因为仅仅几个 problem 本身就 ill-condtioned 的点上的大误差，被判为 inaccurate.

还有一个比 stable 更强的定义

Definition 34.38 : backward stability

给定一个 problem $𝑓$ 和一个对应的 algorithm $\tilde{𝑓}$ , 如果对于每个 $𝑥 \in 𝑋$ 都存在一个 $\tilde{𝑥} \in 𝑋$ , 其满足

$\frac{‖ \tilde{𝑥} - 𝑥 ‖}{‖ 𝑥 ‖} = 𝑂 (𝜀_{machine})$

能够使得 $\tilde{𝑓} (𝑥) = 𝑓 (\tilde{𝑥})$ 那么则称，这个 algorithm $\tilde{𝑓}$ 是backward stable 的.

backward stable 的要求是: 这个 algorithm gives exactly the right answer to nearly the right question. 它蕴含的信息是: 对于这个算法 $\tilde{𝑓}$ , 任意的 output perturbation 其实都等同于某些 input perturbation. (从而可以被 input perturbation 给完全控制.)

backward stable 和 accurate 是 dual 的: accurate 要求的是这个 algorithm gives nearly the right answer to the right question.

Example 34.11

我们用一个例子来阐明 “ $𝑂 (𝜀_{machine})$ ”:

problem: 给定 $𝑏$ , solve system $𝐴 𝑥 = 𝑏$ for $𝐴 \to 𝑥$ . 假设我们有一个 algorithm $\tilde{𝑓} : 𝐴 \to 𝑥$ 是 stable 的, 那么它满足: 对于给定的 $𝑛, 𝑚$ , 存在 uniform bound $𝐶_{1}, 𝐶_{2}$ 使得对于任意的 $𝐴 \in ℂ^{𝑛 \times 𝑚}$ , 都具有 nearly the same question $\tilde{𝐴}$ , 使得 algorithm $\tilde{𝑓}$ 给出的 answer $\tilde{𝑓} (𝐴)$ 几乎就是这个近似问题 $\tilde{𝐴}$ 的正确解 $𝑓 (\tilde{𝐴})$ .

Formally: 对于任意的 $𝐴 \in ℂ^{𝑛 \times 𝑚}$ , 都存在 $\tilde{𝐴} \in ℂ^{𝑛 \times 𝑚}$ s.t.

$\frac{‖ \tilde{𝐴} - 𝐴 ‖}{‖ 𝐴 ‖} \leq 𝐶_{1} 𝜀_{machine}$

使得

$\frac{‖ \tilde{𝑓} (𝐴) - 𝑓 (\tilde{𝐴}) ‖}{‖ 𝑓 (\tilde{𝐴}) ‖} \leq 𝐶_{2} 𝜀_{machine}$

这个 $𝐶_{1}, 𝐶_{2}$ 是和 input 进入的 $𝐴$ 无关的, 它被 problem 的参数固定 (here: $𝑛, 𝑚$ ). For example, $𝐶_{1} = 10, 𝐶_{2} = 100$ .

34.3.2 example: floating point arithmetic

Example 34.12

当然，四种 floating point arithmetic 是有 backward statble 的算法的.

我们以两数相减 from $𝑓 : ℂ^{2} \to ℂ$ 为例: 我们 canonical 的算法就是把这两个数 round 为 float，然后进行 float 的减法.

$\tilde{𝑓} (𝑥_{1}, 𝑥_{2}) = fl (𝑥_{1}) ⊖ fl (𝑥_{2})$

其中，

$fl (𝑥_{1}) = 𝑥_{1} (1 + 𝜀_{1}), fl (𝑥_{2}) = 𝑥_{2} (1 + 𝜀_{2})$

where by def, $| 𝜀_{1} |, | 𝜀_{2} | < 𝜀_{machine}$ .

并且我们知道, float 减法的 error 也是 within machine epsilon 的:

$fl (𝑥_{1}) ⊖ fl (𝑥_{2}) = (fl (𝑥_{1}) - fl (𝑥_{2})) (1 + 𝜀_{3})$

从而

$fl (𝑥_{1}) ⊖ fl (𝑥_{2}) = [𝑥_{1} (1 + 𝜀_{1}) - 𝑥_{2} (1 + 𝜀_{2})] (1 + 𝜀_{3})$

$= 𝑥_{1} (1 + 𝜀_{1}) (1 + 𝜀_{3}) - 𝑥_{2} (1 + 𝜀_{2}) (1 + 𝜀_{3})$

$= 𝑥_{1} (1 + 𝜀_{4}) - 𝑥_{2} (1 + 𝜀_{5})$

where

$| 𝜀_{4} |, | 𝜀_{5} | \leq 2 𝜀_{machine} + 𝑂 (𝜀_{machine}^{2}) = 𝑂 (𝜀_{machine})$

我们把

$\hat{𝑥_{1}} ≔ 𝑥_{1} (1 + 𝜀_{4}), \hat{𝑥_{2}} ≔ 𝑥_{2} (1 + 𝜀_{5})$

从而，这个 canonical algorithm 计算出的是:

$\tilde{𝑓} (𝑥_{1}, 𝑥_{2}) = 𝑓 (\hat{𝑥_{1}}, \hat{𝑥_{2}})$

where for all $(𝑥_{1}, 𝑥_{2})$ , 它对应的这个 $(\hat{𝑥_{1}}, \hat{𝑥_{2}})$ 和它在 $ℂ^{2}$ 中的 relative distance, within any norm 都是 $𝑂 (𝜀_{machine})$ 的.

34.3.3 example: inner/outer product

For inner product: problem is $𝑓 : ℂ^{𝑚} \times ℂ^{𝑚} \to ℝ$ , given vectors $𝑥, 𝑦 \in ℂ^{𝑚}$ , wish to compute the inner product $𝛼 = 𝑥^{*} 𝑦$ .

显然, canonical algorithm: compute the pairwise products ${| 𝑥 |}_{𝑖} 𝑦_{𝑖}$ with $\otimes$ and add them with $\oplus$ to obtain a computed result $\tilde{𝛼}$ .

这个算法是 backward stable 的.

但是, for outer product: problem is $𝑓 : ℂ^{𝑚} \times ℂ^{𝑛} \to ℂ^{𝑚 \times 𝑛}$ .

我们想要计算 $𝐴 = 𝑥 𝑦^{*}$ , for vectors $𝑥 \in ℂ^{𝑚}, 𝑦 \in ℂ^{𝑛}$ .

Canonical algorithm: compute the $𝑚 𝑛$ products $𝑥_{𝑖} {| 𝑦 |}_{𝑗}$ with $\otimes$ and collect them into a matrix $\tilde{𝐴}$ .

它是 stable 的，但却不是 backward stable 的. 因为直观而言: 我们每个 entry 的计算有不同的乘法误差，导致: $\tilde{𝐴}$ will 不太可能 have rank exactly 1, 因而无法真的被写作 written in the form $(𝑥 + 𝛿 𝑥) {(𝑦 + 𝛿 𝑦)}^{*}$ .

34.3.4 theorem: what backward stabililty implies about the accuracy

Theorem 34.43

Suppose a backward stable algorithm is applied to solve a problem $𝑓 : 𝑋 \to 𝑌$ with condition number $𝜅$ , 那么 relative errors:

$\frac{‖ \tilde{𝑓} (𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖} = 𝑂 (𝜅 (𝑥) 𝜀_{machine})$

(notice: 这说明如果 $𝜅$ of this problem bounded，那么 backward stable algorithm 一定是 accurate 的)

Proof

By backward stability, we have $\tilde{𝑓} (𝑥) = 𝑓 (\tilde{𝑥})$ for some $\tilde{𝑥} \in 𝑋$ satisfying

$\frac{‖ \tilde{𝑥} - 𝑥 ‖}{‖ 𝑥 ‖} = 𝑂 (𝜀_{machine})$

我们把 $\tilde{𝑥} - 𝑥$ 作为 $𝛿 𝑥$ , 从而有:

$\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖} = 𝑂 (𝜀_{machine})$

而由于这里 $\frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖} = 𝑂 (𝜀_{machine})$ 已经是 numerically 最小的 error. 从而 By definition of $𝜅 (𝑥)$ :

$\frac{‖ 𝑓 (𝑥 + 𝛿 𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖} \leq (𝜅 (𝑥) + 𝑜 (1)) \cdot \frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}$

这个不等式是因为: $𝑓$ 的相对变化和 $𝑥$ 的相对变化的比例，其上极限就是 $𝜅 (𝑥)$ . 从而 this implies

$\frac{‖ \tilde{𝑓} (𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖} = \frac{‖ 𝑓 (𝑥) - 𝑓 (\tilde{𝑥}) ‖}{‖ 𝑓 (𝑥) ‖} = \frac{‖ 𝑓 (𝑥 + 𝛿 𝑥) - 𝑓 (𝑥) ‖}{‖ 𝑓 (𝑥) ‖} \leq (𝜅 (𝑥) + 𝑜 (1)) \cdot \frac{‖ 𝛿 𝑥 ‖}{‖ 𝑥 ‖}$ □

这一 theorem 表明: backward stability + good conditioning $\Rightarrow$ accuratcy

35 backward error analysis of NLA algorithms

Source attribution in the selected TeX chapter title: doi:10.1137/1.9780898719574.ch3.

我们在上一个 Ch 中介绍了 conditioning 和 stability. 现在我们用它们对经典的 NLA algorithms 进行 backward error analysis.

35.1 Stability of Householder Triangularization

我们 set $𝑅, 𝑄$ to be random upper triangular 和 orthogonal matrices (by orthogonizing 一个 random matrix)，并 set $𝐴 ≔ 𝑄 𝑅$ .

R = triu (randn(50));
[Q,X] = qr(randn(50));
A = Q*R

然后我们再对 $𝐴$ 进行 QR 分解, via Household (Matlab 自带使用 Household), 看看 relative error:

[Q2,R2] = qr(A);
norm (Q2 - Q);
    ans = 0.00889
norm (R2-R) / norm(R);
    ans = 0.00071

我们发现 $𝑄, 𝑅$ 的 relative error 其实很大.

但是，当我们用这个 $𝑄 𝑅$ 计算 $𝐴$ 时:

norm (A - Q2*R2) / norm(A);
    ans = 1.432e-15

我们发现一个惊人的事实: 这个 QR 分解的 error

$Q3 = 𝑄 + 1 e - 4 * randn (50)$

$R3 = 𝑅 + 1 e - 4 * randn (50)$

$\frac{norm (𝐴 - Q3 * R3)}{norm (𝐴)}$

$ans = 0.00088$

1 Introduction

2 Linear equations, vectors, and matrices

2.1 WS 1, p. 1: vectors and vector spaces

2.2 WS 1, p. 2: augmented matrices and RREF

3 Linear systems and matrices

4 Linear combinations

4.1 WS 3, p. 1: geometry and number of solutions

4.2 Number of solutions of a linear system

4.3 WS 3, pp. 2–3: matrix algebra and linear combinations

5 Linear transformations

5.1 WS 4, p. 1: coordinate encoding and standard matrices

5.2 WS 4, p. 2: linearity, bases, and transition matrices

6 Geometry of linear transformations

6.1 WS 5, p. 1: examples, scaling, and projection

6.2 WS 5, p. 2: rotations and shearing

6.3 5. Shearing

7 Gram–Schmidt and QR factorization

7.1 WS 17, p. 1: proof of QR factorization QR factorization

7.2 Proof of matrix product theorem

8 Orthogonal transformations

8.1 WS 18, p. 1: definition and matrix criteria

8.2 WS 18, p. 2: products and change of basis

8.3 WS 18, p. 3: conclusion

9 Least squares

9.1 WS 19, p. 1: projection and image/kernel theorem

9.2 WS 19, p. 2: normal equations

10 Inner-product spaces

10.1 WS 20, p. 1: examples and generalized Gram–Schmidt

10.2 WS 20, p. 2: coordinates and matrix inner products

11 Diagonalization and eigenspaces

11.1 WS 23, p. 1: four theorems

12 Eigenvalues and eigenspaces

12.1 WS 24, p. 1: characteristic polynomial and multiplicities

13 Complex eigenvalues

13.1 WS 26, p. 1: complex roots and real 2×2 matrices

14 Spectral theorem

14.1 WS 27, p. 1: theorem and first two claims

14.2 WS 27, p. 2: induction proof

14.3 Homework 1 — submitted work

14.3.1 Source p. 1 — Exercise 20

14.3.2 Source p. 2 — Exercise 20; Exercise 32 begins

14.3.3 Source p. 3 — Exercise 32

14.3.4 Source p. 4 — Exercise 34

14.3.5 Source p. 5 — Exercise 34; Exercise 44

14.3.6 Source p. 6 — Exercise 12

14.3.7 Source p. 7 — Exercise 12

14.3.8 Source p. 8 — Exercise 12

14.3.9 Source p. 9 — Exercise 36

14.3.10 Source p. 10 — Exercise 44

14.3.11 Source p. 11 — Exercise 44

14.3.12 Source p. 12 — Exercise 44

14.3.13 Source p. 13 — Problem 1(a)–(c)

14.3.14 Source p. 14 — Problem 1(d)–(e); Problem 2(a)

14.3.15 Source p. 15 — Problem 2(b)–(d)

14.3.16 Source p. 16 — Problem 2(e)–(g)

14.3.17 Source p. 17 — Problem 3

14.3.18 Source p. 18 — Problem 4

14.3.19 Source p. 19 — Problem 5

14.3.20 Source p. 20 — Problem 6(a)

14.3.21 Source p. 21 — Problem 6(b)

14.3.22 Source p. 22 — Problem 7

14.3.23 Source p. 23 — Problem 7

14.4 Homework 2 — submitted work

14.4.1 Source p. 1 — Exercise 26

14.4.2 Source p. 2 — Exercise 26; Exercise 34 begins

14.4.3 Source p. 3 — Exercise 34

14.4.4 Source p. 4 — Exercise 48(a)–(b)

14.4.5 Source p. 5 — Exercise 48(c)

14.4.6 Source p. 6 — Exercise 48(c)

14.4.7 Source p. 7 — Exercise 6

14.4.8 Source p. 8 — Exercise 38; Exercise 44 begins

14.4.9 Source p. 9 — Exercise 44

14.4.10 Source p. 10 — Exercise 46

14.4.11 Source p. 11 — Part B, Problem 1(a)–(b)

14.4.12 Source p. 12 — Part B, Problem 1(c)–(d)

14.4.13 Source p. 13 — Part B, Problem 2(a)–(b)

14.4.14 Source p. 14 — Part B, Problem 2(b)–(c)

14.4.15 Source p. 15 — Part B, Problem 2(d)–(e)

14.4.16 Source p. 16 — Part B, Problem 2(e)

14.4.17 Source p. 17 — Part B, Problem 3(a)

13.1 WS 26, p. 1: complex roots and real $2 \times 2$ matrices

14.7.2 Problem 2 - the vector space $ℱ︀ (𝑆, 𝑉)$

14.8.5 Exercise 28 - commuting $2 \times 2$ matrices