Kaczmarz Kac walk

The Kaczmarz method is a way to iteratively solve a linear system of equations <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="upper A x equals b"> <mml:semantics> <mml:mrow> <mml:mi>A</mml:mi> <mml:mi>x</mml:mi> <mml:mo>=</mml:mo> <mml:mi>b</mml:mi> </mml:mrow> <mml:annotation encoding="application/x-tex">Ax = b</mml:annotation> </mml:semantics> </mml:math> </inline-formula>. One interprets the solution <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="x"> <mml:semantics> <mml:mi>x</mml:mi> <mml:annotation encoding="application/x-tex">x</mml:annotation> </mml:semantics> </mml:math> </inline-formula> as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="upper A Superscript left-parenthesis k right-parenthesis Baseline x equals b Superscript left-parenthesis k right-parenthesis"> <mml:semantics> <mml:mrow> <mml:msup> <mml:mi>A</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mo stretchy="false">(</mml:mo> <mml:mi>k</mml:mi> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> </mml:msup> <mml:mi>x</mml:mi> <mml:mo>=</mml:mo> <mml:msup> <mml:mi>b</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mo stretchy="false">(</mml:mo> <mml:mi>k</mml:mi> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> </mml:msup> </mml:mrow> <mml:annotation encoding="application/x-tex">A^{(k)} x = b^{(k)}</mml:annotation> </mml:semantics> </mml:math> </inline-formula> which is fast to compute, preserves the original solution and whose small singular values grow like <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="sigma Subscript script l Baseline left-parenthesis upper A Superscript left-parenthesis k right-parenthesis Baseline right-parenthesis tilde exp left-parenthesis k slash n squared right-parenthesis dot sigma Subscript script l Baseline left-parenthesis upper A right-parenthesis"> <mml:semantics> <mml:mrow> <mml:msub> <mml:mi>σ</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mi>ℓ</mml:mi> </mml:mrow> </mml:msub> <mml:mo stretchy="false">(</mml:mo> <mml:msup> <mml:mi>A</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mo stretchy="false">(</mml:mo> <mml:mi>k</mml:mi> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> </mml:msup> <mml:mo stretchy="false">)</mml:mo> <mml:mo>∼</mml:mo> <mml:mi>exp</mml:mi> <mml:mo>⁡</mml:mo> <mml:mo stretchy="false">(</mml:mo> <mml:mi>k</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mo>/</mml:mo> </mml:mrow> <mml:msup> <mml:mi>n</mml:mi> <mml:mn>2</mml:mn> </mml:msup> <mml:mo stretchy="false">)</mml:mo> <mml:mo>⋅</mml:mo> <mml:msub> <mml:mi>σ</mml:mi> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mi>ℓ</mml:mi> </mml:mrow> </mml:msub> <mml:mo stretchy="false">(</mml:mo> <mml:mi>A</mml:mi> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> <mml:annotation encoding="application/x-tex">\sigma _{\ell }(A^{(k)}) \sim \exp (k/n^2) \cdot \sigma _{\ell }(A)</mml:annotation> </mml:semantics> </mml:math> </inline-formula>.