Question

For 1D, the Newton method is

\[x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\]

We can generalize to \(k\) dimensions by

\[x_{n+1} = x_n - J^{-1} f(x_n)\]

where \(x\) and \(f(x)\) are now vectors, and \(J^{-1}\) is the inverse Jacobian matrix. In general, the Jacobian is not a square matrix, and we use the generalized inverse \((J^TJ)^{-1}J^T\) instead, giving

\[x_{n+1} = x_n - (J^TJ)^{-1}J^T f(x_n)\]

In multivariate nonlinear estimation problems, we can find the vector of parameters \(\beta\) by minimizing the residuals \(r(\beta)\),

\[\beta_{n+1} = \beta_n - (J^TJ)^{-1}J^T r(\beta_n)\]

where the entries of the Jacobian matrix \(J\) are

\[J_{ij} = \frac{\partial r_i(\beta)}{\partial \beta_j}\]