EngNote - Kalman Filtering

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Kalman Filtering.

A Kalman filter is a filtering mechanism that estimates the state of something using a process that describes how the state changes over time and uses measurements that are mathematically related to the state to update the estimate.  WORK IN A DESCRIPTION OF PROCESS.  REMINDER THE PROCESS INCLUDES THE 'A' MATRIX BUT ALSO IN A MORE GENERAL SENSE INCLUDES THE CONTROL INPUT AND PROCESS NOISE. 

A common application of a Kalman filter is radar target tracking.  In this application, the state being estimated is the target being tracked.  The process that describes how the state changes is a set of kinematic equations that relate position, velocity, acceleration, etc.  Common measurements a radar periodically makes that are mathematically related to the state/target are range, azimuth, elevation.

The Kalman filter process is recursive.  Only the previous filter estimates and a new measurement are needed to make new filter estimates.  As a result, the Kalman filter is computationally efficient, making it attractive for many real-time applications.  Other estimation methods (example: linear least squares fitting) require all previous measurements to estimate a state which can be computationally and memory intensive.

Types of Kalman Filters

There are a number of Kalman filters implementations. They all perform the same function but differ in the types of process and measurement relationships they support.

• "Plain" Kalman Filter (KF)
• Extended Kalman Filter (EKF)
• Unscented Kalman Filter (UKF)

"Plain" Kalman Filter

The simplest, the Kalman Filter, assumes that a state's process and the relationship between the state and the measurements used to update the state estimate are linear.  This makes the process of updating state estimates very easy because applying linear operations to the state estimate (a Gaussian random variable) is a linear operation itself that results in another Gaussian random variable.  Simple enough.

Linear Operation On Gaussian

Assume $\vec{x}$ is a multivariate Gaussian random variable:

$\vec{x} = \mathcal{N}\left ( \vec{\mu} ,V \right )$

And $\vec{y}$ is a linear transformation of $\vec{x}$:

$\vec{y}=\vec{a}+B\vec{x}$

Then expected value of $\vec{y}$ is:

$E\left [ \vec{y} \right] = \vec{a}+B\vec{\mu}$

And the covariance of $\vec{y}$ is:

$COV\left [ \vec{y} \right ]=BVB^{T}$

Proof of covariance:

$\begin{matrix} COV\left [ \vec{y} \right ] & = & E\left [ \left ( \vec{y} - \mu_{\vec{y}} \right ) \left ( \vec{y}-\mu_{\vec{y}} \right )^{T} \right ] \\ & = & E\left [ \left( \left( \vec{a}+B\vec{x} \right)-\left( \vec{a}+B\mu_{\vec{x}} \right) \right) \left( \left( \vec{a}+B\vec{x} \right)-\left( \vec{a}+B\mu_{\vec{x}} \right) \right)^{T} \right] \\ & = & E \left[ \left( B \left(\vec{x}-\mu_{\vec{x}} \right) \right) \left( B \left(\vec{x}-\mu_{\vec{x}} \right) \right) ^{T} \right] \\ & = & E \left[ B \left(\vec{x}-\mu_{\vec{x}} \right) \left( \vec{x}-\mu_{\vec{x}} \right)^{T} B^{T} \right] \\ & = & B E \left[ \left(\vec{x}-\mu_{\vec{x}} \right) \left( \vec{x}-\mu_{\vec{x}} \right)^{T} \right] B^{T} \\ & = & BVB^{T} \end{matrix}$

A graphical way to think of a linear transformation of a Gaussian random variable is in the context of a coordinate system transformation. The linear transformation is changing the coordinate system of the variables.  Below are explicit transformations, both linear and non-linear, of a Gaussian random variable.

Linear Transformation of Gaussian Random Variable X: $y=Bx$

$f_{X} \left( x \right) = \frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{\left(x-\mu\right)^{2}}{2\sigma^{2}}}$ $x=\frac{y}{B}$
$\left| \frac{dx}{dy} \right| = \frac{1}{B}$
$\begin{matrix} f_{Y} \left( y \right) & = & f_{X}(x(y))\left|\frac{dx}{dy}\right| \\ & = & \frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{\left(\frac{y}{B}-\mu\right)^{2}}{2\sigma^{2}}}\frac{1}{\left|B\right|} \\ & = & \frac{1}{\sqrt{2\pi(B\sigma)^{2}}}e^{-\frac{(y-B\mu)^{2}}{2\left(B\sigma\right)^{2}}} \end{matrix}$
The result is a Gaussian with mean $B\mu$ and variance $B^{2}\sigma^{2}$

Nonlinear Transformation of Gaussian Random Variable X: $y=x^{2}$

$f_{X} \left( x \right) = \frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{\left(x-\mu\right)^{2}}{2\sigma^{2}}}$ $x=\pm\sqrt{y}\,,\;y>0$
$\left| \frac{dx}{dy} \right| = \frac{1}{2\sqrt{y}}$
$\begin{matrix} f_{Y} \left( y \right) & = & f_{X}(x(y))\left|\frac{dx}{dy}\right| \\ & = & 2\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{\left(\sqrt{y}-\mu\right)^{2}}{2\sigma^{2}}}\frac{1}{2\sqrt{y}} \\ & = & \frac{1}{\sqrt{2\pi\sigma^{2}y}}e^{-\frac{\left(\sqrt{y}-\mu\right)^{2}}{2\sigma^{2}}} \end{matrix}$
The result is not a Gaussian.  In this case, it is a non-central (non-zero mean) chi-squared distribution.

Extended & Unscented Kalman Filters

The Kalman filter works with linear relationships.  Applying non-linear operations/transformations to Gaussian random variables does not (in general) result in new Gaussian random variables.  What happens when the process and/or the measurement relationship to the state are non-linear?  We must linearize the problem.  This is where the EKF and the UKF come into play.

The Extended Kalman Filter and the Unscented Kalman Filter are extensions of the Kalman Filter that do not require linear relationships.  These filter types allow the process and/or the measurement relationship to the state to be a non-linear function.  Since many real-world relationships are non linear, these filter types are common and extremely popular.  Often and confusingly, the adjective Extended or Unscented is frequently left off when using these filter types.

Both the EKF and the UKF work very well when the transformed distribution is nearly Gaussian.  Loosely defined, non-linear transformations of Gaussian random variables result in random variables with nearly Gaussian distributions when the transformation is approximately linear near the mean of the variable being transformed and the variance of the variable being transformed is small such that its values are concentrated in a small neighborhood near the mean.

Extended Kalman Filter

The Extended Kalman filter uses derivatives (Calculus, gasp!) to linearize the process and/or the measurement relationship to the state at the current estimate.  This approach tends to work well for relationships that are close to linear over the timescales involved but can break down if they are not.

Unscented Kalman Filter

The Unscented Kalman Filter uses what is known as an Unscented Transform to calculate the statistics of a random variable that undergoes a nonlinear transformation.  Samples of the state undergoing a non-linear transformation are chosen such that the sample mean and sample covariance match the currently predicted mean and covariance of the process.  The non-linear process is applied to each of the sample points and the resulting mean and covariance of the transformed points is used to represent a Gaussian approximation of the transformed Gaussian random variable.  Note, this method approximates the transformed random variable's distribution whereas the EKF approximates the non-linear function/transformation.

Simple Visual Description

Visually thinking about what a Kalman Filter is doing can be very handy. I find it helps see the forest through the trees.  The filter can be boiled down into a data fusion problem.  A filter starts with an initial seed that has a relatively high uncertainty.  Measurements that have their own uncertainty are taken that give us an estimate of the state.  Each measurement enhances the filter's knowledge of the state.  How much it enhances the state estimate depends on the noise (covariance) associated with the measurements.  At the beginning of the filter process, the measurements and the state estimates may have similar certainty.  If designed correctly, as more measurements are made, the state estimate certainty should increase and the state prediction should become more accurate.  Below is animation of this process.  In this simple case, the filter is attempting to estimate a signal whose true value is 100. 

Below are animations of the same process as above except I've introduced some "issues" with the filters.  The filter producing the illustration on the left has a poor estimate of the measurement covariance that is input to the filter.  In the case illustrated, the measurement covariance is underestimated which causes the state estimate to vary rapidly and to a much larger degree than the state's estimated covariance suggests.  The illustration on the right shows the impact of a bias or error on the initial state estimate or equivalently, an improbable measurement seeding the initial state estimate.  As more and more measurements are taken and used to update the filter, the state estimate bias slowly shrinks.

Kalman Filter With Incorrect Measurement Covariance	Kalman Filter With Bias on Initial State Estimate

Visual Example of Linear and Non-Linear Transformations

Visualizing how a transformation changes a random variable's probability density function is helpful in intuitively understanding how the KF works.  After all, state and measurement random variables are constantly being transformed by linear or non-linear functions as part of the KF.  Below is a derivation of how to determine the transformed random variable's probability density function.

Derivation of Probability Density Function Transformation

X is a random variable with pdf f(x)
$ P \left( a \leqslant X < b \right) = \int_{a}^{b} f\left( x \right) dx $
Y is a random variable derived from X
$ Y = y \left( X \right) $
$ P \left( y \left( a \right) \leqslant Y < y \left( b \right) \right) = P \left( a \leqslant X < b \right) $
Using integration by substitution
$ P \left( y \left( a \right) \leqslant Y < y \left( b \right) \right) = \int_{y \left( a \right)}^{y \left(b\right)} f\left(x\left(y\right)\right)\frac{dx}{dy}dy  $
If we define g(y) as
$ g \left( y \right) = f \left ( x \left ( y \right ) \right )  \frac{dx}{dy} $
Then we can re-write x.x
$ P \left( y \left( a \right) \leqslant Y < y \left( b \right) \right) = \int_{y \left( a \right)}^{y \left(b\right)} g \left( y \right) dy  $
By definition $ g \left( y \right) $ is the transformed pdf associated with the random variable Y $ g \left ( y \right ) = f \left ( x \left ( y \right ) \right ) \left | \frac{dx}{dy} \right | $

Example illustrations of transformed pdfs are shown below.  The grey dotted lines across the subplots illustrate how points in the original pdf are stretched/scaled by the transformation function and subsequently mapped to the new pdf.  The first example is a linear transformation of a Gaussian random variable X.  Since the transformation is linear, the resulting transformed variable Y is also Gaussian.  $ \mathcal{N}(0,0.25) \rightarrow \mathcal{N}(0,0.5) $

The next example is a linear transformation with a shifted mean.  The resulting transformed random variable is still Gaussian.  $ \mathcal{N}(0,0.25) \rightarrow \mathcal{N}(-1,0.5) $

The last example is a non-linear transformation.  Notice the resulting distribution is no longer Gaussian.

Visualizing How the EKF and UKF Linearize the Transformation Process

The EKF and the UKF approximate non-linear transformations of a Gaussian random variable.  In general, the true distribution of the resulting random variable is no longer Gaussian.  However, both the EKF and the UKF linearize the problem and estimate the transformed variable as a Gaussian random variable.  The EKF estimates the transformed distribution by approximating the non-linear function as a linear function and then applying that linear transformation to the original Gaussian distribution.  The UKF creates sample points (sigma points) of the original distribution, explicitly evaluates the non-linear function on these sample points, and then estimates a transformed Gaussian distribution using the resulting mean and standard deviation of the transformed sample points.

Below is an illustration of a cosine transformation function on a Gaussian random variable.  The resulting true distribution, linearized distribution estimate (akin to EKF), and sampled distribution estimate (akin to UKF) are shown.  There are noticeable differences between the estimates and the true distribution.   

The illustration below is the same as above except the random variable being transformed has a smaller variance.  The resulting transformed pdfs more closely match each other compared to the above illustration.  This behavior is intuitive.  As the variance of the variable being transformed shrinks the functional span of the non-linear transformation shrinks and more closely approximates a linear transformation.   

Algorithm Descriptions

Kalman Filter

Kalman Filter Variables

Variable	Size	Description
x	n	State predictions
z	m	Measurements
R	m x m	Measurement covariance
A	n x n	Process (or difference equation), relates state at previous step to state are the current step
H	m x n	Measurement equation relates measurements to the state
P	n x n	State estimate covariance
K	n x m	Kalman filter gain, weight that controls how much measurements vs predictions influence the filter estimate
u	l	Control signal
B	n x l	Control process, describes how the control signal changes the state
Q	n x n	Process noise covariance

State Prediction $\hat{x}_{\hat{k}} = A\hat{x}_{k-1}+Bu_{k-1}$
State Covariance Prediction $P_{\hat{k}}=AP_{k-1}A^{T}+Q$
Filter Gain Calculation $K_{k}=P_{\hat{k}}H^{T}(HP_{\hat{k}}H^{T}+R)^{-1}$
Updated State Estimate $\hat{x}=\hat{x}_{\hat{k}}+K_{k}(z_{k}-H\hat{x}_{\hat{k}})$
Updated State Estimate Covariance $P_{k}=(I-K_{k}H)P_{\hat{k}}$

Extended Kalman Filter

Extended Kalman Filter Variables

Variable	Size	Description
x	n	State predictions
z	m	Measurements
R	m x m	Measurement covariance
a		Non-linear process (or difference) equations that relate state at previous step to state are the current step
A	n x n	Jacobian of the process (or difference equations), matrix of partial derivatives that provide a linear approximation of the process functions at the current state prediction
h		Non-linear measurement equations that relate the measurements to the state
H	m x n	Jacobian of the measurement equations, matrix of partial derivatives that provide a linear approximation of the measurement functions at the current state predictions
P	n x n	State estimate covariance
K	n x m	Kalman filter gain, weight that controls how much measurements influence the state
u	l	Control signal
B	n x l	Control process, describes how the control signal changes the state
Q	n x n	Process noise covariance
W	n x n	Process noise jacobian

State Prediction $\hat{x}_{\hat{k}} = a \left ( \hat{x}_{k-1},u_{k-1}, 0 \right )$
State Covariance Prediction $P_{\hat{k}}=A_{k}P_{k-1}A^{T}_{k}+W_{k}Q_{k-1}W^{T}_{k}$
Filter Gain Calculation $K_{k}=P_{\hat{k}}H^{T}_{k}(H_{k}P_{\hat{k}}H^{T}_{k}+V_{k}R_{k}V^{T}_{k})^{-1}$
State Prediction Update $\hat{x}_{k}=\hat{x}_{\hat{k}}+K_{k}(z_{k}-h \left( \hat{x}_{\hat{k}},0 \right) )$
State Covariance Prediction Update $P_{k}=(I-K_{k}H_{k})P_{\hat{k}}$

Unscented Kalman Filter

Extended Kalman Filter Variables

Variable	Size	Description
x	n	State predictions
X	n x 2n+1	Sigma points, a "cloud" of points that sample the random variable around its estimate, undergo a non-linear transformation, then used to estimate the transformed random variable's mean and covariance
mW	2n+1	Weight vector used to calculate the transformed distribution's mean from the transformed sigma points
cW	2n+1	Weight vector used to calculate the transformed distribution's covariance from the transformed sigma points
z	m	Measurements
R	m x m	Measurement covariance
a		Non-linear process (or difference) equations that relate state at previous step to state are the current step
h		Non-linear measurement equations that relate the measurements to the state
P	n x n	State estimate covariance
K	n x m	Kalman filter gain, weight that controls how much measurements influence the state
u	l	Control signal
B	n x l	Control process, describes how the control signal changes the state
Q	n x n	Process noise covariance
W	n x n	Process noise jacobian

$\begin{matrix} x_{0} & = & \mu_{x} & W_{0} & = & \frac{k}{n+k}\\ x_{i} & = & \mu_{x}+\left(\sqrt{\left( n+k \right)P_{x}} \right) & W_{i} & = & \frac{1}{2(n+k)} \\ x_{i+n} & = & \mu_{x}-\left(\sqrt{\left( n+k \right)P_{x}} \right) & W_{i+n} & = & \frac{1}{2(n+k)} \end{matrix}$ $y_{i}=f \left(x_{i} \right)$ $\hat{y}=\sum_{i=0}^{2n}W_{i}y_{i}$ $P_{y}=\sum_{i=0}^{2n} \left( y_{i}-\hat{y} \right ) \left( y_{i}-\hat{y} \right )^{T}$



In the equations below $x$ represents a random variable and $\theta$ represents the set of parameters that describe the distribution.

Probability Density Function:
$ f \left ( x | \theta \right ) $
The probability of observing $x$ given a distribution described by $\theta$

Joint Probability Density Function:
$ f \left ( x_{1},...,x_{n} | \theta \right ) = P \left ( X_{1} = x_{1},...,X_{n}=x_{n} | \theta \right ) = f \left ( x_{1} | \theta \right ) \cdot ...  f \left ( x_{n} | \theta \right ) = \prod \limits_{i=1}^{n} f \left ( x_{i} | \theta \right ) $
The probability of observing ($x_{i}$) multiple independent and identically distributed random variables ($X_{i}$) given by a distribution described by $\theta$.

Likelihood Function:
$ L\left ( \theta |x \right ) = \prod\limits_{i=1}^{n} f \left ( x_{i} | \theta \right ) $
If we swap the independent and dependent variables from the joint probability density function above we arrive at what is called the likelihood function.  In this equation, the distribution parameters ($\theta$) are treated as the unknowns and the observations ($x_{i}$) are known.


In the likelihood function, we want to find the distribution parameters ($\theta$) that maximize the probability likelihood with respect to the observations ($x_{i}$) that we have seen.  In order to do this we use a bit of calculus.  First, calculate the derivative (or partial derivatives) of the likelihood function with respect to the distribution parameter(s).  Next set the derivative or partial derivatives to zero and solve for the distribution parameters.

A trick normally used is to perform the above steps against the natural log of the likelihood function, also known as the loglikelihood function.  This is a trick because the loglikelihood often has a much simpler form than the likelihood, is also often easier to differentiate, and because the natural log is an increasing function, maximizing the loglikelihood is the same as maximizing the likelihood.

Loglikelihood Function:
$ L_{ln}\left ( \theta |x \right ) = ln \left ( \prod\limits_{i=1}^{n} f \left ( x_{i} | \theta \right ) \right ) $


For a Gaussian distribution, $\theta$ is comprised of a mean ($\mu$) and variance ($\sigma$), $\theta = \left( \mu,\sigma \right )$

In the above equations, it is assumed there are multiple iid (independent and identically distributed) random variables (measurements).


The Kalman filter combines or fuses two Gaussian distributed data sources, the filter's predicted measurements and the actual measurements, to estimate the filter's current state.  Combining these two sources is done by calculating the joint probability distribution of the two sources and then maximizing the resulting likelihood function




 prediction of a measurement the previous state estimate's  (a Guassian distribution) with that of a

, the estimate of the filter state is accomplished by

the random variables are independent but are usually not identically distributed. The state estimated measurement(s) and the sensor measurement(s) are independent Gaussian random variables however their pdfs are rarely the same. In this case the likelihood looks like the following:

$ L\left ( \theta |x \right ) = f_{pdf} \left ( x_{meas}, \mu, \sigma_{meas} \right ) \times f_{pdf} \left ( x_{stateEstMeas}, \mu, \sigma_{stateEstMeas} \right ) $


where “log” means natural log (logarithm to the base e). 

Code

GitHub

Good References

http://cecas.clemson.edu/~ahoover/ece854/ [1] An Introduction to the Kalman Filter [Local Mirror]
http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf

[2] Understanding the Basis of the Kalman Filter [Local Mirror]
http://www.cl.cam.ac.uk/~rmf25/papers/Und erstanding the Basis of the Kalman Filter.pdf

[3] Kalman Filter For Dummies [Local Mirror]
http://bilgin.esme.org/BitsBytes/KalmanFilterforDummies.aspx

[4] First-Hand:The Unscented Transform [Local Mirror]
http://ethw.org/First-Hand:The_Unscented_Transform

[5] The Unscented Kalman Filter for Nonlinear Estimation [Local Mirror]
http://www.cslu.ogi.edu/nsel/ukf/

[6] Reduced Sigma Point Filters for the Propagation of Means and Covariances Through Nonlinear Transformations [Local Mirror]

[7] A New Extension of the Kalman Filter to Nonlinear Systems [Local Mirror]

[8] Unscented Filtering and Nonlinear Estimation [Local Mirror]

[9] A more robust unscented transform [Local Mirror]
https://www.mitre.org/sites/default/files/pdf/vanzandt_unscented.pdf

[10] How a Kalman filter works, in pictures [Local Mirror]
http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/

Making the Kitchen Table

This past winter (2015-2016) I decided to make a dining table for our new house.  The good news is I got to make it exactly how Sarah and I wanted. It is all hardwood and I got to learn a ton making it.  It also gave me an excuse to buy some new tools.  The bad news is it was slow going and it wasn't cheaper than buying a table from a store.  I don't care, it has been a labor of love.  Below is documentation of the build.  About halfway through I stopped logging consistently but I did take pictures.

The Proverbial Specs and Finished Product Pictures

Dimensions:  42" x 84", 30" tall

Top: Walnut, 8 1x6s butt joined
Breadboard: Hard White Maple
Legs: Red Oak, 4 1x4s glued, cut to exactly 3x3, then tapered from 3" to 1.5"
Apron: Red Oak, 2 1x3s glued

The Play-By-Play

(01/04/2016) I picked up the S4S walnut lumber tonight from Owl Hardwood Lumber.  Dad (and Mom) met me at Owl, we loaded it in Dad's car, and drove it to my house.  All total I bought 8 S4S 1" x 6" x 8' boards (64bdft) at $5.55 per board foot.  I cut up some scrap 2x6s and heavy duty paper rolls to properly sticker the boards.  Here is hoping the boards acclimate in the basement and don't warp much.  The temperature is approximately 63F and 35-40% relative humidity in the basement.

Boards "stickered" to allow air to circulate evenly around all sides of the boards

(01/10/2016)  The painstaking and tedious task of edge gluing or butt joining the 1x6s started today.  I laid out the boards and picked the orientation I wanted.

The dark areas above are from me dumping mineral spirits on the boards to try and get a feel for how the gnarly grain on the two boards would look after applying a finish.  The spots got pretty dark and the basement lighting is terrible but at the right angle the funky gain pattern looks nice.  We shall see how the finish looks in better light

The work table the boards are laid on is made from a 4x8 sheet of particle board covered with melamine, two saw horses, and two 2x4s screwed length wise across the particle board to minimize sagging.

How to join the boards took me a while to figure out.  The glue doesn't set instantly but the set timeline doesn't allow for much fidgeting.  I wound up with 7 cauls, 14 clamps for the cauls, and 6 pipe clamps.  Thanks to Jon for letting me borrow some clamps, I didn't have 14 at the ready.  In order to get comfortable and speedy with the process, I dry ran a glue-up without the glue.  That proved very useful.  Placement of the cauls, clamps, and the direction of the pipe clamps was important.  It also became obvious I needed Sarah's help to work quickly enough.

Two key points I figured out while doing the first glue-up.  Putting packing tap on the surface of the cauls is important to keep the cauls from getting glued to the boards and painters tape along the edges works really well at keeping squeeze-outs from making a mess of the boards.  When I removed the painters tape it didn't leave a residue and only lightly raised the grain.

Edges tapped.  Will definitely do again.

Glue-up from the side.  There was no support in the middle so the boards sagged.  I later added support.

Glue-up straight on.

(01/24/2016)  There has been a good amount of activity.  The whole top is glued up!  I learned a few things along the way.  I started the glue-up using standard PVA wood glue.  When I started the second round of glue-ups I found the glue was setting up too quickly for me to work with.  Even with Sarah helping and dry-running the process I had to frantically tear apart a joint when it was way out of flush.  At this point some google-fu convinced me to switch to polyurethane (Gorilla) glue.  This switch helped a lot in terms of working time with the glue up.  To make sure the glue took a nice and long time to set, I didn't moisten the edges.  The basement was also nice and dry, 30% humidity, since this is the dead of winter.

The switch in glue did present a few new problems.  The tape didn't work as well at controlling squeeze-out as it did with wood glue.  It did help, but I did get a little bit of glue seeping below the tape.  The glue also kept getting on my hands and ending up on the boards wherever I pressed my fingers during the glue up.  Another PITA, I had to wait until the glue was completely set before scraping off the squeeze-out.  Trying to scrape it off before it set made a sticky mess.  The actual removal was also much harder since I was now waiting for the glue to completely set.  It was incredibly hard to scrap off.

First Glue-up Using Gorilla Glue

Entire Tabletop Glue-up

Entire Tabletop Glue-up

(February)  I decided I wanted something unique for the table.  After all, Sarah and I get to call all the shots.  After disagreeing on a light colored maple base Sarah and I both agreed Maple breadboard ends would offer a bit of interesting contrast.  I also happen to be a sucker for nicely figured Maple.  I read up on breadboard ends and decided to take a simple route - a mortise and tenon joint using a router bit.  Being new and nervous to more sophisticated joinery I decided to take it simple and glue the joint in the middle instead of dowels and channels.

Routing the ends of the walnut top was nerve racking.  Using the router table, a makeshift fence, and the help of Jon I was able to successfully route a tenon into the edges of the table and a mortise into the maple breadboard ends.  The only thing I really regret on the table came next.  I messed up gluing the middle of the mortise and tenon joint twice in a row.  I dry-fit the joints and everything lined up really well.  While gluing the joints I put too much glue in the middle of the breadboards.  When I tried to use my mallet to tap the ends flush the glue got in the way.  I didn't have clamps long enough to span the table and couldn't apply enough pressure to squeeze the glue out and ended up with a small gap between the top and the breadboards.  Somewhere along the lines of 1/64" to 1/32".  A bummer but my only real regret.  Lesson learned for next time!

Router bit installed and fence setup

The tenon done.  This really made me sweat.

Mortise and tenon.  Perfect mates. . . except for the glue.

All glued up.

Closeup of the joint on the finished table.

(March)  Five 1x2 red oak cross beams were attached to and run perpendicular to the table top boards for support.  Slots were drilled into the beams and screws with washers were run through the the middle of the slots and into the table top.  This should make for a secure attachment that allows for seasonal wood movement.  I missed getting pictures of this but it isn't all that exciting.  In the design, these cross beams fit very snug inside the apron, spanning the entire inner width of the apron and keep the top in place in the apron.

The legs and apron are made of red oak.  Each leg is made of 4 1x4 red oak boards laminated together.  Laminating boards together was significantly cheaper than buying 4x4 oak.  I cut the laminated 4x4s to a true 3x3. To make the legs look less chunky I added a taper to the legs by using a taper jig on the table saw.  They start at 3" and end at 1.5".

The apron is made from 1x3s laminated to 2x3s.  The apron and legs are attached using mortise and tenon joints.

The joints were made with a plunge router, chisels, and a good amount of sanding.  After dry fitting the joints, I used gorilla glue to permanently attach the joints.

Dry Fitting the Joints

The mortise and tenon joints are solid but I decided to add more reinforcement using kerf mount corner brackets.

Kerf Mount Corner Brackets

Attaching the lag bolts took a little bit of thought.  Note to future self, thread two nuts on to the machine thread, tighten them very tightly against each other, and turn the outside nut to drive the lag bolt.

Tighten the inside nut against the outside nut

With the two bolts tight against each other, drive the lag bolt into the leg

(April)  Finishing always takes longer than I anticipate.  Often it takes longer than everything else. I decided to go for a mirror finish on the top which guaranteed a ton of work.  Beginning in March and through all of April I spent a lot of time applying, waiting, sanding (oh so much sanding), cleaning, and repeating.

First up is the easy part.  The legs and apron got two coats of General Finishes Java Gel Polyurethane.  It was a cinch to apply on the vertical surfaces as it doesn't really drip and it didn't stink up the house as much as a regular polyurethane finish.

The top was a real endeavor.  The final ROS pass on the top was 220.  After that I applied four coats of pure tung oil spaced out a few days to a week in between coats.  Woodcraft carries the real deal.  Note to future self, pure tung oil is really thick and has a pleasant nutty smell that is not at all like mineral spirits.  After letting the tung oil dry for a week, I switched to Waterlox Original Gloss.  This particular product is a phenolic resin oil varnish that claims to have a tung oil base.  Prior to using this stuff, I had only used polyurethane based varnishes.  I spent a ton of time researching finishes and my google-fu told me that I wanted an oil based finish to highlight the figure of the wood and that a phenolic resin based varnish like Waterlox rubs out easier and better than poly.  A ton of people rave out the stuff too.  At 35 dollars a pop for a quart it better be great.

I didn't fill the grain with anything.  I made test samples on the underside of the table and on each sample where I tried to fill the grain by wet sanding with oil or varnish I didn't like the outcome.  As a result, I had to put on a ton of coats of Waterlox to fill the grain and build up enough thickness to level the finish.  In total I think I applied five or six coats.  I don't quite remember, it all blurred together.  Each coat got approximately a week of rest before the next although I think I rushed one of the earlier coats because I was getting antsy.

The first two or three coats were rough sanded with the ROS at 320 grit.  I then moved to wet sanding with 400 and 800 grit.  The final coat that was going to be rubbed out got an initial 800 grit wet sand, than a 2000 grit wet sand, and then I moved to automotive polishing compounds and a random orbital buffer.

Lesson learned, not all compounds are good for rubbing out.  Not knowing any better I started with Meguiars M9 Mirror Glaze.  Apparently, the abrasive in this stuff breaks down.  This means that the abrasive particles shrink while being worked and polishes of this sort are referred to as diminishing abrasives.  It should be worked until it completely breaks down for the best result.  Once it breaks down, you can rub all you want but it won't do anything more for you.  I had terrible luck with the stuff and it seemed to leave an oil residue that would eventually dry and become hazy.

After being disappointed and doing some research, I decided to try Meguiars M105 & M205 which are "super micro abrasive" or SMAT polishing compounds.  They don't break down and have very small abrasives that cut quickly but very finely.  The results were much better.

In the end, the Waterlox finish looks great.  It is the best finish I've had to date.  I guess it was worth the 70 dollars.  I am really happy with how the table turned out.  The finish really brings out the walnut figure, the table is sturdy, and the design should accommodate wood movement.  There is one regret which I won't write down but a lesson was definitely learned that won't be forgotten.

Links 

Breadboard construction
Breadboard construction #2

Planner Jig
Planner Jig 2 - Nick Offerman!

Group Delay

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Group Delay.

I had one of those brain fart moments at work recently where you start to question everything and suddenly feel very small.  Why is the phase slope of a delay line negative!?  After a five minute mental battle I decided to give up and ask someone else - it was late and my brain had turned to mush.  After some really simple equations on a white board my confidant was stumped too.  All the usual sources confirmed what I had memorized, that it is negative, but I couldn't rationalize it to myself.

To my future self who begins to doubt again, don't think of phase slope induced by group delay this way:

If you increase frequency, the amount of phase traversed over a particular amount of time increases.  Therefore phase slope of a delay line is positive.

If you are stuck thinking the above, you can even write out an equation that is so simple and appears to contradict god and his uncle.  The statement above has a problem with its frame of reference.  Below is the right way to think about the problem:

Phase is always a relative measurement.  When characterizing the phase slope of a delay line, you are characterizing the phase of the output relative to the input at a particular point in time.  Think transfer function.  The output phase of a delay line will be retarded relative to the input.  Another way to put it is the output phase of a delay line will be behind (negative) relative to phase at the input of the delay line.

Equations to help set my brain straight for next time:

$ \phi(t)_{output} = 2 \Pi f(t-t_{0}) $

$ \phi(t)_{input} = 2 \Pi ft $

$ \tau = -\frac{\Delta\phi}{2\Pi \Delta f} $

Notes of an Aspiring Woodworker

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Woodworking.  This is a living post, I will be updating it as I learn more.

Rules of the Road

1. Always sand with the grain
• Random orbital sanders help allow you to sand any direction due to the random nature
2. Wood swells/expands cross/perpendicular to the grain
• Boards tend to get wider, not longer
3. In general, hardwoods are from deciduous trees and softwoods are from conifers
• Hardwoods are not automatically denser than softwoods, although it is common
4. Oil based polyurethane tends to yellow the finish, water based polyurethane doesn't
5. Use natural brushes for oil-based finishes and synthetic brushes for latex, acrylic or water-based finishes. Rollers and rags can work for either type of finish.
6. Always sticker lumber that has been finished to the final dimension until you get ready to use it so that air can circulate evenly around all sides of the boards.

Tips and Tricks

1. Put painters tape on edges to help contain glue squeeze-out
2. Use packing tape or wax paper on surfaces that may stick to squeeze-out (like clamps)
3. Cauls are used to keep a surface straight while clamping in the other direction
4. Use a tack cloth to clean and remove dust
• Typically cheesecloth soaked in varnish to make it sticky
5. Janka is the wood hardness scale.

Glues

1. Traditional wood glue is PVA glue
• Sets up relatively quickly, 5-10min.
• Tightbond Type III takes twice as long to set as Typebond I and II
2. Polyurethane glue
• Gorilla glue is a common brand
• Longer cure time than PVA glue
• Expands as it dries
• Requires moisture to cure
• Can come from moisture in the air
• To accelerate cure, wet the surfaces to be bonded
3. Hide glue
• From animal hides 
• Water soluble
• Don't use on things that need to be water resistant 

Finishes

1. Oils
• Drying oils cure in the presence of oxygen
• Boiled Linseed Oil (BLO), Tung Oil
• Boiled linseed cures much faster than tung because it has additives
• These oils heat up when curing.  Old rags can spontaneously combust.
• Non-drying oils do not dry in the presence of oxygen
• Vegetable (peanut, olive), mineral oil
2. Varnish is a very general term, consists of a resin dissolved in something
• Polyurethane, alkyd, phenolic are common resins
• Often varnish can be mixed with oils 
3. Lacquer
• Hardens through evaporation of solvent, not chemical reaction
• Can be redissolved by a solvent
4. Shellac is bug poop
• Like lacquer, it hardens through evaporation of a solvent
• Alcohol is a common solvent
• Since alcohol is a solvent, its not recommended for surfaces like bar tops
• Can be affected by heat
5. Lacquer and shellac tend to be harder but more brittle than varnish
6. Coats of shellac and lacquer "melt into" previous coats because application of a new coat  partially dissolves the existing coats
7. Coats of varnish build distinct layers
8. Rubbing out varnish is more difficult than lacquer or shellac because you have to make sure you don't rub through the final coat.  If you do and reach a coat below, you can end up with "halos" at the edges of the rub-through.
• Don't use a thinned or wiping varnish for the final coat of varnish that is to be rubbed out

Lumber Yard Lingo

1. All 1/2 measurements are rounded up
• Almost always true, example: A 2x4 is 1.5" x 3.5"
2. Rough wood is ordered and priced in board feet
3. Thickness is measured in quarter inches
• Example: 4/4 is 1", 8/4 is 2"
4. Milling
• Edges
• Specified by the number of edges that are straight
• Examples: SL1E (straight line 1 edge), SL2E (straight line 2 edges)
• Faces
• H/M (Hit and Miss) the board will be rough planed on both faces.  This only scrapes off the rough and high spots.  There will be low spots left.
• S1S (Surfaced 1 Side) the board will be planed flat on one face
• S2S (Surfaced 2 Sides) the board will be planed flat on both faces
• S3S (Surfaced 3 Sides) essentially S2S + SL1E
• S4S (Surfaced 4 Sides) S2S + SL2E. All 4 sides will be milled and planed true
5. Grade refers to the expected yield (dimensional lumber) expected from rough stock
• FAS (First and Seconds), highest grade. min size 6in x 8ft, yield is 83% and up
• Sel (Select) this is virtually the same as FAS, but the min board size is lesser (4in x 6ft)
• 1 Com (#1 Common) often called Cabinet Grade, min board size is 3-in x 4-ft, yield is 66% and up
• 2 Com (#2 Common) often called Economy Grade
6. Formula for calculating Board Feet: Length(in) x Width(in) x Thickness(in) / 144
• When calculating board feet, anything under one inch is rounded up to 1 inch
• A board that's 3/4 x 5 x 17 would be calculated as (1 x 5 x 17)/144
• Over an inch is left as-is, 2-5/8 would be left as 2-5/8

condensed from Reddit post and comments

EngNote - J/S Ratio

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Jam to Signal Ratio.

The Jam to Signal Ratio (J/S) is a useful quantity when evaluating an EW scenario.  Below is a simplistic derivation of how to calculate the Jam to Signal Ratio.

Note that the isotropic antenna effective area is present twice in the jammer's equation and only once in the target's equation.  This is due to the fact that the jammer must "convert" the received power density into power before re-transmitting.


$ \begin{matrix}
P_{target.rx} & = & P_{radar}Gain_{radar.tx}(\frac{1}{4\pi Range^{2}})\sigma _{rcs}(\frac{1}{4\pi Range^{2}})(\frac{\lambda ^{2}}{4\pi})Gain_{radar.rx} \\
P_{jam.rx} & = & P_{radar}Gain_{radar.tx}(\frac{1}{4\pi Range^{2}})(\frac{\lambda ^{2}}{4\pi})Gain_{sys}(\frac{1}{4\pi Range^{2}})(\frac{\lambda ^{2}}{4\pi})Gain_{radar.rx} \\
\frac{Jam}{Signal} & = & \frac{P_{jam.rx}}{P_{target.rx}} \\
 & = & (\frac{\lambda ^{2}}{4\pi})Gain_{sys}(\frac{1}{\sigma _{rcs}}) \\
\end{matrix} $
Where:

$ (\frac{\lambda ^{2}}{4\pi}) = IsotropicAntenna_{EffectiveArea} $

EngNote - Polarization

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Polarization.

Textbooks and online references often don't give a complete and practical discussion of how to calculate the impact of a wave's polarization on the magnitude and phase of a received signal.  As a result, I made a polarization toolbox for Matlab to serve as a quick reference.  There is a link to the toolbox in the Code section below.

Cross Polarization Discrimination for a Circular Antenna

$ XPD(dB) = 20log_{10}\frac{axRatio+1}{axRatio-1}$

Code

GitHub

Good References

[1] Polarization in Electromagnetic Systems, Warren Stutzman, 1992

The best practical resource on polarization I have found

EngNote - Linear Least Squares

"EngNotes" are my engineering notes.  This is my way of creating a digital version of an engineering notebook.  This particular entry is on Linear Least Squares.



I often find myself using linear least squares curve fitting to estimate a data set.  Below is a quick write-up of linear-least squares that includes a derivation AND some sample Matlab code.  After the write-up I've also included more advanced versions of linear least squares curve fitting - weighted, constrained, and regularized linear least squares.  Often times I have found these variants to be necessary for the problem at hand.  At the very end there are some links to reference material that is invaluable.

Linear Least Squares Write-Up

The Problem
Formulating an equation to describe how one variable depends on another given a data set (x_i,y_i) i = 1,2,3,...,n where x_i is the independent variable and y_i is the dependent variable is known as curve fitting. One quantitative way to assess the quality of an equation's fit to the data set is by finding the sum of the squares of the offset (or residual) between the data and the describing equation.

Illustration 1: Example Data Set and Fit Equation

Introduction
The quality of an equation's fit to a data set is important. Linear least squares is a common and straightforward method used to find a function that best fits a data set. Keep in mind that linear least squares assumes the data set being modeled can be represented as a linear combination of basis functions. The basis functions do not need to be linear in nature, but the combination of them must.

Consider trying to fit a polynomial to a data set. The method of linear least squares will attempt to find the coefficients ( a,b,c,... ) of the polynomial p = a + bx + cx² + … that minimize the sum of the squares of the offsets:

$S=\sum\limits_{i=1}^{n}(a+bx_{i}+cx^{2}_{i}+...-y_{i})^{2}$

1, x, x², … compose the basis functions of the fit. Notice that the basis functions are not confined to be linear functions. The coefficients ( a,b,c,... ) form a weighting for the linear combination of the basis functions.

Derivation

When S, the sum of the squares of the offsets (errors), is viewed as a function of the polynomial's coefficients ( a,b,c,... ), S is minimized when its gradient is 0. Solving for the gradient results in n gradient equations:

$\frac{\partial S}{\partial a}=0, \frac{\partial S}{\partial b}=0, \frac{\partial S}{\partial c}=0, ...$

By combining all of the gradient equations the coefficients ( a,b,c,... ) that minimize S can be solved for. Below is an example of the steps involved in fitting a first-order polynomial (a line) to a data set.

Example
Data points (x_i,y_i) generated by:

$y_{i} = x_{i} + noise$

First order polynomial to be fit to the data set:

$a + bx_{i}$

Equation for the sum of the squares of the error:

$S=\sum\limits_{i=1}^{n}(a+bx_{i}-y_{i})^{2}$

Solving for the two gradient equations:

$\frac{\partial S}{\partial a}=-2\sum\limits_{i=1}^{n}(y_{i}-(a+bx_{i}))=0$

$na+b\sum\limits_{i=1}^{n}x_{i} = \sum\limits_{i=1}^{n}y_{i}$

$\frac{\partial S}{\partial b}=-2\sum\limits_{i=1}^{n}(y_{i}-(a+bx_{i}))x_{i}=0$

$a\sum\limits_{i=1}^{n}x_{i}+b\sum\limits_{i=1}^{n}x_{i}^{2} = \sum\limits_{i=1}^{n}x_{i}y_{i}$

Converting the two equations into matrix form gives:

$ \begin{bmatrix} n & \sum\limits_{i=1}^{n}x_{i} \\
\sum\limits_{i=1}^{n}x_{i} & \sum\limits_{i=1}^{n}x_{i}^{2} \end{bmatrix}

\begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} \sum\limits_{i=1}^{n}y_{i} \\ \sum\limits_{i=1}^{n}x_{i}y_{i} \end{bmatrix} $

Rearranging and solving for (a,b) yields:

$ \begin{bmatrix} a \\ b \end{bmatrix} =
\begin{bmatrix} n & \sum\limits_{i=1}^{n}x_{i} \\ \sum\limits_{i=1}^{n}x_{i} & \sum\limits_{i=1}^{n}x_{i}^{2} \end{bmatrix}^{-1}
\begin{bmatrix} \sum\limits_{i=1}^{n}y_{i} \\ \sum\limits_{i=1}^{n}x_{i}y_{i} \end{bmatrix} $

The general form of linear least squares is almost always given in the form of the following equation.  Reference [1] contains a detailed derivation.

$ a = (X^{T}X)^{-1}X^{T}y $

Examples

Linear Least Squares

$ a = (X^{T}X)^{-1}X^{T}y $

X contains basis functions in the columns of the matrix and different measurement states in the rows.  The vector 'a' is the coefficient vector being sought.  The vector 'y' is the measurement vector being fit.

% Linear least squares example
% ----------------------------

% Create the independent variable
x = [0:25].';

% Create the dependent variable (linear line of slope 1 in this example)
yTrue = x+5;

% Create observations of the dependent variable
% The rand call is adding noise to the observations
yMeas = x + rand(size(x)) + 5;

% Create the basis functions (model) that will be used to estimate the true 
% system. In this case, the basis functions are a constant function and a
% linear function (matches true system).
basisFunc = [ones(size(x)), x];

% Calculate the linear least squares fit
coefs = (basisFunc' * basisFunc)^-1 * basisFunc' * yMeas;
llsFit = basisFunc*coefs;

% Make a plot illustrating the fit
figure; plot(x,yMeas,'o','LineWidth',2);
hold all; plot(x,llsFit,'LineWidth',2);
grid on; legend('Measured Data','Predicted Fit');
text(1,25, ['Fit Equation:  '  num2str(coefs(2)) 'x + ' num2str(coefs(1))])
text(1,26.25, ['True Equation: y = x + 5']);

Weighted Linear Least Squares

$ a = (X^{T}WX)^{-1}X^{T}Wy $
$ W = diag(w) $

Weighted linear least squares allows one to assign less importance to the fit of particular measurement points.  A common reason to do this is if the statistics of some of the measurements you are trying to fit to are different from those of others.  An exaggerated case is given below for illustration.

Applying weights is a somewhat obvious extension.  Simply multiply the measurements and the basis function matrix by a diagonal matrix of the weight (importance) vector.

% Weighted linear least squares
% -----------------------------

% Create a dependent variable
x = [0:99].';
% Define a function to be measured/estimated (5*x)
% Add some noise to it (randn) to simulate a measurement process
y = 5*x + randn(size(x))*5;
basisFunc = [ ones(size(x)) x ];

% Calculate a plain-jane linear-least squares
coefs = (basisFunc'*basisFunc)^-1 * basisFunc' * y;

% Plot the results
figure;
subplot(3,1,1);
plot(x,y,'o');
hold all; plot(x,basisFunc*coefs,'LineWidth',2);
title('Linear Least Squares Fit With No Gross Measurement Error');
ylim([0 525]);

% Pick a measurement(s) and give it a gross error.  I strategicly picked
% two points to make it easier to see the impact.
y(100) = 1;
y(1) = 500;
% Calculate the plain-jane linear least squares with the gross error
coefs = (basisFunc'*basisFunc)^-1 * basisFunc' * y;

% Plot the results to illustrate how the gross error "pulls off" the fit
subplot(3,1,2);
plot(x,y,'o');
hold all; plot(x,basisFunc*coefs,'LineWidth',2);
title('Linear Least Squares Fit With A Gross Measurement Error');
ylim([0 525]);

% Create a weight vector - in this case the weight vector is relatively
% arbitrary to prove a point.  In general, determining the weight vector is
% the secret sauce of the weighted linear least squares.
weight = ones(size(y));
weight(100) = 0;
weight(1) = 0;
weight = diag(weight);

% Calculate the weighted linear least squares solution
coefs = (basisFunc'*weight*basisFunc)^-1 * basisFunc' * weight * y;

% Plot the weighted result to illustrate how the gross error no longer
% "pulls off" the fit
subplot(3,1,3);
plot(x,y,'o');
hold all; plot(x,basisFunc*coefs,'LineWidth',2);
title('Weighted Linear Least Squares Fit With A Gross Measurement Error');
ylim([0 525]);

Regularized Linear Least Squares

$ a = (X^{T}X + \mu I)^{-1}X^{T}y $
where $\mu$ is known as the regularization weight and $I$ is the identity matrix

If your basis functions don't model the process being fit, some are nearly dependent, or the number of basis functions is large, linear least squares can over-fit.  An over-fit solution can look good on paper but it can cause the fit to do "weird" things when evaluating a fit between measured points  (interpolation).  Said another way, it can increase a fit's sensitivity to input noise.  A common symptom of over-fitting is that the coefficients of the fit begin to grow rapidly.

One way around over-fitting is to use regularized linear least squares.  This technique adds a second objective to the least squares solution - minimizing the squared sum of the coefficients themselves.  The trick to this technique is in picking a regularization weight.  Too large of a weight and the fit favors shrinking the coefficients too much, resulting in a poor fit.  Too small of a weight and you wind up over-fitting.  Over-fitting and the balancing act of a regularization weight versus fit is illustrated by an L-curve.

Constrained Linear Least Squares

Often there are parts of the fit process I would like to control.  At one point in time I wanted the sum of the coefficients I was finding to sum to a specific value.  This can be easily accomplished by adding a faux value to all of the basis functions (adding an extra row to the X matrix) and a corresponding faux measurement to the measurement vector (adding an extra element to the y vector).

Keep in mind, this isn't a hard constraint but is part of the fitting process.  As a result, it isn't necessarily or exactly met.  You can increase the importance of the constraint over the fit by increasing the value of the faux values added.

% Constrained linear least squares
% -----------------------------

% Make up a measurement vector
meas = [-6 ; 0.7 ; 0.0];
% Define a regularization weight (not necessarily needed)
lambda = 0.00005;

basisFunc = [0.5 -3.0 0.0 ; 0.3 -1 0.0 ; 0.8 0.3 0.0 ; 0.71 2.3 0.0].';

for weightNdx = 0:99
  % Add the constraint, making the number larger increases the importance of 
  % the constraint
  meas(4) = weightNdx/10;
  basisFunc(4,:) = weightNdx/10;
  
  % Calculate the regularized linear least squares solution
  coefs = (basisFunc' * basisFunc + lambda .* eye(size(basisFunc,2)))^-1 * ...
    basisFunc' * meas;
  
  % Debug evaluation
  coefSize(weightNdx+1) = sum(coefs);
  
end

% Plot the sum of the coeficients to show it approaching one
figure; plot(linspace(0,9.9,100),coefSize,'LineWidth',2);
title('Sum of the Coefficients of the Fit');
ylabel('Sum of Coefficients');
xlabel('Weight Assigned to Sum');

Good References

[1] The Mathematical Derivation of Least Squares [Local Mirror]
http://isites.harvard.edu/fs/docs/icb.topic515975.files/OLSDerivation.pdf

[2] Least-Squares [Local Mirror]

http://stanford.edu/class/ee103/lectures/least-squares/least-squares_slides.pdf

[3] Regularized Least-Squares and Gauss-Newton Method [Local Mirror]
http://see.stanford.edu/materials/lsoeldsee263/07-ls-reg.pdf

[4] Constrained Linear Least Squares [Local Mirror]
http://people.duke.edu/~hpgavin/cee201/constrained-least-squares.pdf

[5] The L-curve and its use in thenumerical treatment of inverse problems [Local Mirror]
https://www.sintef.no/globalassets/project/evitameeting/2005/lcurve.pdf

[6] Choosing the Regularization Parameter [Local Mirror]
http://www2.compute.dtu.dk/~pcha/DIP/chap5.pdf