Variance of the root mean square value of the residuals of sine fitting in the presence of additive noise

The least-squares fitting of a sinusoidal model to a set of data points is a common procedure in signal processing algorithms. A residual is the difference between the value of one data points and the estimated value of that point given by the sinusoidal model. The root mean square (RMS) value of all the residuals is a common metric used in many applications to quantify the goodness of fit. In analog-to-digital conversion, for example, the RMS value is used to compute the number of effective bits. In other applications the RMS value is used to compute the signal-to-noise ratio which measures the amount of noise generated by an electronic circuit such as an amplifier, for instance. Due to the presence of different random non-ideal phenomena affecting the data points, like stimulus signal phase noise, sampling jitter or quantization error, the estimative of the RMS value is uncertain and whose statistical properties are important to evaluate. In this work we focus on the effect that additive noise has on the variance of the RMS value of the residuals. A first exact analytical expression is derived and two easier to use and simpler approximations are proposed. The results presented are validated using numerical simulations employing a Monte Carlo type procedure.