Comparison of air refractive index formulae when used in telemetry for two-wavelength refractivity compensation

Abstract A two-wavelength absolute distance meter operating at 780 nm and 1560 nm has been developed. It measures two optical path lengths simultaneously, the difference of which enables an in-line refractivity compensation. The result is a dispersion-based air-index-compensated distance independent of temperature and pressure along the optical paths. In previous works, the instrumental errors were evaluated and an uncertainty on the measured optical path lengths was assessed. However, the dispersion-based air-index-compensated distance is not traceable to the SI metre, as its calculation requires parameters derived from the air refractive index formula, the dispersion K ( λ ) and humidity g ( λ ) terms, which are not always associated with uncertainties nor defined for infrared wavelengths. The current article reviews how the refractive index has been defined by various authors, such as Edlén, Ciddor, or Voronin and Zheltikov. We first show that whatever formula is used, it can be applied to two-wavelength telemetry for refractivity compensation. The equivalences between the formulae, as well as their differences, are highlighted. Secondly, we compare the formulae using experimental measurements at 780 nm and 1560 nm for distances up to 6.5 km. For these wavelengths, the Voronin and Zheltikov equation is the one that compensates best changes in air refractive index when a fixed distance is measured over several days, with standard deviations better than 280 µm over 2.6 km or 5.4 km. When other formulae are used for the same measurements, the standard deviations exceed 500 µm. Furthermore, when distances obtained by our two-wavelength ADM are compared with those of a GNSS-based distance meter, the measurements of the two systems are compatible within their uncertainties only if the Voronin and Zheltikov equation is used. Otherwise, there is a scale error higher than 2 ppm. This article shows that these errors arise mainly from the humidity term g ( λ ) in the infrared.