Preprocessing API Reference

CORAL(X1, X2; lambda = 1.0)

Performs CORAL to facilitate covariance based transfer from X1 to X2 with regularization parameter lambda. Returns a CORAL object.

Correlation Alignment for Unsupervised Domain Adaptation. Baochen Sun, Jiashi Feng, Kate Saenko. https://arxiv.org/abs/1612.01939

(C::CORAL)(Z)

Applies a the transform from a learned CORAL object to new data Z.

(T::MultiplicativeScatterCorrection)(Z)

Applies MultiplicativeScatterCorrection from a stored object T to Array Z.

MultiplicativeScatterCorrection(Z::Array)

Creates a MultiplicativeScatterCorrection object from the data in Z

Martens, H. Multivariate calibration. Wiley

OrthogonalSignalCorrection(X, Y; Factors = 1)

Performs Thomas Fearn's Orthogonal Signal Correction to an endogenous X and exogenous Y. The number of Factors are the number of orthogonal components to be removed from X. This function returns an OSC object.

Tom Fearn. On orthogonal signal correction. Chemometrics and Intelligent Laboratory Systems. Volume 50, Issue 1, 2000, Pages 47-52.

(OSC::OrthogonalSignalCorrection)(Z; Factors = 2)

Applies a the transform from a learned orthogonal signal correction object OSC to new data Z.

TransferByOrthogonalProjection(X1, X2; Factors = 1)

Performs Thomas Fearns Transfer By Orthogonal Projection to facilitate transfer from X1 to X2. Returns a TransferByOrthogonalProjection object.

Anne Andrew, Tom Fearn. Transfer by orthogonal projection: making near-infrared calibrations robust to between-instrument variation. Chemometrics and Intelligent Laboratory Systems. Volume 72, Issue 1, 2004, Pages 51-56,

(TbOP::TransferByOrthogonalProjection)(X1; Factors = TbOP.Factors)

Applies a the transform from a learned transfer by orthogonal projection object TbOP to new data X1.

ALSSmoother(X; lambda = 100, p = 0.001, maxiters = 10)

Applies an assymetric least squares smoothing function to a 2-Array X. The lambda, p, and maxiters parameters control the smoothness. See the reference below for more information.

Paul H. C. Eilers, Hans F.M. Boelens. Baseline Correction with Asymmetric Least Squares Smoothing. 2005

ConvFilter1DFFT(a, filter)

Performs a 1D convolution of vector filter onto a via the FFT definition of a convolution. This method implicitly zeropads and truncates the result.

Note: This isn't highly optimized for performance. It was made when DSP.jl broke, and broke the ChemometricsTools.jl package.

DirectStandardization(InstrumentX1, InstrumentX2; Factors = minimum(collect(size(InstrumentX1))) - 1)

Makes a DirectStandardization object to facilitate the transfer from Instrument #2 to Instrument #1 . The returned object can be used to transfer unseen data to the approximated space of instrument 1. The number of Factors used are those from the internal orthogonal basis.

Yongdong Wang and Bruce R. Kowalski, "Calibration Transfer and Measurement Stability of Near-Infrared Spectrometers," Appl. Spectrosc. 46, 764-771 (1992)

FirstDerivative(X)

Uses the finite difference method to compute the first derivative for every row in X. Note: This operation results in the loss of a column dimension.

FourierUpsample(X, newlength)

zero pads a vector X in the frequency domain then converts it back to the time domain.

FractionalDerivative(Y, X = 1 : length(Y); Order = 0.5)

Calculates the Grunwald-Leitnikov fractional order derivative on every row of Array Y. Array X is a vector that has the spacing between column-wise entries in Y. X can be a scalar if that is constant (common in spectroscopy). Order is the fractional order of the derivative. Note: This operation results in the loss of a column dimension.

The Fractional Calculus, by Oldham, K.; and Spanier, J. Hardcover: 234 pages. Publisher: Academic Press, 1974. ISBN 0-12-525550-0

LinearResample(X, newsize)

Resamples a vector X to be of size newsize via linear interpolation.

Noise(data, level; scale = :percent_max, type = :gaussian )

Adds noise (uniform, or gaussian) to an array of data. The scale can be made relative to the percentmax, percentaverage, or the absolute level values.

Note: for Gaussian noise it is likely convenient to scale by an inverse quantile(say 0.999)

PerfectSmoother(X; lambda = 100)

Applies an assymetric least squares smoothing function to a a 2-Array X. The lambda parameter controls the smoothness. See the reference below for more information.

Paul H. C. Eilers. "A Perfect Smoother". Analytical Chemistry, 2003, 75 (14), pp 3631–3636.

SavitzkyGolay(X, Delta, PolyOrder, windowsize)

Performs SavitskyGolay smoothing across every row in an Array X. The window size is the size of the convolution filter, PolyOrder is the order of the polynomial, and Delta is the order of the derivative.

Savitzky, A.; Golay, M.J.E. (1964). "Smoothing and Differentiation of Data by Simplified Least Squares Procedures". Analytical Chemistry. 36 (8): 1627–39. doi:10.1021/ac60214a047.

Scale1Norm(X)

Scales the columns of X by the 1-Norm of each row. Returns the scaled array.

Scale2Norm(X)

Scales the columns of X by the 2-Norm of each row. Returns the scaled array.

ScaleByIntensity(X, index)

Scales the columns of X by the value of a peak at a specified index in each row. Returns the scaled array.

ScaleFNorm(X)

Scales EACH entry of X by the Frobenius Norm of. Returns the scaled array.

ScaleInfNorm(X)

Scales the columns of X by the Inf-Norm of each row. Returns the scaled array.

ScaleMinMax(X)

Scales the columns of X by the Min and Max of each row such that no observation is greater than 1 or less than zero. Returns the scaled array.

FirstDerivative(X)

Uses the finite difference method to compute the second derivative for every row in X. Note: This operation results in the loss of two columns.

SincInterpolation(Y, S, Up)
Y - vector of a line shape
S - Sampled domain of Y
Up - Upsampled X vector

Refactored from: https://gist.github.com/endolith/1297227#file-sinc_interp-m

StandardNormalVariate(X)

Scales the columns of X by the mean and standard deviation of each row. Returns the scaled array.

boxcar(X; windowsize = 3, fn = mean)

Applies a boxcar function (fn) to each window of size windowsize to every row in X. Note: the function provided must support a dims argument/parameter.

offsetToZero(X)

Ensures that no observation(row) of Array X is less than zero, by ensuring the minimum value of each row is zero.

(DSX::DirectStandardizationXform)(X; Factors = length(DSX.pca.Values))

Applies a the transform from a learned direct standardization object DSX to new data X.