Robust response transformations for generalized additive models via additivity and variance stabilization

The AVAS (Additivity And Variance Stabilization) algorithm of Tibshirani provides a non-parametric transformation of the response in a linear model to approximately constant variance. It is thus a generalization of the much used Box-Cox transformation. However, AVAS is not robust. Outliers can have a major effect on the estimated transformations both of the response and of the transformed explanatory variables in the Generalized Additive Model (GAM). We describe and illustrate robust methods for the non-parametric transformation of the response and for estimation of the terms in the model and report the results of a simulation study comparing our robust procedure with AVAS. We illustrate the efficacy of our procedure through a simulation study and the analysis of real data.