"
Suppose we compute dimensions sequentially. Instead of requiring dimension p to be orthogonal to 1,...,p-1 we require the dimension to be orthogonal to all polynomials of degree less than or equal to q of dimensions 1,...,p-1.
If we have a, say, two-dimensional object score plot we can fit a nonparametric regression of the second dimension on the first, often by using a linear smoother. We want this regression function to be zero. Thus we can find the second dimension requiring the regression function to be zero, or penalizing for a regression function deviating from zero. We can also fit two dimensions simultaneously, requiring both regression functions to be zero (or small).
More generally, the idea of least squares penalty functions to be added to the Gifi loss function could be explored.