LayerNormalization shape check error

[yuanyao-nv]

For the following LayerNormalization node,

there seems to be a regression since v1.21 where it complains

Status Message: Size of X.shape[axis:] must be larger than 1, got 1

Granted, computing LayerNorm on just 1 value is pointless since the standard deviation become 0+eps and you're essentially rescaling by 1/sqrt(eps), but it should still be a valid computation and the ONNX spec does not forbit it. Imposing this additional constraint in onnxruntime causes some older models to fail on this. Any thoughts?

[yuanyao-nv]

cc @tianleiwu

[xadupre]

LayerNormlization is equivalent to this:

Mean = ReduceMean<axes=normalized_axes>(X)
D = Sub(X, Mean)
DD = Mul(D, D)
Var = ReduceMean<axes=normalized_axes>(DD)
VarEps = Add(Var, epsilon)
StdDev = Sqrt(VarEps)
InvStdDev = Reciprocal(StdDev)
Normalized = Mul(D, InvStdDev)

With one value, D = DD = 0, so the normalized value is 0 in that case but maybe what we would like to have in this case is X not 0. What value would you prefer? We should update onnx as well.

[yuanyao-nv]

LayerNormlization is equivalent to this:

Mean = ReduceMean<axes=normalized_axes>(X)
D = Sub(X, Mean)
DD = Mul(D, D)
Var = ReduceMean<axes=normalized_axes>(DD)
VarEps = Add(Var, epsilon)
StdDev = Sqrt(VarEps)
InvStdDev = Reciprocal(StdDev)
Normalized = Mul(D, InvStdDev)

With one value, D = DD = 0, so the normalized value is 0 in that case but maybe what we would like to have in this case is X not 0. What value would you prefer? We should update onnx as well.

Not sure I understood you. Regardless of the value of X, we always have D=DD=Var=0, VarEps=epsilon,and Normalized will always be zero. I think any backend should just return 0 instead of erroring out?