Definition of terms:
It you want to consider the rotation only then the code for this is shown here.
Alternatively if we want to consider the possibility that there is also a translation from the centre, and that the rotation may not be about the centre but may be about some arbitrary point, then we need to extend the notation as follows:
- A 3D vector has to be added to the euler angles so that together they can specify the position and orientation.
- The matrix has to be increased from 3x3 to 4x4 so that it can specify the position and orientation. (as explained here).
This conversion is better avoided, since it is non-linier and has singularities at + & - 90 degrees, so if already working in terms of matricies its better to continue using matrices if possible.
So it we look at the Euler to Matrix conversion we can see that:
m01/m00 = s1 / c1 = tan(heading)
m12/m22 = s3 / c3 = tan(bank)
m02 = -s2 = -sin(attitude)
so this gives:
heading = atan(m01/m00)
bank = atan(m12/m22)
attitude = asin(-m02)
Note this only applies to a martix which represents a pure rotation. The equations for heading and bank should be independent of uniform scaling as it will cancel out in the division. It would be better to find an expression for attitude which is also independent of scaling.
Since there are several ways to produce the same rotation using heading, bank and attitude then the solution is not unique. tan(0) is 0, tan(90 degrees) is infinity, tan(-90 degrees) is -infinity. So the results will depend on whether arctan processes a result between -90 and 90 or 0 and 180.