Flipping a quaternion from right to left handed coordinates

I don’t think any of these answers is correct.

Andres is correct that quaternions don’t have handedness (*). Handedness (or what I’ll call “axis conventions”) is a property that humans apply; it’s how we map our concepts of “forward, right, up” to the X, Y, Z axes.

These things are true:

• Pure-rotation matrices (orthogonal, determinant 1, etc) can be converted to a unit quaternion and back, recovering the original matrix.
• Matrices that are not pure rotations (ones that have determinant -1, for example matrices that flip a single axis) are also called “improper rotations”, and cannot be converted to a unit quaternion and back. Your mat_to_quat() routine may not blow up, but it won’t give you the right answer (in the sense that quat_to_mat(mat_to_quat(M)) == M).
• A change-of-basis that swaps handedness has determinant -1. It is an improper rotation: equivalent to a rotation (maybe identity) composed with a mirroring about the origin.

To change the basis of a quaternion, say from ROS (right-handed) to Unity (left-handed), we can use the method of .

mat3x3 ros_to_unity = /* construct this by hand */;
mat3x3 unity_to_ros = ros_to_unity.inverse();
quat q_ros = ...;
mat3x3 m_unity = ros_to_unity * mat3x3(q_ros) * unity_to_ros ;
quat q_unity = mat_to_quat(m_unity);

Lines 1-4 are simply the method of https://stackoverflow.com/a/39519079/194921: “How do you perform a change-of-basis on a matrix?”

Line 5 is interesting. We know mat_to_quat() only works on pure-rotation matrices. How do we know that m_unity is a pure rotation? It’s certainly conceivable that it’s not, because unity_to_ros and ros_to_unity both have determinant -1 (as a result of the handedness switch).

The hand-wavy answer is that the handedness is switching twice, so the result has no handedness switch. The deeper answer has to do with the fact that similarity transformations preserve certain aspects of the operator, but I don’t have enough math to make the proof.

Note that this will give you a correct result, but you can probably do it more quickly if unity_to_ros is a simple matrix (say, with just an axis swap). But you should probably derive that faster method by expanding the math done here.

(*) Actually, there is the distinction between Hamilton and JPL quaternions; but everybody uses Hamilton so there’s no need to muddy the waters with that.