Distinct differences of singular moduli

Let $E_1, E_2 / \mathbb{C}$ be non-isomorphic elliptic curves with complex multiplication. We prove that the pair $(E_1, E_2)$ is characterised, up to isomorphism, by the difference $j(E_1) - j(E_2)$ of the respective $j$-invariants. In other words, we show that if $x_1, x_2, x_3, x_4$ are singular moduli such that $x_1 - x_2 = x_3 - x_4$, then either $(x_1, x_2) = (x_3, x_4)$ or $(x_1, x_3) = (x_2, x_4)$.