One way I can tell them apart is James has more feminine features.
That's a really good reference picture. Oliver (on the right) has a mole on the right side of his neck, and his features are sharper, thinner. James (on the right) has a bit more of a pronounced chin, and his cheeks are a little bit chunkier, like he hasn't lost all the baby fat yet :3
I'm so in love with the both of them, and I'm really good with finding differences between twins lol