I say play both. I play both. I'm retired and have time. I play horn in quintet and trumpet in a community band. I'm looking to put together a small group for jazz and I'll play cornet/trumpet/flugelhorn in that. If I can find a brass band, I'll also play Eb cornet. I've noticed that playing all of the different brass has actually improved my playing. Certainly playing Eb cornet improved my range. The mouthpiece thing bothered me for a while. I tried matching diameters but that didn't work for one or the other. I now play Curry 8.5 size on trumpet/cornet/flugel/soprano and Wick Paxman 5.5 for horn. That is 16.23 mm to 17.4 mm. But rim profiles are so different and switching seems to be no problem.
One of the advantages to playing different horns is playing opportunities. A good horn player is hard for groups to find so there are opportunities that are unique to horn. But trumpet means lots of opportunities too.