You yourself said "Like an idiot, I said yes."
Ok, tell him you changed your mind. It will do you ZERO good to play the french horn.
Stick to the trumpet and master that, it IS your passion.
Why mess with your passion and end up with no passion for anything?