It is usually difficult to get a non professional orchestra to play the tempo that you want. It is much easier to get a single pianist to do it (most of the time). There is also the common problem of the conductor thinking he/she knows how your part should be played better than you do.
It shouldn't cause too many difficulties physically. A band is harder to deal with because they play too
LOUD! If you think that the accompaniment is too loud during rehearsal just turn around and face the orchestra. They should be able to hear you well. If they can't, they need to play softer.