In my opinion, the most important thing you can do in jazz music is to listen to as much as you can. You need to have a great feel for the music. Its always great to find a great solo and listen to it everyday and be able to sing along and play some of the licks back on your horn.
The rest of it, like you said, is matching scales/arpeggios with a chord type or group of chords. Thinking of scales in modes will help. That is, starting a major scale on different scale degrees.
I'd also recommend picking up an Aebersold CD @
http://www.aebersold.com/
-Matt