gridjam

Listen a lot, play a lot, ask MANY questions and play with people who are speaking the language fluently.
How many hours have you put in as a passionate, focused listener?
How much time have you spent playing along with your favorite recordings
(I don't mean play along CDS)
It seems to me that when something is difficult, it is either completely out of our reach because we have no clue to how to access it due to a massive lack of skills OR we are confusing the word difficult with unfamiliar. If improv is difficult for you, start by taking something familiar and start getting creative with it. It can be anything. From your favorite Beethoven melody/motif to a groove from the latest Radiohead CD. It doesn't matter, what matters is that you begin getting familiar with creating a new path through familiar territory and start expanding on your own impressions of that material based on related and seemingly unrelated material. Next step, a Blues or a two chord modal tune. Get playing with good players asap.
As Freddie Hubbard said- "this music takes a long time to learn". If you can get busy enjoying the process, you will have a better chance at overcoming your not-so-odd situation. Eventually playing will become more of a fun adventure game and less of a drag.