I have to say, even though my parents probably listen to more classical music than most, I never got into it until I started my AS level music classes (that's age.. 17? The second last year of school)
Then, having been given some of the tools of how to understand the music I really started to enjoy it and get a LOT more out of it.
At the moment Daniel Barenboim is doing a series of lectures for the BBC (
http://www.bbc.co.uk/radio4/reith2006/ if you want to listen) and one of the things he was severely critical of in his last one I heard was the way people are encouraged just to go and sit in a concert hall, and almost ignore the music (thinking of extra-musical things, etc) in order to "get through" the concert, and perhaps absorb something cultural.
I have to agree - if you're exposed to say a Mahler symphony without having any understanding of its context, or the history of the genre, or the history of the symphony orchestra, or even a basic understanding of the forms and structures he inherited, then you're going to be pretty lost at sea.
Bit of a tangent, sorry! I'm just quite interested by this stuff.