If you had asked me 3 years ago my opinion of the likes of Scriabin, Prokofiev, and heck even some of the guys here on YC, you would have gotten a flat response at best. But it didn't take too long for me to start liking it more and more. It seems to me that the more I listen to something (assuming it has any merit to begin with) the more I can appreciate it.
If I understand what you're getting at correctly, I think you're right. I think the music that is the most special to me is the music that took me a bit longer to get into (mostly, I mean does anybody really need a second listen to love Tchaikovsky?). There's a feeling of satisfaction after having learned to appreciate something (it's as if you actually did learn something) that can surpass the feeling you get when you like something right away.