My first real hurdle was becoming a Mahler-ite. At the beginning (meaning when I started really getting into classical music, at age 12-13) I didn't know anything about him but his First Symphony, which I tended to confuse at times with Beethoven's Pastoral or Ninth. But within a year and a half I got to realize how wonderful his works are, even if long. As of now, the only work by him I still struggle with a bit is the Third Symphony.
Next step was Shostakovich, which I took a bit longer to love. But I totally do now. Prokofiev was never that hard for me, though.
I have begun to come into terms with Alban Berg lately, but I still can't stand most of the atonal/serialist/modernist guys, no matter how long and hard I've tried. Let's not lose hope, though!