The answer to your question is, in principle, no; however you must be able to devote yourself to learning your craft as a performer and/or composer, something which may be trickier than starting as a child or teen as you will have to also find a means of supporting yourself and funding your education. I often have to point out the importance of learning an instrument in order to learn how to compose, for numerous reasons which I will not go into much detail on here, suffice to say that I cannot think of a single 'classical' composer to this day who did not also play an instrument to near-professional standard (and a good few were active professionals), and that doing so will give you a knowledge of existing repertoire that will aid your compositional development. I would consider electro-acoustic and 'media' music the field in which playing ability is least vital (although certainly not unnecessary), as it tends to demand more knowledge of software and hardware than actual instruments. Community college is a good first step, but be aware the entry requirements for university are not there to catch you out but because you will need a good all-round ability to cope with specialising in any segment of musical study. Ideally you would play piano and one other orchestral instrument to near-professional standard, be able to sing a capella, have a working knowledge of music history and theory, be able to arrange and compose in several styles including having studied tonal harmony and counterpoint, have some practical pop and jazz knowledge, have a working knowledge of all common instruments and have performance and possibly directing experience
There is, of course, no reason why with a little pointing you could not become an excellent self-taught composer (Elgar is the shining example) but again we come back to this being a product of having abilities as a performer (Elgar was also an excellent violinist and pianist) and exposure to repertoire and other musicians. We haven't mentioned the lifestyle of a professional musician either: for me I can summarise this as the joy of being able to play, compose and teach to a high standard offsetting the many pressures and difficulties that come with this career choice.