It is not a huge problem, it is just that a cold instrument will be significantly out of tune, and the player would then have to adjust his embouchure a lot to become in-tune with the rest of the ensamble. Of course, this isn't a problem in Stravinsky's three pieces because it's for solo clarinet.
The problem has nothing to do with the actual changing, simply that you can't tune the clarinet before you begin and expect it to remain in tune while you play on the other clarinet. It will get cold, becoming shorter, thus making it higher.
Also, the cold clarinet that you have changed to, will become warmer, and this will also affect the intonation, so you will gradually have to alter your embouchoure: not a very pleasant task if it can be avoided.
In orchestral music you have to change clarinets all the time, so you have to be able to cope with it, but as few changes as possible(considering technical difficulties) is always preferred.