You won't need to spend all that much. Sample libraries usually take quite a bit of time to load in almost every computer, as you have noted, so what really changes is the capacity of the pc to render the audio without losing quality, glitching, or god forbid me, crashing.
Now, it will all depend on the music you compose as well. If you write a 200 track orchestral track with a tutti every 20 seconds, then yes, you will probably need a monster of a computer. However, my pc is from 2015, has an Intel Core I7 3.60GHz, 16gb of RAM, with a pretty decent graphics card, and I've never had a problem with it. Recently I'm only using NotePerformer, so I haven't needed that much processing power anyways.
Now, if you have the option, I'd recommend getting a desktop, as you can get a whole lot more procecssing power for a lot less money. I don't know if you have a preferencec over Mac or pc, but I'm a huge fan of Dell. Building your own pc also has huge advantages, succh as a lowered price.
At the end of the day, you just have to find the best deal for what you want to do, which simply takes looking at your options/budget and picking whaat seems to have the biggest value/money.