Here is the key part again:
What this essentially means, is he wants a negative definition for music. I.e. "Everything on this list is not-music, so everything left off of the list is music". Instead of a literal list of "not-music", a formula could be used to check if something would be on the list. It doesn't even have to return yes or no, it could return "percentage not-music", and anything over 10% might make cats scratch their eyes and ears out. Maybe 50% would be Cage music (jokes! :D)
It just so happens that this is exactly how we define prime numbers. There is no definitive list of prime numbers; only a list of "numbers with more than one non-one factors" (from which we can find the prime numbers out). That's why you will read about prime number searching - we actually have to seek them out. Prime number searching is the analogue of an algorithm to make all the music....!....except it's not. There are no prime number formulas that can generate all the prime numbers (I think it was proven, that with our current methods to be impossible to make such a formula). There are things like Mersenne primes, which have been generated by a formula, but even then, not every Mersenne number is a prime (there's actually very few generated this way, 47 so far to be exact out of tens of thousands of candidates).
The point of this excursion into prime numbers is to demonstrate that there are "spaces" (i.e. sets of all prime numbers, sets of all music) that are unable to be mapped positively (this would mean we have a formula that generates all the music, or all the prime numbers), and only can be mapped negatively (which means we have to sift through the list of "everything" to find something that is part of the "space" - we don't get an automatic list).
So, to answer your question, you can't generate a definitive positive formula for music, and we already have negative formulas - of which the rules of counterpoint in the Gradus, Serialism, are examples of. Just like how there are primes that are not Mersenne numbers, your negative formulas can usually only catch a certain amount of everything fitting your definition - you will need the combination of many negative formulas to flesh out the positive space (i.e. you will need many composition techniques and rules to make all the music).