Not cruel at all.
Consider that if you're leaving them unneutered/unspayed, but you don't have any other pets around and are not allowing them to breed, you're giving them a sex drive but artificially preventing them from following through on those urges, which would be even more cruel. Plus, as others have mentioned, if you DO let them run loose and they breed, you're bringing unwanted pets into the world with all the cruelty that goes with that.
Kill 1 man: You are a murderer. Kill 10 men: You are a monster.
Kill 100 men: You are a hero. Kill 10,000 men, you are a conqueror!