It's the same issue you encounter with audio mixing. You have to clamp out-of-range values, even though they don't occur a lot. If you don't you get awful artifacts, and have to lower everything so that it can never overflow your range.
Gladly this isn't that much of an issue anymore since most audio mixing nowadays is done using floating point numbers (either 32 or 64 bits).
Last week I made a test in Bitwig where I boosted a 0dBFS sine wave by 64dB into "clipping", exported it to 32 bit float wav, imported it again, reduced gain by 64dB and there the old sine was in all it's glory. Theoretically now with rounding errors and a loss of precision, but nothing audible.
Of course as soon as you go into 24 or 16 bit fixed point representations (and you will have to eventually) that clipping becomes a problem.
There should be little loss of precision exactly because floating point is by definition precision-invariant over most of its domain – ie. 1.23456*10^-30 and 1.23456*10^30 are essentially the "same" number. Precision loss normally only occurs when combining numbers of different magnitudes.
To be pedantic (apologies), I’m certain you know this, but not everyone thinks about floats - precision loss is almost always occurring in practice with every floating point operation you do, and happens often even when combining numbers of exactly the same magnitude (exponent). They’re just very small losses on the order of 0.5 ULP on average, and it would take many many of those to add up to audible artifacts. It is sometimes possible, but quite rare to do any float math and have no precision loss, that only works if you can change the exponent without touching the mantissa, or in the narrow range of add & subtract operations where the exponents of the operands and the result are all the same. Multiplication is always rounding even when all input & output exponents are identical.
This is of course why we have audio compressors that lower everything while it’s loud and bring it back up when it’s quiet and try to do it quickly but without anyone noticing too much. It’s theoretically possible the same trick could work on the N64, except it would require & cost extra math ops that weren’t there.
