I would beg to differ. Maybe it's a question of "how hi" is hi-fi, but
I think you should try listening to the system with one of the output
cables disconnected from the "Y", and then connect the second output
cable to see if you can tell the difference. I can tell quite a
difference when I do this. The impedance mismatch resulting from
connecting the two output cables in parallel really drags down the
signal's amplitude and causes noticeable distortion. If you want to do
it right insert, for example, a $30.00 Behringer mixer at this point
just to isolate the amp output from the two devices that you want to
drive. Another possibility is to construct a proper terminator to
eliminate the impedance mismatch, but that will involve some signal
loss, so the little mixer seems like the way to go.