I've done something with a studio talkback mic where it would duck as
soon as signal from some instrument that would did not have a mic or
was in the same room as the talkback mic started playing.I believe I
used a Valleypeople Dynamite(more of a compressor than a gate but
still in the dynamic proccessing family) -hook the instrument signal
into the key input and set it to duck. Most compressors with key ins
would make this possible. I wouldn't recomend this for live to 2
sessions however.

But isn't this ultimately what most talkback switches do to a mixers
control room outs? Ducking the signal to the monitors when the
talkback button is pressed so that the monitors don't feed back
through the talkback mic? Or is it normally just a complete bypass
without the ducking?