Since there isn't a standard for speaker connections, I don't know
that there's ever been a convention for polarity vs. color or even
number if the terminals are numbered. Seems to me that I've heard
that the terminal that has the positive voltage on it when the speaker
cone moves out (toward the listener) is defined as positive. If you
put a microphone in front of a speaker with the cone moving in that
direction, it should make pin 2 go positive with respect to pin 3.

This is the consensus -- except when the drivers don't have the same polarity.
In a three-way system, you'd go with the midrange. In a two-way system, it would
probably be the woofer, as the woofer (in such systems) generally caries most
(or all) of the midrange.