.. To optimize
performance/reliability, should i install PT/FCP software on a separate
hard drive from work files? Will it make any difference?

It will make a difference in that whatever resources are being used to
run the operating system and programs will not be available to run
tracks if you install everything one a single drive.

You will get more tracks if you use a seperate drive dedicated to
audio.If you plan on recording at higher sampling rates, a dedicated
drive will be even more desireable to play back high res multitrack
projects.

Or should i just put software, OSX and work files on separate
partitions on the same (120G) drive?

I would opt for seperate operating system/ audio drives.
Your 20 gig drive may work fine for the OS if it is not too old.
If it dont, get another new drive to optimize the system.

Randall