Do I really have to remove the pan?

[chrger1967]

My money is on the shifter shaft seal.

Nope. It was the dipstick tube. I put an aftermarket one that has the grommet that goes into the hole for the tube, then you push in the dipstick tube. I tightened the upper bracket up and now no leaks. The new tube does not have the lower mounting tab, but its not leaking and not moving around, so all is good. At least for now.