I have successfully installed and used Ti-Connect on a Mac G4 running OS X 10.4.11 and on a Macbook running OS X 10.5.8.
I unfortunately rarely have access to these two computers now, and can install connect on my macbook running OS X 10.6.8, but have the same problem you do: It either doesn't recognize that a calculator is plugged in, or it produces an error saying that it detected an unknown device but was unable to connect.
Both of the older computers run on the PowerPC processors, while my new one runs on an Intel one. This might be the cause—I've had similar problems with other programs caused by this.
I did contact TI about this several months ago and never got a reply.