To configure notifications for new Google Talk messages

From the Friends list, tap the Menu Key > Settings.

NOTE

Each Google Account uses its own settings. If you use more than one Google Account on your device, tap the account you want to modify.

Checkmark Invitation notifications to receive a notification in the Status Bar when friend invitation(s) are received.
Tap the IM notifications (Video chat notifications) setting to set how to be notified when a new chat message arrives.
Tap the Notification ringtone (Video chat ringtone) setting to set the ringtone when a new chat message arrives.
Tap the Vibrate setting in the Chat settings and Voice & video chat settings to set when to vibrate as a notification for new chat messages.