I use this command:
sudo /Developer/Library/uninstall-devtools --mode=all
Edit (1 year later):
If you’ve downloaded Xcode from the App Store, it’s self-contained, as @mediaslave suggests. You can just drag it to the trash or use AppZapper (or a similar utility) to remove the developer tools.
If you’re looking to update it, you can also do that straight through the App Store.