EDIT: Brian Slesinsky just documented the protocol (by reverse-engineering the code): https://docs.google.com/document/d/1eG0YocsYYbNAtivkLtcaiEE5IOF5u4LUol8-LL0TIKU/edit
First, GWT-RPC protocol is asymmetric so that it’s always optimized for the client-side: fast to deserialize something coming from the server, and fast to serialize something to send to it.
It’s obviously not binary, as you suspected, but text-based. client-to-server protocol is pipe-delimited while server-to-client is based on JSON (with a //OK or //EX prefix to tell whether the request succeeded or failed). Both use the common knowledge of the serializable classes to serialize/deserialize; for instance, both sides know that class X has two fields, an integer and a String, serialized in that order, so they both write/read an integer, and then a String, with no need to specify in the encoded format which field it’s about.
GWT-RPC protocol is versionned (it changes regularly as new GWT versions are released), and uses hashes of the class and serializable fields’ names to ensure the client and server both use the same versions of the classes (which means you have to recompile and redeploy your client code each time you change a serializable class).
The best documentation is the code, but you’ll find an overview of the request format in these slides: https://www.owasp.org/images/7/77/Attacking_Google_Web_Toolkit.ppt
RequestFactory, contrary to GWT-RPC, uses a symmetric JSON-based protocol (based on AutoBean’s JSON serialization) where client and server can communicate even when not compiled from the same code (well, depending on the changes you made between versions, of course), because they pass around class and property names.