An informal protocol was, as Jonnathan said, typically a category declared on NSObject with no corresponding implementation (most often — there was the rare one that did provide dummy implementations on NSObject).
As of 10.6 (and in the iPhone SDK), this pattern is no longer used. Specifically, what was declared as follows in 10.5 (and prior):
@interface NSObject(NSApplicationNotifications)
- (void)applicationWillFinishLaunching:(NSNotification *)notification;
...
@interface NSObject(NSApplicationDelegate)
- (NSApplicationTerminateReply)applicationShouldTerminate:(NSApplication *)sender;
...
Is now declared as:
@protocol NSApplicationDelegate <NSObject>
@optional
- (NSApplicationTerminateReply)applicationShouldTerminate:(NSApplication *)sender;
...
- (void)applicationWillFinishLaunching:(NSNotification *)notification;
...
That is, informal protocols are now declared as @protocols with a bunch of @optional methods.
In any case, an informal protocol is a collection of method declarations whereby you can optionally implement the methods to change behavior. Typically, but not always, the method implementations are provided in the context of delegation (a table view’s data source must implement a handful of required methods and may optionally implement some additional methods, for example).