It’s not possible to trigger a different rendering mode in a child iframe in IE9, as officially documented here: http://msdn.microsoft.com/en-us/library/gg558056(v=vs.85).aspx (emphasis added):
Although the newer rendering engine is only used when Windows Internet
Explorer detects that an HTML page has requested the highest level of
support for standards, the same is not always true for child pages
that might be loaded within frame and iframe elements. Because only
one rendering engine can be active at a time, IE9 Mode also includes
emulation for Quirks Mode.
However, as it says, you can trigger “quirks mode emulation” which leaves the IE9 rendering engine active but alters its behavior in several ways to match the old quirks mode.
JSBin demo: http://jsbin.com/ozejuk/1/
This example has a div with style background: #ff0000; background: 00ff00; border-radius: 30px … in quirks mode, hex colors without # are accepted. In IE9 mode they are not. Loading the demo in IE9 will show a red div in the parent page, and a green div (but still with rounded corners) in the iframe.
How to trigger quirks mode emulation in an iframe: http://msdn.microsoft.com/en-us/library/gg558096(v=vs.85).aspx
Short version: omit DOCTYPE, add: <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7" />
Complete list of effects quirks mode emulation has on rendering: http://msdn.microsoft.com/en-us/library/gg558047(v=vs.85).aspx