It is fully valid to do
<!-- this, -->
<!DOCTYPE html>
However, it brings all versions of IE into quirks-mode (unless it is forced into no-quirks mode — see the Gotchas section below). The simplest is to move the comment below the DOCTYPE.
<!DOCTYPE html>
<!-- this, -->
But another way is to “upgrade” the comment into a suitable conditional comment, such as this:
<!--[if !IE]> this <![endif]-->
<!DOCTYPE html>
Explanation: a conditional comment does not count as a comment, in IE’s world.
Alternative syntax: To forget/remember that conditional comments are a Microsoft intrusion into the HTML standard, one could for instance do
<!--[if anybrowser]> this <![endif]-->
<!DOCTYPE html>
Likewise, to target IE in particular, one could do
<!--[if !anybrowser]> this <![endif]-->
<!DOCTYPE html>
Gotchas
A comment inside a conditional comment will bring IE into quirks-mode if IE sees it (that is: if one uses an [if IE] condition, or an equivalent to [if IE] — such as the [if !anybrowser] condition that I mentioned above.). So, for example, this would bring IE in quirks-mode:
<![if IE]><!-- this --><![endif]>
<!DOCTYPE html>
As would
<!--[if IE]><!--><!-- this <![endif]-->
<!DOCTYPE html>
and many other variants. Whereas for example
<!--[if IE]><!DOCTYPE html><!--><!-- this <![endif]-->
<!DOCTYPE html>
would not cause quirks-mode, because here the conditional comment has a DOCTYPE before any other content, and thus IE considers that the first content of the page is a DOCTYPE.
Finally, the newest IE versions, IE8 and IE9, can be forced to standards-mode (and to quirks-mode as well) by the use of another Microsoft invention — the x-ua-compatible directive. See http://msdn.microsoft.com/en-us/library/cc288325(v=vs.85).aspx In that case, then
<!-- this -->
<!DOCTYPE html>
<!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=8" ><![endif]-->
will force IE8 and IE9 into no-quirks mode, while IE6 and IE7 will remain in quirks mode. Whereas, in contrast, this
<!--[if gte IE 8]><meta http-equiv="X-UA-Compatible" content="IE=8" ><![endif]-->
<!DOCTYPE html>
would force IE8 and IE9 into standards mode, despite that the content of the conditional comment does not start with a DOCTYPE. And IE6 and IE7 will also remain in no-quirks mode since the conditional comment doesn’t target them.