The HTML5 @pattern attribute regex is based on JavaScript regex, so you can use hexadecimal notation or unicode notation for that char as you would in JavaScript:
" = \x22 OR \u0022
' = \x27 OR \u0027
So that you can easily use
pattern="[^'\x22]+"
TADA! No JavaScript libraries involved! 😉