Since nobody mentioned that, I believe this question needs an update. The way I managed to solve the problem was using the “preload” option supported by modern browsers.
In case someone does not need to support old browsers.
<link rel="preload" href="https://stackoverflow.com/questions/4712242/assets/fonts/xxx.woff" as="font" type="font/woff" crossorigin>
some useful links with more details:
https://developer.mozilla.org/en-US/docs/Web/HTML/Preloading_content
http://www.bramstein.com/writing/preload-hints-for-web-fonts.html