In my case this was caused by having
xdebug.remote_autostart = 1
set in php.ini. That causes xdebug to try to connect to remote debugger on every request. I had some PHP handled styles, auto_preppend_file and other PHP files in the request and for each of them it waited approximately 2secs, which added up to sth. like 15 seconds or so. Setting
xdebug.remote_autostart = 0
solved the problem completely. xdebug connects only when debug cookie is present. Please note you need to remove the debug cookie/param if you are not in debug session for this fix to work.
Here is my config that I use to setup xdebug.