I’m taking a guess here and suspect that you have the following setting configured in your web.config
file:
<modules runAllManagedModulesForAllRequests="true">
This means that every request, including those for static content is hitting the pipeline.
Change this setting to:
<modules runAllManagedModulesForAllRequests="false">
This is assuming your application is running under ASP.NET 4.0 and MVC3.
For this to work you need to install KB980368 (requires a reboot) or Windows 2008R2 SP1 (which includes this hotfix). The reason for this is explained in this excellent article:
How ASP.NET MVC Routing Works and its Impact on the Performance of Static Requests