I was surprised by this since the ajax calls really should be asynchronous. The problem turned out to be the use of session. When session is passed with the ajax call, the call blocks other calls that passes the same session. Because if the first call alters the session it should have an impact on the other calls so they have to wait.
The solution to this was to add this attribute on the controller.
When the controller is marked with that attribute asp.net knows that the code in the controller can't alter the session state so the other calls don't have to wait for it.
Here's the blog post that got us on the right track.