How does server prioritize which type of web.xml error page to use?

This is not specific to Tomcat. This is specific to the Servlet API. How the error page is determined is specified in chapter 9.9.2 of Servlet API specification 2.5. Here’s an extract of relevance:

SRV.9.9.2 Error Pages

If no error-page declaration containing an exception-type fits using the
class-hierarchy match, and the exception thrown is a ServletException or
subclass thereof, the container extracts the wrapped exception, as defined by
the ServletException.getRootCause method. A second pass is made over the error
page declarations, again attempting the match against the error page
declarations, but using the wrapped exception instead.

So, your SpecificExceptionA was likely wrapped in a ServletException and thus the java.lang.Throwable is the closest match on 1st pass. When you remove this entry, a 2nd pass will be made with the wrapped exception and thus your SpecificExceptionA get a match.

The right way to define a general HTTP 500 error page is to map it on error-code instead of exception-type:

<error-page>
    <exception-type>org.SpecificExceptionA</exception-type>
    <location>/WEB-INF/views/error/timedout.jsp</location>
</error-page>
<error-page>
    <error-code>500</error-code>
    <location>/WEB-INF/views/error/error.jsp</location>
</error-page>

If this is not an option for some unclear reason, one of the ways to workaround this is to create a Filter which listens on an url-pattern of /* and does basically the following:

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws ServletException, IOException {
    try {
        chain.doFilter(request, response);
    } catch (ServletException e) {
        Throwable rootCause = e.getRootCause();
        if (rootCause instanceof SpecificExceptionA) {
            throw (SpecificExceptionA) rootCause;
        } else {
            throw e;
        }
    }
}

It only has to extend from RuntimeException to get it to work.