After some digging and asking some questions to a few colleagues, we finally solved the problem.
Important to understand is there are 2 aspects of security in this case. The IIS security and the WCF security.
IIS security: Enable SSL & enable Basic Authentication. Disable Anonymous Authentication.
(Of course, create a windows account/group and set the permissions on your application in IIS.)
WCF security: Because the binding is only a BasicHttpBinding, the service doesn’t require to valid anything. IIS is responsible for this.
The binding configuration of the service:
<bindings>
<basicHttpBinding>
<binding>
<security mode="Transport">
<transport clientCredentialType="Basic" />
</security>
</binding>
</basicHttpBinding>
And finally, to resolve the first error, we deleted the mex Endpoint. This endpoint requires a HTTP binding.
Deleted:
<endpoint address="mex" binding="mexHttpsBinding" contract="IMetadataExchange"/>