I got it working by adding cbrandolino‘s suggested mimetypes to the config/mimes.php (great tip jljohnstone). So the csv property of my $mimes looks like this now:
'csv' => array('application/vnd.ms-excel', 'text/anytext', 'text/plain', 'text/x-comma-separated-values', 'text/comma-separated-values', 'application/octet-stream', 'application/vnd.ms-excel', 'application/x-csv', 'text/x-csv', 'text/csv', 'application/csv', 'application/excel', 'application/vnd.msexcel')