Edit (because this answer is still getting upvotes): This answer works to mitigate the problem but the likely root cause for the issue is addressed in @jfeldman’s answer.
I found the solution as an answer to a completely different question.
The issue is the movieFragmentInterval property in AVCaptureMovieFileOutput.
The documentation for this property explains what these fragments are:
A QuickTime movie is comprised of media samples and a sample table
identifying their location in the file. A movie file without a sample
table is unreadable.In a processed file, the sample table typically appears at the
beginning of the file. It may also appear at the end of the file, in
which case the header contains a pointer to the sample table at the
end. When a new movie file is being recorded, it is not possible to
write the sample table since the size of the file is not yet known.
Instead, the table is must be written when recording is complete. If
no other action is taken, this means that if the recording does not
complete successfully (for example, in the event of a crash), the file
data is unusable (because there is no sample table). By periodically
inserting “movie fragments” into the movie file, the sample table can
be built up incrementally. This means that if the file is not written
completely, the movie file is still usable (up to the point where the
last fragment was written).
It also says:
The default is 10 seconds. Set to kCMTimeInvalid to disable movie
fragment writing (not typically recommended).
So for some reason my recording is getting messed up whenever a fragment is written. I just added the line movieFileOutput.movieFragmentInterval = kCMTimeInvalid; (where movieFileOutput is the AVCaptureMovieFileOutput I’ve added to the AVCaptureSession) to disable fragment writing, and the audio now works.