My assumption would be that the video has default encoding – usually with an mp4 video the final step is to write the MOOV atom that contains the metadata and information about the frame locations etc at the end. This means until the file has been completely read the browser doesn’t have all the information available to it.
You can fix this using FFMPEG (open source, free, downloads available for most platforms) and doing a second pass on the video to move the MOOV atom to the front:
./ffmpeg -y -i source.mp4 -movflags faststart dest.mp4