For Xcode 8.2 or later

You can take videos and screenshots of Simulator using the xcrun simctl, a command-line utility to control the Simulator

1. Run your app on the simulator

2. Open a terminal

3. Run the command

   • To take a screenshot

     xcrun simctl io booted screenshot <filename>.<file extension>

     For example:

     xcrun simctl io booted screenshot myScreenshot.png

   • To take a video

     xcrun simctl io booted recordVideo <filename>.<file extension>

     For example:

     xcrun simctl io booted recordVideo appVideo.mov

4. Press ctrl + C to stop recording the video.

The default location for the created file is the current directory.

Xcode 11.2 and later gives extra options.

From Xcode 11.2 Beta Release Notes

simctl video recording now produces smaller video files, supports HEIC
compression, and takes advantage of hardware encoding support where
available. In addition, the ability to record video on iOS 13, tvOS
13, and watchOS 6 devices has been restored.

You could use additional flags:

xcrun simctl io --help
Set up a device IO operation.
Usage: simctl io <device> <operation> <arguments>

...

    recordVideo [--codec=<codec>] [--display=<display>] [--mask=<policy>] [--force] <file or url>
        Records the display to a QuickTime movie at the specified file or url.
        --codec      Specifies the codec type: "h264" or "hevc". Default is "hevc".

        --display    iOS: supports "internal" or "external". Default is "internal".
                     tvOS: supports only "external"
                     watchOS: supports only "internal"

        --mask       For non-rectangular displays, handle the mask by policy:
                     ignored: The mask is ignored and the unmasked framebuffer is saved.
                     alpha: Not supported, but retained for compatibility; the mask is rendered black.
                     black: The mask is rendered black.

        --force      Force the output file to be written to, even if the file already exists.

    screenshot [--type=<type>] [--display=<display>] [--mask=<policy>] <file or url>
        Saves a screenshot as a PNG to the specified file or url(use "-" for stdout).
        --type       Can be "png", "tiff", "bmp", "gif", "jpeg". Default is png.

        --display    iOS: supports "internal" or "external". Default is "internal".
                     tvOS: supports only "external"
                     watchOS: supports only "internal"

                     You may also specify a port by UUID
        --mask       For non-rectangular displays, handle the mask by policy:
                     ignored: The mask is ignored and the unmasked framebuffer is saved.
                     alpha: The mask is used as premultiplied alpha.
                     black: The mask is rendered black.

Now you can take a screenshot in jpeg, with mask (for non-rectangular displays) and some other flags:

xcrun simctl io booted screenshot --type=jpeg --mask=black screenshot.jpeg