Syntax error: “(” unexpected — with !(*.sh) in bash script

This is not a “sh file” — it’s a bash script. If you run it with sh yourscript, it will not work (as extglobs, the shell feature you’re trying to use, aren’t supported in POSIX sh); it needs to be run only with bash yourscript, or with ./yourscript when starting with #!/bin/bash (as it does). Describing it as a “sh file” is thus misleading. Moreover, even with bash, the extended globbing feature needs to be turned on.

---

Your immediate issue is that !(*.sh) is not regular glob syntax; it’s an extglob extension, not available by default. You may have a .bashrc or similar configuration file which enables this extension for interactive shells, but that won’t apply to scripts. Run:

shopt -s extglob

…to enable these features.

---

Cleaned up, your script might look like:

#!/bin/bash

shopt -s extglob

# putting settings in an array allows unescaped newlines in definition
# also sorted to make it easier to find things.
settings=(
  -b:v 3000k
  -bf 2
  -c:v libx264
  -level 3.1
  -movflags faststart
  -pix_fmt yuv420p
  -preset:v slow
  -profile:v Main
  -r 29.97
  -s 1280x720
  -threads 0
  -vf yadif=0:-1
)

for f in !(*.sh); do
    ffmpeg "${settings[@]}" -i "$f" \
      /mnt/media/out-mp4/"${f%.mxf}.mp4" && rm -- "$f"
done

Note the following changes, above and beyond formatting:

• shopt -s extglob is on its own line, before the glob is expanded.
• The rm is only run if ffmpeg succeeds, because the separator between those commands is &&, rather than either ; or a bare newline.
• The -- argument passed to rm tells it to treat all future arguments (in this case, the content of "$f") as a filename, even if it starts with a dash.
• The "$f" argument to rm is inside double quotes.