NASM/YASM is easy to distinguish from MASM/TASM/emu8086. YASM uses NASM syntax, with a few minor differences in what it accepts for constants and directives.
I don’t know how to distinguish MASM from TASM, or TASM from emu8086, or FASM, so I’ll leave that for another answer to address.
In NASM, explicit sizes on things like memory operands use
byte. In TASM/MASM style, you have to write
dword ptr or
In MASM (and I think TASM/emu8086), a bare symbol name referes to the contents. You have to use
offset foo to get the address of foo. In NASM, you have to use
[foo] to create a memory operand, and
foo is the address.
There are probably other differences in syntax, too (e.g. in segment overrides), but these should be enough to tell by looking whether something is NASM-style or MASM-style.
global foo foo: ; a function called foo() add dword [ecx], 2 add dword [counter], 1 ; Error without "dword", because neither operand implies an operand-size for the instruction. And the  is required. mov eax, [static_var] mov eax, [static_array + ecx*4] ; Everything *must* be inside the  mov esi, static_var ; mov esi,imm32 with the address of the static_var ret section .data static_var: dd 0xdeadbeef ; NASM can use 0x... constant. MASM only allows 0DEADBEEFh style section .bss counter: resd 1 ; reserve space for one dword (initialized to zero) buf: resb 256 ; reserve 256 bytes
: after label names here, even for data. This is recommended but not required: any unknown token at the start of a line is assumed to be a label so
counter resd 1 will assemble. But
loop resd 1 won’t because
loop is a valid instruction mnemonic.
MASM/TASM (I may have some of this wrong, I don’t use MASM or TASM):
.intel_syntax noprefix is mostly the same, but without the magic operand-size association for labels. And GAS directives / pseudo-instruction are totally different, like
.byte 0x12 vs.
.CODE foo PROC ; PROC/ENDP definitely means not NASM add dword ptr [ecx], 2 add counter, 1 ; operand-size magically implied by the dd after the counter label.  is optional mov eax, static_var ; mov eax, [static_var] is the same, and recommended by some for clarity mov eax, static_array[ecx*4] ; [ static_array + ecx*4 ] is also allowed, but not required. mov esi, OFFSET static_var ; mov esi,imm32 with the address. ret ENDP .data ; no SECTION directive, just .data directly static_var dd 0deadbeefH ;;; With a : after the name, it would be just a label, not a "variable" with a size associated. .bss ; (In most OSes, the BSS is initialized to zero. I assume MASM/TASM allows you to write dd 0 in the BSS, but I'm not sure) counter: dd 0 ; reserve space for one dword (zeroed) buf db 256 dup(?) ; reserve 256 bytes (uninitialized).
Except where I commented otherwise, any of these differences are a guaranteed sign that it’s NASM/YASM or MASM/TASM/emu8086
e.g. if you ever see a bare symbol as the destination operand (e.g.
mov foo, eax), it’s definitely not NASM, because
mov imm32, r32 makes no sense. Unless the symbol is actually a macro definition for a register, e.g.
%define result eax would allow
mov result, 5. (Good catch, @MichaelPetch). If the source is full of macros, then look for the defs.
%define means NASM, while
MACRO means MASM/TASM.
MASM/TASM doesn’t have
resd directives. Instead, they have
count DUP(value), where value can be
times 30 db 0x10 to repeat the byte
0x10 30 times. You can use it on anything, even instructions. It also has
%rep directives to repeat a block.
MASM and NASM have significant macro capabilities, but they use different syntax.
The x86 tag wiki has links to assembler manuals and much more.
Other random things when assembling code with the wrong assembler:
dword by itself (not dword ptr) evaluates as the number
4, because that’s the width of a dword. So
mov dword [foo], 123 will disastrously assemble as
mov 4[foo], 123 which is the same as
[foo+4]. And the operand-size will be whatever size is implied by how you declared
foo db 1,2,3,4 is an array of bytes, so
mov dword [foo], 123 assembled by MASM is actually
mov byte ptr:foo, 123.
See also Confusing brackets in MASM32 for the disaster of syntax-design that is MASM.
mov eax, [const] is a mov-immediate if
const was declared like