Regex is not the right tool for the job. Scan a string manually.
Pseudo-code:
depth = 0
for character in some_string:
depth += character == '('
depth -= character == ')'
if depth < 0:
break
if depth != 0:
print "unmatched parentheses"