The square bracket means that the contents are optional, but everything outside of square brackets is compulsory.
With your notation:
RegexObject.match(string, [pos], [endpos])
I would expect to have to write:
r.match("foo",,)
The nesting is required because if you supply the third parameter then you must also supply the second parameter even though it is an optional parameter. The following non-nested alternative would be ambiguous:
RegexObject.match(string[, pos][, endpos])