Why does object.__new__ work differently in these three cases

You are using an older Python version; the error message has since been updated:

>>> object.__new__(testclass1, 56)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object() takes no parameters

Python will only complain about __init__ not supporting arguments if neither __new__ nor __init__ have been overridden; e.g. when you inherit both from object. testclass1 fits that case, testclass3 does not because it has an __init__ method.

This is to support implementing immutable types that don’t have a use for __init__ (which would be inherited from object in that case), and mutable types, where __new__ should not care about what arguments __init__ expects (which usually would be more arguments).

See issue 1683368 where Guido van Rossum explains his motivations for this.

The typeobject.c source code has this to say:

You may wonder why object.__new__() only complains about arguments
when object.__init__() is not overridden, and vice versa.

Consider the use cases:

1. When neither is overridden, we want to hear complaints about
   excess (i.e., any) arguments, since their presence could
   indicate there’s a bug.

2. When defining an Immutable type, we are likely to override only
   __new__(), since __init__() is called too late to initialize an
   Immutable object. Since __new__() defines the signature for the
   type, it would be a pain to have to override __init__() just to
   stop it from complaining about excess arguments.

3. When defining a Mutable type, we are likely to override only
   __init__(). So here the converse reasoning applies: we don’t
   want to have to override __new__() just to stop it from
   complaining.

4. When __init__() is overridden, and the subclass __init__() calls
   object.__init__(), the latter should complain about excess
   arguments; ditto for __new__().

Use cases 2 and 3 make it unattractive to unconditionally check for
excess arguments. The best solution that addresses all four use
cases is as follows: __init__() complains about excess arguments
unless __new__() is overridden and __init__() is not overridden
(IOW, if __init__() is overridden or __new__() is not overridden);
symmetrically, __new__() complains about excess arguments unless
__init__() is overridden and __new__() is not overridden
(IOW, if __new__() is overridden or __init__() is not overridden).

However, for backwards compatibility, this breaks too much code.
Therefore, in 2.6, we’ll warn about excess arguments when both
methods are overridden; for all other cases we’ll use the above
rules.

Note that the .__init__() method itself will still complain! When you create an instance, both __new__ and __init__ are called; your code only calls __new__ directly and does not invoke __init__! Creating an instance of testclass1 and testclass3 both fails if you pass in arguments:

>>> testclass1(56)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object() takes no parameters
>>> testclass3(56)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() takes exactly 1 argument (2 given)

The only difference is that for testclass1 it is the default methods for object() that complain instead a specific error for the custom __init__.