Improve ObjectProxy

[pyhacks]

There are many dunder methods documented here but are missing in ObjectProxy. Some of them like __call__ is defined in classes inheriting ObjectProxy. But for the sake of reusability of ObjectProxy, they should also be defined in it. Also, there are dunder methods like __next__ that are not even defined in inheriting classes. All of them should be defined in ObjectProxy.

[GrahamDumpleton]

It is not that simple.

Concerning __iter__, it is arguable that it was a mistake to add it to ObjectProxy in the first place as it has broken some code where people change behaviour based on a check of hasattr(obj, "__iter__"). That is, this check would always return True even though iter() would fail and raise an AttributeError. So in hindsight it perhaps shouldn't have been added, even if the hasattr check is sort of an anti Python and people should just capture errors when doing iteration itself.

As to __call__ is was never added because there callable() always existed for checking if something was callable and was clear that adding __call__ in base object proxy would cause issues straight away since people relied on callable(). That people were relying on hasattr(obj, "__iter__") was not so obvious.

Because of the issues with __iter__, this is why the asynchronous versions were never added later when that feature was added to Python.

As to __next__, what is the specific use case since __iter__ would call the __iter__ of the wrapped object and so the returned object which provides __next__ would not be a proxy object. Thus I don't see a direct reason why __next__ should appear in ObjectProxy as a default.

If you are overriding __iter__ on a custom object proxy to wrap the result when iter() is called, then you are already dealing with a mess of custom object proxies and so should be using such a custom object proxy to override __next__.

As to the statement that "there are many dunder methods documented but are missing in ObjectProxy", they would fall into two cases. The first is that they didn't exist in older Python versions when wrapt was first written. The second is that it isn't appropriate for them to be added on an object proxy, or at least not as default behaviour. In other words, would be more appropriate to add them in a custom object proxy.

So can you provide some example use case of what you are trying to do where you need __next__ to be proxied.

As to other dunder methods, list those you believe should be added by default on object proxy and provide specific examples so can understand why. In some cases where I learn of new dunder methods added in Python 3 they have been added, but only when asked and it makes sense to add them. I don't troll through each Python release looking for new dunder methods.

[pyhacks]

In order to solve problems of hasattr(obj, "__iter__") and callable(obj) you can define a function generating a ObjectProxy and before instantiating it, modify it depending on the needs of obj. Example for __call__:

def object_proxy(wrapped):
    class ObjectProxy(with_metaclass(_ObjectProxyMetaType)):  # type: ignore[misc]

        __slots__ = "__wrapped__"

        def __init__(self, wrapped):
            object.__setattr__(self, "__wrapped__", wrapped)

            # Python 3.2+ has the __qualname__ attribute, but it does not
            # allow it to be overridden using a property and it must instead
            # be an actual string object instead.

            try:
                object.__setattr__(self, "__qualname__", wrapped.__qualname__)
            except AttributeError:
                pass

            # Python 3.10 onwards also does not allow itself to be overridden
            # using a property and it must instead be set explicitly.

            try:
                object.__setattr__(self, "__annotations__", wrapped.__annotations__)
            except AttributeError:
                pass

        def __self_setattr__(self, name, value):
            object.__setattr__(self, name, value)

        @property
        def __name__(self):
            return self.__wrapped__.__name__

        @__name__.setter
        def __name__(self, value):
            self.__wrapped__.__name__ = value

        @property
        def __class__(self):
            return self.__wrapped__.__class__

        @__class__.setter
        def __class__(self, value):
            self.__wrapped__.__class__ = value

        def __dir__(self):
            return dir(self.__wrapped__)

        def __str__(self):
            return str(self.__wrapped__)

        def __bytes__(self):
            return bytes(self.__wrapped__)

        def __repr__(self):
            return f"<{type(self).__name__} at 0x{id(self):x} for {type(self.__wrapped__).__name__} at 0x{id(self.__wrapped__):x}>"

        def __format__(self, format_spec):
            return format(self.__wrapped__, format_spec)

        def __reversed__(self):
            return reversed(self.__wrapped__)

        def __round__(self, ndigits=None):
            return round(self.__wrapped__, ndigits)

        def __mro_entries__(self, bases):
            return (self.__wrapped__,)

        def __lt__(self, other):
            return self.__wrapped__ < other

        def __le__(self, other):
            return self.__wrapped__ <= other

        def __eq__(self, other):
            return self.__wrapped__ == other

        def __ne__(self, other):
            return self.__wrapped__ != other

        def __gt__(self, other):
            return self.__wrapped__ > other

        def __ge__(self, other):
            return self.__wrapped__ >= other

        def __hash__(self):
            return hash(self.__wrapped__)

        def __nonzero__(self):
            return bool(self.__wrapped__)

        def __bool__(self):
            return bool(self.__wrapped__)

        def __setattr__(self, name, value):
            if name.startswith("_self_"):
                object.__setattr__(self, name, value)

            elif name == "__wrapped__":
                object.__setattr__(self, name, value)
                try:
                    object.__delattr__(self, "__qualname__")
                except AttributeError:
                    pass
                try:
                    object.__setattr__(self, "__qualname__", value.__qualname__)
                except AttributeError:
                    pass
                try:
                    object.__delattr__(self, "__annotations__")
                except AttributeError:
                    pass
                try:
                    object.__setattr__(self, "__annotations__", value.__annotations__)
                except AttributeError:
                    pass

            elif name == "__qualname__":
                setattr(self.__wrapped__, name, value)
                object.__setattr__(self, name, value)

            elif name == "__annotations__":
                setattr(self.__wrapped__, name, value)
                object.__setattr__(self, name, value)

            elif hasattr(type(self), name):
                object.__setattr__(self, name, value)

            else:
                setattr(self.__wrapped__, name, value)
            
        def __getattr__(self, name):
            # If we are being to lookup '__wrapped__' then the
            # '__init__()' method cannot have been called.

            if name == "__wrapped__":
                raise WrapperNotInitializedError("wrapper has not been initialised")

            return getattr(self.__wrapped__, name)

        def __delattr__(self, name):
            if name.startswith("_self_"):
                object.__delattr__(self, name)

            elif name == "__wrapped__":
                raise TypeError("__wrapped__ must be an object")

            elif name == "__qualname__":
                object.__delattr__(self, name)
                delattr(self.__wrapped__, name)

            elif hasattr(type(self), name):
                object.__delattr__(self, name)

            else:
                delattr(self.__wrapped__, name)

        def __add__(self, other):
            return self.__wrapped__ + other

        def __sub__(self, other):
            return self.__wrapped__ - other

        def __mul__(self, other):
            return self.__wrapped__ * other

        def __truediv__(self, other):
            return operator.truediv(self.__wrapped__, other)

        def __floordiv__(self, other):
            return self.__wrapped__ // other

        def __mod__(self, other):
            return self.__wrapped__ % other

        def __divmod__(self, other):
            return divmod(self.__wrapped__, other)

        def __pow__(self, other, *args):
            return pow(self.__wrapped__, other, *args)

        def __lshift__(self, other):
            return self.__wrapped__ << other

        def __rshift__(self, other):
            return self.__wrapped__ >> other

        def __and__(self, other):
            return self.__wrapped__ & other

        def __xor__(self, other):
            return self.__wrapped__ ^ other

        def __or__(self, other):
            return self.__wrapped__ | other

        def __radd__(self, other):
            return other + self.__wrapped__

        def __rsub__(self, other):
            return other - self.__wrapped__

        def __rmul__(self, other):
            return other * self.__wrapped__

        def __rtruediv__(self, other):
            return operator.truediv(other, self.__wrapped__)

        def __rfloordiv__(self, other):
            return other // self.__wrapped__

        def __rmod__(self, other):
            return other % self.__wrapped__

        def __rdivmod__(self, other):
            return divmod(other, self.__wrapped__)

        def __rpow__(self, other, *args):
            return pow(other, self.__wrapped__, *args)

        def __rlshift__(self, other):
            return other << self.__wrapped__

        def __rrshift__(self, other):
            return other >> self.__wrapped__

        def __rand__(self, other):
            return other & self.__wrapped__

        def __rxor__(self, other):
            return other ^ self.__wrapped__

        def __ror__(self, other):
            return other | self.__wrapped__

        def __iadd__(self, other):
            self.__wrapped__ += other
            return self

        def __isub__(self, other):
            self.__wrapped__ -= other
            return self

        def __imul__(self, other):
            self.__wrapped__ *= other
            return self

        def __itruediv__(self, other):
            self.__wrapped__ = operator.itruediv(self.__wrapped__, other)
            return self

        def __ifloordiv__(self, other):
            self.__wrapped__ //= other
            return self

        def __imod__(self, other):
            self.__wrapped__ %= other
            return self

        def __ipow__(self, other):  # type: ignore[misc]
            self.__wrapped__ **= other
            return self

        def __ilshift__(self, other):
            self.__wrapped__ <<= other
            return self

        def __irshift__(self, other):
            self.__wrapped__ >>= other
            return self

        def __iand__(self, other):
            self.__wrapped__ &= other
            return self

        def __ixor__(self, other):
            self.__wrapped__ ^= other
            return self

        def __ior__(self, other):
            self.__wrapped__ |= other
            return self

        def __neg__(self):
            return -self.__wrapped__

        def __pos__(self):
            return +self.__wrapped__

        def __abs__(self):
            return abs(self.__wrapped__)

        def __invert__(self):
            return ~self.__wrapped__

        def __int__(self):
            return int(self.__wrapped__)

        def __float__(self):
            return float(self.__wrapped__)

        def __complex__(self):
            return complex(self.__wrapped__)

        def __oct__(self):
            return oct(self.__wrapped__)

        def __hex__(self):
            return hex(self.__wrapped__)

        def __index__(self):
            return operator.index(self.__wrapped__)

        def __len__(self):
            return len(self.__wrapped__)

        def __contains__(self, value):
            return value in self.__wrapped__

        def __getitem__(self, key):
            return self.__wrapped__[key]

        def __setitem__(self, key, value):
            self.__wrapped__[key] = value

        def __delitem__(self, key):
            del self.__wrapped__[key]

        def __getslice__(self, i, j):
            return self.__wrapped__[i:j]

        def __setslice__(self, i, j, value):
            self.__wrapped__[i:j] = value

        def __delslice__(self, i, j):
            del self.__wrapped__[i:j]

        def __enter__(self):
            return self.__wrapped__.__enter__()

        def __exit__(self, *args, **kwargs):
            return self.__wrapped__.__exit__(*args, **kwargs)

        def __iter__(self):
            return iter(self.__wrapped__)

        def __copy__(self):
            raise NotImplementedError("object proxy must define __copy__()")

        def __deepcopy__(self, memo):
            raise NotImplementedError("object proxy must define __deepcopy__()")

        def __reduce__(self):
            raise NotImplementedError("object proxy must define __reduce__()")

        def __reduce_ex__(self, protocol):
            raise NotImplementedError("object proxy must define __reduce_ex__()")

    if callable(wrapped):
        def __call__(self, *args, **kwargs):
            return self.__wrapped__(*args, **kwargs)
        ObjectProxy.__call__ = __call__
    return ObjectProxy(wrapped)        

Usage example:

So for example, if inspect.isawaitable(obj) is the reason you didn't implement __await__, you can solve it the same way.

As to __next__, what is the specific use case

Proxying an iterator instead of a iterable:

As to other dunder methods, list those you believe should be added by default on object proxy

E.g There are __enter__ and __exit__ in ObjectProxy but their async versions are missing (__aenter__ and __aexit__). And if you decide to implement __next__, there is also __anext__. This is not the complete list. There are some other missing methods too.

[GrahamDumpleton]

You can do that inside of the __init__ method itself, but that technique is only of use for the pure Python implementation of ObjectProxy.

The pure Python implementation would rarely be used as it exists only as a fallback for when the C implementation isn't used for some reason.

In a C implementation of a Python object that sort of trick will not work because Python when it sees an object implemented in C, will only look in the tp_ slots of the C structure defining the type, and will ignore __iter__ and __call__ which are added as an attribute on the C object __dict__.

To be able to use the trick that works for the pure Python object implementation it would be necessary to change wrapt so that the existing C implementation of ObjectProxy is renamed to _ObjectProxy and then create a small ObjectProxy Python implementation shim that inherits from C based _ObjectProxy and in the Python implementation of the derived class __init__ then you could do it.

I will need to research though what the implications of doing this will be though on other dunder attributes which have weird rules such as __qualname__ and __annotations__, plus the meta class hacks. These may have to be replicated in that derived ObjectProxy for the C implementation as well as that in the C base class may no longer have the desired effect.

Although I expect it may be minimal enough that not a concern, adding the extra Python class deriving from C implementation may have extra overhead as a result.

Anyway, your response didn't answer the main question I put to you, which was the example you had where having __next__ as a default on ObjectProxy would be useful.

[pyhacks]

You can't do that inside __init__ because magic methods aren't automatically called when they are object attributes instead of class attributes. See this code:

class A:
    def __init__(self):
        def __call__(self, *args, **kwargs):
            print("hello")

        self.__call__ = __call__
    
x = A()
x()

This gives: TypeError: 'A' object is not callable
You also can't monkeypatch ObjectProxy inside __init__ because that change will be reflected on all ObjectProxy instances and it is possible that one instance should be callable and another shouldn't be. So that every ObjectProxy instance should have its own type.

your response didn't answer the main question I put to you

I don't have a real world example but i tried to answer you by giving an example of proxying an iterator. Isn't being unable to correctly proxy an iterator bad enough?

Lastly, you also didn't answer a question of mine. Why didn't you implement __aenter__ and __aexit__?

[GrahamDumpleton]

Using:

self.__call__ = __call__

would not have desired effect since that would set __call__ on the wrapped object and not the wrapper because of how __setattr__ is defined. Would have had to use object.__setattr__() instead.

Anyway, I keep forgetting that __call__ is one of those methods that needs to be set on the class and not on the instance even in pure Python code. Not all dunder methods necessarily need to be unless things have changed from early Python 2 days. The rules around them can be non obvious.

Rather than create a full class definition nested inside of a normal function, which would prohibit easily creating derived custom ObjectProxy classes, a better approach may be a minimal ObjectProxy derived class shim inside of a __new__ method that inherits from the cls argument passed to __new__. This will though need to be tolerant of a derived class already overriding __call__ and also want to ensure same type name comes through, the latter meaning can't use normal nested class and so construct it manually.

class A:
  def __init__(self, wrapped):
    self.__wrapped__ = wrapped

  def __new__(cls, wrapped):
    print("orig", cls)

    namespace = {}

    if callable(wrapped) and "__call__" not in dir(cls):
      def __call__(self, *args, **kwargs):
        print("base")
        return self.__wrapped__(*args, **kwargs)
      namespace["__call__"] = __call__

    NewClass = type(cls.__name__, (cls,), namespace)

    print("new", NewClass)

    return super().__new__(NewClass)

class B(A):
  def __call__(self, *args, **kwargs):
    print("derived")
    return self.__wrapped__(*args, *kwargs)

def func():
  print("func")

a = A(func)

print(type(a))

a()

b = B(func)

print(type(b))

b()

Next problem to consider then is that __wrapped__ can technically be assigned to and the wrapped object changed after the wrapper has been initialised. One would have to document a caveat that the behaviour of the class for the wrapper would adopt what was required only for the wrapped object initially supplied when the wrapper is created, and overriding __wrapped__ would not change it if requirement was different. Since expect is unlikely anyone would be reassigning __wrapped__ this would be acceptable.

[GrahamDumpleton]

Oh, and regard question "why didn't you implement __aenter__ and __aexit__", because they never existed when wrapt was first written.

[pyhacks]

We can handle the case of assigning to __wrapped__ in __setattr__ so that class structure will adapt to the new object:

class A:
    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

    def __new__(cls, wrapped):
        NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
        obj = super().__new__(NewClass)
        if "__call__" in NewClass.__dict__:
            obj._self_call_backup = type(obj).__call__
        else:
            def __call__(self, *args, **kwargs):
                print("base")
                return self.__wrapped__(*args, **kwargs)
            obj._self_call_backup = __call__
        obj.__init__(wrapped) # Must manually call since NewClass != cls
        return obj

    def __setattr__(self, name, value):
        if name.startswith("_self_"):
            object.__setattr__(self, name, value)        
        elif name == "__wrapped__":
            if not ("__call__" in type(self).__dict__) and callable(value):
                type(self).__call__ = self._self_call_backup
            elif ("__call__" in type(self).__dict__) and not callable(value):
                del type(self).__call__
            object.__setattr__(self, name, value) 
                

class B(A):
    def __call__(self, *args, **kwargs):
        print("derived")
        return self.__wrapped__(*args, *kwargs)

def func():
    pass

class C:
    pass

def test():
    a = A(func)
    print(callable(a))
    a.__wrapped__ = C()
    print(callable(a))
    a.__wrapped__ = func
    print(callable(a))
    a()
    print()

    a = B(func)
    print(callable(a))
    a.__wrapped__ = C()
    print(callable(a))
    a.__wrapped__ = func
    print(callable(a))
    a()

test()

Since expect is unlikely anyone would be reassigning __wrapped__

Actually I'm working on a code that requires overloading operators like += and in there this feature is pretty important

[pyhacks]

@GrahamDumpleton I just found this issue from 2016 and it's still not fixed. If you are busy with other things these days, I can also update the python part of ObjectProxy and create a pull request. Would you want it or you already working on these?

[GrahamDumpleton]

I have already implemented a variant of dynamically adding a new class in __new__ as discussed to add __call__, __iter__, __next__, __aiter__ and __anext__. I also already added __aenter__, __aexit__ and __await__ direct to ObjectProxy.

I am still not keen on trying to change the class definition after the fact in rare case that __wrapped__ is assigned as there is more complexity to it than the code you suggested. Specifically, you should not delete any of the attributes if they were defined in a derived class, only if they were added dynamically. So have not made that change as yet and still thinking about it.

I haven't yet pushed changes up as I need to add more tests and have been busy with life stuff so haven't had time. I also still need to look more at dunder methods for matrix multiplication, which is another that currently isn't supported.

[pyhacks]

I modified my previous code snippet to check whether we are working with a dynamically added __call__ attribute or not before deleting it. This will not delete the __call__ attribute if it is defined by a derived class:

class A:
    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

    def __new__(cls, wrapped):
        NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
        obj = super().__new__(NewClass)
        if "__call__" in NewClass.__dict__:
            obj._self_call_backup = type(obj).__call__
            obj._self_delete_call = False
        else:
            def __call__(self, *args, **kwargs):
                print("base")
                return self.__wrapped__(*args, **kwargs)
            obj._self_call_backup = __call__
            obj._self_delete_call = True
        obj.__init__(wrapped) # Must manually call since NewClass != cls
        return obj

    def __setattr__(self, name, value):
        if name.startswith("_self_"):
            object.__setattr__(self, name, value)        
        elif name == "__wrapped__":
            if not ("__call__" in type(self).__dict__) and callable(value):
                type(self).__call__ = self._self_call_backup
            elif ("__call__" in type(self).__dict__) and not callable(value):
                if self._self_delete_call:
                    del type(self).__call__
            object.__setattr__(self, name, value) 
                

class B(A):
    def __call__(self, *args, **kwargs):
        print("derived")
        return self.__wrapped__(*args, *kwargs)

def func():
    pass

class C:
    pass

def test():
    a = A(func)
    print(callable(a))
    a.__wrapped__ = C()
    print(callable(a))
    a.__wrapped__ = func
    print(callable(a))
    a()
    print()

    a = B(func)
    print(callable(a))
    a.__wrapped__ = C()
    print(callable(a))
    a.__wrapped__ = func
    print(callable(a))
    a()

test()

Is there any other problem with it now?

[pyhacks]

Great news you already implemented many dunder methods. But just like you gave the example of __matmul__, there are still many other unimplemented dunder methods. Example: __get__, __set__, __delete__

[GrahamDumpleton]

In respect of __get__ you would normally have:

• obj.attr → Python calls attr.__get__(obj, type(obj))

If you have basic proxying of __get__ then result is:

• proxy.attr → Python calls proxy.__get__(obj, type(obj)) → calls wrapped.__get__(obj, type(obj))

The issue is that self in the wrapper functions refers to the proxy object, not the original wrapped object. This can break descriptor behaviour because:

• Identity issues: The descriptor receives the proxy as the instance parameter instead of the actual object it was designed to work with.
• Type confusion: owner parameter might not match what the descriptor expects.
• Binding context: Descriptors often rely on being bound to specific class hierarchies.

So I can't see that there is a generic proxy wrapper implementation one can have for special descriptor methods and binding protocol.

Binding for methods is complicated enough that wrapt has a special FunctionWrapper derived proxy type already to handle it all and that code is a mess of special cases.

[pyhacks]

In your sentence:

• proxy.attr → Python calls proxy.__get__(obj, type(obj)) → calls wrapped.__get__(obj, type(obj))

I think you made a typo by writing proxy.attr instead of obj.attr, or even better, obj.proxy right?

Anyway, why would you pass self as the instance parameter? If you pass instance as the instance parameter and owner as the owner parameter there will be no problems. Like in this code:

class Descriptor:
    def __get__(self, instance, owner = None):
        print(instance)


class Proxy:
    def __new__(cls, wrapped):
        NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
        obj = super().__new__(NewClass)
        if "__get__" in NewClass.__dict__:
            obj._self_get_backup = type(obj).__get__
            obj._self_delete_get = False
        else:
            def __get__(self, instance, owner = None):
                return self.__wrapped__.__get__(instance, owner)        
            obj._self_get_backup = __get__
            obj._self_delete_get = True
        obj.__init__(wrapped) # Must manually call since NewClass != cls
        return obj
    
    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

    def __setattr__(self, name, value):
        if name.startswith("_self_"):
            object.__setattr__(self, name, value)        
        elif name == "__wrapped__":
            if not ("__get__" in type(self).__dict__) and hasattr(value, "__get__"):
                type(self).__get__ = self._self_get_backup
            elif ("__get__" in type(self).__dict__) and not hasattr(value, "__get__"):
                if self._self_delete_get:
                    del type(self).__get__
            object.__setattr__(self, name, value)


class A:
    var1 = Descriptor()
    var2 = Proxy(Descriptor())

    @Proxy
    def func1(self):
        print(self)

a = A()
a.var1
a.var2
a.func1()

It prints the A object 3 times. No proxy object is printed.

[GrahamDumpleton]

That will teach me to bash out a response when running out the door and still groggy from a late night and lack of sleep. 🤣

So yes, I messed up badly and forgot the instance when doing binding comes from class the descriptor is used in. I thought I had something wrong, but was in a rush. Still not sure of a valid use case for having an object proxy wrapping a data descriptor. Non data descriptions, ie., binding behaviour as implemented by FunctionWrapper and as used in decorators, is more obvious.

[pyhacks]

Having an object proxy wrapping a data descriptor would be as easy as having an object proxy wrapping a non data descriptor. Just add __set__ and __delete__ dynamically in __new__ just like i showed how to for __get__. This way, you don't have to worry about what to do in __set__ and __delete__ in case __wrapped__ doesn't define them.