Python Descriptor

Article Info

Contributed by
2 authors

Last updated on
2022-02-15 11:57:03

Article Versions

5 2022-02-15 11:57:03
3398,3025 5,3398

By devbot5S

Migrating blockquotes to markdown syntax
4 2022-01-08 12:35:25
3025,2829 4,3025

By arvindpdmn

Minor fix in milestone.
3 2021-12-25 13:35:09
2829,2787 3,2829

By arvindpdmn

Completing main content. Two more milestone added. Sample Code added.
2 2021-12-22 04:37:30
2787,2684 2,2787

By arvindpdmn

Added Milestones and See Also.
1 2021-12-09 03:25:26
1,2684

By arvindpdmn

Questions identified.

Chat Room

Submitting ...

You are editing an existing chat message.

Python objects have attributes that can be accessed, modified or deleted. Python provides a default manner in which these operations are performed but what if we wish to customize these operations? This is where descriptors are helpful.

Descriptors can be used to manage access to an attribute, validate attribute values, customize error messages, track data corruption bugs, or perform dynamic computations on attribute lookup. Descriptors are commonly defined at class level but when defined at module level they can be used to display deprecation warnings or implement lazy loading.

It's been said that a good understanding of descriptors can improve a developer's coding skills. Descriptors are commonly used by library developers.

Discussion

What are some use cases of Python descriptors?
An example descriptor that computes size on demand. Source: Hettinger 2021.
Suppose there's a Directory class that has a size attribute to reflect the number of files in that directory. Naturally, its value can change dynamically as files are added or removed. Without descriptors, we might implement this as a method get_size(). The method will query the file system and obtain the current size.
Descriptors allow us to get the size dynamically while also accessing the size as a data attribute. Thus, we can say d.size rather than d.get_size(). Moreover, the behaviour of querying the file system was previously within Directory.get_size(). With descriptors, the behaviour is nicely encapsulated within the descriptor class DirectorySize.
Another use case of descriptors is to manage data access. For example, public attribute age is exposed in a Person class but managed privately as _age within a descriptor class. The descriptor class could also perform validations. For example, when assigning a value to age it can check if the value is a non-negative integer.
A real-world example of a descriptor is GenericForeignKey in Django.
How can I implement Python's descriptor protocol?
Illustrating __get__ and __set__ methods. Source: Adapted from Beaumont 2013, slides 16-17.
Assume one or more instances of TextWrapper class are instantiated as members of Email class. TextWrapper is the descriptor class and Email is the owner class. To support the descriptor protocol, TextWrapper implements the following methods (but not necessarily all):
- __get__(self, instance, owner=None): Called when the attribute of the owner class or its instance is accessed. Eg. print(e.sender).
- __set__(self, instance, value): Called to set the attribute on an instance of the owner class. Eg. e.sender = 'foo'.
- __delete__(self, instance): Called to delete the attribute on an instance of the owner class. Eg. del(e.sender).
- __set_name__(self, owner, name): Called when the owner class is created. This allows descriptor instances to access their own name as defined in the owner class.
An object that has defined one or more of __get__, __set__ or __delete__ is a descriptor. Descriptors work only as class variables of the owner class.
The statement e.sender += 'bar' will automatically call both __get__ and __set__.
What's the difference between data and non-data descriptors?
A descriptor that defines __set__ and/or __delete is a data descriptor. A descriptor that defines only __get__ is a non-data descriptor.
Data descriptors override redefinitions in instance dictionary whereas instances can override non-data descriptors.
For a descriptor that doesn't define __get__, attribute access will return the value from the instance dictionary. If this doesn't exist, the descriptor object is returned.
Non-data descriptors are typically used for methods. In fact, Python methods are implemented as non-data descriptors, including those decorated with @staticmethod and @classmethod. Instances can override these methods. On the other hand, property() function is implemented as a data descriptor, meaning that instances can't override its behaviour.
Why do we need the descriptor protocol when we have __getattr__, __setattr__ and __delattr__?
The descriptor method __set__ is better than __setattr__. Source: Adapted from Egan 2018, 4:50, 16:17.
Methods __getattr__, __setattr__ and __delattr__ allow us to customize the behaviour of attribute access. These methods are defined on the owner class. If the owner class has many attributes to manage, these methods could become complex to maintain. Method implementations may have ugly if-else constructs to manage the different attributes.
Descriptors offer a more modular, maintainable and extensible approach. Instead of the owner class controlling how attributes should be managed, the control is now with the class of the attribute being managed. In other words, descriptors have a say on how their instances should be managed.
In the figure, we see how any value assigned to a name attribute is capitalized. Method __setattr__ of Person class can be used for this but requires the attribute be named "name". The descriptor protocol offers a better way. Class CapitalizedValue is a data descriptor that does the capitalization. Person class defines an attribute of name "name" but any other name could have been given.
How do Python descriptors work under the hood?
Python internals of attribute get/set/delete. Source: Adapted from Cohen 2017b.
How Python descriptors work under the hood is basically about how the language does attribute access. Python applies a bunch of rules in a well-defined order of precedence. This is shown in the figure for both get and set/delete operations. These rules check whether class overrides the access methods __getattribute__ and __getattr__, whether class or instance access is performed, and whether attribute is a descriptor object and what parts of the descriptor protocol are implemented.
We can see that for a get operation, data descriptor has higher precedence than instance attribute, which in turn has higher precedence than non-data descriptor.
We can summarize the lookups for instance and class attribute accesses:
- Get: obj.x ⇒ Cls.__dict__['x'].__get__(obj, Cls)and Cls.x ⇒ Cls.__dict__['x'].__get__(None, Cls)
- Set: obj.x = v ⇒ Cls.__dict__['x'].__set__(obj, val) and Cls.x = v ⇒ Regular override
- Delete: del obj.x ⇒ Cls.__dict__['x'].__delete__(obj) and del Cls.x ⇒ Regular deletion
What other features of Python are implemented by descriptors?
Built-in functions classmethod(), staticmethod(), property(), and functools.cached_property() are all implemented as descriptors.
A property attribute is one that triggers method calls when accessed. Property is implemented as a data descriptor. It has an easier interface than a descriptor and a different abstraction. The methods are typically defined in the same class in which the attribute resides. There are two ways to link methods to a property: (i) using the property(fget=None, fset=None, fdel=None, doc=None) built-in function; (ii) using decorators @property, @x.setter and @x.deleter where x is the name of the property and also the name of the methods so decorated.
Another Python feature is slots, available as a class variable __slots__. This variable can be assigned all allowed attributes of the class and its instances. Thus, instances can't add new attributes. Instances also don't contain __dict__ and __weakref__ unless explicitly included in __slots__. Every item in __slots__ is implemented as a descriptor.
Could you share some tips for working with Python descriptors?
For most use cases, or at least for beginners, property is a simpler interface than defining a descriptor from scratch. In fact, property has been called a "high-level descriptor builder" or a "descriptor factory".
A read-only data descriptor can be constructed by defining both __get__ and __set__ but the latter is coded to raise an AttributeError exception.
We can use a descriptor to implement a one-time initialization. This is done by setting an internal flag in __set__.
Descriptors enable many other use cases elegantly. Logging all accesses of an attribute can be implemented by a descriptor. When an attribute changes, we can update all dependent attributes, such as, updating area when circle radius changes. Lazy evaluation is another use case in which behaviour is invoked only when necessary.

Milestones

Jan
1994

Python 1.0 is released. Descriptors don't exist in this initial release.

Dec
2001

Python 2.2 is released. This introduces descriptors into the language via PEP 252. Descriptors themselves have the following attributes: __name__, __doc__, __get__(), __set__() and __delete__(). Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors.

Jun
2011

Python 2.7.2 is released. In the accompanying documentation, Hettinger states in the Descriptor HowTo Guide that,

Learning about descriptors not only provides access to a larger toolset, it creates a deeper understanding of how Python works and an appreciation for the elegance of its design.

Feb
2015

In PEP 487, Teichmann proposes the use of __set_name__() initializer for descriptors. While a descriptor knows its owner when __get__() is called, it doesn't know its own name, that is, the instance name that the owner has defined. Method __set_name__() solves this problem. This feature is accepted for Python 3.6 (December 2016).

Sep
2017

In PEP 549, Hastings proposes instance descriptors, since currently Python supports descriptors only as members of the type of an object. This proposal is rejected in preference to a simpler proposal by Levkivskyi in PEP 562. The idea is to support __getattr__() and __dir__() at module level. These allow customization of module attribute access. Two use cases for this are deprecation warnings and lazy loading. This feature is accepted for Python 3.7 (June 2018).

Aug
2018

Use of the descriptor protocol in public repositories at GitHub. Source: Egan 2018, 8:08.

At PyCon Australia, Egan reveals his analysis of the use of the descriptor protocol in public GitHub repositories. Method __get__() is implemented 1.2 million times. Method _set_name__() is implemented the least since it's been around only since Python 3.6.

Sample Code

# Source: https://docs.python.org/3/howto/descriptor.html
# Accessed: 2021-12-25
# An example of managed access in which obj._age is not exposed publicly
 
import logging
 
logging.basicConfig(level=logging.INFO)
 
class LoggedAgeAccess:
 
    def __get__(self, obj, objtype=None):
        value = obj._age
        logging.info('Accessing %r giving %r', 'age', value)
        return value
 
    def __set__(self, obj, value):
        logging.info('Updating %r to %r', 'age', value)
        obj._age = value
 
class Person:
 
    age = LoggedAgeAccess()             # Descriptor instance
 
    def __init__(self, name, age):
        self.name = name                # Regular instance attribute
        self.age = age                  # Calls __set__()
 
    def birthday(self):
        self.age += 1                   # Calls both __get__() and __set__()
 
>>> mary = Person('Mary M', 30)         # The initial age update is logged
INFO:root:Updating 'age' to 30
>>> dave = Person('David D', 40)
INFO:root:Updating 'age' to 40
 
>>> vars(mary)                          # The actual data is in a private attribute
{'name': 'Mary M', '_age': 30}
>>> vars(dave)
{'name': 'David D', '_age': 40}
 
>>> mary.age                            # Access the data and log the lookup
INFO:root:Accessing 'age' giving 30
30
>>> mary.birthday()                     # Updates are logged as well
INFO:root:Accessing 'age' giving 30
INFO:root:Updating 'age' to 31
 
>>> dave.name                           # Regular attribute lookup isn't logged
'David D'
>>> dave.age                            # Only the managed attribute is logged
INFO:root:Accessing 'age' giving 40
40

# Source: https://docs.python.org/3/whatsnew/3.6.html
# Accessed: 2021-12-21
# Example showing how to use __set_name__()
# that allows access to the instance's name
 
class IntField:
    def __get__(self, instance, owner):
        return instance.__dict__[self.name]
 
    def __set__(self, instance, value):
        if not isinstance(value, int):
            raise ValueError(f'expecting integer in {self.name}')
        instance.__dict__[self.name] = value
 
    # this is the new initializer:
    def __set_name__(self, owner, name):
        self.name = name
 
class Model:
    int_field = IntField()
 
try:
    m = Model()
    m.int_field = 'hello'
except ValueError as e:
    print(e) # print would include the name "int_field"

# Source: https://www.python.org/dev/peps/pep-0562/
# Accessed: 2021-12-21
# Example showing how to show deprecation warnings
 
# lib.py
 
from warnings import warn
 
deprecated_names = ["old_function", ...]
 
def _deprecated_old_function(arg, other):
    ...
 
def __getattr__(name):
    if name in deprecated_names:
        warn(f"{name} is deprecated", DeprecationWarning)
        return globals()[f"_deprecated_{name}"]
    raise AttributeError(f"module {__name__} has no attribute {name}")
 
# main.py
 
from lib import old_function  # Works, but emits the warning

References

Article Stats

1532

Words

Authors

Edits

Chats

Likes

2789

Hits

Cite As

Devopedia. 2022. "Python Descriptor." Version 5, February 15. Accessed 2023-11-13. https://devopedia.org/python-descriptor

Contributed by
2 authors

Last updated on
2022-02-15 11:57:03

languages python object-oriented

Python Descriptor

Discussion

Milestones

Sample Code

References

Further Reading

Article Stats

Author-wise Stats for Article Edits

Cite As

See Also

Login