Magic Methods in Python

A selection of Python magic methods with examples. Source: Adapted from Kettler 2011.
A selection of Python magic methods with examples. Source: Adapted from Kettler 2011.

Magic methods are special methods that are called implicitly by the Python interpreter to execute corresponding operations. They help customize common operators or constructs. They make code more intuitive and Pythonic.

For example, to add two Python objects we might write a + b. The actual execution would call the magic method a.__add__() implicitly to perform this addition. If this magic method doesn't exist, the Python interpreter would call b.__radd__(). Application code where addition is invoked is itself simple and readable.

There are several conceptual groups of magic methods such as object lifecycle, object representation, object comparison, object attributes, class creation, callability, containers, numerics, context management, and more. This article describes many of these and shares how to use them.

Discussion

  • What are the characteristics of magic methods?
    Some characteristics of magic methods. Source: Devopedia 2022.
    Some characteristics of magic methods. Source: Devopedia 2022.

    Magic methods are simply special methods. They have system-defined names starting and ending with double underscores, informally known as dunder names. Not all of these are magic methods.

    Several built-in classes define various magic methods. Output of dir() can be filtered to see only dunder names but not all are magic methods. Further, callable() can be used to introspect these names, but not all callables are magic methods. Ultimately, Python documentation clarifies which attributes are magic methods. For example, as illustrated in the figure, __doc__ isn't one and __class__ despite being callable isn't one.

    The code above shows implicit calls to __str__() and __repr__() magic methods by str() and repr() respectively. These return suitable string representations of the Point2d instances. Magic methods thus serve these purposes:

    • Enable classes to define their own behaviour corresponding to Python operators or constructs.
    • Enable developers to write Pythonic code. A developer could have written to_string() method but magic methods provide a uniform convention. This reduces technical debt.
    • Enable code reuse and design patterns. For example, custom objects can be compared and sorted based on magic methods without extra effort.
  • Could you give an overview of the main magic methods?
    A near-complete list of magic methods in Python 3.10.1. Source: Adapted from Python Docs 2022a.
    A near-complete list of magic methods in Python 3.10.1. Source: Adapted from Python Docs 2022a.

    Python 3.10.1 has dozens of magic methods. We discuss them within a few conceptual groups:

    • Object Lifecycle: These relate to creation, initialization and destruction of objects. Eg. __new__(), __init__(), __del__().
    • String Representation: These represent objects as strings. Eg. __repr__(), __str__().
    • Comparisons: To compare objects via operators <, ==, >=, etc. Eg. __lt__(), __eq__(), __ge__().
    • Math Operations: Arithmetic operators +, *; bit operators &, <<; and math functions. Eg. __add__(), __rshift__(), __floor__().
    • Type Conversions: Convert from one type to another. Eg. __bool__(), __int__().
    • Type Check: Check class hierarchy. Eg. __instancecheck__(), __subclasscheck__().
    • Context Manager: To set up and clean up objects within a context, such as open/close of files. Eg. __enter__(), __exit__().
    • Attribute Access: Control access or inspect instance attributes. Eg. __getattr__() , __dir__() , __set__() .
    • Operations on Sequences: Find length, get/set/delete items or iterate through items of a sequence. Eg. __len__(), __getitem__(), __contains__().
    • Asynchronous: Pertaining to asynchronous operations. Eg. __await__(), __aiter__(), __aenter__().
    • Miscellaneous: Not covered above. Eg. __call__(), __mro_entries__().

    Some Python modules add their own magic methods. Examples include copy.__deepcopy__() and pickle.__reduce__().

  • Are magic methods Python's approach to operator overloading?

    Indeed, magic methods are Python's approach to operator overloading.

    However, there are many magic methods that don't correspond to an operator. For example, __str__(), __bool__() and __repr__() are magic methods corresponding to functions rather than operators.

    Further, for example, the object class implements __eq__(), which corresponds to the == operator. When we implement this magic method we're technically overriding a base class method in OOP parlance. However, we're actually overloading the == operator.

    This is why it's better to think of magic methods as intuitive interfaces.
  • What magic methods differ between Python 2 and Python 3?

    For object comparisons, Python 2's magic method __cmp__() is replaced with rich comparison methods in Python 3: __lt__(), __le__(), __eq__(), __ne__(), __gt__(), __ge__(). Because it can be extra work to implement all these magic methods for a class, Python 3 provides the decorator @functools.total_ordering. We can provide implementations for __eq__() and one of the ordering operators. The decorator will supply the rest.

    Python 2's magic method __nonzero__() is replaced with __bool__(). Method __div__() is replaced with __truediv__(). Method __unicode__() is removed since strings in Python 3 are Unicode by default.

    Python 2's __coerce__ is no longer needed since other magic methods do this in Python 3. For example, 2 + 3.5 would do type coercion in Python 2 but in Python 3, __radd__() on the float object is called to add the integer.

  • Which magic methods are available for object lifecycle constructs?

    __new__(), __init__() and __del__() are magic methods that allow customization of object creation, initialization and destruction respectively.

    __new__() is mainly intended to allow subclasses of immutable types to customize instance creation and to customize class creation in custom metaclasses. __init__() is mainly intended for object customization before returning to the caller; that is, __new__() deals with creation and is a static class method while __init__() deals with initialization and is an instance method. __del__() is for runtime clean up just before an instance is destroyed, which only happens if the corresponding reference count becomes zero.

    There are magic methods to customize class creation itself. Classes themselves are objects: a class is created from its metaclass. Classes themselves use memory, can be assigned, copied and passed as parameters, and even dynamically created. Magic method __init_subclass__() influences the behaviour of future subclasses as this method is called whenever the class is subclassed. Related magic methods include __set_name__(), __prepare__(), __instancecheck__(), __subclasscheck__(), and __class_getitem__().

  • Which magic methods are available for object representation?

    Objects of type str are inherently strings but any Python object can be given a string representation. Built-in functions str(), repr(), bytes(), format(), and print() serve this purpose. However, their default behaviours can be customized by defining suitable magic methods.

    Built-in function repr() calls magic method __repr__() to obtain the official string representation. Built-in functions str() and print() call magic method __str__() to obtain an informal and printable string. Since __repr__() is typically used for debugging, it should return an information-rich and unambiguous string. If possible, it should be a valid Python expression string for data serialization so that the instance can be recreated from the string.

    Built-in function bytes() calls magic method __bytes__() to obtain a byte-string representation, which is primarily useful for storage and network transfers.

    Built-in function format() and string method str.format() call magic method __format__() to evaluate formatted string literals, also known as format-strings. If this magic method is not defined, __str__() is called instead.

  • Which magic methods are available for object comparison?

    Comparison magic methods allow custom classes to implement specific comparison behaviours. Python defines six object comparison magic methods: __lt__(), __le__(), __eq__(), __ne__(), __gt__() and __ge__(), corresponding to <, <=, ==, !=, >, and >= respectively.

    The magic methods should return the singleton NotImplemented if the comparison is not implemented for the requested operands. They should return a boolean value. Otherwise, Python will call bool() on the returned object.

    If __ne__() is undefined, its default behaviour is to call __eq__() and invert the result. No such relation exists among other comparison operators. For example, Python doesn't enforce that x < y is same as y > x. However, implementations should maintain semantic consistency.
  • How can I customize object conversions?

    Any Python object can be used in boolean context, such as if obj: pass. This operation is trivial for a bool type object. For other types, we need a way to evaluate the object in a boolean context. Magic method __bool__() does this.

    Evaluating an object in a boolean context or directly calling the built-in function bool() calls __bool__(), which must return either True or False. Any other return value will cause Python to raise an exception. If __bool__() is undefined, bool() calls __len__(). The result is considered True if nonzero and False otherwise. If a class doesn't define either of these, all its instances are considered True.

    Likewise, built-in functions complex(), int() and float() trigger other magic methods. A call of int() checks for __int__(), __index__() and __trunc__() in that order; float() checks for __float__() and __index__(); complex() checks for __complex__(), __float__() and __index__().

    When present, __index__() indicates that the object is of integer type. It must return an integer. Python slicing and functions operator.index(), bin(), hex() and oct() rely on this.

  • Which magic methods emulate numeric objects?
    Magic methods specific to numeric operations. Source: Souza 2020.
    Magic methods specific to numeric operations. Source: Souza 2020.

    Numbers can be added, multiplied or bit-shifted. What if we wish to emulate these behaviours on non-numeric objects? For example, we could add two RGB colour objects or left shift beads on an abacus object. Magic methods associated with numeric operators enable this.

    There are three categories of magic methods (we describe only the addition operator):

    • Left Operand: Given a + b, the call a.__add__(b) is made.
    • Right Operand: Given a + b, the call a.__add__(b) is attempted first. If this doesn't exist or returns NotImplemented, b.__radd__(a) is called. For example, 3 + 5.5 will call 3.__add__(5.5). Since an integer doesn't know how to add a float, this returns NotImplemented. Then 5.5.__radd__(3) is called and this succeeds with the addition.
    • In-Place Operation: Given a += b, the call a.__iadd__(b) is made. This method should modify the object in-place. If this method doesn't exist, a.__add__(b) and b.__radd__(a) are tried in that order.

    Related to these are unary operators + and - that call __pos__() and __neg__(); built-in function abs() calls __abs__(); similar built-in functions (some in math module) call __round__(), __trunc__(), __floor__(), and __ceil__().

  • Which magic method makes class instances callable?

    Given a class A and its initializer __init__(self, i), we can instantiate an object by doing a = A(2). Now a is an instance. But it can't be called like a method or function. However, if the class A includes the magic method __call__(), we can write a(). Thus, the instance becomes callable (with or without arguments) when the magic method __call__() is defined.

    Class instantiation itself is just calling the corresponding class object. With __call__(), we're enabling a similar syntax on instances.

    Callable instances have practical use cases. With __init__() an object can be initialized in a fundamental way. Suppose it has only one specific behaviour. Hence it defines no methods other than __call__(). Different types of instances are then created and called. An example is initializing a Hasher object with an algorithm: md5 = Hasher(hashlib.md5) and sha1 = Hasher(hashlib.sha1). Actual hashing happens later by calling these instances: md5('somefile.txt') or sha1('somefile.txt').

    Implementing decorators or presenting a consistent API are other use cases of __call__().

  • Which magic methods are available for customization of attribute access?

    __getattr__(), __getattribute__(), __setattr__() and __delattr__() allow us to customize attribute access for class instances. This includes use of, assignment to or deletion of a.x, where x is an attribute of instance a.

    Built-in function dir() calls magic method __dir__() to customize the list of valid instance attributes. Moreover, __getattr__() and __dir__() can be used to customize access to module attributes.

    A feature of the language called descriptor introduces further magic methods: __get__(), __set__() and __delete__(). These are more powerful and present a more modular way to control attribute access.

  • How can I make a custom object hashable?
    An example implementation of custom object hashing. Source: Millikin 2010.
    An example implementation of custom object hashing. Source: Millikin 2010.

    Certain data structures in Python such as sets, frozensets and dictionaries are implemented using hash tables for efficient lookups and insertions. This requires computing a hash value on the object by calling the built-in function hash(). Internally, this calls the magic method __hash__().

    By defining this magic method on a class, we can compute the hash value for objects of that class. It must return an integer. Objects that compare equal via the method __eq__() must have the same hash value. One suggested way to compute a proper hash is to pack comparison-sensitive components of an object into a tuple and hash that tuple. For example, return hash((self.x, self.y)) as the body of __hash__() method on a class Point2d. Note that __hash__() in general is tightly coupled with __eq__() and has atypical behaviour related to __hash__() implementations of str and bytes types.

Milestones

Sep
2000

Python 1.6 is released. Magic method __contains__() is introduced to allow overriding the in operator.

Apr
2001

Python 2.1 is released. Magic methods __lt__(), __le__(), __eq__(), __ne__(), __gt__() and __ge__() are introduced to allow overriding the corresponding comparison operators.

Dec
2001

Python 2.2 is released. This introduces descriptors into the language via PEP 252. Descriptors themselves have the following attributes: __name__, __doc__, __get__(), __set__() and __delete__(). Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors.

Sep
2006

Python 2.5 is released. This introduces the with statement and its associated magic methods __enter__() and __exit__(). A common use case of this is to open a file in __enter__() when the with context is entered and close the file in __exit__() as soon as execution comes out of the with context. Another new method is __index__() that solves a problem faced by NumPy users. It's now possible to use NumPy types as slice indexes.

Oct
2008

Python 2.6 is released. Some features of Python 3.0 are backported to Python 2.6. One of these is the __complex__() magic method. Magic method __dir__() is introduced. Built-in function dir() now looks for it on the object it receives.

Dec
2008

Python 3.0 is released. Magic method __cmp__() is removed. Instead use __lt__() for sorting, __eq__() with __hash__() and other rich comparisons. Operator != returns the opposite of ==. Methods __getslice__(), __setslice__() and __delslice__() are removed. The slice a[i:j] is implemented as a.__getitem__(slice(i, j)). Methods __setitem__() and __delitem__() are introduced. Converters oct() and hex() now use __index__(). Method __nonzero__() is now __bool__().

Sep
2015

Python 3.5 is released. Magic methods for asynchronous programming are introduced: __await__(), __aiter__(), __anext__(), __aenter__(), and __aexit__().

Jun
2018
An example use of @dataclass decorator. Source: Adapted from Python Docs 2022g.
An example use of @dataclass decorator. Source: Adapted from Python Docs 2022g.

Python 3.7 is released. Module-level methods introduced are __getattr__() and __dir__(). These can be used to implement module attribute deprecation and lazy loading. New methods __class_getitem__() and __mro_entries__() bring 7x improvement to performance in relation to types and the typing module. New module dataclasses has decorators and functions to automatically add some magic methods to user-defined classes.

Oct
2019

Python 3.8 is released. Constructors of int, float and complex will now use __index__() if their corresponding magic methods are not available.

Oct
2021

Python 3.10 is released. If __ipow__() returns NotImplemented, __pow__() and __rpow__() are the fallbacks.

Sample Code

  • # Source: Devopedia 2022
# Showing the use of __str__() and __format__()
 
def print_compare(obj):
  print('str', str(obj))
  print('format', format(obj))
  print('-', obj)
 
class Point2d:
    def __init__(self, x, y):
        self.x, self.y = x, y
 
print("\nPoint2d without Magic ...")
print_compare(Point2d(2, 3))
# Point2d without Magic ...
# str <__main__.Point2d object at 0x7f0411a38320>
# format <__main__.Point2d object at 0x7f0411a38320>
# - <__main__.Point2d object at 0x7f0411a38320>
 
class Point2dMagic(Point2d):
    def __str__(self):
        return "str magic = ({0}, {1})".format(self.x, self.y)
    def __format__(self, format_spec):
        return "format magic = ({0}, {1})".format(self.x, self.y)
 
print("\nPoint2d with Magic ...")
print_compare(Point2dMagic(2, 3))
# Point2d with Magic ...
# str str magic = (2, 3)
# format format magic = (2, 3)
# - str magic = (2, 3)
 
print("\nPoint2d with Magic but __format__ removed ...")
del(Point2dMagic.__format__)
print_compare(Point2dMagic(2, 3))
# Point2d with Magic but __format__ removed ...
# str str magic = (2, 3)
# format str magic = (2, 3)
# - str magic = (2, 3)
 

References

  1. Fedora Python SIG. 2018. "Comparing and Sorting." In: The Conservative Python 3 Porting Guide, Fedora Python SIG, January 1. Accessed 2022-01-08.
  2. Hettinger, Raymond. 2019. "What’s New in Python 3.8." Release Notes, The Python Software Foundation, October 14. Accessed 2022-01-08.
  3. Hunner, Trey. 2022. "Python Doesn't Have Type Coercion." Topic series: Overlooked Fundamentals, Python Morsels. Accessed 2022-01-08.
  4. Kettler, Rafe. 2011. "A Guide to Python's Magic Methods." Version 1.17. Updated 2016-01-19. Accessed 2022-01-08.
  5. Kuchling, A.M. 2002. "What’s New in Python 2.2." Release Notes, v2.2.2, The Python Software Foundation, October 14. Updated 2021-12-21. Accessed 2021-12-21.
  6. Kuchling, A.M. 2006. "What’s New in Python 2.5." Release Notes, The Python Software Foundation, September 19. Accessed 2022-01-08.
  7. Kuchling, A.M. 2008. "What’s New in Python 2.6." Release Notes, The Python Software Foundation, October 1. Accessed 2022-01-08.
  8. Louis, Pila. 2020. "Constructors in Python (__init vs __new__)." Dev.to, September 16. Accessed 2022-01-08.
  9. Millikin, John. 2010. "What's a correct and good way to implement __hash__()?" StackOverflow, May 25. Updated 2019-07-07. Accessed 2022-01-13.
  10. Pranskevichus, Elvis. 2018. "What’s New in Python 3.7." Release Notes, The Python Software Foundation, June 27. Accessed 2022-01-08.
  11. Python Docs. 2022a. "Data Model." Section 3, The Python Language Reference, v3.10.1, January 7. Accessed 2022-01-08.
  12. Python Docs. 2022b. "Lexical analysis." Section 2, The Python Language Reference, v3.10.1, January 7. Accessed 2022-01-08.
  13. Python Docs. 2022c. "Glossary." Documentation, v3.10.1, January 7. Accessed 2022-01-08.
  14. Python Docs. 2022d. "Classes." Section 9, The Python Tutorial, v3.10.1, January 7. Accessed 2022-01-08.
  15. Python Docs. 2022e. "functools — Higher-order functions and operations on callable objects." The Python Standard Library, v3.10.1, January 7. Accessed 2022-01-08.
  16. Python Docs. 2022f. "Built-in Functions." The Python Standard Library, v3.10.1, January 7. Accessed 2022-01-08.
  17. Python Docs. 2022g. "dataclasses — Data Classes." The Python Standard Library, v3.10.1, January 13. Accessed 2022-01-13.
  18. Salgado, Pablo Galindo. 2021. "What’s New in Python 3.10." Release Notes, The Python Software Foundation, October 4. Accessed 2022-01-08.
  19. Souza, Fernando. 2020. "Python Dunder Methods." Level Up Coding, GitConnected, June 29. Accessed 2022-01-08.
  20. StackOverflow. 2008. "What are metaclasses in Python?" StackOverflow, September 19. Accessed 2022-01-08.
  21. StackOverflow. 2011. "Python __call__ special method practical example." StackOverflow, April 28. Accessed 2022-01-08.
  22. StackOverflow. 2014. "Difference between python set and dict internally." StackOverflow, June 11. Updated 2019-08-20. Accessed 2022-01-08.
  23. The Python Software Foundation. 2000. "Python 1.6." Release Notes, The Python Software Foundation, September 5. Accessed 2022-01-08.
  24. The Python Software Foundation. 2021. "Python Documentation by Version." The Python Software Foundation, December 6. Accessed 2022-01-08.
  25. Wikipedia. 2021a. "Object-oriented programming." Wikipedia, December 31. Accessed 2022-01-08.
  26. Wikipedia. 2021b. "Method overriding." Wikipedia, December 7. Accessed 2022-01-08.
  27. van Rossum, Guido. 2001. "PEP 252 -- Making Types Look More Like Classes." Python.org, April 19. Updated 2021-12-21. Accessed 2021-11-21.
  28. van Rossum, Guido. 2008. "What’s New In Python 3.0." Release Notes, v3.0, The Python Software Foundation, December 4. Accessed 2022-01-08.
  29. van Rossum, Guido, and David Ascher. 2000. "PEP 207 -- Rich Comparisons." Python.org, July 25. Updated 2021-02-09. Accessed 2022-01-08.

Further Reading

  1. Python documentation for magic methods
  2. Python documentation for format()
  3. Python documentation for format specification mini-language
  4. Python documentation for formatted string literals
  5. An almost comprehensive but dated guide to Python's magic methods

Article Stats

Author-wise Stats for Article Edits

Author
No. of Edits
No. of Chats
DevCoins
User avatar anon2776
30
28
3145
Avatar of user arvindpdmn arvindpdmn
13
25
2336
DevCoins due to articles, chats, their likes and article hits are included.
2432
Words
2
Authors
43
Edits
53
Chats
3
Likes
5350
Hits

Cite As

Devopedia. 2022. "Magic Methods in Python." Version 43, January 13. Accessed 2023-11-12. https://devopedia.org/magic-methods-in-python
Contributed by
2 authors

Last updated on
2022-01-13 15:18:50
languages python object-oriented

See Also