Reflection and decorators

Problem

The field that str forgot

The order's __str__ is written by hand. Later, someone adds a field for the discount:

class Order:
    def __init__(self):
        self.number = 1042
        self.customer = "Maria Petrova"
        self.total = 86.40
        self.discount = 10  # added later

    def __str__(self):
        return f"Order[number={self.number}, customer={self.customer}, total={self.total}]"


print(Order())

Order[number=1042, customer=Maria Petrova, total=86.4]

The discount isn't in the output, because nobody remembered to update __str__. With ten classes, this happens all the time.

There's another question from the start of the module, too. When you write a test_… method, unittest finds it and runs it, even though it has never seen your class. How? Both things work through the same idea: a program can look at its own classes while it runs.

The idea

The class as an object

In Python, everything is an object, classes too: Order is an object of type type that describes the class: name, parents, methods. Looking at classes and objects while the program runs is called reflection (the program sees itself as in a mirror). There are three ways to get the class:

Order, when you know the class while you write the code: the name of the class is the object itself;
type(obj), when you have an object;
getattr(importlib.import_module("collections"), "Counter"), when you only have the names as text.

import collections.abc
import importlib

module = importlib.import_module("collections")
cls = getattr(module, "Counter")
print("Name:", cls.__module__ + "." + cls.__qualname__)
print("Short name:", cls.__name__)
print("Parent:", cls.__base__.__name__)
for parent in cls.__mro__:
    print("In the chain:", parent.__name__)
print("Is it a Mapping:", issubclass(cls, collections.abc.Mapping))

Name: collections.Counter
Short name: Counter
Parent: dict
In the chain: Counter
In the chain: dict
In the chain: object
Is it a Mapping: True

__mro__ is the chain of classes it inherits from, in the order in which Python looks for methods. Python has no interfaces. Instead, there are abstract classes like collections.abc.Mapping, and issubclass checks whether the class is one of them. If there's no module or class with that name, import_module raises ModuleNotFoundError, and getattr raises AttributeError.

See it

The fields of any object

vars(obj) gives the fields of the object as a dictionary: name to value. The type of each field is the type of its value, type(value): in Python, fields have no declared type.

def dump(obj):
    cls = type(obj)
    print(f"{cls.__name__}:")
    for name, value in vars(obj).items():
        print(f"  {name} ({type(value).__name__}) = {value}")

class Room:
    def __init__(self):
        self.number = 214
        self.sea_view = True


dump(Order())
dump(Room())

Order:
  number (int) = 1042
  customer (str) = Maria Petrova
  total (float) = 86.4
  discount (int) = 10
Room:
  number (int) = 214
  sea_view (bool) = True

One function prints any object, and the new discount field is already there. Python has no private fields: a leading _ or __ is only a convention, and vars sees everything. That's why you don't need anything like setAccessible.

def dump(obj):
    cls = type(obj)
    print(f"{cls.__name__}:")
    for name, value in vars(obj).items():
        print(f"  {name} ({type(value).__name__}) = {value}")

Step by step, in words

dump gets an Order object and knows nothing about its class.
type(obj) gives the Order class. It's an object too.
The first line is the short name of the class.
vars(obj) gives a dictionary with 4 fields: name and value.
Python has no private fields: number comes straight with its value, 1042. The type belongs to the value: int.
customer (str) = Maria Petrova.
total (float) = 86.4.
discount: the field that __str__ forgot shows up by itself, because reflection gets all the fields.
The function knows nothing about Order. For Room it prints number and sea_view the same way.

More precisely, if you are curious

vars(obj) gives only the fields of the object, without the methods and without the attributes of the class. dir(obj) gives all names: fields, methods, inherited ones and the special ones with __.
The order is the order in which the fields got a value. Dictionaries in Python keep their order, and that's guaranteed.
A field with two underscores, self.__price, gets renamed by Python to _Product__price. It's still visible in vars, just under another name.
A class with __slots__ has no dictionary for its fields, and vars raises TypeError.

Try it

Whose class?

What will the program print?

class Document:
    pass


class Invoice(Document):
    pass


doc = Invoice()
print(type(doc).__name__)
print(type(doc).__base__.__name__)
print(Document.__base__.__name__)

Answer

Document
object
object

Invoice
Document
None

Invoice
Document
object

type(doc) gives the real class of the object: Invoice. Its parent (__base__) is Document. Document doesn't list a parent in parentheses, so its parent is object, the common parent from the inheritance module. For object itself, __base__ is None: there's nothing above it.

See it

Object, method and private field by name

Through reflection you can create an object, call a method and change a field, knowing only their names:

class Product:
    def __init__(self, name, price):
        self.name = name
        self._price = price

    def apply_discount(self, percent):
        self._price = self._price * (100 - percent) / 100

    def _label(self):
        return f"{self.name}: €{self._price}"

cls = globals()["Product"]
coffee = cls("Coffee", 10.0)

discount = getattr(coffee, "apply_discount")
discount(20)

print("Price after discount:", getattr(coffee, "_price"))
setattr(coffee, "_price", 9.5)

label = getattr(coffee, "_label")
print(label())

Price after discount: 8.0
Coffee: €9.5

globals()["Product"] finds the class by its name in the module. You call the class like a function, and that's the constructor.
getattr(coffee, "apply_discount") gives the method, already bound to coffee. All that's left is to call it with the arguments.
getattr and setattr read and write any field, _price too. _label can be called too: "private" in Python only means "don't call it from outside".

You don't pass the types of the parameters: in Python, methods are found by name only.

More precisely, if you are curious

A list, created and filled without list() or .append appearing in the code:

import builtins

items = getattr(builtins, "list")()
add = getattr(items, "append")
add("Coffee")
add("Tea")
print(items)

['Coffee', 'Tea']

Built-in classes like list are in the builtins module.

Try it

The price that isn't there

The program starts, but it stops. Which line has the bug?

Pick the line with the bug.

The field is called _price, but here it says "_prise". This is ordinary text, and nobody checks it before the program gets to it. The error only shows up at run time:

AttributeError: 'Product' object has no attribute '_prise'. Did you mean: '_price'?

Python 3.12 adds the "Did you mean" hint when it prints the error.

This is the main price of reflection:

nobody checks in advance the names and types, and the editor doesn't find them when you rename something;
the code is slower than an ordinary call and harder to read;
getattr and setattr get around encapsulation: you can break a rule that the class protects.

The benefit is that you can write code that works with any class without knowing it in advance. That's why reflection is mostly for libraries: unittest and pytest, dataclasses, SQLAlchemy (it reads the fields of the class and fills in the objects), Django. You rarely write it in ordinary code.

The idea

Field labels

In Java, labels with @ on fields and methods are called annotations. In Python, "annotation" means something else: the type after the colon, guest: str. Python has no @ labels on fields. For a dataclass field, a label is the metadata of field. The label function makes such a field, and print_labeled prints only the fields with a label:

def label(text, default):
    return field(default=default, metadata={"label": text})

@dataclass
class Booking:
    number: int = label("Number", 501)
    guest: str = label("Guest", "Ivan Georgiev")
    nights: int = label("Nights", 3)
    internal_note: str = "VIP"

def print_labeled(obj):
    for f in fields(obj):
        text = f.metadata.get("label")
        if text is not None:
            print(f"{text}: {getattr(obj, f.name)}")

Number: 501
Guest: Ivan Georgiev
Nights: 3

field(default=…, metadata={…}) makes a dataclass field with a default value and a dictionary with whatever you want.
fields(obj) gives the fields of the dataclass as Field objects: each one has name, type and metadata.
f.metadata.get("label") returns the label, or None if the field doesn't have one. internal_note has no label and isn't printed.

More precisely, if you are curious

Python's annotations (the types of the fields) are read with reflection too: inspect.get_annotations(Booking) gives {'number': <class 'int'>, 'guest': <class 'str'>, …}. That's how dataclass knows which fields there are.
Labels on methods and classes in Python are the decorators with @. You'll see them in the next card.

See it

How unittest finds the tests

Here's a tiny test runner. The marker is a decorator: @check above the method calls the check function with the method, and the function gives it an is_check attribute and returns it. The test class checks the discount from the first lesson of the module (with the old bug price > 100):

def check(method):
    method.is_check = True
    return method


class DiscountChecks:
    @check
    def ten_percent_off(self):
        self.expect(45.0, apply_discount(50, 10))

    @check
    def extra_five_percent_from_100(self):
        self.expect(85.0, apply_discount(100, 10))

    def expect(self, expected, actual):
        if expected != actual:
            raise AssertionError(f"expected {expected}, but was {actual}")

def run_checks(cls):
    passed = 0
    failed = 0
    for name in sorted(dir(cls)):
        if not getattr(getattr(cls, name), "is_check", False):
            continue
        instance = cls()
        try:
            getattr(instance, name)()
            print(f"PASS  {name}")
            passed += 1
        except Exception as e:
            print(f"FAIL  {name}: {e}")
            failed += 1
    print(f"Checks: {passed + failed}, passed: {passed}, failed: {failed}")

FAIL  extra_five_percent_from_100: expected 85.0, but was 90.0
PASS  ten_percent_off
Checks: 2, passed: 1, failed: 1

unittest does the same, only with many more details:

it gets the names in the class with dir(cls) and sorts them, so the order is always the same;
it skips the ones without the marker. unittest doesn't use a marker but the name: it runs the methods whose names start with test;
it makes a new object for each test, so the tests don't get in each other's way;
it gets the method from the object with getattr and calls it. If the test raises, the exception comes straight through, without a wrapper as in Java.

assertEqual in unittest is also a method that raises AssertionError when the values differ. Exactly like expect here.

Try it

The marker that disappeared

check has no return method at the end, and everything else is the same:

What will run_checks print?

def check(method):
    method.is_check = True


class DiscountChecks:
    @check
    def ten_percent_off(self): ...

    @check
    def extra_five_percent_from_100(self): ...


run_checks(DiscountChecks)

Answer

The same as before: one passes, one fails

Checks: 0, passed: 0, failed: 0

The program stops with an error

@check above def is the same as ten_percent_off = check(ten_percent_off). Without return, check returns None, and None takes the place of the method. getattr(None, "is_check", False) is False, and not one check runs. The program runs, it just doesn't do anything. That's why every decorator returns the function it got (or a new one in its place).

Exercise

Code that works with any class

Solve the tasks in your editor.

1. All the fields, class by class. Write print_fields(obj) for a dataclass, which prints every field as name = value, including the inherited ones, going class by class. Start from type(obj).__mro__ from the back, and for each class take only its own fields with inspect.get_annotations(cls); the parent's fields come first.

@dataclass
class Vehicle:
    plate: str = "CA 1234 BK"
    year: int = 2019


@dataclass
class Truck(Vehicle):
    load_tons: float = 12.5

plate = CA 1234 BK
year = 2019
load_tons = 12.5

2. Mini JSON. Write to_json(obj), which returns the fields as JSON: the names in quotes, text in quotes, numbers and true/false without quotes. Use vars, isinstance(value, str) and isinstance(value, bool): str(False) is False, but JSON needs false. (The ready-made version is json.dumps(vars(obj)).)

class Ticket:
    def __init__(self):
        self.seat = 14
        self.event = "Concert"
        self.price = 45.0
        self.vip = False

{"seat": 14, "event": "Concert", "price": 45.0, "vip": false}
{"number": 214, "sea_view": true}

3. Your own test runner. Make the decorators my_test and my_before_each (like check) and a runner run(cls). For each method with my_test: a new object, then the method with my_before_each (if there is one), then the test. If the test raises, print the kind (type(e).__name__) and the message of the exception. Try it with these tests:

def my_test(method):
    method.is_test = True
    return method


def my_before_each(method):
    method.is_before_each = True
    return method


class Account:
    def __init__(self, balance):
        self.balance = balance

    def withdraw(self, amount):
        if amount > self.balance:
            raise ValueError(f"Short by €{amount - self.balance}")
        self.balance -= amount


class AccountChecks:
    @my_before_each
    def fresh(self):
        self.account = Account(100)

    @my_test
    def withdraw_40(self):
        self.account.withdraw(40)
        if self.account.balance != 60:
            raise AssertionError(f"expected 60, but was {self.account.balance}")

    @my_test
    def withdraw_70(self):
        self.account.withdraw(70)
        if self.account.balance != 30:
            raise AssertionError(f"expected 30, but was {self.account.balance}")

    @my_test
    def withdraw_too_much(self):
        self.account.withdraw(130)

PASS  withdraw_40
PASS  withdraw_70
FAIL  withdraw_too_much: ValueError: Short by €30
Tests: 3, passed: 2, failed: 1

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·

Reflection and decorators

Cookies