Untrusted input and input validation in Python

Problem

Headphones for -177 €

The shop sells headphones for 59 €. The form on the page does not let you enter a quantity below 1 or above 10, and the price sits in a hidden field:

<form method="post" action="/orders">
  <input type="hidden" name="product" value="17">
  <input type="hidden" name="price" value="59">
  <input type="number" name="qty" min="1" max="10" value="1">
  <button>Buy</button>
</form>

The server takes the fields and works out the total:

def place_order(form):
    price = int(form["price"])
    qty = int(form["qty"])
    return f"Order: {qty} pcs at {price} € = {qty * price} €"


# From the page: the browser has checked the form
print(place_order({"product": "17", "price": "59", "qty": "2"}))
# The same request, sent by hand
print(place_order({"product": "17", "price": "1", "qty": "2"}))
print(place_order({"product": "17", "price": "59", "qty": "-3"}))

Order: 2 pcs at 59 € = 118 €
Order: 2 pcs at 1 € = 2 €
Order: -3 pcs at 59 € = -177 €

The first request comes from the page. The other two do not. The attacker opened the browser’s developer tools, or sent the request with curl, and wrote whatever they liked:

POST /orders HTTP/1.1
Content-Type: application/x-www-form-urlencoded

product=17&price=59&qty=-3

The payment provider gets a total of -177 € and refunds 177 € to the card. The min="1" check was not there: the request never went through the form.

The idea

The trust boundary

The browser belongs to the buyer. Everything in it, from the form to the page’s JavaScript, the attacker can change or skip. Your code starts on the server. The place where data comes from outside in is the trust boundary.

Whatever crosses it is untrusted input until you check it:

the fields of the form, hidden ones too;
the address and the parameters in it;
the request’s headers and cookies;
the name, type and size of an uploaded file;
a reply from someone else’s system, for example from the payment provider.

Your site put the hidden field and the cookie there. But the buyer’s browser sends them back, and they may have been changed on the way.

Thinking like the attacker means asking about every field: what if it is negative? Empty? Huge? Text instead of a number? The id of someone else’s order? 10 MB instead of 10 characters?

See it

What the server lets in

The fixed handler trusts nothing in the request. Here is how the two orders go through it:

def place_order(form):
    product = form.get("product", "")
    qty = form.get("qty", "")
    if product not in PRICES:
        return "Refused: no such product"
    if not re.fullmatch(r"[1-9]|10", qty):
        return "Refused: the quantity is 1 to 10"
    count = int(qty)
    price = PRICES[product]  # from the catalogue
    total = count * price
    return f"Order: {count} pcs at {price} € = {total} €"

Step by step, in words

Two orders arrive. One comes from the page, where the browser checked qty ≥ 1. The other was sent by hand: price 1, quantity -3. To the server both are only text.
They cross the trust boundary. From here on nothing has been checked. place_order reads only product and qty; the price from the request is never read.
First check: is the product in the catalogue? 17 is, for both. An unknown id would stop here.
Second check: the quantity must be one of "1" … "10". "2" is on the list. "-3" is not: refused, and the order goes no further.
The price comes from the catalogue: 59 €. What the request says about the price does not matter.
The honest order: 2 × 59 € = 118 €. The forged one stopped at the boundary checks. Only what passed every check gets in.

Three rules:

Do not read what you know. The price is in the catalogue. The request says only which product and how many.
Check against an allow-list. The product must be in the catalogue. The quantity must be exactly one of the texts "1" to "10".
If something does not pass, refuse. Every path that does not reach the order is a refusal.

import re


PRICES = {"17": 59, "23": 9}


def place_order(form):
    product = form.get("product", "")
    qty = form.get("qty", "")
    if product not in PRICES:
        return "Refused: no such product"
    if not re.fullmatch(r"[1-9]|10", qty):
        return "Refused: the quantity is 1 to 10"
    count = int(qty)
    price = PRICES[product]  # from the catalogue
    total = count * price
    return f"Order: {count} pcs at {price} € = {total} €"


print(place_order({"product": "17", "price": "59", "qty": "2"}))
print(place_order({"product": "17", "price": "1", "qty": "2"}))
print(place_order({"product": "17", "price": "59", "qty": "-3"}))
print(place_order({"product": "99", "price": "1", "qty": "1"}))

Order: 2 pcs at 59 € = 118 €
Order: 2 pcs at 59 € = 118 €
Refused: the quantity is 1 to 10
Refused: no such product

re.fullmatch checks the whole text against the pattern: [1-9]|10 means one digit from 1 to 9, or exactly 10.

Try it

Where the price comes from

Finish the line where the handler decides the price:

def place_order(form):
    product = form.get("product", "")
    qty = form.get("qty", "")
    if product not in PRICES:
        return "Refused: no such product"
    if not re.fullmatch(r"[1-9]|10", qty):
        return "Refused: the quantity is 1 to 10"
    count = int(qty)
    price = 
    total = count * price
    return f"Order: {count} pcs at {price} € = {total} €"

Try it

Three requests

What does the fixed handler print for these three requests?

print(place_order({"product": "23", "price": "0", "qty": "10"}))
print(place_order({"product": "17", "qty": "11"}))
print(place_order({"product": "17", "qty": " 3"}))

Answer

Order: 10 pcs at 9 € = 90 €
Refused: the quantity is 1 to 10
Order: 3 pcs at 59 € = 177 €

Order: 10 pcs at 0 € = 0 €
Refused: the quantity is 1 to 10
Refused: the quantity is 1 to 10

Order: 10 pcs at 9 € = 90 €
Refused: the quantity is 1 to 10
Refused: the quantity is 1 to 10

The idea

Allowed, not forbidden

Many people’s first idea is to forbid the bad: “the quantity cannot be negative or zero”. That is a deny-list. The other way is to describe exactly the good: “the quantity is one of the texts from 1 to 10”. That is an allow-list.

Here are both on the same inputs:

import re


def deny_list(qty):
    return not qty.startswith("-") and qty != "0"


def allow_list(qty):
    return re.fullmatch(r"[1-9]|10", qty) is not None


def word(ok):
    return "accepted" if ok else "refused"


print(f"{'input':<6}{'deny-list':<11}allow-list")
for qty in ["2", "-3", "0", "500", "2.5", "1e3"]:
    print(f"{qty:<6}{word(deny_list(qty)):<11}{word(allow_list(qty))}")

input deny-list  allow-list
2     accepted   accepted
-3    refused    refused
0     refused    refused
500   accepted   refused
2.5   accepted   refused
1e3   accepted   refused

The deny-list let 500, 2.5 and 1e3 through. Nobody thought of them when writing it. The attacker tries exactly what you did not think of. With an allow-list, the unknown is refused without you having to foresee it.

So: describe what a correct input looks like (type, length, characters, range, a value from a known list) and refuse everything else.

More precisely, if you are curious

Valid does not mean yours. 1042 is a perfectly correct order number, but it may belong to another buyer. A format check is not enough: the server also checks that the order belongs to the logged-in buyer. Without that you have one of the most common holes in real sites (IDOR, insecure direct object reference).
Normalise, then check. If the system further on reads the input differently from the check (spaces, upper and lower case, Unicode, encoding in the address), check the input after it has been normalised. Otherwise the check looks at one thing and the system gets another.
Limit the size. Every field has an upper limit on its length, and every uploaded file on its size. Otherwise a 50 MB “name” eats the server’s memory.
Regular expressions and the end of the text. re.search and re.match do not check the whole text: they also let "500" through. $ also accepts a \n at the end, which is why this uses re.fullmatch.
In a real project validation is declarative. pydantic or the forms of Django and Flask-WTF: conint(ge=1, le=10) and a type for every field. The rules are the same: an allow-list, refuse on error, check on the server.

Problem

The check that lets things through on an error

The sale has a limit: at most 10 headphones per order. The check asks a table of limits, and for an unknown product it throws an exception. The code catches the exception, writes it to the log and carries on:

LIMITS = {"17": 10, "23": 50}


def limit_of(product):
    if product not in LIMITS:
        raise ValueError(f"unknown product {product}")
    return LIMITS[product]


def may_buy(product, qty):
    allowed = True
    try:
        allowed = qty <= limit_of(product)
    except ValueError as e:
        print(f"log: {e}")
    return allowed


print(f"17 x 3: {'yes' if may_buy('17', 3) else 'no'}")
print(f"17 x 500: {'yes' if may_buy('17', 500) else 'no'}")
print(f"99 x 500: {'yes' if may_buy('99', 500) else 'no'}")

17 x 3: yes
17 x 500: no
log: unknown product 99
99 x 500: yes

500 items of product 99 went through. The error in the check became “allowed”.

Try it

Find the line

Which line makes the order pass when there is an error?

Pick the line with the bug.

This is the rule fail closed. The unknown, the empty, an error in the check and an exception you did not expect all lead to “no”. The opposite (failing open) is convenient until someone finds the input that breaks the check.

Exercise

The discount code

Solve the task in your editor.

The coupon. The cart is 120 €. Write discount(code), which returns the total after the discount, or “refused”. The codes are AUTUMN26 (25%) and STUDENTS (10%). Let through only exactly 8 characters from A to Z and 0 to 9, and only a code that is in the table. Everything else is refused. Try these inputs:

"AUTUMN26": 90 €
"autumn26": refused
"AUTUMN26' OR '1'='1": refused
"": refused
"STUDENTS": 108 €
"NOPE1234": refused

autumn26 is refused: the rule says capital letters, not “fix it for the buyer”. The shop may decide to accept lower case, but then it writes that into the rule and checks that instead.

More tasks

The delivery address. Describe with rules what a postcode (4 digits in Bulgaria), a town and a phone number look like. Which of them can be an allow-list, and which only a limit on length and characters?
Someone else’s order. GET /orders/1042 returns order 1042. The input 1042 passes every format check. What else must the server check?

Test

Check yourself

1. The quantity field on the page is <input type="number" min="1" max="10">. Does the server need to check the quantity?No: the browser will not send a number outside 1…10Yes: the request can come without the page, and the check in the browser is done by the buyer’s browserOnly if the page has no JavaScript

2. Which of these is untrusted input for the shop’s server?Only the fields the buyer types intoOnly requests from unknown IP addressesEverything that comes in the request: fields, hidden fields, headers, cookies, names of uploaded files

3. Why is an allow-list safer than a deny-list?Everything you did not think of stays refusedBecause it runs fasterBecause it is shorter

4. The limit check throws an exception because it does not know the product. What should the code decide?Refuse: if the check did not pass, the order does not passLet the order through so the buyer is not annoyedLet the order through and write the error to the log

5. The product’s price sits in a hidden field of the form. What does the server do with it?It compares it with the catalogue price and lets it through if the difference is smallIt does not read it: it takes the price from the catalogue by the product’s idIt checks that it is a positive number

Untrusted input

Cookies