Skip to content
wannacode

Secrets in code and in git

In short: A secret (a key or password) committed to git stays in the history even after it is deleted: rotate it first, then clean the history, and have the code read it from environment variables.

Problem

The key you deleted

The shop takes payments through a provider. The program sends the provider an API key: a long string that proves "this is the shop". Whoever has the key can start payments and refunds in the shop’s name.

A colleague is in a hurry and writes the key straight into the code:

settings.py
PAYMENT_API_KEY = "pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd"

Somebody spots it in code review. The colleague deletes it with a new commit, "Remove the key from the code". The file is clean now.

Three months later the repo becomes public because of the shop’s open SDK. The next night refunds to strangers’ cards start going out from the shop’s account. Somebody ran one command:

$ git log -p -S pay_live_
commit 9c41e07  Remove the key from the code
-            "pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd";
commit 3a9f2d1  Add payments
+            "pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd";

Deleting did not remove the key. It added one more place where it shows.

The idea

Git remembers everything

Every commit is a snapshot of the files. A new commit does not change the old one; it sits on top of it. That is why git log -p shows both adding the key and deleting it.

A secret that went into git and left your machine has leaked. Not "maybe": there are copies in your colleagues’ clones, in forks, in the CI cache, in mirrors. You cannot find them, let alone delete them.

So the order is always the same:

  1. Rotation. In the provider’s dashboard you revoke the old key and issue a new one. Now the old one is useless, wherever it has been copied.
  2. Cleaning the history. git filter-repo rewrites every commit without the key. Then git push --force, and everyone clones again.
  3. The new key never goes into the code. The program reads it from outside: from the environment (environment variables) or from a secret store.

The reverse order is a mistake: while you clean the history, the old key still works.

A secret is anything that grants access: an API key, a database password, a key for signing tokens, the private key of a certificate. In the lesson on hostile input you protected the program from other people’s data. Here you protect other people from your own.

See it

The key in the history

def payment_key():
    key = os.environ.get("PAYMENT_API_KEY", "").strip()
    if not key:
        raise RuntimeError("PAYMENT_API_KEY is not set")
    return key
Step by step, in words
  1. The payment key is in the code. Commit 3a9f2d1 "Add payments" saves it in the history.
  2. The next commit deletes these lines. The file is clean, but 3a9f2d1 still holds the key.
  3. Somebody has a copy of the repo: a clone, a fork, a CI cache. git log -p takes the key out of the old commit.
  4. First, rotation: the provider revokes the old key. The copy out there is now useless.
  5. Then the history: git filter-repo rewrites the commits without the key. The hashes are new, everyone clones again.
  6. The new key lives only in the environment (or a secret store). The code reads it by its name, PAYMENT_API_KEY.
  7. If the name is missing, the program stops at once and says what is missing: fail closed. No default key.
  8. With the variable set the shop starts. No secret in git, and the leaked one no longer works.

The scene starts with the code from the problem and ends with the code that reads the key from the environment. The leaked key stays in the copy out there, but it is revoked. The history is rewritten: the commits have new hashes and the key is not in them.

Terminal
# 1. Rotation: revoke pay_live_…6wHd at the provider and issue a new key.
# 2. The history: replacements.txt contains the line
#    pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd==>***REMOVED***
git filter-repo --replace-text replacements.txt
git push --force --all

The idea

The secret comes from outside, and without it the program stops

The program reads the key from the environment: a table of names and values that the operating system gives every process. On your machine you set it with export PAYMENT_API_KEY=… (in PowerShell $env:PAYMENT_API_KEY = "…"). On the server, the system that runs the application sets it, usually from a secret store.

What matters is what happens when the key is missing. A new server, a misspelled name, a forgotten setting: the program stops at once and says what is missing. This is called fail closed. The opposite, fail open, is to start with an empty or "default" key and blow up at the first customer, or worse, to run with the wrong key.

def payment_key():
    key = os.environ.get("PAYMENT_API_KEY", "").strip()
    if not key:
        raise RuntimeError("PAYMENT_API_KEY is not set")
    return key

The key is never printed in full, not even in the log. If you need to see which key is loaded, show only the beginning and the last four characters:

def masked(key):
    return key[:9] + "…" + key[-4:]
try:
    key = payment_key()
    print(f"Payments are ready, key {masked(key)}")
except RuntimeError as error:
    print(f"The shop does not start: {error}")
Output
The shop does not start: PAYMENT_API_KEY is not set

When the site checks the code there is no such variable, so the output is the refusal. On your machine, with the key set, the program prints Payments are ready, key pay_live_…6wHd.

os.environ.get returns the second argument when the name is not there. os.environ["PAYMENT_API_KEY"] would raise KeyError, which is fail closed too, but with a less clear message. Here except is only for one clean sentence. In a real application you let the exception stop the startup: the process ends with code 1.

See it

A secret scanner

Best of all, the secret never reaches a commit. So a scanner runs before every commit: a program that looks for strings in the changes that look like secrets. Here is a small one with three rules (regular expressions):

  • the provider’s key has a known form: pay_live_ and 24 characters;
  • a URL with a password inside: ://user:password@;
  • assigning a string of at least 8 characters that does not start with $ to password, secret or key.

The last line of the commit is "${PAYMENT_API_KEY}": this is a placeholder that the configuration system fills in from the environment. It is not a secret, and the scanner must not stop it.

RULES = [
    ("payment key", re.compile(r"pay_live_[A-Za-z0-9]{24}")),
    ("password in a URL", re.compile(r"://[^:/\s]+:[^@\s]+@")),
    ("password or key",
     re.compile(r'(password|secret|key)\s*=\s*"[^"$]{8,}"', re.IGNORECASE)),
]




def scan(text):
    found = 0
    for number, line in enumerate(text.splitlines(), start=1):
        for name, pattern in RULES:
            if pattern.search(line):
                print(f"line {number}: {name}")
                found += 1
                break
    return found
diff = """\
+ payment.url = "https://api.pay.example/v1"
+ payment.key = "pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd"
+ db.url = "postgres://shop:Kq7vR2pX@db.internal/shop"
+ mail.password = "winter-sale-2026"
+ payment.key = "${PAYMENT_API_KEY}"
"""
found = scan(diff)
print(f"Found: {found}. The commit stops." if found else "Clean.")
Output
line 2: payment key
line 3: password in a URL
line 4: password or key
Found: 3. The commit stops.

Real scanners have hundreds of rules and also measure how "random" a string is (entropy) to catch keys without a known form. The idea is the same.

Try it

Predict the output

The same scanner, a different configuration file. What will it print?

config = """\
mail.user = "orders@shop.example"
mail.password = "${MAIL_PASSWORD}"
db.url = "postgres://shop:Kq7vR2pX@db/shop"
"""
found = scan(config)
print(f"Found: {found}. The commit stops." if found else "Clean.")
Answer

Try it

Fill in

Where does the key come from?

key = .get("PAYMENT_API_KEY", "").strip()
if not key:
    raise RuntimeError("PAYMENT_API_KEY is not set")

Try it

Find the bug

A colleague does not want the shop to "go down because of one setting" and adds a fallback. Which line is the bug?

Pick the line with the bug.

More precisely, if you are curious

.gitignore and .env.example. If the local secrets are in a .env file, its name is in .gitignore from the very first commit. What goes into git is .env.example with the names and empty values: PAYMENT_API_KEY=. That way a new colleague knows what to fill in without seeing the values.

A scanner before every commit. Doing it by hand does not work: people forget. The scanner runs by itself from a git hook (pre-commit). Ready-made ones are gitleaks, trufflehog and detect-secrets, usually through the pre-commit framework. GitHub also has push protection: it refuses a push with a key of a known service. The scanner is a net under the rope, not a reason to write secrets in the code.

A secret store. Environment variables are better than the code, but they are not ideal. Child processes see them, they sometimes end up in a crash dump, and they are easy to print by mistake. So on the server the secrets come from a store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager. They record who read what and when, rotate the keys on a schedule and hand out short-lived keys.

The python-dotenv package loads .env into os.environ during development. In production the environment comes from the system that runs the application.

The cleaning, in detail. git filter-repo deliberately removes the origin remote after rewriting, so that you do not accidentally send the new history to the wrong place: you add it back and run git push --force --all and --tags. Colleagues have to clone again: an old clone merged back brings the key back into the history. Services like GitHub also keep old pull request references, so there you also write to support. The older tool for the same job is BFG Repo-Cleaner.

Secrets in CI (deploy keys, access to the container registry, the pipeline’s secrets) are a topic of the DevOps course ZeroToOps. This lesson is the part the programmer writes.

Exercise

The key in the log

Solve it in your own editor.

Secrets also leak through logs: the request to the provider is logged in full, with the key inside. Write clean, which replaces every pay_live_ + 24 characters key in a log line with its masked form, and run it on these three lines:

POST /pay 200 key=pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd amount=49.90
POST /refund 200 key=pay_live_7Qm2Xc9LkT4vB8nR1sZp6wHd amount=12.00
GET /health 200
Output
POST /pay 200 key=pay_live_…6wHd amount=49.90
POST /refund 200 key=pay_live_…6wHd amount=12.00
GET /health 200

More tasks

  • One more rule. Add a rule for a private key to the scanner: a line that starts with -----BEGIN and ends with PRIVATE KEY-----.
  • Exceptions. Add a list of allowed values (for example the sample key in the documentation) that the scanner skips.
  • A check at startup. Write a function that checks all the needed names at once (PAYMENT_API_KEY, DB_PASSWORD, MAIL_PASSWORD) and reports every missing one, not just the first.

Test

Check yourself

1. The payment key was in the public repo for one day. A new commit deletes it. What else has to happen, and in what order?
2. Why is a clean history without rotating the key not enough?
3. The environment has no PAYMENT_API_KEY. What is the right behaviour of the shop?
4. Where does the .env file with the real values live?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·