Skip to content
wannacode

Tests in CI

In short: CI runs the tests on every change, on a clean machine, and does not allow a merge while anything is red. A failed test is a signal: a defect, a changed rule, a wrong test or a broken environment. The most dangerous one is the test that fails sometimes: people get used to running it again and miss the real failure. The cause is almost always something the test does not control, such as the time.

Problem

Red at night, green in the morning

Every night at 1 a.m. Bulgarian time the server runs all the tests. Three nights in a row, the test "the promo code is valid on its last day" fails. In the morning the programmer runs it again, and it passes.

"It fails sometimes, run it again" became a habit. A month later nobody was looking at the nightly report. Then a real discount test failed, and nobody saw it until the first customer complained.

The idea

Continuous integration

Continuous integration (CI) is an agreement: on every change, before it gets into the shared code, a server takes the code onto a clean machine, compiles it and runs the tests. If anything is red, the change is not merged.

Here is the configuration for GitHub Actions, one of the most widespread tools. It runs on every push and every pull request:

.github/workflows/tests.yml
name: Tests


on:
  push:
  pull_request:


jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: 21
      # Without the UI tests (marked ui): they run separately, once a night.
      - run: mvn --batch-mode test -DexcludedGroups=ui

The component tests, the scenarios and the API tests from this module run here. The UI tests are separate, once a night against the test environment: they need a browser and a running shop and are too slow for every change.

More precisely, if you are curious

The configuration is shown but not run in the course: it needs GitHub. Other widespread tools are GitLab CI, Jenkins and Azure Pipelines. The idea and the steps are the same: get the code, set up the language, run the tests, save the report.

The rule "red stops the merge" is set up separately, in the protection of the main branch (in GitHub: branch protection, required checks). Without it, CI is only advice.

Try it

The path of one change

Put in order what happens from pushing the change to the merge.

  1. The code is compiled
  2. The scenarios and the API tests
  3. The component tests
  4. CI takes the code onto a clean machine
  5. The merge is allowed only if everything is green
  6. The programmer pushes the change and opens a pull request
  7. The report: what passed and what failed

The idea

Why one test fails

Red in CI is a signal, not a verdict. There are four causes, and each needs something different:

  • a defect in the code: the change broke something. The code gets fixed. This is the regression the tests exist for;
  • the rule changed: the test states the old requirement. The test changes, but according to the new requirement, not "until it passes";
  • a wrong test: the test checks something nobody asked for. The test gets fixed, and the author is told;
  • the environment: the database did not start, the network is slow. Look in the machine, not in the product.

One thing is not a cause: "run it again". If the test passes on the second try with no change, the problem is still there.

Try it

What does this red mean?

1. The free delivery test fails after a change in the promo code logic. The delivery rule has not changed.
2. The owner decided the threshold should be 60 €. The programmer changed the code, and the test for 50.00 € fails.
3. All the API tests fail with "connection refused", and they passed yesterday. The code has not been touched.
4. A test fails once in ten runs, with no change in the code.

The idea

A test that fails sometimes

Here is the test from the start. A promo code is valid up to and including its last day. The code asks "what day is it today" in the shop's time zone, Sofia:

src/main/java/shop/PromoCode.java
public boolean isValidToday() {
    LocalDate today = LocalDate.now(ZoneId.of("Europe/Sofia"));
    return !today.isAfter(lastDay);
}

And the test creates a code whose last day is "today", by the machine's clock:

src/test/java/shop/PromoCodeTest.java
@Test
@DisplayName("is valid on its last day")
void validOnItsLastDay() {
    PromoCode code = new PromoCode("AUTUMN10", LocalDate.now());
    assertTrue(code.isValidToday());
}

On a laptop in Sofia both "todays" are the same day, and the test always passes. The CI server is in the UTC time zone. Between midnight and 2 a.m. (3 a.m. in summer) Sofia time it is already tomorrow in Sofia, but still today on the server. The code says: the last day is over. The test fails exactly when the nightly build runs. (In JavaScript the test always takes the date in UTC, so it fails at that hour on the laptop too.)

Such a test is called flaky: it sometimes passes and sometimes fails, with no change in the code. The causes are almost always something the test does not control:

  • the time and the date, the time zone;
  • the order: the test depends on data left behind by another test;
  • waiting: a fixed pause that is sometimes not enough;
  • the network and external services;
  • random data without a fixed seed.
More precisely, if you are curious

The code and the test on this card are shown and compiled, but not run in the course: their result depends on the moment they run, and a check with an expected output is not possible. That is exactly their problem.

The idea

The fix: the day is an input

The test cannot control the clock, so the day becomes a parameter. The code asks "is it valid on this day":

src/main/java/shop/PromoCode.java
// The code is valid up to and including its last day.
public boolean isValidOn(LocalDate day) {
    return !day.isAfter(lastDay);
}

The test sets specific dates, not "today". And now it can check the boundary precisely: the last day and the day after it:

src/test/java/shop/PromoCodeTest.java
@Test
@DisplayName("is valid on its last day")
void validOnItsLastDay() {
    assertTrue(code.isValidOn(LocalDate.of(2026, 11, 30)));
}

The whole file, run. It passes on every run, on every machine, at every hour:

src/test/java/shop/PromoCodeTest.java
package shop;


import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;


import java.time.LocalDate;


import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;


class PromoCodeTest {
    private final PromoCode code = new PromoCode("AUTUMN10", LocalDate.of(2026, 11, 30));


    @Test
    @DisplayName("is valid before its last day")
    void validBeforeItsLastDay() {
        assertTrue(code.isValidOn(LocalDate.of(2026, 11, 1)));
    }


    @Test
    @DisplayName("is valid on its last day")
    void validOnItsLastDay() {
        assertTrue(code.isValidOn(LocalDate.of(2026, 11, 30)));
    }


    @Test
    @DisplayName("is not valid the day after")
    void notValidTheDayAfter() {
        assertFalse(code.isValidOn(LocalDate.of(2026, 12, 1)));
    }
}
Output
PASS  is not valid the day after
PASS  is valid before its last day
PASS  is valid on its last day
Tests: 3, passed: 3, failed: 0

And if someone writes "before the last day" instead of "up to and including the last day", the test fails every time, not sometimes:

src/main/java/shop/PromoCode.java
// The code is valid up to and including its last day.
public boolean isValidOn(LocalDate day) {
    return day.isBefore(lastDay);
}
src/test/java/shop/PromoCodeTest.java
package shop;


import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;


import java.time.LocalDate;


import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;


class PromoCodeTest {
    private final PromoCode code = new PromoCode("AUTUMN10", LocalDate.of(2026, 11, 30));


    @Test
    @DisplayName("is valid before its last day")
    void validBeforeItsLastDay() {
        assertTrue(code.isValidOn(LocalDate.of(2026, 11, 1)));
    }


    @Test
    @DisplayName("is valid on its last day")
    void validOnItsLastDay() {
        assertTrue(code.isValidOn(LocalDate.of(2026, 11, 30)));
    }


    @Test
    @DisplayName("is not valid the day after")
    void notValidTheDayAfter() {
        assertFalse(code.isValidOn(LocalDate.of(2026, 12, 1)));
    }
}
Output
FAIL  is valid on its last day: expected: <true> but was: <false>
PASS  is not valid the day after
PASS  is valid before its last day
Tests: 3, passed: 2, failed: 1
More precisely, if you are curious

Java and C# also have a standard way to pass the clock in from outside: java.time.Clock and TimeProvider (from .NET 8). The code then asks the clock, and the test gives it a fixed one. In Python and JavaScript there are libraries that freeze time in a test (freezegun, the fake timers of the test tools). The simplest way, though, is the one on this card: the day is a parameter.

The answer to "which day is today for the promo code" is a business decision, not a programmer's one. Here it was decided: in the shop's time zone.

The idea

The report

The report from CI is the first thing someone sees when something is red. A good report says, without asking anyone:

  • which test failed, with a name that states the rule ("is valid on its last day"), not test17;
  • expected and actual: "expected: true, actual: false";
  • on which change: who pushed what;
  • the history: is it failing for the first time, or yesterday and the day before too.

The tools write the report in a format CI understands (most often JUnit XML), and CI shows it next to the change.

If a test fails sometimes and the fix will take time, the honest move is to set it apart from the required tests (quarantine), with a defect report and a deadline, and do the check by hand. Not to delete it, and not to run it until it passes.

Exercise

Red in CI

1. Causes and fixes. The questions are about the promo code and the shop's CI.

1. The test "is valid on its last day" fails only in the nightly build. Which question do you ask first?
2. How do you fix a test that depends on today's date?
3. A test fails sometimes, and the fix will take days. What is the honest thing to do in the meantime?
4. Test A passes on its own but fails when test B runs before it. What is the most likely cause?

Test

Check yourself

1. What is continuous integration (CI)?
2. Why does CI run the tests on a clean machine and not on the programmer's laptop?
3. Why is a test that fails sometimes dangerous?
4. What should the report say about a failed test?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·