Flaky Tests

What is this document?

This document is designed to indicate different aspects related to the problem that occurs at the time of running the tests on Time Tracker UI

Problem

Currently, when attempting to run the tests, some of them randomly fail, and show problems such as the following:

First time when running tests.

Second time when running tests.

As can be seen in the images above, each time the tests are run, the errors they produce are different.

Why Tests Fails Randomly?

Tests randomly fail for these two main reasons:

1. Karma-jasmine uses a seed to execute the tests, every time the tests are executed, karma will generate a new seed.

Example:

2. When using asynchronous functionalities in our tests such as 'waitForAsync()', It causes the code to run in a special asynchronous test zone, for this reason, this type of functionality must be used appropriately.

How seed works?

The seed works as follows:

As can be seen in the previous image, each time the tests are executed, a new seed is generated and it alters the order in which the test files are executed.

Where do these errors happen?

After an analysis, it was possible to identify the components, services, providers involved in the function causing the problem.

It can be seen in the following image:

Possible Solutions

Several approaches have been proposed for a solution:

First Approach

To apply this approach, the intention is to modify getFilterFromConfiguration() method inside FeatureFilterProvider class and also modify karma.conf.js.

getFilterFromConfiguration()

Before:

import { Injectable } from "@angular/core";
import { AzureAdB2CService } from "src/app/modules/login/services/azure.ad.b2c.service";
import { FeatureFilterConfiguration } from "./feature-filter-configuration";
import { FeatureFilterTypes } from "./feature-filter-types";
import { FeatureFilterModel } from "./feature-filter.model";
import { TargetingFilterParameters } from "./targeting/targeting-feature-filter-parameters";
import { TargetingFeatureFilterModel } from "./targeting/targeting-feature-filter.model";

@Injectable({
  providedIn: "root",
})
export class FeatureFilterProvider {
  constructor(private userService: AzureAdB2CService) {}

  getFilterFromConfiguration(
    featureFilterConfiguration: FeatureFilterConfiguration
  ): FeatureFilterModel {
    const featureName = featureFilterConfiguration.name;
    switch (featureName) {
      case FeatureFilterTypes.TARGETING: {
        const appContext = {
          username: this.userService.getUserEmail(),
          group: this.userService.getUserGroup(),
        };
        const filter = new TargetingFeatureFilterModel(
          featureFilterConfiguration.parameters as TargetingFilterParameters,
          appContext
        );
        return filter;
      }
      default: {
        break;
      }
    }
  }
}

After:

import { Injectable } from "@angular/core";
import { AzureAdB2CService } from "src/app/modules/login/services/azure.ad.b2c.service";
import { FeatureFilterConfiguration } from "./feature-filter-configuration";
import { FeatureFilterTypes } from "./feature-filter-types";
import { FeatureFilterModel } from "./feature-filter.model";
import { TargetingFilterParameters } from "./targeting/targeting-feature-filter-parameters";
import { TargetingFeatureFilterModel } from "./targeting/targeting-feature-filter.model";

@Injectable({
  providedIn: "root",
})
export class FeatureFilterProvider {
  constructor(private userService: AzureAdB2CService) {}

  getFilterFromConfiguration(
    featureFilterConfiguration: FeatureFilterConfiguration
  ): FeatureFilterModel {
    const featureName = featureFilterConfiguration.name;
    switch (featureName) {
      case FeatureFilterTypes.TARGETING: {
        let username: string;
        let group: string;
        if (this.userService) {
          try {
            username = this.userService.getUserEmail();
            group = this.userService.getUserGroup();
          } catch (error) {
            username = "[email protected]";
            group = "fake-group";
          }
        }

        const appContext = {
          username,
          group,
        };
        const filter = new TargetingFeatureFilterModel(
          featureFilterConfiguration.parameters as TargetingFilterParameters,
          appContext
        );
        return filter;
      }
      default: {
        break;
      }
    }
  }
}

karma.conf.js

Before:

client: {
  clearContext: false
},

After:

client: {
  clearContext: false,
  jasmine: {
    random: true,
    seed: '90967',
  },
},

With this approach, so far it has been verified more than a hundred times and none of them has failed.

Second Approach

To apply this approach, the intention is to modify only karma.conf.js.

karma.conf.js

Before:

client: {
  clearContext: false
},

After:

client: {
  clearContext: false,
  jasmine: {
    random: true,
    seed: '90967',
  },
},

With this approach, maintains a more stable test execution, however, it fails sometimes with the same error as initially indicated.

Third Approach

The third approach is the most ideal and at the same time the most complicated, it consists of detecting the root of the whole problem, however, to achieve this objective, each of the tests must be verified and corroborating that it is well done and does not contain errors.

Selected Approach

After a technical meeting with Time Tracker's development team, it was decided to opt for First Approach and it was also decided to develop a plan for achieving the Third Approach, execute the next steps:

1. Quality Strategy

2. Test Strategy

Notes

The first approach is currently in production, you can see the related PR.