Skip to content

plugins.md

Latest commit

 

History

History
795 lines (598 loc) · 19.2 KB

plugins.md

File metadata and controls

795 lines (598 loc) · 19.2 KB
permalink plugins
sidebarDepth
sidebar auto
title Plugins

analyze

Uses AI to analyze test failures and provide insights

This plugin analyzes failed tests using AI to provide detailed explanations and group similar failures. When enabled with --ai flag, it generates reports after test execution.

Usage

// in codecept.conf.js
exports.config = {
  plugins: {
    analyze: {
      enabled: true,
      clusterize: 5,
      analyze: 2,
      vision: false
    }
  }
}

Configuration

  • clusterize (number) - minimum number of failures to trigger clustering analysis. Default: 5
  • analyze (number) - maximum number of individual test failures to analyze in detail. Default: 2
  • vision (boolean) - enables visual analysis of test screenshots. Default: false
  • categories (array) - list of failure categories for classification. Defaults to:
    • Browser connection error / browser crash
    • Network errors (server error, timeout, etc)
    • HTML / page elements (not found, not visible, etc)
    • Navigation errors (404, etc)
    • Code errors (syntax error, JS errors, etc)
    • Library & framework errors
    • Data errors (password incorrect, invalid format, etc)
    • Assertion failures
    • Other errors
  • prompts (object) - customize AI prompts for analysis
    • clusterize - prompt for clustering analysis
    • analyze - prompt for individual test analysis

Features

  • Groups similar failures when number of failures >= clusterize value
  • Provides detailed analysis of individual failures
  • Analyzes screenshots if vision=true and screenshots are available
  • Classifies failures into predefined categories
  • Suggests possible causes and solutions

Parameters

  • config Object Plugin configuration (optional, default {})

Returns void

auth

Logs user in for the first test and reuses session for next tests. Works by saving cookies into memory or file. If a session expires automatically logs in again.

For better development experience cookies can be saved into file, so a session can be reused while writing tests.

Usage

  1. Enable this plugin and configure as described below
  2. Define user session names (example: user, editor, admin, etc).
  3. Define how users are logged in and how to check that user is logged in
  4. Use login object inside your tests to log in:
// inside a test file
// use login to inject auto-login function
Feature('Login');

Before(({ login }) => {
   login('user'); // login using user session
});

// Alternatively log in for one scenario.
Scenario('log me in', ( { I, login } ) => {
   login('admin');
   I.see('I am logged in');
});

Configuration

  • saveToFile (default: false) - save cookies to file. Allows to reuse session between execution.
  • inject (default: login) - name of the login function to use
  • users - an array containing different session names and functions to:
    • login - sign in into the system
    • check - check that user is logged in
    • fetch - to get current cookies (by default I.grabCookie())
    • restore - to set cookies (by default I.amOnPage('/'); I.setCookie(cookie))

How It Works

  1. restore method is executed. It should open a page and set credentials.
  2. check method is executed. It should reload a page (so cookies are applied) and check that this page belongs to logged-in user. When you pass the second args session, you could perform the validation using passed session.
  3. If restore and check were not successful, login is executed
  4. login should fill in login form
  5. After successful login, fetch is executed to save cookies into memory or file.

Example: Simple login

auth: {
  enabled: true,
  saveToFile: true,
  inject: 'login',
  users: {
    admin: {
      // loginAdmin function is defined in `steps_file.js`
      login: (I) => I.loginAdmin(),
      // if we see `Admin` on page, we assume we are logged in
      check: (I) => {
         I.amOnPage('/');
         I.see('Admin');
      }
    }
  }
}

Example: Multiple users

auth: {
  enabled: true,
  saveToFile: true,
  inject: 'loginAs', // use `loginAs` instead of login
  users: {
    user: {
      login: (I) => {
         I.amOnPage('/login');
         I.fillField('email', 'user@site.com');
         I.fillField('password', '123456');
         I.click('Login');
      },
      check: (I) => {
         I.amOnPage('/');
         I.see('User', '.navbar');
      },
    },
    admin: {
      login: (I) => {
         I.amOnPage('/login');
         I.fillField('email', 'admin@site.com');
         I.fillField('password', '123456');
         I.click('Login');
      },
      check: (I) => {
         I.amOnPage('/');
         I.see('Admin', '.navbar');
      },
    },
  }
}

Example: Keep cookies between tests

If you decide to keep cookies between tests you don't need to save/retrieve cookies between tests. But you need to login once work until session expires. For this case, disable fetch and restore methods.

helpers: {
   WebDriver: {
     // config goes here
     keepCookies: true; // keep cookies for all tests
   }
},
plugins: {
   auth: {
     users: {
       admin: {
         login: (I) => {
           I.amOnPage('/login');
           I.fillField('email', 'admin@site.com');
           I.fillField('password', '123456');
           I.click('Login');
         },
         check: (I) => {
           I.amOnPage('/dashboard');
           I.see('Admin', '.navbar');
         },
         fetch: () => {}, // empty function
         restore: () => {}, // empty funciton
       }
    }
  }
}

Example: Getting sessions from local storage

If your session is stored in local storage instead of cookies you still can obtain sessions.

plugins: {
   auth: {
    admin: {
      login: (I) => I.loginAsAdmin(),
      check: (I) => I.see('Admin', '.navbar'),
      fetch: (I) => {
        return I.executeScript(() => localStorage.getItem('session_id'));
      },
      restore: (I, session) => {
        I.amOnPage('/');
        I.executeScript((session) => localStorage.setItem('session_id', session), session);
      },
    }
  }
}

Tips: Using async function in the auth

If you use async functions in the auth plugin, login function should be used with await keyword.

auth: {
  enabled: true,
  saveToFile: true,
  inject: 'login',
  users: {
    admin: {
      login: async (I) => {  // If you use async function in the auth plugin
         const phrase = await I.grabTextFrom('#phrase')
         I.fillField('username', 'admin'),
         I.fillField('password', 'password')
         I.fillField('phrase', phrase)
      },
      check: (I) => {
         I.amOnPage('/');
         I.see('Admin');
      },
    }
  }
}
Scenario('login', async ( {I, login} ) => {
  await login('admin') // you should use `await`
})

Tips: Using session to validate user

Instead of asserting on page elements for the current user in check, you can use the session you saved in fetch

auth: {
  enabled: true,
  saveToFile: true,
  inject: 'login',
  users: {
    admin: {
      login: async (I) => {  // If you use async function in the auth plugin
         const phrase = await I.grabTextFrom('#phrase')
         I.fillField('username', 'admin'),
         I.fillField('password', 'password')
         I.fillField('phrase', phrase)
      },
      check: (I, session) => {
         // Throwing an error in `check` will make CodeceptJS perform the login step for the user
         if (session.profile.email !== the.email.you.expect@some-mail.com) {
              throw new Error ('Wrong user signed in');
        }
      },
    }
  }
}
Scenario('login', async ( {I, login} ) => {
  await login('admin') // you should use `await`
})

Parameters

  • config

autoDelay

Sometimes it takes some time for a page to respond to user's actions. Depending on app's performance this can be either slow or fast.

For instance, if you click a button and nothing happens - probably JS event is not attached to this button yet Also, if you fill field and input validation doesn't accept your input - maybe because you typed value too fast.

This plugin allows to slow down tests execution when a test running too fast. It puts a tiny delay for before and after action commands.

Commands affected (by default):

  • click
  • fillField
  • checkOption
  • pressKey
  • doubleClick
  • rightClick

Configuration

plugins: {
   autoDelay: {
     enabled: true
   }
}

Possible config options:

  • methods: list of affected commands. Can be overridden
  • delayBefore: put a delay before a command. 100ms by default
  • delayAfter: put a delay after a command. 200ms by default

Parameters

  • config

consolidateWorkerJsonResults

Consolidates JSON reports from multiple workers into a single HTML report

Parameters

  • config

coverage

Dumps code coverage from Playwright/Puppeteer after every test.

Configuration

plugins: {
   coverage: {
     enabled: true,
     debug: true,
     name: 'CodeceptJS Coverage Report',
     outputDir: 'output/coverage'
   }
}

Possible config options, More could be found at monocart-coverage-reports

  • debug: debug info. By default, false.
  • name: coverage report name.
  • outputDir: path to coverage report.
  • sourceFilter: filter the source files.
  • sourcePath: option to resolve a custom path.

Parameters

  • config

customLocator

Creates a custom locator by using special attributes in HTML.

If you have a convention to use data-test-id or data-qa attributes to mark active elements for e2e tests, you can enable this plugin to simplify matching elements with these attributes:

// replace this:
I.click({ css: '[data-test-id=register_button]');
// with this:
I.click('$register_button');

This plugin will create a valid XPath locator for you.

Configuration

  • enabled (default: false) should a locator be enabled
  • prefix (default: $) sets a prefix for a custom locator.
  • attribute (default: data-test-id) to set an attribute to be matched.
  • strategy (default: xpath) actual locator strategy to use in query (css or xpath).
  • showActual (default: false) show in the output actually produced XPath or CSS locator. By default shows custom locator value.

Examples:

Using data-test attribute with $ prefix:

// in codecept.conf.js
plugins: {
 customLocator: {
   enabled: true,
   attribute: 'data-test'
 }
}

In a test:

I.seeElement('$user'); // matches => [data-test=user]
I.click('$sign-up'); // matches => [data-test=sign-up]

Using data-qa attribute with = prefix:

// in codecept.conf.js
plugins: {
 customLocator: {
   enabled: true,
   prefix: '=',
   attribute: 'data-qa'
 }
}

In a test:

I.seeElement('=user'); // matches => [data-qa=user]
I.click('=sign-up'); // matches => [data-qa=sign-up]

Using data-qa OR data-test attribute with = prefix:

// in codecept.conf.js
plugins: {
 customLocator: {
   enabled: true,
   prefix: '=',
   attribute: ['data-qa', 'data-test'],
   strategy: 'xpath'
 }
}

In a test:

I.seeElement('=user'); // matches => //*[@data-qa=user or @data-test=user]
I.click('=sign-up'); // matches => //*[data-qa=sign-up or @data-test=sign-up]
// in codecept.conf.js
plugins: {
 customLocator: {
   enabled: true,
   prefix: '=',
   attribute: ['data-qa', 'data-test'],
   strategy: 'css'
 }
}

In a test:

I.seeElement('=user'); // matches => [data-qa=user],[data-test=user]
I.click('=sign-up'); // matches => [data-qa=sign-up],[data-test=sign-up]

Parameters

  • config

customReporter

Sample custom reporter for CodeceptJS.

Parameters

  • config

enhancedRetryFailedStep

Enhanced retryFailedStep plugin that coordinates with other retry mechanisms

This plugin provides step-level retries and coordinates with global retry settings to avoid conflicts and provide predictable behavior.

Parameters

  • config

heal

Self-healing tests with AI.

Read more about heaking in Self-Healing Tests

plugins: {
  heal: {
   enabled: true,
  }
}

More config options are available:

  • healLimit - how many steps can be healed in a single test (default: 2)

Parameters

  • config (optional, default {})

pageInfo

Collects information from web page after each failed test and adds it to the test as an artifact. It is suggested to enable this plugin if you run tests on CI and you need to debug failed tests. This plugin can be paired with analyze plugin to provide more context.

It collects URL, HTML errors (by classes), and browser logs.

Enable this plugin in config:

plugins: {
 pageInfo: {
  enabled: true,
}

Additional config options:

  • errorClasses - list of classes to search for errors (default: ['error', 'warning', 'alert', 'danger'])
  • browserLogs - list of types of errors to search for in browser logs (default: ['error'])

Parameters

  • config (optional, default {})

pauseOnFail

Automatically launches interactive pause when a test fails.

Useful for debugging flaky tests on local environment. Add this plugin to config file:

plugins: {
  pauseOnFail: {},
}

Unlike other plugins, pauseOnFail is not recommended to be enabled by default. Enable it manually on each run via -p option:

npx codeceptjs run -p pauseOnFail

reportData

TypeScript: Explicitly type reportData arrays as any[] to avoid 'never' errors

retryFailedStep

Retries each failed step in a test.

Add this plugin to config file:

plugins: {
    retryFailedStep: {
       enabled: true
    }
}

Run tests with plugin enabled:

npx codeceptjs run --plugins retryFailedStep

Configuration:

  • retries - number of retries (by default 3),
  • when - function, when to perform a retry (accepts error as parameter)
  • factor - The exponential factor to use. Default is 1.5.
  • minTimeout - The number of milliseconds before starting the first retry. Default is 1000.
  • maxTimeout - The maximum number of milliseconds between two retries. Default is Infinity.
  • randomize - Randomizes the timeouts by multiplying with a factor from 1 to 2. Default is false.
  • defaultIgnoredSteps - an array of steps to be ignored for retry. Includes:
    • amOnPage
    • wait*
    • send*
    • execute*
    • run*
    • have*
  • ignoredSteps - an array for custom steps to ignore on retry. Use it to append custom steps to ignored list. You can use step names or step prefixes ending with *. As such, wait* will match all steps starting with wait. To append your own steps to ignore list - copy and paste a default steps list. Regexp values are accepted as well.

Example

plugins: {
    retryFailedStep: {
        enabled: true,
        ignoredSteps: [
          'scroll*', // ignore all scroll steps
          /Cookie/, // ignore all steps with a Cookie in it (by regexp)
        ]
    }
}

Disable Per Test

This plugin can be disabled per test. In this case you will need to stet I.retry() to all flaky steps:

Use scenario configuration to disable plugin for a test

Scenario('scenario tite', { disableRetryFailedStep: true }, () => {
   // test goes here
})

Parameters

  • config

safeJsonStringify

Safely serialize data to JSON, handling circular references

Parameters

  • data

screenshotOnFail

Creates screenshot on failure. Screenshot is saved into output directory.

Initially this functionality was part of corresponding helper but has been moved into plugin since 1.4

This plugin is enabled by default.

Configuration

Configuration can either be taken from a corresponding helper (deprecated) or a from plugin config (recommended).

plugins: {
   screenshotOnFail: {
     enabled: true
   }
}

Possible config options:

  • uniqueScreenshotNames: use unique names for screenshot. Default: false.
  • fullPageScreenshots: make full page screenshots. Default: false.

Parameters

  • config

stepByStepReport

step-by-step-report

Generates step by step report for a test. After each step in a test a screenshot is created. After test executed screenshots are combined into slideshow. By default, reports are generated only for failed tests.

Run tests with plugin enabled:

npx codeceptjs run --plugins stepByStepReport

Configuration

"plugins": {
   "stepByStepReport": {
     "enabled": true
   }
}

Possible config options:

  • deleteSuccessful: do not save screenshots for successfully executed tests. Default: true.
  • animateSlides: should animation for slides to be used. Default: true.
  • ignoreSteps: steps to ignore in report. Array of RegExps is expected. Recommended to skip grab* and wait* steps.
  • fullPageScreenshots: should full page screenshots be used. Default: false.
  • output: a directory where reports should be stored. Default: output.
  • screenshotsForAllureReport: If Allure plugin is enabled this plugin attaches each saved screenshot to allure report. Default: false.
  • `disableScreenshotOnFail : Disables the capturing of screeshots after the failed step. Default: true.

Parameters

  • config any

stepTimeout

Set timeout for test steps globally.

Add this plugin to config file:

plugins: {
    stepTimeout: {
       enabled: true
    }
}

Run tests with plugin enabled:

npx codeceptjs run --plugins stepTimeout

Configuration:

  • timeout - global step timeout, default 150 seconds

  • overrideStepLimits - whether to use timeouts set in plugin config to override step timeouts set in code with I.limitTime(x).action(...), default false

  • noTimeoutSteps - an array of steps with no timeout. Default:

    • amOnPage
    • wait*

    you could set your own noTimeoutSteps which would replace the default one.

  • customTimeoutSteps - an array of step actions with custom timeout. Use it to override or extend noTimeoutSteps. You can use step names or step prefixes ending with *. As such, wait* will match all steps starting with wait.

Example

plugins: {
    stepTimeout: {
        enabled: true,
        overrideStepLimits: true,
        noTimeoutSteps: [
          'scroll*', // ignore all scroll steps
          /Cookie/, // ignore all steps with a Cookie in it (by regexp)
        ],
        customTimeoutSteps: [
          ['myFlakyStep*', 1],
          ['scrollWhichRequiresTimeout', 5],
        ]
    }
}

Parameters

  • config

subtitles

Automatically captures steps as subtitle, and saves it as an artifact when a video is found for a failed test

Configuration

plugins: {
 subtitles: {
   enabled: true
 }
}