--- stage: none group: unassigned info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/development/development_processes/#development-guidelines-review. title: End-to-end test pipelines --- ## Common architecture All E2E tests are executed within separate child pipelines. To support different dynamic features of E2E testing pipelines, all child pipeline YAML files are generated by `e2e-test-pipeline-generate` CI/CD job and triggered by respective trigger job. ### e2e-test-pipeline-generate The `e2e-test-pipeline-generate` job generates CI/CD YAML file definitions which are used to trigger child pipelines that run E2E tests. The [`generate_e2e_pipelines`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/tasks/ci.rake) Rake task: 1. Determines which e2e specs should be executed in a particular merge request pipeline. 1. Generates a CI/CD YAML file definition for each E2E test pipeline type. This Rake task: 1. Analyzes changes in a particular merge request and determines which specs must be executed using selective test execution with [these criteria](_index.md#selective-test-execution). Based on that, a `dry-run` of every [scenario](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test) executes and determines if a scenario contains any executable tests. 1. The total run time for each scenario is calculated. 1. Based on the run time value, [dynamic job scaling](_index.md#dynamic-parallel-job-scaling) calculates the necessary number of parallel CI/CD jobs for each scenario type and generates pipeline YAML file with appropriate values. ## `e2e:test-on-cng` The `e2e:test-on-cng` child pipeline runs tests against a [Cloud Native GitLab](https://gitlab.com/gitlab-org/build/CNG) installation. Deployment is managed by the [`orchestrator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/gems/gitlab-orchestrator/README.md) CLI tool, which you can also use to locally recreate CI/CD deployments. The `e2e:test-on-cng` child pipeline is executed in merge requests and is part of pre-merge validation lifecycle. If any test fails, you can't merge introduced code changes. ### Setup This E2E test child pipeline is triggered by the `e2e:test-on-cng` job using dynamically generated CI/CD YAML file stored as artifacts in the `e2e-test-pipeline-generate` CI/CD job. CI/CD YAML files are generated using a [template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/test-on-cng/main.gitlab-ci.yml). ### child pipeline jobs The child pipeline consists of several stages that support E2E test execution. #### .pre - `build-cng-env` job is responsible for setting up all environment variables for [CNG](https://gitlab.com/gitlab-org/build/CNG-mirror) downstream pipeline - `build-cng` job triggers `CNG` downstream pipeline which is responsible for building all necessary images #### test Jobs in `test` stage perform following actions: 1. local k8s cluster setup using [`kind`](https://github.com/kubernetes-sigs/kind) 1. GitLab installation using official [`helm` chart](https://gitlab.com/gitlab-org/charts/gitlab) 1. E2E test execution against performed deployment ##### report This stage is responsible for [allure test report](_index.md#allure-report) generation. ### Debugging To help with debugging: - Each test job prints a list of arguments that you can pass to the [`orchestrator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/gems/gitlab-orchestrator/README.md) to exactly recreate the same deployment for local debugging. - Cluster events log and all pod logs are saved in E2E test job artifacts. - `orchestrator` automatically outputs all cluster events with errors in the case of failed deployment. ## `e2e:test-on-omnibus-ee` The `e2e:test-on-omnibus-ee` child pipeline runs tests against an [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab) installation. This pipeline type is not executed in merge request pipelines by default and can be triggered manually by triggering the `e2e:test-on-omnibus-ee` job. This pipeline type is allowed to fail and even in the case of a manual trigger inside of a merge request pipeline, failing tests will not block the ability to merge. Linux package deployment is managed by [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa). ### Setup This E2E test child pipeline is triggered by the `e2e:test-on-omnibus-ee` job using dynamically generated CI/CD YAML file stored as artifacts in `e2e-test-pipeline-generate` CI/CD jobs. The CI/CD YAML file is generated using a [template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/test-on-omnibus/main.gitlab-ci.yml). ### child pipeline jobs E2E test execution pipeline consists of several stages which all support execution of E2E tests. #### .pre This stage is responsible for the following tasks: - Triggering downstream pipeline which builds the [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) Docker image. #### test Jobs in `test` stage perform following actions: 1. Installs GitLab by using a built Linux package. 1. Executes E2E tests against the Linux package installation. #### report This stage is responsible for [allure test report](_index.md#allure-report) generation. ## `e2e:test-on-gdk` The `e2e:test-on-gdk` child pipeline supports development of the GitLab platform by providing feedback to engineers on end-to-end test execution faster than via `e2e:test-on-omnibus-ee`. This is achieved by running tests against the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) (GDK), which can be built and installed in less time than when testing against [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). The trade-off is that Omnibus GitLab can be used to deploy a production installation, whereas the GDK is a development environment. Tests that run against the GDK might not catch bugs that depend on part of the process of preparing GitLab to run in a production environment, including pre-compiling assets, assigning configuration defaults as part of an official installation package, deploying GitLab services to multiple servers, and more. On the other hand, engineers who use the GDK day-to-day can benefit from automated tests catching bugs that only appear on the GDK. ### Setup This E2E test child pipeline is triggered by the `e2e:test-on-gdk` job using dynamically generated CI/CD YAML file stored as artifacts in the `e2e-test-pipeline-generate` CI/CD job. The CI/CD YAML file is generated by using a [template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/test-on-gdk/main.gitlab-ci.yml). #### `build-gdk-image` [The `build-gdk-image` job](https://gitlab.com/gitlab-org/gitlab/-/blob/07504c34b28ac656537cd60810992aa15e9e91b8/.gitlab/ci/build-images.gitlab-ci.yml#L32) uses the code from the merge request to build a Docker image that can be used in test jobs to launch a GDK instance in a container. The image is pushed to the container registry. The job also runs in pipelines on the default branch to build a base image that includes the GDK and GitLab components. This avoids building the entire image from scratch in merge requests. However, if the merge request includes changes to [certain GitLab components or code](https://gitlab.com/gitlab-org/gitlab/-/blob/24109c1a7ae1f29d4f6f1aeba3a13cbd8ea0e8e6/.gitlab/ci/rules.gitlab-ci.yml#L911) the job will rebuild the base image before building the image that will be used in the test jobs. #### child pipeline jobs The child pipeline consists of several stages that support E2E test execution. ##### test Jobs in the `test` stage perform following actions: 1. Start an instance of GDK by using the Docker image built by the [`build-gdk-image`](#build-gdk-image) job. 1. Run E2E tests against the running instance of GDK. #### report This stage is responsible for [allure test report](_index.md#allure-report) generation. ## Test Licenses For more information on the licenses these pipelines use, see [test licenses](https://gitlab-org.gitlab.io/quality/runbooks/test_licenses/). ## Adding new jobs to E2E test pipelines E2E test pipelines use dynamic scaling of jobs based on their runtime. To create a mapping between job definitions in pipeline definition YAML files and a particular test scenario, `scenario` classes are used. These classes are located in `qa/qa/scenario` folder. A typical job definition in one of the e2e test pipeline definition YAML files would look like: ```yaml my-new-test-job: # ... variables: QA_SCENARIO: Test::Integration::MyNewTestScenario ``` In this example: - `QA_SCENARIO: Test::Integration::MyNewTestScenario`: name of the scenario class that is passed to the `qa/bin/qa` test execution script. While the full class name would be `QA::Scenario::Test:Integration::MyNewTestScenario`, `QA::Scenario` is omitted to have shorted definitions. Considering example above, perform the following steps to create a new job: 1. Create a new scenario `my_new_job.rb` in the [`integration`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test/integration) directory of the [`e2e`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa) test framework. The scenario class should define a pipeline mapping that couples the scenario to a specific job in a specific pipeline type. If job was added to the [test-on-cng](#e2etest-on-cng) pipeline, this scenario would define RSpec tags that should be executed and pipeline mapping: ```ruby module QA module Scenario module Test module Integration class MyNewJob < Test::Instance::All tags :some_special_tag pipeline_mappings test_on_cng: %w[my-new-test-job] end end end end end ``` 1. Add the new job definition in the [`main.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/test-on-cng/main.gitlab-ci.yml) pipeline definition: ```yaml my-new-test-job: extends: - .cng-test variables: QA_SCENARIO: Test::Integration::MyNewTestScenario ``` Such a definition ensures that `my-new-test-job` has automatic parallel job scaling based on predefined runtime threshold.