VladPolskiy/gitlab-foss
1
0
Fork 0
mirror of https://gitlab.com/gitlab-org/gitlab-foss.git synced 2025-08-15 21:39:00 +00:00
Code Issues Packages Projects Releases 1 Wiki Activity

Add latest changes from gitlab-org/gitlab@master

Browse Source
This commit is contained in:
GitLab Bot
2025-01-03 12:35:11 +00:00
parent 60165a7271
commit d9b9e96d7c
46 changed files with 924 additions and 355 deletions
Download Patch File Download Diff File Expand all files Collapse all files

219
doc/development/cloud_connector/configuration.md
View File

@ -13,7 +13,175 @@ A GitLab Rails instance accesses backend services uses a [Cloud Connector Servic
The Cloud Connector **JWT** contains a custom claim, which represents the list of access scopes that define which features, or unit primitives, this token is valid for.
## Configuration structure
## Unit Primitives and Configuration
As per the [Architecture Decision Record (ADR) 003](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/cloud_connector/decisions/003_unit_primitives/),
we have decided to extract the configuration of unit primitives to an external library called [`gitlab-cloud-connector`](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector).
This library serves as the Single Source of Truth (SSoT) for all Cloud Connector configurations and is available as both a Ruby gem and a Python package.
### Configuration format and structure
The configuration in `gitlab-cloud-connector` follows this structure:
```shell
config
├─ unit_primitives/
│ ├─ duo_chat.yml
│ └─ ...
├─ backend_services/
│ ├─ ai_gateway.yml
│ └─ ...
├─ add_ons/
│ ├─ duo_pro.yml
│ └─ ...
├─ services/
│ ├─ duo_chat.yml
│ └─ ...
└─ license_types/
├─ premium.yml
└─ ...
```
#### Unit primitive configuration
We have a YAML file per unit primitive. It contains information on how this unit primitive is bundled with add-ons and license types, and other metadata.
The configuration for each unit primitive adhere to the following schema.
##### Required Fields
| Field | Type | Description |
|-------|------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| `name` | string | Unit primitive name in `snake_case` format (lowercase letters, numbers, underscores). Should follow `$VERB_$NOUN` pattern (for example, `explain_vulnerability`). |
| `description` | string | Description of the unit primitive's purpose and functionality. |
| `group` | string | Engineering group that owns the unit primitive (for example, "group::cloud connector"). |
| `feature_category` | string | Feature category classification (see [categories](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/categories.yml) |
| `documentation_url` | string | URL to the unit primitive's documentation |
##### Optional Fields
| Field | Type | Description |
|-------|------|------------------------------------------------------|
| `milestone` | string | GitLab milestone that introduced the unit primitive. |
| `introduced_by_url` | string | Merge request URL that introduced the unit primitive. |
| `unit_primitive_issue_url` | string | Issue URL proposing the unit primitive introduction. |
| `deprecated_by_url` | string | Merge request URL that deprecated the unit primitive. |
| `deprecation_message` | string | Explanation of deprecation context and reasons. |
| `cut_off_date` | datetime | UTC timestamp when free access ends (if applicable). |
| `min_gitlab_version` | string | Minimum required GitLab version (for example, `17.8`). |
| `min_gitlab_version_for_free_access` | string | Minimum version for free access period (for example, `17.8`). |
##### Access Control Fields
| Field | Type | Description |
|-------|------|-------------------------------------------------------------------------|
| `license_types` | array[string] | GitLab license types that can access this primitive. Possible values must match the name field in corresponding files under `config/license_types` (for example, `premium`).|
| `backend_services` | array[string] | Backend services hosting this primitive. Possible values must match the name field in corresponding files under `config/backend_services` (for example, `ai_gateway`).|
| `add_ons` | array[string] | Add-on products including this primitive. Possible values must match the name field in corresponding files under `config/add_ons` (for example, `duo_pro`). |
Example unit primitive configuration:
```yaml
# config/unit_primitives/new_feature.yml
---
name: new_feature
description: Description of the new feature
cut_off_date: 2024-10-17T00:00:00+00:00 # Optional, set if not free
min_gitlab_version: '16.9'
min_gitlab_version_for_free_access: '16.8'
group: group::your_group
feature_category: your_category
documentation_url: https://docs.gitlab.com/ee/path/to/docs
backend_services:
- ai_gateway
add_ons:
- duo_pro
- duo_enterprise
license_types:
- premium
- ultimate
```
#### Related Configurations
##### Backend Services
Each backend service must have its own YAML configuration under `config/backend_services`. For example:
```yaml
# config/backend_services/ai_gateway.yml
---
name: ai_gateway
project_url: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist
group: group::ai framework
jwt_aud: gitlab-ai-gateway
```
##### Add-ons
Each add-on must have its own YAML configuration under `config/add_ons`. For example:
```yaml
# config/add_ons/duo_pro.yml
---
name: duo_pro
```
##### License Types
Each license type must have its own YAML configuration under `config/license_types`. For example:
```yaml
# config/license_types/premium.yml
---
name: premium
```
### Backward compatibility
To support backward compatibility for customers running older GitLab versions and with the old [legacy structure](#legacy-structure), we provide a mapping from the new to old format,
and soon to be deprecated "service" abstraction.
#### Service configuration
| Field | Type | Description |
|-------|------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `name` | string | The unique name of the service, consisting of lowercase alphanumeric characters and underscores. |
| `basic_unit_primitive` | string | The most fundamental unit primitive representing key configuration values like `cut_off_date` and `min_gitlab_version`. If not set, the first unit primitive in the `unit_primitives` list is used. Used to derive these shared properties across the service. |
| `gitlab_realm` | array[string] | An array of environments where the service is available. Possible values: `gitlab-com`, `self-managed`. |
| `description` | string | A brief description of the service. |
| `unit_primitives` | array[string] | An array of unit primitives associated with the service. |
Example of a new service mapping configuration:
```yaml
# config/services/duo_chat.yml
---
name: duo_chat
basic_unit_primitive: duo_chat
gitlab_realm:
- gitlab-com
- self-managed
unit_primitives:
- ask_build
- ask_commit
- ask_epic
- ask_issue
- ask_merge_request
- documentation_search
- duo_chat
- explain_code
- fix_code
- include_dependency_context
- include_file_context
- include_issue_context
- include_local_git_context
- include_merge_request_context
- include_snippet_context
- refactor_code
- write_tests
```
### Legacy structure
The information about how paid features are bundled into GitLab tiers and add-ons is configured and stored in a YAML file:
@ -38,41 +206,34 @@ services:
- documentation_search
```
### Terms
| Field | Type | Description |
|-------|------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `unit_primitives` | array[string] | The smallest logical features that a permission or access scope can govern. Should follow `$VERB_$NOUN` naming pattern (for example, `explain_vulnerability`). |
| `service` | string | The service name that delivers the feature. Can be standalone or part of an existing service (for example, `duo_chat`). |
| `bundled_with` | object | Map of add-ons that include this feature. A feature can be bundled with multiple add-ons (for example, `duo_pro`, `duo_enterprise`). |
| `cut_off_date` | datetime | UTC timestamp when free access ends. If not set, feature remains free. |
| `min_gitlab_version` | string | Minimum required GitLab version (for example, `17.8`). If not set, available for all versions. |
| `min_gitlab_version_for_free_access` | string | Minimum version for free access period (for example, `17.8`). If not set, available for all versions. |
| `backend` | string | Name of the backend service hosting this feature, used as token audience claim (for example, `gitlab-ai-gateway`). |
The configuration contains the following information:
#### GitLab.com configuration
- `unit_primitives`: Each unit primitive represents the smallest logical feature that a permission/access scope can govern.
It represents the actual feature,
and the value of this attribute is used as a scope when Service Access token is being issued.
Because the GitLab.com deployment enjoys special trust, it can [self-sign and create Instance JSON Web Tokens (IJWTs)](architecture.md#gitlabcom) for every request to a Cloud Connector feature.
The recommended naming convention for the new `unit_primitive` should be `$VERB_$NOUN`. Example: `explain_vulnerability`
- `service`: The feature, or UP, can be delivered as part of the existing service (Duo Chat) or as a stand-alone service.
Example: The **documentation_search** represent an unit primitive that is delivered as part of the Duo Chat service.
- `bundled_with`: Represents the list of add-ons. Each unit primitive is bundled with add-on.
Example: Code Suggestions and Duo Chat were two features sold together under the `DUO PRO` add-on.
Same unit primitive can be bundled with multiple add-ons.
Example: Code Suggestions could be available with both the `DUO PRO` and `DUO ENTERPRISE` add-ons.
- `cut_off_date`: Represents the cut-off date when the feature is no longer available for free (Experimental).
If it's not set, the feature is available for free by default.
- `min_gitlab_version`: Represents the minimum required GitLab version to use the feature.
If it's not set, the feature is available for all GitLab versions.
- `min_gitlab_version_for_free_access`: Represents the minimum required GitLab version to use the service during the free access period.
If it's not set, the service is available for all GitLab versions.
- `backend`: The name of the backend serving this feature. The name is used as a token audience claim.
Example: `gitlab-ai-gateway`.
To issue tokens with the proper scopes, GitLab.com needs access to the configuration.
## CustomersDot configuration
Configuration is stored in the `gitlab-cloud-connector` gem using a [unit primitive configuration format and structure](#configuration-format-and-structure).
GitLab.com still uses the [legacy `available_services` structure](#legacy-structure).
The `Gitlab::CloudConnector::AvailableServicesGenerator` generates legacy structure for compatibility.
To add a new unit primitive, follow [Register new feature for Self-Managed, Dedicated and GitLab.com customers](index.md#register-new-feature-for-self-managed-dedicated-and-gitlabcom-customers)
#### CustomersDot configuration
For GitLab Dedicated and self-managed GitLab instances we are delegating trust to the CustomersDot, the access token issuer.
The configuration is located in [`cloud_connector.yml`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml),
and represents an almost exact copy of the GitLab.com configuration.
## GitLab.com configuration
Because the GitLab.com deployment enjoys special trust, it has the advantage of being able to [self-sign and create IJWTs](architecture.md#gitlabcom) for every request to a Cloud Connector feature.
To issue tokens with the proper scopes, GitLab.com needs access to the configuration.
The configuration is located in [`access_data.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/cloud_connector/access_data.yml),
and it's an almost exact copy of the [CustomersDot configuration](#customersdot-configuration).
We are in the process of removing `cloud_connector.yml` as part of our effort to use the `gitlab-cloud-connector` gem as the Single Source of Truth (SSoT) for unit primitive configuration.
To add a new unit primitive during this transition period, follow the [Transition period from old configuration process](index.md#transition-from-old-configuration).

178
doc/development/cloud_connector/index.md
View File

@ -41,157 +41,77 @@ because we do not currently have interfaces in place to self-service this.
- For GitLab Dedicated and self-managed GitLab instances, the CustomersDot is the **JWT issuer**.
- For GitLab.com deployment, GitLab.com is the **JWT issuer**, because it's able to [self-sign and create JWTs](architecture.md#gitlabcom) for every request to a Cloud Connector feature.
##### Register new feature for Self-Managed, Dedicated and GitLab.com customers
#### Register new feature for Self-Managed, Dedicated and GitLab.com customers
Because the CustomersDot is the **JWT issuer** for GitLab Dedicated and self-managed GitLab instances,
to register new feature, we need to update [CustomersDot configuration](configuration.md#customersdot-configuration).
You must register the new feature as a unit primitive in the [`gitlab-cloud-connector`](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector) repository.
This repository serves as the Single Source of Truth (SSoT) for all Cloud Connector configurations.
Since the GitLab.com is the **JWT issuer**, because it's able to [self-sign and create JWTs](architecture.md#gitlabcom),
to register new feature, we need to update [GitLab.com configuration](configuration.md#gitlabcom-configuration) as well.
To register a new feature:
So, in total, you need to open two separate merge requests: one to [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) and another to [GitLab.com](https://gitlab.com/gitlab-org/gitlab). No synchronization between when they are merged is needed.
1. Create a new YAML file in the `config/unit_primitives/` directory of the `gitlab-cloud-connector` repository.
1. Define the unit primitive configuration, and ensure you follow the [schema](configuration.md#unit-primitive-configuration).
The new feature needs to be added as a new **unit primitive**, so you can include it in the **JWT** (Service Access token).
For example, to add a new feature called `new_feature`:
The new feature is delivered in several ways:
- Some features (such as `documentation_search`) are delivered as part of the existing (`Duo Chat`) service.
- Others are delivered as a stand-alone service (`code_suggestions`).
- It is possible to move existing features (`explain_vulnerability`) to become a part of an existing (`Duo Chat`) service.
- It is also possible to deliver a feature to be both part of a Duo Chat service, and also to act as a stand-alone service.
**Terms**:
- **feature** - `documentation_search`, registered as new **unit primitive** so it could be included in **JWT**.
- **existing service** - `Duo Chat` - The feature is delivered as part of `Duo Chat` service.
- **stand-alone service** - `code_suggestions`, the feature is accessible as a separate service,
probably through its own interface.
- **backend service** - `Ai Gateway`, the backend where the feature is hosted.
##### Register a new feature to the existing service (DuoChat)
The new feature is delivered as part of an existing service (Duo Chat).
As an example, the new feature is:
- Named `new_feature`
- Delivered as part of the existing `duo chat` service and accessible only through the Duo Chat window
- Sold only through the `duo_enterprise` add-on
Update [`cloud_connector.yml`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml) (for Self-Managed and Dedicated) and [`access_data.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/cloud_connector/access_data.yml) (for GitLab.com),
by registering the `new_feature_up` **unit primitive** under the `duo chat` service:
```diff
defaults: &defaults
services:
duo_chat:
cut_off_date: 2024-7-15 00:00:00 UTC
min_gitlab_version: 16.8
min_gitlab_version_for_free_access: 16.8
bundled_with:
duo_pro:
unit_primitives:
- duo_chat
- documentation_search
duo_enterprise:
unit_primitives:
- duo_chat
- documentation_search
+ - new_feature_up
```yaml
# config/unit_primitives/new_feature.yml
---
name: new_feature
description: Description of the new feature
cut_off_date: 2024-10-17T00:00:00+00:00 # Optional, set if not free
min_gitlab_version: '16.9'
min_gitlab_version_for_free_access: '16.8' # Optional
group: group::your_group
feature_category: your_category
documentation_url: https://docs.gitlab.com/ee/path/to/docs
backend_services:
- ai_gateway
add_ons:
- duo_pro
- duo_enterprise
license_types:
- premium
- ultimate
```
- The feature is delivered as part of existing service (Duo Chat), so we are registering new `new_feature_up` unit primitive under the `duo_chat` service.
- The feature is sold only through `duo_enterprise`, so we need to add `new_feature_up` as a unit primitive, under
**bundled with** only for `duo_enterprise` add-on.
##### Backward Compatibility
See [configuration](configuration.md) for more information about configuration structure.
For backward compatibility where instances are still using the old [legacy structure](configuration.md#legacy-structure), consider adding your unit primitive to the [service configuration](configuration.md#service-configuration) as well.
NOTE:
Currently, it is not supported to set `cut_off_date` or `min_gitlab_version` per unit primitive if the unit primitive is delivered as part of the existing service (`Duo Chat`).
[Issue 465221](https://gitlab.com/gitlab-org/gitlab/-/issues/465221) proposes restructuring the current configuration structure and adding `cut_off_date` support per unit primitive.
- If the unit primitive is a stand-alone feature, no further changes are needed, and the service with the same name is generated automatically.
- If the unit primitive is delivered as part of existing service like `duo_chat`, `self_hosted_models` or `vertex_ai_proxy`, add the unit primitive to the desired service in the [`config/services`](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/tree/main/config/services) directory.
##### Register a new stand-alone service
##### Deployment process
The new feature is delivered as stand-alone service.
Follow our [release checklist](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/blob/main/.gitlab/merge_request_templates/Release.md#checklist) for publishing the new version of the library and using it in GitLab project.
As an example, the new feature is:
#### Transition from old configuration
- Named `new_feature`
- Delivered as a stand-alone service and not part of the `Duo Chat` service.
- Sold through both the `duo_pro` and `duo_enterprise` add-ons.
- Should not be accessible for free.
- Minimum required GitLab version is 17.1.
We are transitioning away from separate [CustomersDot](configuration.md#customersdot-configuration) and [GitLab.com](configuration.md#gitlabcom-configuration) configurations, as outlined in our [ADR-003 Migration Path](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/cloud_connector/decisions/003_unit_primitives/).
Update [`cloud_connector.yml`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml) (for Self-Managed and Dedicated) and [`access_data.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/cloud_connector/access_data.yml) (for GitLab.com)
by registering the `new_feature` as a `service`. See [configuration](configuration.md).
##### Current state and ongoing changes
```diff
defaults: &defaults
services:
+ new_feature:
+ cut_off_date: 2024-1-1 00:00:00 UTC
+ min_gitlab_version: 17.1
+ bundled_with:
+ duo_pro:
+ unit_primitives:
+ - new_feature_up
+ duo_enterprise:
+ unit_primitives:
+ - new_feature_up
```
- New features are added as **unit primitives** in the [`gitlab-cloud-connector` configuration](configuration.md#configuration-format-and-structure).
- Migration to the `gitlab-cloud-connector` gem as the Single Source of Truth (SSoT) is in progress in [epic 15949](https://gitlab.com/groups/gitlab-org/-/epics/15949).
- The feature is accessible as a new stand-alone service, probably through its own interface,
so we are registering a new `new_feature` service.
- The feature should not be accessible for free, so we need to set the `cut_off_date` in the past.
- The feature is sold through both `duo_pro` and `duo_enterprise`, so we need to add `new_feature_up`
as a unit primitive, under **bundled with** for both `duo_pro` and `duo_enterprise` add-ons.
- `access_data.yml` was removed in [issue 507518](https://gitlab.com/gitlab-org/gitlab/-/issues/507518).
- `cloud_connector.yml` file is slated for deprecation and removal, as detailed in [issue 11268](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268).
See [configuration](configuration.md) for more information about configuration structure.
##### Transition period guidelines
##### Register a feature as combination of existing service and stand-alone feature
- Maintain both [new](configuration.md#configuration-format-and-structure) and [CustomersDot configuration](configuration.md#customersdot-configuration) configurations.
The new feature is accessible through an existing service (`Duo Chat`) and also through its own user interface.
##### Register new features
As an example, the new feature is:
###### Process
- Named `new_feature`.
- Delivered as part of the existing `duo chat` service and accessible only through the Duo Chat window.
- Also delivered as a stand-alone service. That is, it is also used outside of Duo Chat through its own interface.
- Sold only through the `duo_enterprise` add-on.
After adding a [new unit primitive to `gitlab-cloud-connector`](#register-new-feature-for-self-managed-dedicated-and-gitlabcom-customers),
open a separate merge request to update [CustomersDot configuration](configuration.md#customersdot-configuration) for GitLab Dedicated and self-managed instances.
Update [`cloud_connector.yml`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml) (for Self-Managed and Dedicated) and [`access_data.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/cloud_connector/access_data.yml) (for GitLab.com):
```diff
defaults: &defaults
services:
+ new_feature:
+ bundled_with:
+ duo_enterprise:
+ unit_primitives:
+ - new_feature
duo_chat:
bundled_with:
duo_pro:
unit_primitives:
- duo_chat
- documentation_search
duo_enterprise:
unit_primitives:
- duo_chat
- documentation_search
+ - new_feature_up
```
- The feature is delivered as part of an existing service (Duo Chat), so we are registering
new `new_feature_up` unit primitive under the `duo_chat` service.
- The feature is also delivered as a stand-alone service, so we need to register `new_feature` as a new `service`.
- The feature is sold only through `duo_enterprise`, so we need to add `new_feature_up` as a unit primitive, under
**bundled with** only for `duo_enterprise` add-on for both services.
See [configuration](configuration.md) for more information about configuration structure.
1. Make sure that merge request for adding [new unit primitive to `gitlab-cloud-connector`](#register-new-feature-for-self-managed-dedicated-and-gitlabcom-customers) has been merged.
1. Download the latest generated [`cloud-connector.yml` file](https://gitlab.com/api/v4/projects/58733651/jobs/artifacts/main/raw/config/cloud_connector.yml?job=generate_cloud_connector_yml)
1. Replace the [`cloud-connector.yml` file](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml)
1. Ensure that the generated file reflects the changes you have made. Additionally, carefully verify that no changes have been inadvertently removed in the file.
#### Implement Permission checks in GitLab Rails