VladPolskiy/gitlab-ce
1
0
Fork 0
mirror of https://github.com/gitlabhq/gitlabhq.git synced 2025-07-23 00:45:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
4310bd9b3d275c371bc26045ba52dead17a96260
gitlab-ce/doc/update/versions/gitlab_14_changes.md
GitLab Bot 4310bd9b3d Add latest changes from gitlab-org/gitlab@master
2023-08-30 09:10:38 +00:00

34 KiB
Raw Blame History

stage, group, info
stage group info
Systems Distribution To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments

GitLab 14 changes (FREE SELF)

This page contains upgrade information for minor and patch versions of GitLab 14. Ensure you review these instructions for all versions between your current version and your target version.

For more information about upgrading GitLab Helm Chart, see the release notes for 5.0.

14.10.0

  • Before upgrading to GitLab 14.10, you must already have the latest 14.9.Z installed on your instance. The upgrade to GitLab 14.10 executes a concurrent index drop of unneeded entries from the ci_job_artifacts database table. This could potentially run for multiple minutes, especially if the table has a lot of traffic and the migration is unable to acquire a lock. It is advised to let this process finish as restarting may result in data loss.

  • If you run external PostgreSQL, particularly AWS RDS, check you have a PostgreSQL bug fix to avoid the database crashing.

  • Upgrading to patch level 14.10.3 or later might encounter a one-hour timeout due to a long running database data change, if it was not completed while running GitLab 14.9.

    FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
(gitlab::database_migrations line 51) had an error:
[..]
Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:

    A workaround exists to complete the data change and the upgrade manually.

Linux package installations

  • In GitLab 14.10, Gitaly has introduced a new runtime directory. This directory is intended to hold all files and directories Gitaly needs to create at runtime to operate correctly. This includes, for example, internal sockets, the Git execution environment, or the temporary hooks directory.

    This new configuration can be set via gitaly['runtime_dir']. It replaces the old gitaly['internal_socket_dir'] configuration. If the internal socket directory is not explicitly configured, sockets will be created in the runtime directory.

    Support for gitaly['internal_socket_dir'] will be removed in 15.0.

14.9.0

  • Database changes made by the upgrade to GitLab 14.9 can take hours or days to complete on larger GitLab instances. These batched background migrations update whole database tables to ensure corresponding records in namespaces table for each record in projects table.

    After you upgrade to 14.9.0 or a later 14.9 patch version, batched background migrations must finish before you upgrade to a later version.

    If the migrations are not finished and you try to upgrade to a later version, you see errors like:

    Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':

    Or

    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource

  • GitLab 14.9.0 includes a background migration ResetDuplicateCiRunnersTokenValuesOnProjects that may remain stuck permanently in a pending state.

    To clean up this stuck job, run the following in the GitLab Rails Console:

    Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
  puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
end

  • If you run external PostgreSQL, particularly AWS RDS, check you have a PostgreSQL bug fix to avoid the database crashing.

14.8.0

  • If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the Critical Security Release: 14.8.2, 14.7.4, and 14.6.5 blog post. Updating to 14.8.2 or later resets runner registration tokens for your groups and projects.

  • The agent server for Kubernetes is enabled by default on Omnibus installations. If you run GitLab at scale, such as the reference architectures, you must disable the agent on the following server types, if the agent is not required.

    • Praefect
    • Gitaly
    • Sidekiq
    • Redis (if configured using redis['enable'] = true and not via roles)
    • Container registry
    • Any other server types based on roles(['application_role']), such as the GitLab Rails nodes

    The reference architectures have been updated with this configuration change and a specific role for standalone Redis servers.

    Steps to disable the agent:

    1. Add gitlab_kas['enable'] = false to gitlab.rb.
    2. If the server is already upgraded to 14.8, run gitlab-ctl reconfigure.

  • GitLab 14.8.0 includes a background migration PopulateTopicsNonPrivateProjectsCount that may remain stuck permanently in a pending state.

    To clean up this stuck job, run the following in the GitLab Rails Console:

    Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job|
  puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments)
end

  • If upgrading from a version earlier than 14.3.0, to avoid an issue with job retries, first upgrade to GitLab 14.7.x and make sure all batched migrations have finished.

  • If upgrading from version 14.3.0 or later, you might notice a failed batched migration named BackfillNamespaceIdForNamespaceRoute. You can ignore this. Retry it after you upgrade to version 14.9.x.

  • If you run external PostgreSQL, particularly AWS RDS, check you have a PostgreSQL bug fix to avoid the database crashing.

14.7.0

  • See LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2.

  • If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the Critical Security Release: 14.8.2, 14.7.4, and 14.6.5 blog post. Updating to 14.7.4 or later resets runner registration tokens for your groups and projects.

  • GitLab 14.7 introduced a change where Gitaly expects persistent files in the /tmp directory. When using the noatime mount option on /tmp in a node running Gitaly, most Linux distributions run into an issue with Git server hooks getting deleted. These conditions are present in the default Amazon Linux configuration.

    If your Linux distribution manages files in /tmp with the tmpfiles.d service, you can override the behavior of tmpfiles.d for the Gitaly files and avoid this issue:

    sudo printf "x /tmp/gitaly-%s-*\n" hooks git-exec-path >/etc/tmpfiles.d/gitaly-workaround.conf

    This issue is fixed in GitLab 14.10 and later when using the Gitaly runtime directory to specify a location to store persistent files.

Linux package installations

  • In GitLab 14.8, we are upgrading Redis from 6.0.16 to 6.2.6. This upgrade is expected to be fully backwards compatible.

    If your instance has Redis HA with Sentinel, follow the upgrade steps documented in Redis HA (using Sentinel).

14.6.0

14.5.0

  • When make is run, Gitaly builds are now created in _build/bin and no longer in the root directory of the source directory. If you are using a self-compiled installation, update paths to these binaries in your systemd unit files or init scripts by following the documentation.

  • Connections between Workhorse and Gitaly use the Gitaly backchannel protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, Workhorse can no longer connect. As a workaround, disable the temporary workhorse_use_sidechannel feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, go to this issue.

  • In 14.1 we introduced a background migration that changes how we store merge request diff commits, to significantly reduce the amount of storage needed. In 14.5 we introduce a set of migrations that wrap up this process by making sure that all remaining jobs over the merge_request_diff_commits table are completed. These jobs have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. However, if there are remaining jobs or you haven't already upgraded to 14.1, the deployment may take multiple hours to complete.

    All merge request diff commits automatically incorporate these changes, and there are no additional requirements to perform the upgrade. Existing data in the merge_request_diff_commits table remains unpacked until you run VACUUM FULL merge_request_diff_commits. However, the VACUUM FULL operation locks and rewrites the entire merge_request_diff_commits table, so the operation takes some time to complete and it blocks access to this table until the end of the process. We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. The time it takes to complete depends on the size of the table, which can be obtained by using select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));.

    For more information, refer to this issue.

  • GitLab 14.5.0 includes a background migration UpdateVulnerabilityOccurrencesLocation that may remain stuck permanently in a pending state when the instance lacks records that match the migration's target.

    To clean up this stuck job, run the following in the GitLab Rails Console:

    Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job|
  puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments)
end

  • Upgrading to 14.5 (or later) might encounter a one hour timeout owing to a long running database data change.

    FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
(gitlab::database_migrations line 51) had an error:
[..]
Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:

    There is a workaround to complete the data change and the upgrade manually

  • As part of enabling real-time issue assignees, Action Cable is now enabled by default. For self-compiled (source) installations, config/cable.yml is required to be present.

    Configure this by running:

    cd /home/git/gitlab
sudo -u git -H cp config/cable.yml.example config/cable.yml

# Change the Redis socket path if you are not using the default Debian / Ubuntu configuration
sudo -u git -H editor config/cable.yml

14.4.4

  • For zero-downtime upgrades on a GitLab cluster with separate Web and API nodes, you must enable the paginated_tree_graphql_query feature flag before upgrading GitLab Web nodes to 14.4. This is because we enabled paginated_tree_graphql_query by default in 14.4, so if GitLab UI is on 14.4 and its API is on 14.3, the frontend has this feature enabled but the backend has it disabled. This results in the following error:

    bundle.esm.js:63 Uncaught (in promise) Error: GraphQL error: Field 'paginatedTree' doesn't exist on type 'Repository'

14.4.0

Linux package installations

  • In GitLab 14.4, the provided Grafana version is 7.5, this is a downgrade from the Grafana 8.1 version introduced in GitLab 14.3. This was reverted to an Apache-licensed Grafana release to allow time to consider the implications of the newer AGPL-licensed releases.

    Users that have customized their Grafana install with plugins or library panels may experience errors in Grafana after the downgrade. If the errors persist after a Grafana restart you may need to reset the Grafana db and re-add the customizations. The Grafana database can be reset with sudo gitlab-ctl reset-grafana.

14.3.0

  • Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later.

  • Ensure batched background migrations finish before upgrading to 14.3.Z from earlier GitLab 14 releases.

  • Ruby 2.7.4 is required. Refer to the Ruby installation instructions for how to proceed.

  • GitLab 14.3.0 contains post-deployment migrations to address Primary Key overflow risk for tables with an integer PK for the tables listed below:

    If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful:

    # For Omnibus GitLab
sudo gitlab-rake db:migrate

# For source installations
sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production

  • After upgrading to 14.3, ensure that all the MigrateMergeRequestDiffCommitUsers background migration jobs have completed before continuing with upgrading to GitLab 14.5 or later. This is especially important if your GitLab instance has a large merge_request_diff_commits table. Any pending MigrateMergeRequestDiffCommitUsers background migration jobs are foregrounded in GitLab 14.5, and may take a long time to complete. You can check the count of pending jobs for MigrateMergeRequestDiffCommitUsers by using the PostgreSQL console (or sudo gitlab-psql):

    select status, count(*) from background_migration_jobs
where class_name = 'MigrateMergeRequestDiffCommitUsers' group by status;

    As jobs are completed, the database records change from 0 (pending) to 1. If the number of pending jobs doesn't decrease after a while, it's possible that the MigrateMergeRequestDiffCommitUsers background migration jobs have failed. You can check for errors in the Sidekiq logs:

    sudo grep MigrateMergeRequestDiffCommitUsers /var/log/gitlab/sidekiq/current | grep -i error

    If needed, you can attempt to run the MigrateMergeRequestDiffCommitUsers background migration jobs manually in the GitLab Rails Console. This can be done using Sidekiq asynchronously, or by using a Rails process directly:

    • Using Sidekiq to schedule jobs asynchronously:

      # For the first run, only attempt to execute 1 migration. If successful, increase
# the limit for subsequent runs
limit = 1

jobs = Gitlab::Database::BackgroundMigrationJob.for_migration_class('MigrateMergeRequestDiffCommitUsers').pending.to_a

pp "#{jobs.length} jobs remaining"

jobs.first(limit).each do |job|
  BackgroundMigrationWorker.perform_in(5.minutes, 'MigrateMergeRequestDiffCommitUsers', job.arguments)
end

      NOTE: The queued jobs can be monitored using the Sidekiq admin panel, which can be accessed at the /admin/sidekiq endpoint URI.

    • Using a Rails process to run jobs synchronously:

      def process(concurrency: 1)
  queue = Queue.new

  Gitlab::Database::BackgroundMigrationJob
    .where(class_name: 'MigrateMergeRequestDiffCommitUsers', status: 0)
    .each { |job| queue << job }

  concurrency
    .times
    .map do
      Thread.new do
        Thread.abort_on_exception = true

        loop do
          job = queue.pop(true)
          time = Benchmark.measure do
            Gitlab::BackgroundMigration::MigrateMergeRequestDiffCommitUsers
              .new
              .perform(*job.arguments)
          end

          puts "#{job.id} finished in #{time.real.round(2)} seconds"
        rescue ThreadError
          break
        end
      end
    end
    .each(&:join)
end

ActiveRecord::Base.logger.level = Logger::ERROR
process

      NOTE: When using Rails to execute these background migrations synchronously, make sure that the machine running the process has sufficient resources to handle the task. If the process gets terminated, it's likely due to insufficient memory available. If your SSH session times out after a while, it might be necessary to run the previous code by using a terminal multiplexer like screen or tmux.

  • See Maintenance mode issue in GitLab 13.9 to 14.4.

  • You may see the following error when setting up two factor authentication (2FA) for accounts that authenticate using an LDAP password:

    You must provide a valid current password

  • If you encounter the error, I18n::InvalidLocale: :en is not a valid locale, when starting the application, follow the patching process. Use 122978 as the mr_iid.

14.2.0

14.1.0

  • Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later but can upgrade to 14.1.Z.

    It is not required for instances already running 14.0.5 (or later) to stop at 14.1.Z. 14.1 is included on the upgrade path for the broadest compatibility with self-managed installations, and ensure 14.0.0-14.0.4 installations do not encounter issues with batched background migrations.

  • Upgrading to GitLab 14.5 (or later) may take a lot longer if you do not upgrade to at least 14.1 first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must run the migration in the foreground and therefore take a lot longer to complete.

  • See Maintenance mode issue in GitLab 13.9 to 14.4.

  • If you encounter the error, I18n::InvalidLocale: :en is not a valid locale, when starting the application, follow the patching process. Use 123475 as the mr_iid.

14.0.0

Prerequisites:

Long running batched background database migrations:

  • Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. These batched background migrations update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or later.

  • Due to an issue where BatchedBackgroundMigrationWorkers were not working for self-managed instances, a fix was created that requires an update to at least 14.0.5. The fix was also released in 14.1.0.

    After you update to 14.0.5 or a later 14.0 patch version, batched background migrations must finish before you upgrade to a later version.

    If the migrations are not finished and you try to upgrade to a later version, you see an error like:

    Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':

    See how to resolve this error.

Other issues:

Linux package installations

  • In GitLab 13.0, sidekiq-cluster was enabled by default and the sidekiq service ran sidekiq-cluster under the hood. However, users could control this behavior using sidekiq['cluster'] setting to run Sidekiq directly instead. Users could also run sidekiq-cluster separately using the various sidekiq_cluster[*] settings available in gitlab.rb. However these features were deprecated and are now being removed.

    Starting with GitLab 14.0, sidekiq-cluster becomes the only way to run Sidekiq in Linux package installations. As part of this process, support for the following settings in gitlab.rb is being removed:

    • sidekiq['cluster'] setting. Sidekiq can only be run using sidekiq-cluster now.
    • sidekiq_cluster[*] settings. They should be set via respective sidekiq[*] counterparts.
    • sidekiq['concurrency'] setting. The limits should be controlled using the two settings sidekiq['min_concurrency'] and sidekiq['max_concurrency'].

  • In GitLab 13.0, Puma became the default web server for GitLab, but users were still able to continue using Unicorn if needed. Starting with GitLab 14.0, Unicorn is no longer supported as a webserver for GitLab and is no longer shipped with the Linux package.

    Users must migrate to Puma following the documentation to upgrade to GitLab 14.0.

  • The Consul version has been updated from 1.6.10 to 1.9.6 for Geo and multi-node PostgreSQL installs. Its important that Consul nodes be upgraded and restarted one at a time.

    For more information, see Upgrade the Consul nodes.

  • Starting with GitLab 14.0, GitLab automatically generates a password for initial administrator user (root) and stores this value to /etc/gitlab/initial_root_password.

    For more information, see Set up the initial password.

  • The binaries for PostgreSQL 11 and repmgr have been removed. Prior to upgrading, administrators of Linux package installations must:

    1. Ensure the installation is using PostgreSQL 12.
    2. If using repmgr, convert to using patroni.

  • Two configuration options for Redis were deprecated in GitLab 13 and removed in GitLab 14:

    • redis_slave_role is replaced with redis_replica_role.
    • redis['client_output_buffer_limit_slave'] is replaced with redis['client_output_buffer_limit_replica'].

    Redis Cache nodes being upgraded from GitLab 13.12 to 14.0 that still refer to redis_slave_role in gitlab.rb will encounter an error in the output of gitlab-ctl reconfigure:

    There was an error running gitlab-ctl reconfigure:

The following invalid roles have been set in 'roles': redis_slave_role

Upgrading to later 14.Y releases