同时工作时如何在gitlab-runner上共享缓存

发布于 2025-02-10 15:57:49 字数 2311 浏览 0 评论 0原文

我有一个使用gitlab-ci测试的代码。 以前,我正在使用共享的gitlab-runner,一切都很好。

我最近迁移到了我公司托管的GitLab运营商,并且完全相同的代码不再通过Gitlab-Ci管道有效。 提出的错误是:

Executing "step_script" stage of the job script 00:05
Using docker image sha256:xxx for registry.gitlab.com/me/myproject/myimage with digest registry.gitlab.com/me/myproject/myimage@sha256:xxx ...
$ cd build
/bin/bash: line 125: cd: build: No such file or directory

我调查并发现有关共享缓存的问题。 的确,我的缓存是在同一阶段同时工作的不同作业之间共享的,但是在共享的跑步者中,这不是问题。 因此,我要切换Gitlab-Runner配置,以便没有并发作业(即并发= 1在文件中/etc/gitlab-runner/config.toml) ,导致没有缓存问题。 我得出的结论是,问题更多地是关于Gitlab-runner配置。 我当前的配置文件/etc/gitlab-runner/config.toml is:

concurrent = 8
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "MY-RUNNER-NAME"
url = "https://gitlab.com/"
token = "MY-TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
   Shared = true
   [runners.cache.s3]
   [runners.cache.gcs]
   [runners.cache.azure]
[runners.docker]
   tls_verify = false
   image = "IMAGE:latest"
   privileged = true
   disable_entrypoint_overwrite = false
   oom_kill_disable = false
   disable_cache = false
   volumes = ["/cache"]
   shm_size = 0

那么如何配置我的跑步者以允许并发作业共享其caches(或解决此冲突)?

在这里编辑

是我的gitlab-ci.yml文件:

# GitLab-CI configuration file.

stages:
  - prepare
  - build
  - test

.allConfigs:
  parallel:
    matrix:
      - DISTRIBUTION: ['alpine']
        COMPILER: ['gcc'] 
  image: ${CI_REGISTRY_IMAGE}/${DISTRIBUTION}-${COMPILER}
  cache:
    key: ${DISTRIBUTION}-${COMPILER}-key
    paths:
      - build/
      - external/
  

prepare:
  stage: prepare
  script:
    - git reset --hard HEAD 
    - git clean -d -f -x
    - mkdir build
    - cd build
    - cmake ../
    - make update
  extends: .allConfigs


build-exe:
  stage: build
  script:
    - cd build
    - cmake ../ #-DTESTING=OFF
    - make main
    - make install
  extends: .allConfigs


test-integration:
  stage: test
  script:
    - cd build
    - cmake ../ -DTESTING=ON
    - make integrationTest
  extends: .allConfigs
  
test-unit:
  stage: test
  script:
    - cd build
    - cmake ../ -DTESTING=ON
    - make unitTest
  extends: .allConfigs

当2个测试作业同时执行时,提到的错误出现。

I have a code that I test using gitlab-ci.
Previously, I was using a shared gitlab-runner, and everything was fine.

I recently migrated to a gitlab-runner hosted by my company, and the exact same code is no longer valid through the gitlab-ci pipeline.
The error raised was:

Executing "step_script" stage of the job script 00:05
Using docker image sha256:xxx for registry.gitlab.com/me/myproject/myimage with digest registry.gitlab.com/me/myproject/myimage@sha256:xxx ...
$ cd build
/bin/bash: line 125: cd: build: No such file or directory

I investigate and found a problem about the cache shared.
Indeed, my cache is shared between different jobs working concurrently at the same stage, but on shared runner freely available it was not an issue.
So, I switch my gitlab-runner configuration in order to have no concurrent jobs (ie. concurrent = 1 in the file /etc/gitlab-runner/config.toml), resulting no cache problem anymore.
I conclude that the issue is more about the gitlab-runner configuration.
My current configuration file /etc/gitlab-runner/config.toml is:

concurrent = 8
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "MY-RUNNER-NAME"
url = "https://gitlab.com/"
token = "MY-TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
   Shared = true
   [runners.cache.s3]
   [runners.cache.gcs]
   [runners.cache.azure]
[runners.docker]
   tls_verify = false
   image = "IMAGE:latest"
   privileged = true
   disable_entrypoint_overwrite = false
   oom_kill_disable = false
   disable_cache = false
   volumes = ["/cache"]
   shm_size = 0

So how to configure my runner to allow concurrent jobs to share their caches (or solve this conflict) ?

EDIT

Here is my gitlab-ci.yml file:

# GitLab-CI configuration file.

stages:
  - prepare
  - build
  - test

.allConfigs:
  parallel:
    matrix:
      - DISTRIBUTION: ['alpine']
        COMPILER: ['gcc'] 
  image: ${CI_REGISTRY_IMAGE}/${DISTRIBUTION}-${COMPILER}
  cache:
    key: ${DISTRIBUTION}-${COMPILER}-key
    paths:
      - build/
      - external/
  

prepare:
  stage: prepare
  script:
    - git reset --hard HEAD 
    - git clean -d -f -x
    - mkdir build
    - cd build
    - cmake ../
    - make update
  extends: .allConfigs


build-exe:
  stage: build
  script:
    - cd build
    - cmake ../ #-DTESTING=OFF
    - make main
    - make install
  extends: .allConfigs


test-integration:
  stage: test
  script:
    - cd build
    - cmake ../ -DTESTING=ON
    - make integrationTest
  extends: .allConfigs
  
test-unit:
  stage: test
  script:
    - cd build
    - cmake ../ -DTESTING=ON
    - make unitTest
  extends: .allConfigs

The error mentioned appears when the 2 test jobs are executing concurrently.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文