Skip to content

Blog

Setup docker in Github actions

Building and publishing docker images

.github/workflows/docker.yml
name: Create and publish a Docker image

on:
  push:
    branches: ['main']

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push-image:
    runs-on: ubuntu-latest

    # GITHUB_TOKEN permissions
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - name: Setup Buildx
        uses: docker/setup-buildx-action@v2
      - name: Log in to the Container registry
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Run docker containers in Github actions

Use container keyword

Running jobs in a container in GitHub actions.

.github/workflows/test-container.yml
name: container
on: push

jobs:
  node-docker:
    runs-on: ubuntu-latest
    container:
      image: node:14.15.0-alpine3.12
    steps:
      - name: Log the node version
        run: |
          node -v
          cat /etc/os-release

Docker run command

Use the regular docker run command. Here we use > to fold commands in YAML.

.github/workflows/test-container.yml
name: container
on: push

env:
  IMG: "node:14.15.0-alpine3.12"

jobs:
  node-docker:
    runs-on: ubuntu-latest
    steps:
      - name: Log the node version
        run: >
          docker run -rm -v $(pwd):/tmp -w /tmp
          ${{ env.IMG }}
          bash -c "node -v; cat /etc/os-release"

Julia in GitHub actions

Official Julia GitHub actions

Example workflow:

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: julia-actions/setup-julia@v1
    - uses: julia-actions/cache@v1
    - uses: julia-actions/julia-buildpkg@v1
    - uses: julia-actions/julia-runtest@v1
    - uses: julia-actions/julia-docdeploy@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
    - uses: julia-actions/julia-processcoverage@v1
    - uses: codecov/codecov-action@v2
      with:
        files: lcov.info

Using the Julia shell

Using Julia shell to run Julia scripts is much cleaner than julia -e 'code'.1

For example, the two steps do the same:

- name: Run Julia command
  shell: julia --color=yes --project=. --threads=auto {0}
  run: |
    println("Hello, Julia!")
    println("This is fine.")

- name: Run Julia command
  run: julia --color=yes --project=. --threads=auto -e '
    println("Hello, Julia!")
    println("This is fine.")'

Using PyCall.jl

In GNU/Linux systems like Ubuntu, PyCall.jl may not be able to install python packages for PyCall.jl because it will first try to use the system Python (/usr/bin/python) and pip. It would fail due to lack of superuser privileges.

To solve this, you can set the PYTHON environment variable to where the Python executable is.2

  • Either using a blank (PYTHON:'') variable will force Julia to install a local miniconda distribution.
  • Or using the Python executable from the setup-python action.
- uses: actions/setup-python@v4
  id: py
  with:
    python-version: '3.x'
- uses: julia-actions/setup-julia@v1
  with:
    version: 1
- uses: julia-actions/julia-buildpkg@v1
  env:
    # PYTHON: ''      # Use python from Conda.jl
    PYTHON: ${{ steps.py.outputs.python-path }}  # Use python from setup-python

NodeJS in GitHub actions

The actions/setup-node action installs Node.js and caches package dependencies.

steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
  with:
    node-version: 'lts/*'
    cache: 'npm'
- run: npm ci
- run: npm test

GitHub Pages in GitHub actions

Publish your website to GitHub pages with GitHub actions (CI/CD).

Official workflow

Official GitHub actions

The benefit of using the official workflow is that you do not need an orphan branch to hold the webpages.

.github/workflows/pages.yml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # After the website is built
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          path: ./site

  # Deployment job
  deploy:
    needs: build
    if: ${{ github.ref == 'refs/heads/main' }}
    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write # to deploy to Pages
      id-token: write # to verify the deployment originates from an appropriate source
      actions: read # to download an artifact uploaded by `actions/upload-pages-artifact@v3`
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

In the repository settings => Pages => Build and deployment => Select GitHub actions as the page source.

Publish to another branch

You need to give write permission to GITHUB_TOKEN in the workflow file for the following actions to work

permissions:
  contents: write

Use peaceiris/actions-gh-pages

# After the website was built
- name: Deploy
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}  # You need an SSH deploy key if deploying to another repo
    publish_dir: ./public
    force_orphan: true
    commit_message: ${{ github.event.head_commit.message }}

Or JamesIves/github-pages-deploy-action

# After the website was built
- name: Deploy 🚀
  uses: JamesIves/github-pages-deploy-action@v4
  with:
    folder: ./public # The folder the action should deploy.

In the repository settings => Pages => Build and deployment => Select Deploy from a branch as the page source.

Python in GitHub actions

Pip packages

The actions/setup-python actions installs python with a specific version and could cache downloaded Python packages. (But not the installed environment)

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
  with:
    python-version: '3.x'
    cache: 'pip'
- run: pip install -r requirements.txt

Cache virtual environment

The following workflow caches the virtual environment folder1, which is faster than caching the whole Python environment.

- name: Setup Python
  uses: actions/setup-python@v5
  id: setup-python
  with:
    python-version: '3.x'
- name: Cache virtualenv
  uses: actions/cache@v4
  id: cache-venv
  with:
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}
    path: .venv
- name: Install Python dependencies
  run: |
    python -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements.txt
    echo ".venv/bin" >> $GITHUB_PATH
    echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV
    echo "PYTHON=${VIRTUAL_ENV}/bin/python" >> $GITHUB_ENV
    echo "JULIA_PYTHONCALL_EXE=${VIRTUAL_ENV}/bin/python">> $GITHUB_ENV

Use uv

uv is a drop-in replacement for pip, an extremely fast Python package and project manager written in Rust.

The GitHub actions workflow:

name: UV example

jobs:
  python-linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'
      - name: Set up uv
        run: curl -LsSf https://astral.sh/uv/install.sh | sh
      - name: Install requirements
        run: uv pip install --system -r requirements.txt

Conda packages

The mamba-org/setup-micromamba action installs the micromamba package manager and conda package dependencies. It also caches the Python runtime environment.

- uses: mamba-org/setup-micromamba@v1
  with:
    environment-file: environment.yml
    init-shell: bash
    cache-environment: true
    post-cleanup: 'all'

- name: Run custom command in micromamba environment
  shell: micromamba-shell {0}
  run: python --version

Release in GitHub actions

GitLab

GitLab CI/CD is a tool built into GitLab for software development for Continuous Integration (CI) and Continuous Delivery/Deployment (CD).

Parallel Matrix build

Test and build in parallel with matrix build in Gitlab CI/CD.

For example,

.gitlab-ci.yml
test:
  image: $IMAGE
  script:
    - echo $MSG
    - python -V
  parallel:
    matrix:
      # First cartesian set of parameters
      - IMAGE: ['python:3.6-alpine', 'python:3.7-alpine']
        MSG: ['Test1', 'Test2']
      # Second cartesian set of parameters

This will create 4 jobs with a combination of a custom message and a specific Python image.

See also the blog post by Michael Friedrich for more parallel matrix build with GitLab CI/CD.

Replace old only/except with new rules to include or exclude jobs in pipelines

GitLab CI/CD rules reference

Note

Rules cannot be used together with only/except. Otherwise, GitLab will return a key may not be used with rules error.

only run if this is a scheduled pipeline

.gitlab-ci.yml
scheduled-update:
  # only run if this is a scheduled pipeline
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Run upon push

.gitlab-ci.yml
push-job:
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"

Run upon merge request

.gitlab-ci.yml
merge-request:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Run only for the commits in the default branch

.gitlab-ci.yml
# GitLab pages job
pages:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Run only for tags

.gitlab-ci.yml
pages:
  rules:
    - if: $CI_COMMIT_TAG

Choose a specific runner

Use tags to tun jobs in a specific runner e.g., your self-hosted GitLab runner in the workstation.

.gitlab-ci.yml
run-custom:
  tags:
    - myWS
  script:
    - echo "Running in my workstation."

Create a release

Create a release with GitLab CI/CD pipelines with the release-cli docker image:

.gitlab-ci.yml
release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG                  # Run this job when a tag is created manually
  script:
    - echo "Running the release job."
  release:
    name: "Release $CI_COMMIT_TAG"
    description: "Release created using the release-cli."

Cache Conda Packages

We can cache conda packages by setting CONDA_PKGS_DIRS environment variable inside the project folder (CI_PROJECT_DIR) so that the GitLab runner can cache these dependencies.

.gitlab-ci.yml
image: condaforge/miniforge3:latest

variables:
  CONDA_PKGS_DIRS: "${CI_PROJECT_DIR}/.cache/conda/pkgs"

cache:
  - key:
      files:
        - environment.yml
    paths:
      - .env/
      - .cache/conda/pkgs

before_script:
  - conda env update --prefix ./.env --file environment.yml --prune
  - source activate ./.env

Because GitLab only caches files inside the project folder (CI_PROJECT_DIR)

  • CONDA_PKGS_DIRS is set to ${CI_PROJECT_DIR}/.cache/conda/pkgs to hold the downloaded compressed packages.
  • Extracted environment folder is set to ${CI_PROJECT_DIR}/.env using the --prefix option.

Conda will create the runtime environment according to environment.yml. The environment folder will be created (if not present) or cached. The option --prune means conda will remove unnecessary packages for subsequent caching.

Git Operations in GitLab CI/CD

Using SSH keys

Warning

Currently the private key cannot be masked and base64 encoding/decoding is needed.

You can use a pair of SSH keys to access a git repository
- The private key would be a CI/CD project variable
- The public key would be a deploy key

You also need additional steps to setup a SSH client in the pipeline.

before_script:
   # apt-get applies to Debian-based images. Change the package manager if needed.
  - 'which ssh-agent || ( apt-get update -qy && apt-get install openssh-client -qqy )'
  - 'which git || ( apt-get update -qy && apt-get install git -qqy )'
  - eval `ssh-agent -s`
  - echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null # add ssh key
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

And replace the default HTTP-based git origin with the SSH one.

script:
  - git remote rm origin && git remote add origin git@gitlab.com:$CI_PROJECT_PATH.git

Using a personal access token (PAT)

Compared to SSH, using a personal access token (PAT) with write repo right might be simpler. In the following example, the PAT is stored as a masked CI/CD variable GIT_PUSH_TOKEN.

script:
  - bash update.sh
  - |
    if [ -n $(git status --porcelain) ]; then
        echo "Committing updates"
        git config --global user.name "${GITLAB_USER_NAME}"
        git config --global user.email "${GITLAB_USER_EMAIL}"
        git add .
        git commit -m "Automated update: $(date '+%Y-%m-%d-%H-%M-%S')"
        git push "https://${GITLAB_USER_NAME}:${GIT_PUSH_TOKEN}@${CI_REPOSITORY_URL#*@}"
        exit;
    else
        echo "no change, nothing to commit"
    fi

For a MR pipeline, GitLab provides git push options for merge request settings.

script:
  - bash update.sh
  - |
    if [ -n $(git status --porcelain) ]; then
        echo "Committing updates"
        NEW_BR=auto-update-$(date '+%Y-%m-%d-%H-%M-%S')
        git config --global user.name "${GITLAB_USER_NAME}"
        git config --global user.email "${GITLAB_USER_EMAIL}"
        git checkout -b ${NEW_BR}
        git add .
        git commit -m "${NEW_BR}"
        git push "https://${GITLAB_USER_NAME}:${GIT_PUSH_TOKEN}@${CI_REPOSITORY_URL#*@}" \
            -o merge_request.create \
            -o merge_request.target="${CI_DEFAULT_BRANCH}" \
            -o merge_request.merge_when_pipeline_succeeds \
            -o merge_request.remove_source_branch \
            -o merge_request.title="${NEW_BR}" \
            -o merge_request.label="automated update" \
            -o merge_request.assign="${GITLAB_USER_NAME}"
        exit;
    else
        echo "no change, nothing to commit"
    fi

Synchronize GitLab repo to GitHub

Assuming you have two identical repositories on GitLab and GitHub each (you can do this by importing one's repo to the other), the following steps show how to mirror GitLab repositories to GitHub with deploy SSH keys.

On the GitLab side
  1. In the GitLab repo, go to Settings/Repository/Mirroring repositories and set Git repository URL as ssh://git@github.com/<namespace>/<repo>.git. e.g. ssh://git@github.com/sosiristseng/docker-python-julia.git

Warning

The GitHub button gives git@github.com:<namespace>/<repo>.git as the repo URL, one should change it to ssh://git@github.com/<namespace>/<repo>.git for GitLab to access the repository.

  1. Set Mirror direction to push.

  2. Set Authentication method to SSH public key. Optionally you can click Detect host keys.

  3. (Optionally) check "Keep divergent refs" to prevent force pushes and/or "Mirror only protected branches" for a cleaner GitHub mirror.

  4. Click Mirror repository.
  5. Copy the SSH public key (the middle button) and go to the GitHub mirror repo.
On the GitHub side

In the Github mirror repository, go to Settings/Deploy keys and add deploy key.

Paste the SSH public key copied from the GitLab source. Give it a title, allow write access, click add key to finish this step, and viola.

Dynamic parallel matrix

Job matrix creates multiple job runs that are based on the combinations of the variables. Sometimes we want a dynamic number of matrix jobs, which requires a JSON array as an output. Here we use json and glob modules in Python to generate that JSON list.12