Skip to content

DevOps

  • in DevOps
  • 1 min read

Artifacts in GitHub actions

Artifacts are data generated by workflows that can be passed to subsequent jobs.

Workflows

Official workflows

The actions/upload-artifact and the actions/download-artifact actions by GitHub for storing and retrieving artifact(s) in the same workflow.

Artifacts across two different workflows

The dawidd6/action-download-artifact action downloads and extracts uploaded artifact(s) associated with a given (different) workflow.

The tonyhallett/artifacts-url-comments action creates comment(s) in pull request and/or associated issues containing the URL to artifacts from the workflow run being watched.

Merge multiple artifacts

merge-multiple: true

- name: Download notebooks
  uses: actions/download-artifact@v4
  with:
    path: path/of/artifacts
    pattern: notebook-*
    merge-multiple: true
- name: Display structure of downloaded files
  run: ls -R path/of/artifacts
  • in DevOps
  • 2 min read

Automatic Dependency Update in GitHub

Updating package dependencies automatically as a part of continuous integration (CI)

Dependabot

Dependabot creates a pull request once there is an update for the dependencies. The pull requests are usually tested by continuous integration (CI).

However, dependabot does not support automerging on its own due to security concerns. The good news is that we could use Kodiak to do the job. See it's quickstart if you are interested.

For example, the dependabot file .github/dependabot.yml

.github/dependabot.yml
version: 2

updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "daily"
    labels:
    - "automerge"

Kodiak bot file: .github/.kodiak.toml

.github/.kodiak.toml
version = 1

[merge]
method = "squash"

And you need additional steps in the Github settings to make Kodiak Bot work

  • Add the automerge tag in the GitHub issue tab.
  • In Options -> Branches, protect the to-be-merged branch (usually the main branch)
  • Also tick "Require status checks to pass before merging" and "Require branches to be up to date before merging"
  • And select which github action job(s) should be passed in order to automerge by using the search bar below.

Renovate

Renovate bot can manage both dependency update checking and automated pull request merging.

Renovate supports a variety of platforms

  • GitHub (.com and Enterprise)
  • GitLab (.com and CE/EE)
  • Bitbucket Cloud / Servee
  • Azure DevOps
  • Gitea

And a variety of programming languages

  • Git submodules
  • GitHub actions
  • Node JS packages
  • Dockerfile
  • Javascript (and node JS)
  • Java
  • And more

Setup for GitHub

Enable the Renovate GitHub APP for GitHub repositories. Renovate bot will open an pull request for reachable repos to begin an interactive setup.

Setup for GitLab

According to the renovate GitLab runner documentation,

  1. Create a repository for the Renovate runner.
  2. Add a GitLab personal access token (PAT) with read_user, api and write_repository scopes as the RENOVATE_TOKEN CI/CD variable,
  3. Add a GitHub PAT as GITHUB_COM_TOKEN. This token allows renovate bot to read information of updated dependencies unhindered.
  4. Create .gitlab-ci.yml to run the pipelines
    .gitlab-ci.yml
    include:
 - project: 'renovate-bot/renovate-runner'
   file: '/templates/renovate-dind.gitlab-ci.yml'
  5. Select what repositories renovate bot could touch by setting up the CI/CD variable RENOVATE_EXTRA_FLAGS : --autodiscover=true --autodiscover-filter=group1/* or configure them in the config.js file.
    config.js
    module.exports = {
    repositories: [
        "group1/repo1",
        "group2/repo2",
    ],
};

    As a plus, it's easier to set up more renovate runner options in the config.js file.
  6. Setup a schedule for the pipeline.

Renovate settings file

The settings file renovate.json example

renovate.json
{
  "extends": [
    "config:recommended",
  ],
  "git-submodules": {
      "enabled": true
  }
}
  • in DevOps
  • 2 min read

Caching in GitHub actions

Caching dependencies

The actions/cache action caches dependencies for the execution environment.

- name: Cache multiple paths
  uses: actions/cache@v4
  with:
    path: |
      ~/cache
      !~/cache/exclude
    key: ${{ runner.os }}-${{ hashFiles('**/Lockfile') }}
    restore-keys: |
      ${{ runner.os }}-
  • The key is the identifier for writing into the cache. If the key stays the same before and after the workflow, the cache will not be updated.
  • The restore-keys are the identifiers for reading the cache besides the key. If there is no matching key but a part of it (restore-keys) matches, the GitHub action will still read the cache and update it after the job. (since the key is different)

Restore and save actions

The cache actions could be split into restore and save steps, leading to a fine-grained behavior.

- name: Restore cached Primes
      id: cache-primes-restore
      uses: actions/cache/restore@v4
      with:
        path: |
          path/to/dependencies
          some/other/dependencies
        key: ${{ runner.os }}-primes
#
# //intermediate workflow steps
#
- name: Save Primes
  id: cache-primes-save
  uses: actions/cache/save@v4
  with:
    path: |
      path/to/dependencies
      some/other/dependencies

Caching for a specific programming language

Some GitHub actions for setting up runtime for programming languages can cache package dependency.

Cleanup PR caches

Clean up PR caches after it closes to save space.

name: Cleanup PR caches
on:
  pull_request:
    types:
      - closed

jobs:
  cleanup:
    permissions:
      actions: write
    runs-on: ubuntu-latest
    steps:
      - name: Cleanup
        run: |
          gh extension install actions/gh-actions-cache

          echo "Fetching list of cache key"
          cacheKeysForPR=$(gh actions-cache list -R $REPO -B $BRANCH -L 100 | cut -f 1 )

          ## Setting this to not fail the workflow while deleting cache keys.
          set +e
          echo "Deleting caches..."
          for cacheKey in $cacheKeysForPR
          do
              gh actions-cache delete $cacheKey -R $REPO -B $BRANCH --confirm
          done
          echo "Done"
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          REPO: ${{ github.repository }}
          BRANCH: refs/pull/${{ github.event.pull_request.number }}/merge
  • in DevOps
  • 2 min read

Git Operations in GitHub actions

Git commands, such as checkout, add, create a branch, make a pull request in Github actions.

Checkout (Clone a repository)

The official actions/checkout action clones the repository to $GITHUB_WORKSPACE. By default it uses built-in GITHUB_TOKEN for authentication.

In most cases, this is what you need:

- uses: actions/checkout@v4

The checkout action also supports pushing a commit to the same repo.

Warning

This may not work on protected branches that need status checks.

on: push
jobs:
  git-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          date > generated.txt
          git config user.name github-actions
          git config user.email github-actions@github.com
          git add .
          git commit -m "generated"
          git push

However, no further workflows will be triggered with the GITHUB_TOKEN. You will need the following steps to trigger workflows.

How to trigger further CI runs

You will need either a Personal access token (PAT) with repo scope access as an action secret.

- uses: actions/checkout@v4
  with:
    token: ${{ secrets.PAT }}

Or a pair of SSH keys; the public key is the deploy key with write access, while the private key is an action secret variable SSH_PRIVATE_KEY.

- uses: actions/checkout@v4
  with:
    ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}

Push changes back to GitHub

The following actions are more convenient for commit and push than the official checkout action.

Create a pull request

The peter-evans/create-pull-request action will commit all files into a new branch and make a pull request to the target (default main) branch.

- name: Create Pull Request
  uses: peter-evans/create-pull-request@v6
  with:
  # token: ${{ secrets.PAT }} # A PAT is required for triggering pull request workflows
    token: ${{ secrets.GITHUB_TOKEN }}  # This will not trigger further workflows

Merge pull requests

  • in DevOps
  • 1 min read

Setup docker in Github actions

Building and publishing docker images

.github/workflows/docker.yml
name: Create and publish a Docker image

on:
  push:
    branches: ['main']

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push-image:
    runs-on: ubuntu-latest

    # GITHUB_TOKEN permissions
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - name: Setup Buildx
        uses: docker/setup-buildx-action@v2
      - name: Log in to the Container registry
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Run docker containers in Github actions

Use container keyword

Running jobs in a container in GitHub actions.

.github/workflows/test-container.yml
name: container
on: push

jobs:
  node-docker:
    runs-on: ubuntu-latest
    container:
      image: node:14.15.0-alpine3.12
    steps:
      - name: Log the node version
        run: |
          node -v
          cat /etc/os-release

Docker run command

Use the regular docker run command. Here we use > to fold commands in YAML.

.github/workflows/test-container.yml
name: container
on: push

env:
  IMG: "node:14.15.0-alpine3.12"

jobs:
  node-docker:
    runs-on: ubuntu-latest
    steps:
      - name: Log the node version
        run: >
          docker run -rm -v $(pwd):/tmp -w /tmp
          ${{ env.IMG }}
          bash -c "node -v; cat /etc/os-release"
  • in DevOps
  • 2 min read

Julia in GitHub actions

Official Julia GitHub actions

Example workflow:

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: julia-actions/setup-julia@v1
    - uses: julia-actions/cache@v1
    - uses: julia-actions/julia-buildpkg@v1
    - uses: julia-actions/julia-runtest@v1
    - uses: julia-actions/julia-docdeploy@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
    - uses: julia-actions/julia-processcoverage@v1
    - uses: codecov/codecov-action@v2
      with:
        files: lcov.info

Using the Julia shell

Using Julia shell to run Julia scripts is much cleaner than julia -e 'code'.1

For example, the two steps do the same:

- name: Run Julia command
  shell: julia --color=yes --project=. --threads=auto {0}
  run: |
    println("Hello, Julia!")
    println("This is fine.")

- name: Run Julia command
  run: julia --color=yes --project=. --threads=auto -e '
    println("Hello, Julia!")
    println("This is fine.")'

Using PyCall.jl

In GNU/Linux systems like Ubuntu, PyCall.jl may not be able to install python packages for PyCall.jl because it will first try to use the system Python (/usr/bin/python) and pip. It would fail due to lack of superuser privileges.

To solve this, you can set the PYTHON environment variable to where the Python executable is.2

  • Either using a blank (PYTHON:'') variable will force Julia to install a local miniconda distribution.
  • Or using the Python executable from the setup-python action.
- uses: actions/setup-python@v4
  id: py
  with:
    python-version: '3.x'
- uses: julia-actions/setup-julia@v1
  with:
    version: 1
- uses: julia-actions/julia-buildpkg@v1
  env:
    # PYTHON: ''      # Use python from Conda.jl
    PYTHON: ${{ steps.py.outputs.python-path }}  # Use python from setup-python

  1. https://discourse.julialang.org/t/tip-use-julia-as-custom-shell-in-github-actions/53377 ↩↩

  2. PyCall.jl documentation ↩↩

  • in DevOps
  • 1 min read

NodeJS in GitHub actions

The actions/setup-node action installs Node.js and caches package dependencies.

steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
  with:
    node-version: 'lts/*'
    cache: 'npm'
- run: npm ci
- run: npm test
  • in DevOps
  • 2 min read

GitHub Pages in GitHub actions

Publish your website to GitHub pages with GitHub actions (CI/CD).

Official workflow

Official GitHub actions

The benefit of using the official workflow is that you do not need an orphan branch to hold the webpages.

.github/workflows/pages.yml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # After the website is built
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          path: ./site

  # Deployment job
  deploy:
    needs: build
    if: ${{ github.ref == 'refs/heads/main' }}
    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write # to deploy to Pages
      id-token: write # to verify the deployment originates from an appropriate source
      actions: read # to download an artifact uploaded by `actions/upload-pages-artifact@v3`
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

In the repository settings => Pages => Build and deployment => Select GitHub actions as the page source.

Publish to another branch

You need to give write permission to GITHUB_TOKEN in the workflow file for the following actions to work

permissions:
  contents: write

Use peaceiris/actions-gh-pages

# After the website was built
- name: Deploy
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}  # You need an SSH deploy key if deploying to another repo
    publish_dir: ./public
    force_orphan: true
    commit_message: ${{ github.event.head_commit.message }}

Or JamesIves/github-pages-deploy-action

# After the website was built
- name: Deploy 🚀
  uses: JamesIves/github-pages-deploy-action@v4
  with:
    folder: ./public # The folder the action should deploy.

In the repository settings => Pages => Build and deployment => Select Deploy from a branch as the page source.

  • in DevOps
  • 2 min read

Python in GitHub actions

Pip packages

The actions/setup-python actions installs python with a specific version and could cache downloaded Python packages. (But not the installed environment)

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
  with:
    python-version: '3.x'
    cache: 'pip'
- run: pip install -r requirements.txt

Cache virtual environment

The following workflow caches the virtual environment folder1, which is faster than caching the whole Python environment.

- name: Setup Python
  uses: actions/setup-python@v5
  id: setup-python
  with:
    python-version: '3.x'
- name: Cache virtualenv
  uses: actions/cache@v4
  id: cache-venv
  with:
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}
    path: .venv
- name: Install Python dependencies
  run: |
    python -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements.txt
    echo ".venv/bin" >> $GITHUB_PATH
    echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV
    echo "PYTHON=${VIRTUAL_ENV}/bin/python" >> $GITHUB_ENV
    echo "JULIA_PYTHONCALL_EXE=${VIRTUAL_ENV}/bin/python">> $GITHUB_ENV

Use uv

uv is a drop-in replacement for pip, an extremely fast Python package and project manager written in Rust.

The GitHub actions workflow:

name: UV example

jobs:
  python-linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'
      - name: Set up uv
        run: curl -LsSf https://astral.sh/uv/install.sh | sh
      - name: Install requirements
        run: uv pip install --system -r requirements.txt

Conda packages

The mamba-org/setup-micromamba action installs the micromamba package manager and conda package dependencies. It also caches the Python runtime environment.

- uses: mamba-org/setup-micromamba@v1
  with:
    environment-file: environment.yml
    init-shell: bash
    cache-environment: true
    post-cleanup: 'all'

- name: Run custom command in micromamba environment
  shell: micromamba-shell {0}
  run: python --version

  1. https://adamj.eu/tech/2023/11/02/github-actions-faster-python-virtual-environments/ ↩↩

  • in DevOps
  • 1 min read

Release in GitHub actions