Skip to content

2024-10

  • in Code
  • 1 min read

Make IJulia use Python package provided by CondaPkg.jl

Set the JUPYTER environment variable to CondaPkg.jl-provided jupyter and IJulia.jl will take it to start the kernel. 1

using CondaPkg
CondaPkg.add("jupyter")
ENV["JUPYTER"] = CondaPkg.which("jupyter")

using Pkg
Pkg.add("IJulia") # if IJulia was not yet installed
Pkg.build("IJulia") # To change the configuration of existing installation

  1. https://stackoverflow.com/questions/78130757/how-to-run-ijulia-using-the-pythoncall-condapkg-python-installation-rather-than ↩

  • in System
  • 1 min read

heredoc: Passing multiple lines of string

Use heredoc to pass the string as-is between two delimiters (e.g. EOF)

cat << "EOF" >> ~/.xprofile
# ~/.xprofile
export GTK_IM_MODULE=ibus
export QT_IM_MODULE=ibus
export XMODIFIERS=@im=ibus
ibus-daemon -drx
EOF

Will append the following lines to ~/.xprofile:

.xprofile
export GTK_IM_MODULE=ibus
export QT_IM_MODULE=ibus
export XMODIFIERS=@im=ibus
ibus-daemon -drx
  • in DevOps
  • 1 min read

Setup docker in Github actions

Building and publishing docker images

.github/workflows/docker.yml
name: Create and publish a Docker image

on:
  push:
    branches: ['main']

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push-image:
    runs-on: ubuntu-latest

    # GITHUB_TOKEN permissions
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - name: Setup Buildx
        uses: docker/setup-buildx-action@v2
      - name: Log in to the Container registry
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Run docker containers in Github actions

Use container keyword

Running jobs in a container in GitHub actions.

.github/workflows/test-container.yml
name: container
on: push

jobs:
  node-docker:
    runs-on: ubuntu-latest
    container:
      image: node:14.15.0-alpine3.12
    steps:
      - name: Log the node version
        run: |
          node -v
          cat /etc/os-release

Docker run command

Use the regular docker run command. Here we use > to fold commands in YAML.

.github/workflows/test-container.yml
name: container
on: push

env:
  IMG: "node:14.15.0-alpine3.12"

jobs:
  node-docker:
    runs-on: ubuntu-latest
    steps:
      - name: Log the node version
        run: >
          docker run -rm -v $(pwd):/tmp -w /tmp
          ${{ env.IMG }}
          bash -c "node -v; cat /etc/os-release"
  • in DevOps
  • 2 min read

Julia in GitHub actions

Official Julia GitHub actions

Example workflow:

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: julia-actions/setup-julia@v1
    - uses: julia-actions/cache@v1
    - uses: julia-actions/julia-buildpkg@v1
    - uses: julia-actions/julia-runtest@v1
    - uses: julia-actions/julia-docdeploy@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
    - uses: julia-actions/julia-processcoverage@v1
    - uses: codecov/codecov-action@v2
      with:
        files: lcov.info

Using the Julia shell

Using Julia shell to run Julia scripts is much cleaner than julia -e 'code'.1

For example, the two steps do the same:

- name: Run Julia command
  shell: julia --color=yes --project=. --threads=auto {0}
  run: |
    println("Hello, Julia!")
    println("This is fine.")

- name: Run Julia command
  run: julia --color=yes --project=. --threads=auto -e '
    println("Hello, Julia!")
    println("This is fine.")'

Using PyCall.jl

In GNU/Linux systems like Ubuntu, PyCall.jl may not be able to install python packages for PyCall.jl because it will first try to use the system Python (/usr/bin/python) and pip. It would fail due to lack of superuser privileges.

To solve this, you can set the PYTHON environment variable to where the Python executable is.2

  • Either using a blank (PYTHON:'') variable will force Julia to install a local miniconda distribution.
  • Or using the Python executable from the setup-python action.
- uses: actions/setup-python@v4
  id: py
  with:
    python-version: '3.x'
- uses: julia-actions/setup-julia@v1
  with:
    version: 1
- uses: julia-actions/julia-buildpkg@v1
  env:
    # PYTHON: ''      # Use python from Conda.jl
    PYTHON: ${{ steps.py.outputs.python-path }}  # Use python from setup-python

  1. https://discourse.julialang.org/t/tip-use-julia-as-custom-shell-in-github-actions/53377 ↩

  2. PyCall.jl documentation ↩

  • in DevOps
  • 1 min read

NodeJS in GitHub actions

The actions/setup-node action installs Node.js and caches package dependencies.

steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
  with:
    node-version: 'lts/*'
    cache: 'npm'
- run: npm ci
- run: npm test
  • in DevOps
  • 2 min read

GitHub Pages in GitHub actions

Publish your website to GitHub pages with GitHub actions (CI/CD).

Official workflow

Official GitHub actions

The benefit of using the official workflow is that you do not need an orphan branch to hold the webpages.

.github/workflows/pages.yml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # After the website is built
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          path: ./site

  # Deployment job
  deploy:
    needs: build
    if: ${{ github.ref == 'refs/heads/main' }}
    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write # to deploy to Pages
      id-token: write # to verify the deployment originates from an appropriate source
      actions: read # to download an artifact uploaded by `actions/upload-pages-artifact@v3`
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

In the repository settings => Pages => Build and deployment => Select GitHub actions as the page source.

Publish to another branch

You need to give write permission to GITHUB_TOKEN in the workflow file for the following actions to work

permissions:
  contents: write

Use peaceiris/actions-gh-pages

# After the website was built
- name: Deploy
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}  # You need an SSH deploy key if deploying to another repo
    publish_dir: ./public
    force_orphan: true
    commit_message: ${{ github.event.head_commit.message }}

Or JamesIves/github-pages-deploy-action

# After the website was built
- name: Deploy 🚀
  uses: JamesIves/github-pages-deploy-action@v4
  with:
    folder: ./public # The folder the action should deploy.

In the repository settings => Pages => Build and deployment => Select Deploy from a branch as the page source.

  • in DevOps
  • 2 min read

Python in GitHub actions

Pip packages

The actions/setup-python actions installs python with a specific version and could cache downloaded Python packages. (But not the installed environment)

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
  with:
    python-version: '3.x'
    cache: 'pip'
- run: pip install -r requirements.txt

Cache virtual environment

The following workflow caches the virtual environment folder1, which is faster than caching the whole Python environment.

- name: Setup Python
  uses: actions/setup-python@v5
  id: setup-python
  with:
    python-version: '3.x'
- name: Cache virtualenv
  uses: actions/cache@v4
  id: cache-venv
  with:
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}
    path: .venv
- name: Install Python dependencies
  run: |
    python -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements.txt
    echo ".venv/bin" >> $GITHUB_PATH
    echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV
    echo "PYTHON=${VIRTUAL_ENV}/bin/python" >> $GITHUB_ENV
    echo "JULIA_PYTHONCALL_EXE=${VIRTUAL_ENV}/bin/python">> $GITHUB_ENV

Use uv

uv is a drop-in replacement for pip, an extremely fast Python package and project manager written in Rust.

The GitHub actions workflow:

name: UV example

jobs:
  python-linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'
      - name: Set up uv
        run: curl -LsSf https://astral.sh/uv/install.sh | sh
      - name: Install requirements
        run: uv pip install --system -r requirements.txt

Conda packages

The mamba-org/setup-micromamba action installs the micromamba package manager and conda package dependencies. It also caches the Python runtime environment.

- uses: mamba-org/setup-micromamba@v1
  with:
    environment-file: environment.yml
    init-shell: bash
    cache-environment: true
    post-cleanup: 'all'

- name: Run custom command in micromamba environment
  shell: micromamba-shell {0}
  run: python --version

  1. https://adamj.eu/tech/2023/11/02/github-actions-faster-python-virtual-environments/ ↩