Python & GitHub Actions

Agenda

  • I ❤️ GitHub

  • Introduction to GitHub Actions

  • Using GitHub Actions for Python libraries

  • –//– for Python projects

  • –//– for JavaScript libraries

  • –//– for full stack projects

  • GitHub Actions issues (if any)

I ❤️ GitHub

My GitHub Timeline

2008:

Created account

2008 - 2012:

Open source acitivity

2012 - 2016:

Main work tool

2014 - present:

GitHub PRO member. Home of all my pet projects

Code quality assistance

  • Jenkins

  • Travis CI

  • Circle CI

  • Gitlab

  • Drone.io and other host based solutions

Dude, Where’s my CI?

Introduction to GitHub Actions

https://help.github.com/en/actions

.github/workflows/ci.yml


name: "ci"
on: ["push"]

env:
    ...

jobs:
    ...

When to start job?

  • On push

  • On push to master

  • On pull request to master

  • On specific tag

  • Trigger every hour

  • Trigger for changed files

  • Documentation

Where run the job?

Where run the job?

  • Virtual Machine

    • Linux: ubuntu-latest, ubuntu-1604

    • macOS: macos-latest

    • Windows: windows-latest

  • Docker Container

Job strategy

Virtual machine to use


strategy:
  matrix:
    os: ["ubuntu-latest", "macos-latest", "windows-latest"]

Job strategy

Different Python version


strategy:
  matrix:
    python-version: ["3.6", "3.7", "3.8"]

Steps

Test job

  • Check out

  • Install Python

  • Install poetry

  • Install dependencies

  • Setup cache for pre-commit only on Python 3.8

  • Run pre-commit only on Python 3.8

  • Validate OpenAPI schemas only on Python 3.8

  • Run tests

  • Send report to Coveralls only on Python 3.8

Steps

Package job

  • Check out

  • Setup Python

  • Setup poetry

  • Build package

  • Check package

  • Publish package, if push to tag triggered the job

Reusable Actions

Check out


steps:
  - uses: "actions/checkout@v2.0.0"

Reusable Actions

Install python


steps:
  - name: "Install Python"
    uses: "actions/setup-python@v1.2.0"
    with:
      python-version: "3.8"

Reusable Actions

Install poetry


steps:
  - name: "Install poetry"
    uses: "dschep/install-poetry-action@v1.3"
    with:
      version: "1.0.5"
      create_virtualenvs: true

Conditionals

Run pre-commit only on Python dev version


env:
  DEV_PYTHON_VERSION: "3.8"

steps:
  - name: "Run pre-commit"
    if: "matrix.python-version == env.DEV_PYTHON_VERSION"
    uses: "pre-commit/action@v1.0.1"

Conditionals

Publish package


steps:
  - name: "Publish package"
    if: "github.event_name == ’push’ && startsWith(github.event.ref, ’refs/tags’)"
    uses: "pypa/gh-action-pypi-publish@v1.1.0"
    with:
      user: "${{ secrets.PYPI_USERNAME }}"
      password: "${{ secrets.PYPI_PASSWORD }}"

Services

Definition


services:
  postgres:
    image: "postgres:12.2-alpine"
    env:
      POSTGRES_USER: "project"
      POSTGRES_PASSWORD: "project"
      POSTGRES_DB: "test_project"
    ports:
      - "5432:5432"
    options: "--health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5"

  redis:
    image: "redis:5.0.7-alpine"

Services

Usage


steps:
  - name: "Run Tests"
    run: "make migrate-only test-api-only"
    env:
      LEVEL: "test"
      DB_HOST: "postgres"
      DB_PORT: "${{ job.services.postgres.ports[5432] }}"
      REDIS_HOST: "redis"
      REDIS_PORT: "6379"

More?

Using GitHub Actions

For Python libraries

Based on Hynek article: “Python in GitHub Actions”

  • Run in VM

  • Build job to check library is working on any OS

  • Test library on all supported Python versions

  • Package job to publish on PyPI

For Python libraries

Gotchas

  1. Pin everything

  2. Run build & test in parallel (failed job will stop workflow)

  3. Package job needs build & test

  4. Cache pre-commit hooks

  5. tox-gh-actions

  6. Provide dev Python version:

    • To run pre-commit

    • To send coveralls report

    • etc (to run only once)

For Python libraries

Examples

For Python projects

  • Run in container

  • Provide basic Docker image to reuse

  • Only test & deploy jobs needed

For Python projects

playpauseandstop/docker-python


runs-on: "ubuntu-latest"
container:
  image: "playpauseandstop/docker-python:3.2.0-py38"

For Python projects

Install project


- name: "Cache venv"
  uses: "actions/cache@v1.1.2"
  with:
    path: ".venv"
    key: "venv-${{ hashFiles('poetry.lock') }}"

- name: "Install"
  run: "make install"

For Python projects

Run unit & integrational tests


- name: "Run Unit Tests"
  run: "make test-unit-only"

- name: "Run Integrational Tests"
  if: "github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')"
  env:
    HOBOTNICA_ROOT: "~/.hobotnica-test"
    GITHUB_API_USERNAME: "${{ secrets.GITHUB_API_USERNAME }}"
    GITHUB_API_TOKEN: "${{ secrets.GITHUB_API_TOKEN }}"
  run: "make test-integrational-only"

For Python projects

Gotchas

  • Base image is good idea

  • Cache .venv as well

For JavaScript libraries

  • Run in VM

  • Same rules as for Python libraries

For JavaScript libraries

Publish to NPM


- name: "Publish package"
  if: "github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')"
  run: |
    set -e
    tag=latest
    if [ ${{ contains(github.event.ref, 'alpha') }} = true ]; then
      tag=alpha
    else
      if [ ${{ contains(github.event.ref, 'beta') }} = true ]; then
        tag=beta
      else
        if [ ${{ contains(github.event.ref, 'rc') }} = true ]; then
          tag=rc
        fi
      fi
    fi
    set -x
    npm publish --tag=${tag}
  env:
    NODE_AUTH_TOKEN: "${{ secrets.NPM_TOKEN }}"

For full stack projects

  • Run in container

  • Same rules as for Python project

  • But also provide jobs for test & deploy frontend

For full stack projects

Installing node in Python image


- name: "Install node"
  run: "nodeenv -n ${{ env.NODE_VERSION }} ${HOME}/.node"

- name: "Install UI"
  run: "PATH=${HOME}/.node/bin:${PATH} make install-ui"

Bonus. Deploying this slides

.github/workflows/slides.yml


name: "slides"

on:
  push:
    paths: "slides/**"

jobs:
  slides:
    runs-on: "ubuntu-latest"

    steps:
      - uses: "actions/checkout@v2.0.0"

Bonus. Deploying this slides

Build Slides


- name: "Build slides"
  uses: "ammaraskar/sphinx-action@0.4"
  with:
    docs-folder: "./slides/"
    pre-build-command: "pip install Sphinx==3.0.3 sphinx-revealjs==0.11.0"
    build-command: "sphinx-build -b revealjs ./ _build/"

Bonus. Deploying this slides

Deploy Slides


- name: "Deploy slides"
  if: "github.event_name == 'push' && github.event.ref == 'refs/heads/master'"
  uses: "AEnterprise/rsync-deploy@v1.0"
  env:
    ARGS: "-avuzPt"
    DEPLOY_KEY: "${{ secrets.DEPLOY_SSH_KEY }}"
    FOLDER: "./slides/_build/"
    SERVER_DESTINATION: "Projects/igordavydenko/slides/_build/"
    SERVER_IP: "igordavydenko.com"
    SERVER_PORT: "22"
    USERNAME: "playpauseandstop"

GitHub Actions issues

  • Why GitHub released Actions only in 2019?

  • Documentation is messy at times

  • Tough to configure from the scratch

  • Secrets not shared with forks

  • Often problems with 3rd party reusable actions

  • Failed job email notifications are useless

Use GitHub Actions for great good!

That’s all, folks!

I am Igor Davydenko

GitHub

@playpauseandstop

Twitter

@playpausenstop

Telegram

@playpausenstop

Slides

Made with sphinx-revealjs

Link

https://igordavydenko.com/slides/lvivpy-9/

Questions?