Python & GitHub Actions

Agenda

• I ❤️ GitHub

• Introduction to GitHub Actions

• Using GitHub Actions for Python libraries

• –//– for Python projects

• –//– for JavaScript libraries

• –//– for full stack projects

• GitHub Actions issues (if any)

I ❤️ GitHub

My GitHub Timeline

2008:

Created account

2008 - 2012:

Open source acitivity

2012 - 2016:

Main work tool

2014 - present:

GitHub PRO member. Home of all my pet projects

Code quality assistance

• Jenkins

• Travis CI

• Circle CI

• Gitlab

• Drone.io and other host based solutions

Dude, Where’s my CI?

Introduction to GitHub Actions

https://help.github.com/en/actions

.github/workflows/ci.yml

name: "ci"
on: ["push"]

env:
    ...

jobs:
    ...

When to start job?

• On push

• On push to master

• On pull request to master

• On specific tag

• Trigger every hour

• Trigger for changed files

• …

• Documentation

Where run the job?

• GitHub hosted runner

• Self hosted runner

Where run the job?

• Virtual Machine

  • Linux: ubuntu-latest, ubuntu-1604

  • macOS: macos-latest

  • Windows: windows-latest

• Docker Container

Job strategy

Virtual machine to use

strategy:
  matrix:
    os: ["ubuntu-latest", "macos-latest", "windows-latest"]

Job strategy

Different Python version

strategy:
  matrix:
    python-version: ["3.6", "3.7", "3.8"]

Steps

Test job

• Check out

• Install Python

• Install poetry

• Install dependencies

• Setup cache for pre-commit only on Python 3.8

• Run pre-commit only on Python 3.8

• Validate OpenAPI schemas only on Python 3.8

• Run tests

• Send report to Coveralls only on Python 3.8

Steps

Package job

• Check out

• Setup Python

• Setup poetry

• Build package

• Check package

• Publish package, if push to tag triggered the job

Reusable Actions

Check out

steps:
  - uses: "actions/checkout@v2.0.0"

Reusable Actions

Install python

steps:
  - name: "Install Python"
    uses: "actions/setup-python@v1.2.0"
    with:
      python-version: "3.8"

Reusable Actions

Install poetry

steps:
  - name: "Install poetry"
    uses: "dschep/install-poetry-action@v1.3"
    with:
      version: "1.0.5"
      create_virtualenvs: true

Conditionals

Run pre-commit only on Python dev version

env:
  DEV_PYTHON_VERSION: "3.8"

steps:
  - name: "Run pre-commit"
    if: "${{ matrix.python-version == env.DEV_PYTHON_VERSION }}"
    uses: "pre-commit/action@v1.0.1"

Conditionals

Publish package

steps:
  - name: "Publish package"
    if: "${{ github.event_name == ’push’ && startsWith(github.event.ref, ’refs/tags’) }}"
    uses: "pypa/gh-action-pypi-publish@v1.1.0"
    with:
      user: "${{ secrets.PYPI_USERNAME }}"
      password: "${{ secrets.PYPI_PASSWORD }}"

Services

Definition

services:
  postgres:
    image: "postgres:12.2-alpine"
    env:
      POSTGRES_USER: "project"
      POSTGRES_PASSWORD: "project"
      POSTGRES_DB: "test_project"
    ports:
      - "5432:5432"
    options: "--health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5"

  redis:
    image: "redis:5.0.7-alpine"

Services

Usage

steps:
  - name: "Run Tests"
    run: "make migrate-only test-api-only"
    env:
      LEVEL: "test"
      DB_HOST: "postgres"
      DB_PORT: "${{ job.services.postgres.ports[5432] }}"
      REDIS_HOST: "redis"
      REDIS_PORT: "6379"

More?

• Documentation

• Configuring a workflow

• Marketplace

Using GitHub Actions

For Python libraries

Based on Hynek article: “Python in GitHub Actions”

• Run in VM

• Build job to check library is working on any OS

• Test library on all supported Python versions

• Package job to publish on PyPI

For Python libraries

Gotchas

1. Pin everything

2. Run build & test in parallel (failed job will stop workflow)

3. Package job needs build & test

4. Cache pre-commit hooks

5. tox-gh-actions

6. Provide dev Python version:

   • To run pre-commit

   • To send coveralls report

   • etc (to run only once)

For Python libraries

Examples

• playpauseandstop/rororo

• playpauseandstop/aiohttp-middlewares

• pylotcode/aiohttp-tus

For Python projects

• Run in container

• Provide basic Docker image to reuse

• Only test & deploy jobs needed

For Python projects

playpauseandstop/docker-python

runs-on: "ubuntu-latest"
container:
  image: "playpauseandstop/docker-python:3.2.0-py38"

For Python projects

Install project

- name: "Cache venv"
  uses: "actions/cache@v1.1.2"
  with:
    path: ".venv"
    key: "venv-${{ hashFiles('poetry.lock') }}"

- name: "Install"
  run: "make install"

For Python projects

Run unit & integrational tests

- name: "Run Unit Tests"
  run: "make test-unit-only"

- name: "Run Integrational Tests"
  if: "{{ github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/') }}"
  env:
    HOBOTNICA_ROOT: "~/.hobotnica-test"
    GITHUB_API_USERNAME: "${{ secrets.GITHUB_API_USERNAME }}"
    GITHUB_API_TOKEN: "${{ secrets.GITHUB_API_TOKEN }}"
  run: "make test-integrational-only"

For Python projects

Gotchas

• Base image is good idea

• Cache .venv as well

For JavaScript libraries

• Run in VM

• Same rules as for Python libraries

For JavaScript libraries

Publish to NPM

- name: "Publish package"
  if: "${{ github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags') }}"
  run: |
    set -e
    tag=latest
    if [ ${{ contains(github.event.ref, 'alpha') }} = true ]; then
      tag=alpha
    else
      if [ ${{ contains(github.event.ref, 'beta') }} = true ]; then
        tag=beta
      else
        if [ ${{ contains(github.event.ref, 'rc') }} = true ]; then
          tag=rc
        fi
      fi
    fi
    set -x
    npm publish --tag=${tag}
  env:
    NODE_AUTH_TOKEN: "${{ secrets.NPM_TOKEN }}"

For full stack projects

• Run in container

• Same rules as for Python project

• But also provide jobs for test & deploy frontend

For full stack projects

Installing node in Python image

- name: "Install node"
  run: "nodeenv -n ${{ env.NODE_VERSION }} ${HOME}/.node"

- name: "Install UI"
  run: "PATH=${HOME}/.node/bin:${PATH} make install-ui"

Bonus. Deploying this slides

.github/workflows/slides.yml

name: "slides"

on:
  push:
    paths: "slides/**"

jobs:
  slides:
    runs-on: "ubuntu-latest"

    steps:
      - uses: "actions/checkout@v2.0.0"

Bonus. Deploying this slides

Build Slides

- name: "Build slides"
  uses: "ammaraskar/sphinx-action@0.4"
  with:
    docs-folder: "./slides/"
    pre-build-command: "pip install Sphinx==3.0.3 sphinx-revealjs==0.11.0"
    build-command: "sphinx-build -b revealjs ./ _build/"

Bonus. Deploying this slides

Deploy Slides

- name: "Deploy slides"
  if: "{{ github.event_name == 'push' && github.event.ref == 'refs/heads/master' }}"
  uses: "AEnterprise/rsync-deploy@v1.0"
  env:
    ARGS: "-avuzPt"
    DEPLOY_KEY: "${{ secrets.DEPLOY_SSH_KEY }}"
    FOLDER: "./slides/_build/"
    SERVER_DESTINATION: "Projects/igordavydenko/slides/_build/"
    SERVER_IP: "igordavydenko.com"
    SERVER_PORT: "22"
    USERNAME: "playpauseandstop"

GitHub Actions issues

• Why GitHub released Actions only in 2019?

• Documentation is messy at times

• Tough to configure from the scratch

• Secrets not shared with forks

• Often problems with 3rd party reusable actions

• Failed job email notifications are useless

Use GitHub Actions for great good!

That’s all, folks!

I am Igor Davydenko

GitHub:

@playpauseandstop

Twitter:

@playpausenstop

Telegram:

@playpausenstop

Slides

Made with sphinx-revealjs

Link:

https://igordavydenko.com/slides/lvivpy-9.html

Questions?