Why I’m using poetry
when you aren’t

Agenda

  • History of Python package managers
  • Problems
  • How it done in other languages
  • PyPA initiative
  • poetry – Python pacakging and depndency management made easy
  • poetry Pros & Cons
  • What’s next for poetry?

History

PYTHONPATH

$ curl -O package.tar.gz
$ tar -xzf package.tar.gz --directory=lib/package
$ PYTHONPATH=lib python app.py

import sys
sys.path.append('../lib')

python setup.py

$ curl -O package.tar.gz
$ tar -xzf package.tar.gz
$ python package/setup.py

easy_install

Part of setuptools, now is deprecated

$ curl -O package.tar.gz
$ easy_install package.tar.gz

Is able to install packages from PyPI:

$ easy_install package

pip

pip install packages

$ pip install package
$ pip install -r requirements.txt
  • Active development started at 2008
  • 1.0 version released 8 years ago, 4th of April, 2011
  • Current release: 19.0.3

Problems

Wildcard dependencies

requirements.txt

Django

$ pip install -r requirements.txt
...
Successfully installed Django-2.1.7 pytz-2018.9

Dependency resolver

requirements-dev.txt

awsebcli==3.14.13

$ pip install -r requirements-dev.txt
...
Successfully installed PyYAML-3.13 awsebcli-3.14.13 blessed-1.15.0
botocore-1.12.120 cached-property-1.5.1 cement-2.8.2 certifi-2019.3.9
chardet-3.0.4 colorama-0.3.9 docker-3.7.1 docker-compose-1.23.2
docker-pycreds-0.4.0 dockerpty-0.4.1 docopt-0.6.2 docutils-0.14 future-0.16.0
idna-2.7 jmespath-0.9.4 jsonschema-2.6.0 pathspec-0.5.9 python-dateutil-2.8.0
requests-2.20.1 semantic-version-2.5.0 six-1.11.0 termcolor-1.1.0
texttable-0.9.1 urllib3-1.24.1 wcwidth-0.1.7 websocket-client-0.56.0

Virtual environment management

$ python -m venv .venv
$ .venv/bin/activate
(.venv)$ pip install ...

Others

  • Few of the other problems…
  • Hash check not enabled by default
  • Adding requirements via pip add?
  • Updating requirements via pip update?

Other languages

Rust

  • cargo new to create new package
  • cargo install CRATE to install external crate (package)
  • cargo build to build the package
  • And…
  • cargo run
  • cargo test
  • cargo doc
  • cargo publish
  • Behind the scene cargo stores everything in Cargo.toml and Cargo.lock

JavaScript. npm

  • Default package manager for JavaScript
  • Before npm@5 did not have lock file
  • Had problem with dependency resolving, which partially fixed by pinning deps

JavaScript. yarn

  • Package manager for JavaScript from Facebook devs
  • Introduced lock file: yarn.lock
  • Do not care about node_modules, while managing deps
  • Cause of yarn, npm got package-lock.json
  • Cause of yarn, npm@6 fixed many issues, but not all

PyPA initiative

pip improvements

  • Wheels
  • Environment markers (PEP 508)
  • Build dependencies (PEP 518)

pipenv

  • Attempt to make something new
  • Pipfile to store package metadata
  • Pipfile.lock to store dependencies versions
  • Many additional commands

poetry

poetry goals

  • Official site
  • Fix issues with pip & pipenv
  • Have tools for packaging Python libraries
  • Better dependency resolver management
  • Better virtual environment management
  • Originally inspired by composer & cargo

Installation & Upgrade

Install with:

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

Upgrade with:

$ poetry self:update

What’s included?

poetry new

Create new package at path

shep:Projects playpauseandstop$ poetry new test-project
Created package test-project in test-project
shep:Projects playpauseandstop$ exa -l test-project/
.rw-r--r-- 297 playpauseandstop 23 Mar 12:20 pyproject.toml
.rw-r--r--   0 playpauseandstop 23 Mar 12:20 README.rst
drwxr-xr-x   - playpauseandstop 23 Mar 12:20 test_project
drwxr-xr-x   - playpauseandstop 23 Mar 12:20 tests

poetry init

Add pyproject.toml to the existing project

shep:Projects playpauseandstop$ mkdir test-project
shep:Projects playpauseandstop$ cd test-project/
shep:test-project playpauseandstop$ poetry init

This command will guide you through creating your pyproject.toml config.

...

Would you like to define your dependencies (require) interactively? (yes/no) [yes] no
Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes] no

Generated file

...

Do you confirm generation? (yes/no) [yes] yes

shep:test-project playpauseandstop$ exa -l .
.rw-r--r-- 297 playpauseandstop 23 Mar 12:30 pyproject.toml

poetry add

Add dependency or dev dependency to the project

shep:test-project playpauseandstop$ poetry add aiohttp
Creating virtualenv test-project-py3.7 in /Users/playpauseandstop/Projects/test-project/.venv
Using version ^3.5 for aiohttp

...

Writing lock file

  - Installing idna (2.8)
  - Installing multidict (4.5.2)
  - Installing async-timeout (3.0.1)
  - Installing attrs (19.1.0)
  - Installing chardet (3.0.4)
  - Installing yarl (1.3.0)
  - Installing aiohttp (3.5.4)
shep:test-project playpauseandstop$ exa -l
.rw-r--r-- 6.4k playpauseandstop 23 Mar 12:33 poetry.lock
.rw-r--r--  314 playpauseandstop 23 Mar 12:32 pyproject.toml

poetry add

Dependency resolver included

shep:test-project playpauseandstop$ poetry add async-timeout==1.1.0

Updating dependencies
Resolving dependencies... (15.1s)

[SolverProblemError]
Because no versions of aiohttp match >3.5,<3.5.1 || >3.5.1,<3.5.2 || >3.5.2,<3.5.3 || >3.5.3,<3.5.4 || >3.5.4,<4.0
 and aiohttp (3.5.0) depends on async-timeout (>=3.0,<4.0), aiohttp (>=3.5,<3.5.1
 || >3.5.1,<3.5.2 || >3.5.2,<3.5.3 || >3.5.3,<3.5.4 || >3.5.4,<4.0) requires
 async-timeout (>=3.0,<4.0).
And because aiohttp (3.5.1) depends on async-timeout (>=3.0,<4.0)
 and aiohttp (3.5.2) depends on async-timeout (>=3.0,<4.0), aiohttp (>=3.5,<3.5.3
 || >3.5.3,<3.5.4 || >3.5.4,<4.0) requires async-timeout (>=3.0,<4.0).
And because aiohttp (3.5.3) depends on async-timeout (>=3.0,<4.0)
 and aiohttp (3.5.4) depends on async-timeout (>=3.0,<4.0), aiohttp (>=3.5,<4.0) requires async-timeout (>=3.0,<4.0).
So, because test-project depends on both aiohttp (^3.5) and async-timeout (=1.1.0), version solving failed.

poetry remove

Remove dependency and all its dependencies

shep:test-project playpauseandstop$ poetry remove aiohttp
Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 7 removals

Writing lock file

  - Removing aiohttp (3.5.4)
  - Removing async-timeout (3.0.1)
  - Removing attrs (19.1.0)
  - Removing chardet (3.0.4)
  - Removing idna (2.8)
  - Removing multidict (4.5.2)
  - Removing yarl (1.3.0)

poetry update

Update dependencies


shep:project playpauseandstop$ poetry show -o
decorator           4.3.2   4.4.0   Better living through Python with decorators
ipython             7.3.0   7.4.0   IPython: Productive Interactive Computing
uvloop              0.12.1  0.12.2  Fast implementation of asyncio event loop on top of libuv
shep:project playpauseandstop$ poetry update
Updating dependencies
Resolving dependencies... (3.2s)

Package operations: 0 installs, 3 updates, 0 removals

Writing lock file

  - Updating decorator (4.3.2 -> 4.4.0)
  - Updating ipython (7.3.0 -> 7.4.0)
  - Updating uvloop (0.12.1 -> 0.12.2)

poetry run

Run command inside virtual environment

shep:test-project playpauseandstop$ $ poetry run python -m aiohttp.web app:create_app
DEBUG:asyncio:Using selector: KqueueSelector
======== Running on http://localhost:8080 ========
(Press CTRL+C to quit)

And more…

  • poetry build to build package to wheel or sdist
  • poetry publish to publish package to PyPI or private repository
  • poetry shell to spawns a shell within a virtual environment

Pros & Cons

Pro. Dependency Resolver

  • Using dependency lock is a real thing
  • It is slower then pip or pipenv, but it resolve dependencies
  • You will not install a wrong dependency
  • poetry update also cares about resolving dependencies

Pro. Simplify Dependency Maintainment

  • poetry update rules

Pro. Dotenv + poetry

dotpoetry

#!/bin/bash
#
# Run poetry as usual, but before load values from dotenv file(s).
#

set -e

dotenv () {
    filename=$1
    if [ -f "$filename" ]; then
        set -a
        source "$filename"
        set +a
    fi
}

LEVEL=${LEVEL:-dev}

dotenv .env
dotenv ".${LEVEL}.env"
dotenv ".env.${LEVEL}"

$HOME/.pyenv/versions/`cat .python-version`/bin/python $HOME/.poetry/bin/poetry "$@"

Pro. Publishing Packages to PyPI

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

You don’t need twine to publish packages

Pro. Building Packages Included

  • poetry build rules

Con. Do not supported by Read the Docs

You need to have setup.py & docs/requirements.txt

update-setup-py: .install
  rm -rf dist/
  $(POETRY) build
  tar -xzf dist/$(PROJECT)-*.tar.gz --directory dist/
  cp dist/$(PROJECT)-*/setup.py .
  rm -rf dist/

Con. Do not supported by Amazon ElasticBeanstalk

You need to generate requirements.txt

generate-requirements:
  -rm -rf .venv/ .install
  $(POETRY) install --no-dev
  .venv/bin/pip uninstall -y $(PROJECT)
  .venv/bin/pip freeze > requirements.txt
  rm -rf .venv/

Con. Custom Docker image

poetry not available at python official image

FROM michalmazurek/python-poetry

Con. GitHub Dependencies

poetry not yet available on GitHub deps page

What’s next?

poetry 1.0

python-poetry

GitHub Organization

Conclusion

poetry is for real

Questions?

Twitter: @playpausenstop
GitHub: @playpauseandstop