async/await in Python

One year later

I am...

asyncio in Python

A brief history

Async programming in Python

asyncio

  • Everything started back Dec 2012 with PEP 3156
  • asyncio has been included into standard library 2 years ago, in Python 3.4
  • Min version to use asyncio: Python 3.3
  • asyncio has been ported to Python 2 as trollius

What is included in asyncio?

  • Pluggable event-loop with various system-specific implementations
  • Transport and protocol abstractions
  • Concrete support of TCP, UDP, SSL, subprocess pipes, delayed calls, and others
  • A Future class adapted for use with event loop
  • Coroutines and tasks based on yield from
  • Cancellation support for Futures and coroutines
  • And few other features

Hello world! in asyncio

import asyncio
import random

@asyncio.coroutine
def hello_world(idx):
    yield from asyncio.sleep(random.uniform(.1, .5))
    print('Hello, world!', idx)

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait([hello_world(idx) for idx in range(10)]))
loop.close()

Hello, world! in asyncio

Hello, world! 2
Hello, world! 9
Hello, world! 4
Hello, world! 5
Hello, world! 1
Hello, world! 3
Hello, world! 8
Hello, world! 7
Hello, world! 6
Hello, world! 0

Remove asyncio magic

import random
import time

def hello_world(idx):
    time.sleep(random.uniform(.1, .5))
    print('Hello, world!', idx)

[hello_world(idx) for idx in range(10)]

Remove asyncio magic

Hello, world! 0
Hello, world! 1
Hello, world! 2
Hello, world! 3
Hello, world! 4
Hello, world! 5
Hello, world! 6
Hello, world! 7
Hello, world! 8
Hello, world! 9

Add asyncio magic back

import asyncio

@asyncio.coroutine

yield from asyncio.sleep(...)

Add asyncio magic back

loop = asyncio.get_event_loop()

asyncio.wait([...])

loop.run_until_complete(...)

loop.close()

async/await statements

yield from is so 1998

Welcome async/await!

  • Again everything started with PEP, PEP 492
  • First draft: last April
  • Received warm greetings from community
  • Included in Python 3.5, last September

Initial implementation

@asyncio.coroutine
def hello_world(idx):
    yield from asyncio.sleep(random.uniform(.1, .5))
    print('Hello, world!', idx)

async def instead @asyncio.coroutine

async def hello_world(idx):
    yield from asyncio.sleep(random.uniform(.1, .5))
    print('Hello, world!', idx)

await instead yield from

async def hello_world(idx):
    await asyncio.sleep(random.uniform(.1, .5))
    print('Hello, world!', idx)

What else?

Before

with (yield from engine) as conn:
    yield from conn.excute(...)

After

async with engine.acquire() as conn:
    await conn.execute(...)

What else?

Before

for item in (yield from conn.execute(...)):
    ...

After

async for item in conn.execute(...):
    ...

asyncio stack

aio-libs

  • aiohttp
    HTTP client/server for asyncio.
  • aiopg
    PostgreSQL client for asyncio. With SQLAlchemy core support.
  • aioredis
    Redis client for asyncio.
  • aio…
    There is also a clients for MySQL, Memcached, ZMQ, AMQP, ...

Other Frameworks

  • Asphalt
    asyncio based microframework for network oriented applications
  • autobahn
    WebSocket and WAMP in Python for Twisted and asyncio
  • API Hour
    Write efficient network daemons (HTTP, SSH...) with ease
  • nautilus
    A framework for building event-driven microservice applications

Other Frameworks

  • muffin
    Fast, simple and asyncronous web-framework for Python 3
  • Growler
    Micro web framework built atop of asyncio
  • albatross3
    A modern, fast, simple async python3 web framework
  • Kyōkai
    Fast asynchronous Python server-side web microframework

More

  • curio
    The coroutine concurrency library
  • aioh2
    HTTP/2 implementation with hyper-h2 on Python 3 asyncio
  • aiotg
    Framework for building Telegram bots
  • aiofiles
    File support for asyncio
  • aioredx
    Pythonic redux
  • Agent
    Async generators for humans
  • twtxt
    Decentralised, minimalist microblogging service for hackers

asyncio in production

Web Applications

aiohttp.web

Scrapers

aiohttp.client

Multiprocessing

asyncio…

What's next?

Is Python fast enough?

  • Why to write Python, if node.js is faster?
  • Why to write Python, if Go is faster?
  • Is asyncio slow?

Welcome, uvloop

uvloop makes asyncio 2-4x faster

uvloop makes asyncio 2-4x faster

uvloop is fast

aiohttp + Gunicorn

gunicorn -b 0.0.0.0:8000 -k aiohttp.worker.GunicornUVLoopWebWorker -w 5 \
path.to.app:app

asyncio

import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

Welcome, asyncpg

3x faster than psycopg2 & aiopg

3x faster than psycopg2 & aiopg

asyncpg is fast

async def run():
    conn = await asyncpg.connect(user='user', password='password',
                                 database='database', host='127.0.0.1')
    values = await conn.fetch('SELECT * FROM mytable')
    await conn.close()

Python 3.6

  • All changes to asyncio in 3.6 will be backported to 3.5
  • Main changes related to support of custom loop as uvloop

FAT Python

  • Efficient guards using versionned dictionaries to check if something was modified
  • Guards are used to decide if the specialized bytecode of a function can be used or not
  • fatoptimizer project
  • fat module

PyPy

The Future is Now

async/await is
Python next big thing

Questions?