Discussion forum for David Beazley

Thredo's evil twin, or mixing sync and async in reentrant kernel


#1

Hello!

I could not find anything about this, except that this is not right and goes against the curio philosophy, so I gave it a shot.

import curio

async def socket_recv():
    """Mocks some kind of blocking operation, working via curio"""
    print('Async socket_recv: working')
    await curio.sleep(5)
    print('Async socket_recv: done')
    return 32


def lib_func(arg):
    """Mocks some kind of sync library, that calls something blocking using curio"""
    print('Sync lib_func: working')
    res = curio.AWAIT_SYNC(socket_recv)  # restarts curio's kernel
    print('Sync lib_func: socket_recv returned', res)
    res += 10
    print('Sync lib_func: done')
    return res


async def counter():
    n = 0
    while True:
        print(f'Async counter: {n}')
        n += 1
        await curio.sleep(1)


async def main():
    print('Async main: started')
    print('Async main: launching counter')
    await curio.spawn(counter, daemon=True)
    await curio.sleep(2)

    print('Async main: launching lib_func')
    res = await curio.run_sync(lib_func, 'hello')  # pauses curio's kernel
    print('Async main: lib_func returned', res)


if __name__ == '__main__':
    curio.run(main)

Result:

*** Kernel started  # in meta.running
*** Kernel yielded  # above yield in Main Kernel Loop
*** Kernel resumed  # below yield in Main Kernel Loop
Async main: started
Async main: launching counter
Async counter: 0
Async counter: 1
Async main: launching lib_func
Async counter: 2
*** Kernel yielded
Sync lib_func: working
*** Kernel resumed
Async socket_recv: working
Async counter: 3
Async counter: 4
Async counter: 5
Async counter: 6
Async socket_recv: done
*** Kernel yielded
Sync lib_func: socket_recv returned 32
Sync lib_func: done
*** Kernel resumed
Async main: lib_func returned 42
*** Kernel yielded
*** Kernel stopped  # in meta.running
*** Kernel started  # that's shutdown, i guess
*** Kernel resumed
*** Kernel yielded
*** Kernel stopped

WHY‽

I tried to write a curio server that interacts with docker. Unfortunately docker-py is as anti-sans-io as it could be, and so are a bunch of other libraries. The socket calls are buried underneath the whole logic :frowning:

I’m afraid that sans-io is not gonna take over the world in the near future, and converting the whole docker api to sans-io or curio for my non-commercial project is infeasible. So I tried to create some way of integrating synchronous code with my async framework of choice.

Creating run_in_thread wrapper around the whole docker api or using async threads is probably the curio-way, but why then use curio if most of the code is running in threads? All of the asyncs, AWAITs and awaits just add noise, increase complexity and degrade performance. And besides, non-blocking code is the same, wherever it is in function or in coroutine, all we need to worry about is the blocking stuff.

Right now I can hardly imagine what horrible bugs would result from using this in practice, since I’ve barely managed to get this toy example to work (and make a semblance of a sense in my head). For example: socket_recv works inside curio that works inside synchronous lib_func that works inside Kernel.run, but outside kernel loop, despite being scheduled from inside the loop. That’s seriously messed up.

I don’t know if this is a way forward, most likely it is not, but the all-in nature of coroutines needs to be somehow worked around, and if this workaround involves basically spawning per-client thread (despite how cool thredo looks), then why are we even bothering with the async?

I’d like to hear your thoughts about this: is it even possible that this approach is worth working on, or I should rm -rf the code before Cthulhu awakens?


#2

Aaaand I’ve understood why my idea is rubbish.

For posterity: if first run_sync code schedules short operation, then another async function awakens (inside AWAIT_SYNC) and shedules longer async operation - the first piece of sync code is blocked until the second is done. Also, you can easily break the whole system, if tasks are becoming progressively longer. They will create a lot of nested stack frames and could go past the recursion limit.

If you prohibit calls to run_sync nested in AWAIT_SYNC - you disallowed simultaneous waiting for resources and defeated the whole premise of async programming.

If you limit the numer of nested calles - all are blocked on the latest one if they get data earlier.

If you spawn thread per run_sync - you’ve reinvented run_in_thread, only worse.

In the end, I shouldn’t have written about my idea before fully comprehending it. Also stacks are evil and David is smart :sweat_smile: