Discussion forum for David Beazley

Curio based on libuv


#1

To make it clear, not uvloop, but pure libuv.

I am thinking now, that it would be great (not a small effort though) to make curio work on top of libuv. Alternative kernel?

There are multiple obvious benefits, like speed. And I do not want to discuss all of them here, but to me it is also native support for filesystem events (inotify, filesystem watcher) which python historically lacks and which allow very effective caching of file content.

Now, why pure libuv. First of all, asyncio loop provided by uvloop is not good base for curio. Second, uvloop is, because of reasons unknown, Linux specific, while libuv is highly portable. So libuv will allow high performance Python servers under Windows too. If CFFI and not Cython will be used, it will be even better, because PyPy JIT + libuv will make curio unbelievably fast.

I understand that curio is not for [high load] production right now, but I love it so much and sad it’s not the best in everything.


#2

I haven’t checked in detail for curio, but I looked at this for trio a while ago. At the time I concluded that it was impossible to implement trio on top of libuv, because libuv’s support for cancellation was too limited. Curio may have a similar issue.


#3

@njsmith I’m quite interesting in this as well, I’m trying to use C++ asio (no boost) and pybind11 though instead of libuv and my progress are very very slow (just managed to see how to actually call a coroutine and get a result). Why do you think libuv’s cancellation support is too limited? What does trio need which you figured out can’t be done with libuv? It would be good to know so I can actually figure this out for asio as well.
Thanks


#4

I don’t remember the details, but IIRC most operations in libuv don’t support cancellation at all. (Presumably because node.js doesn’t really have a cancellation system.)


#5

But is it required in the first place? Is cancelling sending to socket really that important? I do not know any case of such behavior except right before connection termination. Can’t we implement cancel, if not possible otherwise, as wait for completion and discard result? In my very unprofessional tests libuv is 2.5 times faster than curio.


#6

Curio just uses the selectors module, by default whatever is its default. That, in turn, is a thin abstraction right on top of The OS’s low-level interface (epoll, kqueue, etc.). So, I’m skeptical that libuv even adds any value at all. Python already has all the parts you need, and Curio adds the coroutine runner on top of that.


#7

On my PC

11850±50 TPS with UVLoop
5600±50 TPS with Curio
3400±50 TPS with AsyncIO (we knew it’s bad)

So it’s more than two times faster.