Discussion forum for David Beazley

Different backend for curio


#1

Hi,
I’m learning curio internals, I’ve found it’s a great source to understand how async I/O programming works. One think I’d like to tinker with is to write a c++ extension to replace the curio kernel and use something like the C++ networking TS (currently I’m probably going to use non-boost asio which implements all the relevant part and it’s the source of inspiration for the TS) with pybind11. I’m still scouring through the code, but my understanding is that I can have something working just by replacing the Kernel class, am I right? For an additional boost in performance I think that rewriting the I/O using the c++ library could also help, but I would certainly keep the things simple for the first iteration.
Any guidance, suggestions or advices?


#2

Well, you may have stumbled onto Curio’s most powerful yet most secret feature–which is the fact that the backend kernel is largely irrelevant and can be swapped out by something else ;-). There are a few things you’ll need to focus on. First, the kernel needs to implement the required traps. See curio/traps.py. There are 13 of them, but most are pretty simple to implement. Second, the kernel does make use of a few data structures in Python. For example, the Task class and the scheduling primitives in curio/sched.py. There might be a number of places where the interaction between the kernel and other Python code is wonky or could be made better (i.e., activations and the timer wheel stuff). Nevertheless, what you propose is a doable project.

About two years ago, I experimented with making a pure C-extension implementation of the Curio kernel. It was purely proof of concept and the results were mixed. Because I’m often pressed for time, I decided to not pursue it any further. However, I just made the Git repo for it public for you at https://github.com/dabeaz/colonel. That code does NOT work with the current version of Curio. However, you might take a look at it for inspiration and ideas.

I’d definitely be interested in any results you produce from a project like this.


#3

Hi Dave,
Thank you for the pointers. I’ll have a look at your C extension soon. As soon as I get something working, I’ll post an update here, just don’t hold your breath, it’s a night-time project for me as my job is quite absorbing my time lately!