Discussion forum for David Beazley

Curio for producer/consumer?


I am interested in implementing a single producer/multiple consumer application for processing data in parallel. I’m wondering if Curio is appropriate for this task, and whether you think it would provide any speed advantages over both asyncio and multiprocessing. Here are reference implementations of both that I’ve written: https://github.com/jdidion/producer_consumer_test/tree/master/python.


I cant speak to the performance implications of using Curio over the other options, but I do believe Curio can at least provide a working solution to your problem. Take a look at the documentation for Queues, especially the producer-consumer example at the end of the class Queue section.

There is no reason you couldn’t create multiple consumers and have all of them pulling from the same Queue.


Curio uses multiprocessing and similar techniques as asyncio to perform its work. I can’t say whether it would be faster or slower as I haven’t measured it. However, given the implementation, my guess is that it would likely be roughly comparable in performance.