What's the best way to make interfaces with async generators?


#1

If I have some remote resource which continuously produces one or more items and writes them somewhere, what is the best way to deliver the data to the user?

def parse_rawdata(rawdata):
    return rawdata.split(b'SPLITTOKEN')

async def res():
    rawdata = await connection.read()
    data_list = parse_rawdata(rawdata)
    return data_list

What is the best interface for an user to consume the resources? One would be to simply return the whole data list in a loop:

class MyObj:
    def __aiter__(self):
        return self
    async def __anext__(self):
        return await res() 

So the user is expected to do an async loop followed by a normal loop to iterate over the data (not very nice). Another would be async generators, so instead I would do:

async def res():
    rawdata = await connection.read()
    data_list = parse_rawdata(rawdata)
    for ele in data_list:
        yield ele

Unfortunately this is also not very nice, because as of the latest version of curio, the async for loop needs to be wrapped in curio.finalize…


#2

Short answer—having that curio.finalize() in there is annoying. I agree on that. We’re currently discussing options on this here: https://github.com/dabeaz/curio/issues/176

Ultimately, I think the async generator is going to be the best approach. I’m just not sure how it’s going to be packaged up yet.


#3

Curio’s strictness on async generators has been relaxed. Give it a try and see if it works better.


#4

Great! Seems to be working for me. Thanks!