support pluggable http clients/coroutine frameworks (anyio/httpx)

[graingert]

I'd like to be able to use aiobotocore in a project that uses trio and httpx

anyio provides a generic interface to curio/trio/asyncio that libraries can use to provide generic support for all async/await eventloops

aiobotocore would require users to pass a "request" coroutine function that provides all the http functionality that aiobotocore needs for http requests

[graingert]

As far as I'm aware aiobotocore would need to replace all asyncio calls with the relevant anyio equivalent, and any gather/ensure_future with a nursary/spawn pattern instead

[thehesiod]

should be done with #619 which will cause a TON of refactoring and perhaps re-thinking this whole project

[graingert]

I don't think #619 and this issue are really that coupled and could be completed in any order

[dazza-codes]

botocore/boto3 already supports an event callback for extensions and aiobotocore inherits that behavior without any modifications. See

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
• use aiobotocore as a starting point to create a new base class and hook it up to a boto3 event when it creates clients.
  • https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#creating-client-class

To focus only on the request component, if that's possible, consider details in #755 that describe how moto uses the event system to mock requests. It hooks into the before-send event. So long as the http_response returned is compatible with the rest of the response parsing, it should work. If it needs a custom response parser, register a parser-generator as a component in the client.

All of this is easily understood (mostly) from reading

• https://github.com/boto/botocore/blob/develop/botocore/client.py

[thehesiod]

the reason I said #619 (#659) is because it's going to greatly expand this codebase, and thus parts which would need to be refactored to support whats mentioned in this issue. IIRC there are sleeps/locks for example in the new credentials file we need to replace components in. @dazza-codes that's great insight for that one particular event, I just logged #774 so we don't forget about it and apply it where possible. Given #659 moves us to async events a lot more will be possible.

[ediskandarov-cur]

What is the roadmap here. I would contribute if there's a plan.

aiohttp dependency is discouraged in projects that use a different server web framework.

[graingert]

@toidi step 1 would be to find any imports/calls to asyncio and replace it with the anyio equivalent

[ediskandarov-cur]

@thehesiod are you ok with the proposal above?

[thehesiod]

Let me take a look at anyio and report back

[thehesiod]

ok looking at anyio that's not going to solve the http client portion, that tries to solve asyncio event loop plugging, which actually is already designed to be globally replaceable (see for example how uvloop works). From what I can see botocore is already working on a pluggable http client session, see: https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L280 however it's not piped to either Config or create_client yet from what I can see, seems like a WIP. So I think we need to wait to see what botocore is going to do. Another factor is that it's not as simple as just allowing a aiohttp replacement, you need to port all the exceptions as well (https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/endpoint.py#L156), I just went through that in fact on the most recent botocore update. looping in @terrycain .

[graingert]

that tries to solve asyncio event loop plugging, which actually is already designed to be globally replaceable

I'm looking to replace the coroutine framework not the event loop

[thehesiod]

ah I c, sorry this is a two prong request, probably should be split as they're not necessarily related