Skip to content

Conversation

@james-gray
Copy link
Contributor

@james-gray james-gray commented Apr 2, 2020

This branch adds async support to all API endpoints to make them non-blocking, using asyncio and aiohttp.

  • Used aiohttp to make the two requests for data sources instead of requests, as requests is not compatible with asyncio A key piece of this is the addition of a global aiohttp.ClientSession instance, which exists for the entire application run.
  • Updated methods up the call chain with async/await as necessary.
  • Updated the endpoint methods to be async as per https://fastapi.tiangolo.com/async/.
  • Updated the tests to handle asynchronous calls.
  • Added tests for the setup and teardown of the global ClientSession.

Added dependencies:

  • aiohttp
  • asyncache
  • async-asgi-testclient
  • pytest-asyncio
  • importlib-metadata (backported from 3.8)
  • idna_ssl (backported from 3.7)
  • dataclasses (backported from 3.7)

#232

(Reopening this on my personal GH account to avoid authorship confusion.)

These are required to be able to test asyncio functionality, as well as making ASGI test clients play nice asynchronously.
- Move test fixtures into tests/fixtures.py
- Update test fixtures to mock aiohttp.ClientSession.get, instead of requests.get
- Add a context manager to replace the global httputils.client_session with an AsyncMock
- Misc. cleanup
I didn't see this file when I first added the new fixtures.
One can still define a separate responder function here, but since this is the
only response defined for the tests thus far, this seems like a reasonable default.
@james-gray james-gray changed the title Async endpoints #232: Make all endpoints async Apr 2, 2020
@james-gray
Copy link
Contributor Author

Builds are currently failing for 3.6 and 3.7, I wrote this PR under 3.8 - looks like some of the async test fixtures are new in the standard lib as of 3.7 and 3.8. I'll take a look for backport options for these.

Using these as decorators causes a lot of problems when combined with Pytest fixtures, often resulting in the mock not applying.
@Kilo59 Kilo59 added the performance Issue related to performance and optimizations label Apr 2, 2020
@Kilo59 Kilo59 linked an issue Apr 2, 2020 that may be closed by this pull request
@Kilo59 Kilo59 added this to the Performance Optimizations milestone Apr 3, 2020
@Kilo59 Kilo59 self-assigned this Apr 3, 2020
Copy link
Collaborator

@Kilo59 Kilo59 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

[dev-packages]
async-asgi-testclient = "*"
async_generator = "*"
asyncmock = "*"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These back-ports should be adjusted so that they are only installed when required.
https://pipenv.readthedocs.io/en/latest/basics/#specifying-versions-of-a-package
https://pipenv.readthedocs.io/en/latest/advanced/#specifying-basically-anything

I'll try to do this tonight if I have a chance.
Thanks for all the notes, should make this easy.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem, happy to help!

@Kilo59 Kilo59 merged commit 2b2fc19 into ExpDev07:master Apr 3, 2020
@Kilo59
Copy link
Collaborator

Kilo59 commented Apr 3, 2020

@all-contributors please add @james-gray for code

@allcontributors
Copy link
Contributor

@Kilo59

I've put up a pull request to add @james-gray! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Issue related to performance and optimizations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make all endpoints async

3 participants