Async download and caching in `op` module

[Mr-Milk]

Description of feature

Thanks for developing such a nice package.

The download of a large resource in the op module can be a bit time consuming. I wonder if we could make the download asynchronously and return a future object instead.

Potential API design:

markers = dc.op.resource('MSigDB', as_future=True)  # Non-blocking
markers = markers.result()  # Get the result

For caching, since the database is from OmniPath and hosted by your team, the etag would be a graceful solution. This is also how huggingface handle file caching.

I'm happy to submit a PR for the async download part.

[deeenes]

Hello Yimin, Thanks for the great suggestion! We don't really plan to add new features to this module, because its design is a fascinating showroom of extravagant Python tricks, which makes it difficult to maintain and extend. Instead, our longer term plans is to implement a new OmniPath client, using our recently developed download manager, where the async i/o should be implemented. Still, that will take some months to realize, and if you see an easy, low effort solution that could improve the situation here (maybe only in decoupler), we'll be glad to receive a PR from you.

Using etag also is a good idea, we're working on a completely new API for OmniPath and about to deploy a new implementation of the current, legacy one, we can easily implement it in both.

[PauBadiaM]

Hi @Mr-Milk,

The async download makes a lot of sense, yes please go ahead!

Regarding caching, I'd rather not use it for now. The thing is that omnipath gets constantly updated, so if we implement caching people can end up using older versions of the databases without knowing. I'd rather let people always download the latest version and save it as a csv locally if they want to have reproducible results.

[Mr-Milk]

Thanks for the responses @deeenes @PauBadiaM , I'll go ahead and submit a PR.