Improving time turnaround for activity requests about chembl_webresource_client HOT 6 CLOSED

Hi,
This week I'm working on improving the speed so I'm happy to have a look at your example and experiment with different settings both on the client and server side. Can you please give a list of ~5k targets that give take you 2m45s to retrive activities?

from chembl_webresource_client.

It appears that I goofed, and the slow part isn't the data retrieval, but the conversion/looping-through the list of dicts to just a list of lists of the relevant properties:

for x in chembl_ids:
    if x not in existing_files:
        print("Checking CHEMBL id {}".format(x))
        res = activity.filter(target_chembl_id='{}'.format(x), relation='=', assay_type='B', standard_type="IC50")
        print("Data retrieved")

        for i,z in enumerate(res):
            if z['parent_molecule_chembl_id'] and z['canonical_smiles'] and z['standard_value'] is not None:
                temp.append([str(z['parent_molecule_chembl_id']), str(z['canonical_smiles']), float(z['standard_value'])])

        write_csv(x, temp)

The data retrieval itself looks to take ~1-2s, but the conversion takes the remaining few minutes, at least.

from chembl_webresource_client.

OK, in that case is there anything I can help you here?

from chembl_webresource_client.

BTW: if you just rerun your script and noticed it's much faster it can because now the results are cached on the client side so you may concider clearing cache or configuring the client not to use the cache during profiling.

from chembl_webresource_client.

Ah, thanks for the clarification. Didn't know things were getting cached on our side--it still seems that that bulk of the time is in processing of the dict being returned, so I'll close this out!

from chembl_webresource_client.

No worries, I'm still optimising the API performance, next week it should work smoother but don't expect any dramatic improvements.

from chembl_webresource_client.