Need some way to rate limit requests for sitemap_to_df about advertools HOT 3 CLOSED

I did download them manually, but the advertools was useless for me. 1 worker is still too fast, and there is no way to specify rate limiting without doing it myself.On 2023-03-05, at 14:39, Elias Dabbas ***@***.***> wrote: You can do one of the following: Use the max_workers parameter: This defaults to 8, and can be modified to change the number of concurrent requests made. You might set it to 1 to make a single request at a time. Download sitemaps individually by first setting recursive=False: import advertools as adv sitemap = adv.sitemap_to_df(SITEMAP_URL, recursive=False) Now you have all the URLs of the sub-sitemaps, and you can download them individually while introducing intervals between them. import time sitemap_df_list = [] for sitemap_url in sitemap['loc']: tempdf = adv.sitemap_to_df(sitemap_url) sitemap_df_list.append(tempdf) time.sleep(60) all_sitemaps = pd.concat(sitemap_df_list, ignore_index=False) If your user agent is not being accepted, you can change it (together with any request header) as follows: adv.sitemaps.headers['User-Agent'] = 'YOUR-USER-AGENT' Hope this solves it. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>