How to get initial url in output.jl ? about advertools HOT 2 CLOSED

Thanks a lot @caroheymes
Glad you're using it!

This is a very important issue. Good news, the solution already exists (in all crawl files you have already). Maybe I should do a better job documenting and explaining.

In the crawl df, url refers to the crawled URL (downloaded, parsed, and its elements saved to a file on disk), but as you know it might not be the URL that was requested. This typically happens where there are redirects. The issue is that we might have one, two, three, or even more redirects.

Example:

import pandas as pd
crawl_df = pd.read_json('output_file.jl', lines=True)
crawl_df.filter(regex='^url$|redirect_').dropna()

The redirect_urls column contains all URLs requested and redirected, leading finally to url. In the first row we have one redirect (301), and in the second we have two. We also have the full redirect chain in both rows (one in the first and two in the second). So you can also see intermediate pages if any.

As all other columns, multiple values are separated by @@.

I hope this clarifies it.

from advertools.

Assuming the question is answered. Feel free to re-open if you have other questions on this topic.

from advertools.