trivio / common_crawl_index Goto Github PK
View Code? Open in Web Editor NEWIndex URLs in Common Crawl
Index URLs in Common Crawl
Not sure if this is the right place to create a ticket, but I'm wondering if it's possible to update the url index to the March 2014 crawl?
I think I found incorrectly sorted URLs in the index. For example, in block 4, net.about-plumbing.www/... comes after net.absolutely.www/...:
'my.com.mahsuri.www/blog/rawatan/ti\x00',
'name.armando.francesco.www/gallery/andrea_e_vera/slides/DSCN4325\x00',
'net.123tools.www/audio_multimedia/audio_file_players/index-n-123tools-3\x00',
+ 'net.about-plumbing.www/new-hampshire/colebrook-m\x00',
"net.absolutely.www/event/Premiere_of_'Land_of_the_Lost'/land_of_the_lost_02_wenn24377\x00",
- 'net.about-plumbing.www/new-hampshire/colebrook-m\x00',
'net.adiochiropractic.www/templates20/article/1296\x00',
'net.agilpage.www/index~m~1~w~11276\x00',
'net.alblasserdam.www/nieuws/2009-07-02-6051-internetprovider-proserve-bouwt-regionaal-datacenter-in-alblasserdam.html:http\x00',
(The above is a diff of the actual list of prefixes in the block and the sorted list of prefixes in the block.)
This could cause a failure of the bisection lookup.
Using the remote_read script I think there is a typo in the item_keys. Shouldn't arcFileParition be arcFilePartition?
The documentation recommends using a specific instance, m1.xlarge EC2 instance "for a fast connection to S3"
https://github.com/trivio/common_crawl_index#using-the-remote_copy-utility-script
Since then that instance is now considered a "previous generation" amazon instance (they are on m3 now).
They also have a bunch of other instances that might not have been available when the documentation and script was initially written, some optimized for different things, as opposed to the M series, which provides a balance.
http://aws.amazon.com/ec2/instance-types/
It's also suggested that the CC data be hosted on s3, in which case network performance might be less critical for the machine?
Do you have any recommendations about what machine might be optimal with and without CC hosted on s3?
Input:
./bin/remote_copy check "org.domain.www"
Output:
Traceback (most recent call last):
File "./bin/remote_copy", line 152, in <module>
mmap = BotoMap(s3_anon, src_bucket, '/common-crawl/projects/url-index/url-index.1356128792')
File "./bin/remote_copy", line 35, in __init__
self.key = bucket.get_key(key_name)
AttributeError: 'NoneType' object has no attribute 'get_key'
Hello,
I am experimenting with your Python script which seems very promising for a separate project requiring lookups of Common Crawl URLs. I have installed your current version from Git on my Ubuntu machine and ran:
$ bin/index_lookup_remote france.fr Traceback (most recent call last): File "bin/index_lookup_remote", line 26, in <module> import boto ImportError: No module named boto
Can you please tell me what boto
is supposed to be?
Thanks,
Dan
Hello,
Could this script retrieve urls belonging to a country tld eg *.fi?
Function reversehost() in some cases makes original url recovery ambiguous,
i.e ua.com.book-hunter.www/book/view/231/page:16:http represents both
reversehost('http://www.book-hunter.com.ua/book/view/231/page:16')
== reversehost('http://www.book-hunter.com.ua:16/book/view/231/page')
== 'ua.com.book-hunter.www/book/view/231/page:16:http'
which makes such revers hostname order ambiguous and reversehost() procedure not always invertible to the actual URL.
Right now CC contains some urls with username:password in them.
On index creation they were transformed by function reversehost() and as a result they are not searchable.
reversehost('http://123456:[email protected]/')
== '123456/:[email protected]:http'
reversehost('http://Dennis:[email protected]/members/index.shtml')
== 'Dennis/members/index.shtml:[email protected]:http'
I am trying to copy a file in text mode, but it is not working. The URL is com.wordpress.alinebessa/2011/06/11/documenting-accerciser-first-impressions/:http
which exists in CommonCrawl. When I check it out here: http://urlsearch.commoncrawl.org/page/1346876860454/1346973204444/3513/41986721/13163
It gets loaded correctly, but this does not happen when I try to fetch it in remote_copy (method copy_arc_files) by making:
if src_key:
print src_key.get_contents_as_string(headers=headers, encoding="iso-8859-1")
It comes back to me as bytes. Can you folks please help me in retrieving the actual text? Thanks!
The reason i believe it should return something is:
url -r 1048584-1048744 http://s3.amazonaws.com/aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792
which is the second index block, which contains prefixes for this domain
I expected all of these to give the same first result:
$ bin/remote_read org.wikipedia.en | head -1
org.wikipedia.en/wiki/1525:http {'compressedSize': 15889, 'arcSourceSegmentId': 1346876860777, 'arcFilePartition': 4752, 'arcFileDate': 1346910706993, 'arcFileOffset': 75900503}
$ bin/remote_read org.wikipedia.en/wiki | head -1
org.wikipedia.en/wiki/1647_in_literature:http {'compressedSize': 9294, 'arcSourceSegmentId': 1346876860777, 'arcFilePartition': 1900, 'arcFileDate': 1346910123817, 'arcFileOffset': 77716338}
$ bin/remote_read org.wikipedia.en/wiki/1 | head -1
org.wikipedia.en/wiki/1942:_Joint_Strike:http {'compressedSize': 10488, 'arcSourceSegmentId': 1346823846039, 'arcFilePartition': 1724, 'arcFileDate': 1346872155599, 'arcFileOffset': 10475238}
$ bin/remote_read org.wikipedia.en/wiki/1942 | head -1
[no output]
The last one didn't even find the existing URL. Am I doing something wrong?
Is this project deprecated? I see there are no commits since 2013, and there appears to be a new index scheme available since 2015: http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/
Since the example in the README.md is not working, I'm guessing that this project is not being (and does not need to be) maintained. If that's the case, I think we could help some people by updating the README.md file to indicate that the project is deprecated by the new index.
If that's true, I am happy to create a pull request for it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.