tosdr / tosback2 Goto Github PK
View Code? Open in Web Editor NEWThis project forked from pde/tosback2
Reimplementing TOSBack with Ruby and using git to see TOS changes!
Home Page: http://tosback.org
License: GNU General Public License v2.0
This project forked from pde/tosback2
Reimplementing TOSBack with Ruby and using git to see TOS changes!
Home Page: http://tosback.org
License: GNU General Public License v2.0
The current list is
500px
Amazon
Apple
AT&T
Comcast
Couchsurfing
Delicious
Dropbox
DuckDuckGo
Facebook
Flickr
Github
Google
Microsoft
MySpace
Netflix
Rapidshare
Skype
SoundCloud
Spotify
Steam
Tumblr
Twitpic
Twitter
Wikipedia
Wordpress.com
Yahoo!
YouTube
Would you expand the TOS list to include Instagram, Reddit, Xvideos, Pornhub and Baidu?
Just to ensure the Terms of Services of the top 10 websites in the world are covered.
Reference
TOSBACK Website
Top 100 Websites in the World
Right now, phoenix only looks at documents in the crawl_reviewed folder. It would be good if the two folders could be merged so there is only one place we have to look? Then services like ResearchGate (whose terms are in crawl and not in crawl_reviewed) can also be reviewed using Phoenix's annotate view.
It is very often the case that a service will simply copy parts of another service's ToS.
For this case and other cases, it would be nice to analyse the similarities between different documents hosted there, so we can see for instance when documents are 95% the same. This will also make it easy for ToS;DR to avoid duplicating reviews :-)
I'm in the process of creating a rules set and found that setting reviewed="false"
for a URL has the same effect as setting it to true.
The tosback.org server ran out of diskspace. I removed the git history of this repo which was taking up 15Gb out of the 20Gb disk that server has, by doing a git shallow clone. The disk issue seems solved now, but I found that the script had stopped working.
When running it manually, I found it helps if I comment out https://github.com/tosdr/tosback2/blob/master/rubycode/main.rb#L12
I guess that will mean the git repo will still be updated, but the information on the web page will no longer get updated - I guess it uses mysql to communicate with https://github.com/tosdr/ToSBack3?
In any case, it's running now, I'll see if the website gets updated once the script is finished.
Not sure if this is just git on my machine, or if there's something interesting happening in the repo somehow, but no matter how I try to refresh my repo, I get this:
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: crawl_reviewed/amazon.com/AMAZON SILK TERMS & CONDITIONS.txt
no changes added to commit (use "git add" and/or "git commit -a")
If I use git reset --hard
, then, only the capitalization changes:
modified: crawl_reviewed/amazon.com/Amazon Silk Terms & Conditions.txt
The actual diff—which is quite large—remains the same.
Any insights?
The following policies are linked from the main facebook.com page:
Of these, we're only crawling the third one. We're also crawling the Full Data Use Policy, which still exists as a URL, but is not linked from the main signup form.
Let's keep crawling the existing ones, but add the Cookie and Data policies.
For instance, I remember we got a question from a journalist when they started to mention that they might do research on users, and then we found out that it was on a page which people had to agree to, but which we were not crawling.
Please import rules that we have in tosdr.org/services/
I know that everyone hates such reports: you would think that localised ToS should change with the English "master". Well, nope.
Today I noticed that Google Play have changed their ToS. Or, more precisely:
https://play.google.com/intl/en-us_us/about/play-terms.html - English - December 18, 2017
https://play.google.com/intl/hu_hu/about/play-terms.html - Hungarian - February 5, 2018
possibly similar to
https://play.google.com/intl/en_uk/about/play-terms.html - UK English, Febr 5, 2018
So, first, it seems changes shall be tracked for localised languages, which may be kind of hundredfold amount of data.
Then there is the funny part, when multiple different English versions exist, where they could be happily diffed against one another, to show regional differences between the terms.
(As a sidenote: all the websites are in horrible shape: tosdr.org and especially tosback.org. They are desperately in need of lots of love.)
I noticed links to commits from tosback.org after August 30 are missing from github.
For instance when visiting cnet.com you agree to the cbsinteractive.com privacy policy. Should we crawl that as rules/cnet.com.xml or as rules/cbsinteractive.com.xml? I opted for the first now, but this issue deserves more attention.
As noted by @hugoroy and discussed via
email, 27f3076?diff=split#diff-ca787fbf273ad31090b8a57a734c51c8L437 looks like the previous crawl got the GDPR version of the Privacy Policy, and the current one got the non-GDPR one. Maybe we can change the URL to force one or the other (e.g. I've seen some services have something like ?eu=true
at the end of the URL).
Trying:
<sitename name="cloudsight.ai">
<docname name="Privacy Policy">
<url name="https://cloudsight.ai/privacy-policy" xpath="//[@class='container']" reviewed="true">
<norecurse name="arbitrary" />
</url>
</docname>
<docname name="Terms of Service">
<url name="https://cloudsight.ai/terms" xpath="//div[@class='container']" reviewed="true">
<norecurse name="arbitrary" />
</url>
</docname>
</sitename>
but i'm seeing a reCaptcha error :(
$ ruby main.rb ../rules/cloudsight.ai.xml
https://cloudsight.ai/privacy-policy
ReferenceError: Can't find variable: Set
reCAPTCHA couldn't find user-provided function: vueRecaptchaApiLoaded
https://cloudsight.ai/terms
ReferenceError: Can't find variable: Set
reCAPTCHA couldn't find user-provided function: vueRecaptchaApiLoaded
The Twitter TOS and Privacy Policy updated this week:
We have made revisions to these Terms of Service, effective May 25th, 2018. You can see the new Terms of Service here [https://twitter.com/tos#update]. By continuing to use our services after May 25th, you agree to the new Terms of Service. A summary of changes can be found on our Help Center [https://help.twitter.com/rules-and-policies/update-privacy-policy].
However the version in this repository hasn't been updated in 7 months: crawl_reviewed/twitter.com
The URL is the same: https://twitter.com/en/tos
But I think the xpath
rule no longer is valid. (At least I couldn't find main-content
from rules/twitter.com.xml
when looking at the source.)
Am I the only one who gets a 404 error using the commit link in the daily email?
My July 15 link:
These were changed in last night's crawl. Please have a look at the commit at dcaa695?diff=split to see the differences!
I am new to Ruby, so please tell me if I there are any gotchas around running old versions.
When I run bundle install --without production
, I get the error message Your Ruby version is 2.5.1, but your Gemfile specified 2.3.1
.
Should I try to downgrade my Ruby version or is it reasonable to keep the project’s Ruby version up-to-date? (latest version is currently: 2.6.1)
I installed master/45995736 on OS X 10.8.5 today. I'm using ruby 1.9.3-p429 via rbenv, and when I try to run the crawler from the "rubycode" directory, I get a gem specification failure:
$ ruby main.rb ../rules/dropbox.com.xml
/Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems/specification.rb:1637:in `raise_if_conflicts': Unable to activate mail-2.5.4, because mime-types-2.1 conflicts with mime-types (~> 1.16) (Gem::LoadError)
from /Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems/specification.rb:746:in `activate'
from /Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems.rb:212:in `rescue in try_activate'
from /Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems.rb:209:in `try_activate'
from /Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems/custom_require.rb:59:in `rescue in require'
from /Users/myuser/.rbenv/versions/1.9.3-p429/lib/ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
The same error persists in a clean install of 1.9.3-p429, plus the mechanize, nokogiri, sanitize, and mail gems. gem search mime-types
indicates I have versions 2.1 and 1.25.1 installed, which should satisfy the requirement for ~> 1.16, but appears not to. I can't uninstall mime-types 2.1 because mechanize requires ~> mime-types 2.0.
I'm curious to know how other people are bypassing this problem.
https://tosback.org/ shows a directory listing, suggesting it's running Apache without Passenger, probably? The root cause seems to be disk quota:
tosback3@dragon:/var/www/ToSBack3/public$ touch test.txt
touch: cannot touch 'test.txt': Disk quota exceeded
I often see <norecurse name="arbitrary" />
, but I'm not sure what it does. Could we put a section in the README that explains it?
The ToS;DR icon in the header of http://tosback.org/ only links to eff.org like the EFF logo next to it. Would be good to actually link the ToS;DR icon to tosdr.org
Also, since EFF is in the footer as well, we should add ToS;DR there also. Or alternatively just remove the footer.
See https://github.com/tosdr/tosback2/blame/master/rules/amazon.com.xml - most docs seem to be commented out? I'll try to run tosback2 locally, so I can learn how to contribute xPath entries.
Not sure if this is the proper place to report this, but it should be fixed ASAP.
All the "View the changes on Github" links on https://tosback.org/ that indicate change on the same day, point to the same commit which includes changes on multiple websites. This is a bit confusing. Perhaps it would be a good idea to have separate commit for each website.
Stripe has Terms split over 4 countries: CA, IE, GB, US. How should this be reflected in the rules?
One idea is to add a new attribute country
on the url
element and expand the definition to allow multiple url
elements under a single docname
, like so:
<sitename name="stripe.com">
<docname name="Terms of Service">
<url name="https://stripe.com/ca/terms"
xpath="//article[1]" lang="en" country="ca">
<norecurse name="arbitrary"/>
</url>
<url name="https://stripe.com/ie/terms"
xpath="//article[1]" lang="en" country="ie">
<norecurse name="arbitrary"/>
</url>
</docname>
</sitename>
This could even be useful for multi-lingual cases. Such as this fictitious site:
<sitename name="multilingual.tld">
<docname name="Terms of Service">
<url name="http://multilingual.tld/be/nl"
xpath="//main" lang="nl" country="be">
<norecurse name="arbitrary"/>
</url>
<url name="http://multilingual.tld/be/fr"
xpath="//main" lang="fr" country="be">
<norecurse name="arbitrary"/>
</url>
<url name="http://multilingual.tld/ca/fr"
xpath="//main" lang="fr" country="ca">
<norecurse name="arbitrary"/>
</url>
<url name="http://multilingual.tld/ca/en"
xpath="//main" lang="en" country="ca">
<norecurse name="arbitrary"/>
</url>
</docname>
</sitename>
(I wanted to discuss this in #tosback but couldn’t find the channel on either freenode or OFTC.)
Their documents have just changed and I was asked to accept those changes while trying to install an app. But neither tosback nor tosdr had an entry for windows phone.
The URLs for those documents is
http://www.windowsphone.com/**LOCALE**/store/terms-of-service
where LOCALE is a fully specified locale such as in the example of
http://www.windowsphone.com/en-us/store/terms-of-service
http://www.windowsphone.com/**LOCALE**/legal/wp8/windows-phone-privacy-statement
where LOCALE is a fully specified locale such as in the example of
http://www.windowsphone.com/en-us/legal/wp8/windows-phone-privacy-statement
One may like or dislike the social media site that is promoting OpenGraph API, but nevertheless I would recommend to implement those metatags for tosback.org to make it more appealing to share the site.
Above all I would suggest to implement an og:image tag which links to the logo of the service (or possible more appealing "marketing images"). As of now logos of the services that are tested are displayed in the preview, which is rather misleading.
https://developers.facebook.com/docs/sharing/opengraph/object-properties
https://trello.com/security: Ambiguous match, found 2 elements matching visible xpath "//div[@Class='layout-centered-content']"
but i can't see the problem when i execute
a=document.evaluate("//div[@class='layout-centered-content']", document, null, XPathResult.ANY_TYPE, null)
a.iterateNext()
a.iterateNext()
in the browser console on https://trello.com/legal/security
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.