synzvato / decentraleyes Goto Github PK

With this addon these tools are broken as they will always redirect to the home page:

• https://report-uri.io/home/analyse
• https://report-uri.io/home/pkp_analyse

More information: https://twitter.com/rugkme/status/675972938110210048

CC: @ScottHelme

Removing the en-US component from all links that point to AMO pages, will result in automatic language detection and an attempt to redirect the user in question to an appropriate, localized, page.

Since localization support is part of the upcoming v1.3.0-update, this should be, too.

AMO mentions that Web Font Loader is covered by Decentraleyes, but the FAQ doesn't make clear whether I can or should start carving out exceptions in uBO for Fanboy's anti-font list. Can you clear this up?

Also, although I understand that Decentraleyes is intercepting the WFL code itself, am I correct in believing that continuing to allow remote fonts to load still poses its own metadata leakage risk?

If so, should I look into implementing a low-impact proxy for media elements for mitigation? I can't think of another way to handle leakage via media elements, short of asking for massive scope creep in Decentraleyes—which I'm not and won't. 😁

PS: I _love_ the idea of this extension. Thanks so much!

Plenty of sites use these. Could we mirror them in decentraleyes?

I guess currenly the addon contains library files inside it.

How about if it worked this way:

1. There is a cache folder for thirdparty files. It's empty in the beginning.
2. When a CDN request is made, addon checks if it exists in cache folder.
3. If yes return that.
4. If no, download it, cache and return.

So that no addon update is required in case of library updates. What do you think?

On Android 5.1.1 with uBlock Origin

i modified the install.rdf and simply added the needed:

 <!-- SeaMonkey -->
    <em:targetApplication>
      <Description>
        <em:id>{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}</em:id>
        <em:minVersion>2.0</em:minVersion>
        <em:maxVersion>2.39</em:maxVersion>
      </Description>
    </em:targetApplication>

and it works perfectly with 2.39 release.

will you add google recaptcha?
and script from script.aculo.us
http://madrobby.github.io/scriptaculous/
css from maxcdn.bootstrapcdn.com
https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css
css from jsdelivr.net
http://cdn.jsdelivr.net/bootstrap/3.3.4/css/bootstrap.min.css

and for addon name will better like "cache content" or "local content"

thank you

Mozilla recently put in SRI support into Firefox. If a script tag has a SRI attached, and it matches the hash of a known, then it can be assumed there is no need to download the resource.

Edit: To be clear, this would allow the same library hosted anywhere to be caught. I don't know the current usage of SRI, but hopefully it grows.

Decentraleyes should be installable, and usable, on mobile devices capable of running Fennec. At this point the extension downloads, but the installation silently fails in the background. We should run some tests and, if the add-on runs well, greenlight mobile installations.

Do you think it would be a good idea, when a request is intercepted, or intercepted and served from a local resource, to log that activity to the browser console? (Ctrl+Shift+J in Firefox)

I just loaded a website (flattr.com) and got an ugly page. When I allowed policeman to contact to maxcdn.bootstrapcdn.com it worked fine.

But I think it loaded https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css directly from the server and not through the plugin, I think it should.

So this is from my Firefox dev-tools:

<link href="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css" rel="stylesheet">
<!-- (…) -->
<script type="text/javascript" src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>

In the end I am not totally sure if it's loaded by the plugin or directly, can I test/see/prove/investegate if a ressource is loaded from a cdn or hdd? If not maybe it's a good idea to do this in the HTTP-header… like X-DECENTRAL = "eyes".

See http://www.mono-project.com/download/ with/without.

Sorry for the terse report. What's the preferred format? I didn't do any digging, but assuming it's a missing lib, do you happily accept PRs for missing libraries?

Hey!

I use Decentraleyes along with Requestpolicy. Right now it is necessary to whitelist the domains mentioned here by hand.
It would be useful, if there was a whitelist, which is updatet, with the progress of the Decentraleyes addon.

Best wishes,
Sammy

xref: #12

Toward avoiding incessant disk access (updating prefs.js), I'm suggesting that yet another pref (which would be read at startup and stored) should be introduced, enabling the user to opt-out (disable the counter).

Alternatively, after reading the cumulative total to-date tally at start of session, increment its value in memory only -- updating prefs.js value at shutdown only.

I just had a problem with it. Twitch login prompt is invisible with Decentraleyes active.

twitch.tv
Firefox 45.0
Decentraleyes 1.2.2

It should be possible for users to make Decentraleyes allow all Content Delivery Network requests for specific websites. The whitelist should be configurable from within the Add-ons Manager.

Would this be possible so I could install a newer developer edition for the temporary fixes for some websites?

Ah, well, JPM didn't turn out to be that difficult, just kept getting hung up on the bit that required me to be on linux. Got it working just fine. You can close this, ty.

The title explains it all. I'm using Firefox 44.0.1 x64 on Ubuntu.

I think it's important to find out which versions of which libraries are most commonly used on websites, so that somewhat less popular resources can be removed from the default bundle.

For example, jQuery 1.5.0 is currently included in Decentraleyes. However, according to W3Techs, statistically, out of 10.000 websites, only around 6 depend on this particular version of the library. If these numbers are accurate, I'm sure we could put its spot to better use.

Would this be a good source to go by, or does anyone have a better suggestion?

Needs React.js files: https://cdnjs.com/libraries/react/
Should include react, react-dom (0.14+) and react-with-addons

In agreement with one of the Essential Next Steps...

• To keep this add-on from turning into bloatware, it's important to find out which versions of which libraries are most commonly used on websites, so that less popular resources can be removed from the default bundle.

...would a library automatically be saved upon its first encounter (assuming it's from a correct CDN?..) and if so, then could the user be provided Save for future intercepts. or Ignore any future intercepts. options?

Both Noscript and RequestPolicy offer standard menu buttons with very user-friendly prompts and controls. These menus are also very powerful and in-depth, potential I believe decentraleyes also has. Better to show than tell, but if I'm intercepting jQuery 3.0.0 from MaxCDN and out of nowhere some site is using jQuery 2.0.0 from MaxCDN, it'd be nifty to allow the user to decide whether or not to save and intercept for later (especially if they plan of visiting that site often or for whatever other reason.)

Configurations could also be made to automatically save and intercept newly encountered libraries from either white-listed domains or CDNs. Maybe we could scrape together some resource digests of as many libraries we can find and simply host them from this repository... the user's add-on could then verify the resource integrity before saving.

There's a lot available!

Decentraleyes does not yet have a fully fledged user interface. To give users at least some feedback on what the add-on does, a counter should be added to the Add-ons Manager (under preferences).

Now that Decentraleyes has proper support for localizations, it's time to move this forward. Do you master a non-supported language? Please help out by translating this add-on on Crowdin.

Language requests: If your preferred language is not in the current list, please post a request here or send me an email. I'll happily add any language you'd like to help maintain.

If you don't feel like signing up for Crowdin, feel free to ask for a source file package.

I have found that this extension broke the Google Play Store (http://play.google.com/) for me. It doesn't show the menu on the left, the search bar doesn't work, and clicking on install on an app's page doesn't work either. Seems like a JS script is broken, but I don't have time to figure out which one.

Deactivating the extension fixes the problem as expected.

On https://stefansundin.github.io/2fa-qr/, it produces the following error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://code.jquery.com/jquery-2.1.4.min.js. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

and breaks the site.

I was intrigued when reading your essential next steps, esp. " to find out which versions of which libraries are most commonly used on websites". Since I'm comfy using HTTP Archive + Google BigQueries I ran some initial queries:

I've compared year-over-year growth of the most common JS libraries I could think of (angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype) and ran the query on HTTP Archive's January 1st runs for 2014, 2015 & 2016. Here's a screenshot of the resulting table:

And here's the SQL query to generate this result from Google Bigqueries:
SELECT year, type, CONCAT(STRING(INTEGER(100 * (COUNT - prev) / prev)), '%') year_growth, count, prev prev_count from( SELECT COUNT, LAG(COUNT) OVER( PARTITION BY type ORDER BY year ) prev, type, year FROM ( SELECT type, COUNT, year FROM ( SELECT REGEXP_EXTRACT(url, r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type, COUNT(DISTINCT(pageid)) COUNT, '2016' year FROM [httparchive:runs.2016_01_01_requests] WHERE REGEXP_MATCH(url, r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype') GROUP BY type), ( SELECT REGEXP_EXTRACT(url, r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type, COUNT(DISTINCT(pageid)) COUNT, '2015' year FROM [httparchive:runs.2015_01_01_requests] WHERE REGEXP_MATCH(url, r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype') GROUP BY type), ( SELECT REGEXP_EXTRACT(url, r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type, COUNT(DISTINCT(pageid)) COUNT, '2014' year FROM [httparchive:runs.2014_01_01_requests] WHERE REGEXP_MATCH(url, r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype') GROUP BY type), )) WHERE prev IS NOT NULL ORDER BY year, type

I hope this helps. Let me know how I can assist further.

Thanks for this great plugin!

redundancy:
(using jqueryui as an example, but applicaple to many other libs)
By convention (if not outright definition), the minified copy of a js library file must not substantively differ from the non-minified file. Therefore, no need to mirror both ~~ mappings for both forms of a given library file can (should, sez me) point to a single locally cached item. Yes, I recognize that "auditability" necessitates the redundancy. Hold that thought...

omissions:
(using jqueryui as an example; I have not yet examined other other libs)
refer to
http://code.jquery.com/ui/1.11.0-beta.2/jquery-ui.js
and search in file for:
"buttonImage", "_attachments: function(input, inst) {", "attr({ src: buttonImage"

Currently, decentraeyes "bundle" does not consider the imagefiles employed as jqueryui buttons -- therefore is ineffectual in preventing contact with the CDN.

walk with me... (and, don't shoot the messenger, eh)
http://code.jquery.com/ui/
Effectively mirroring jqueryui would require bundling a BOATLOAD of additional resources.
Permutations: css stylesheet plus imagefiles for each available jqueryuiTHeme X libVersionPath

Next (still referring to http://code.jquery.com ) note the in-page declaration
"powered by MaxCDN"
Click the MaxCDN link, then go read about their "Origin Shield" offering.
Yeah, browser will often (still) be contacting the CDN ~~ decentraleyes will just be providing a panacea each time an apparent "first-party" subdomain is actually DNS aliased to a CDN-hosted server (which, due to geo-targeting and load balancing, is increasingly commonplace).

Clearly, the current "many-to-many" mappings construct not only begs ongoing maintenance, it prevents decentraleyes from effectively fulfilling its goal. For many of the covered libraries, I believe that efficacious handling requires an "ANY-to-ONE" approach (based on filename match). If a site developer is so stupid (or so devious) to name his custom script "jquery-1.2.1.min.js" ...so sad , too bad. We should demand that our sole permacached copy of "jquery-1.2.1.min.js" is injected, regardless of the remote URL specified by a webpage author.

Yes, I've read that you are concerned about the potential performance impact from mappings employing regex. Fuggedaboutit. With attention to what I've outlined above, along with attention to the limited storage space available on client mobile devices, the necessity of employing text.replace() regex matching should be clear.

Following in that train of thought, a one-size-fits-all (mobile, vs desktop) permacache doesn't seem reasonable. Each decentraleyes user will be saddled with the overhead of libraries/versions which will NEVER be specified for his device's platform.

I have also read that you intend decentraleyes to serve as an "install-n-forget" extension. Why so? Feed a man a fish vs teach a man to fish... Howabout let's give personal responsibility (and choice) its due. Each member of the expected userbase has chosen to use an alternative browser, has chosen to install this 'privacy' extension. So, the following is a perfectly reasonable first-run setup scenario: user is presented with a (yes, looong) treeview displaying "known" libraries for which mappings are available. All entries (perhaps 2 groupings, mobile-relevant vs general) are tickmarked by default. User self-selects which libs he wishes to permacache and, at that juncture, the extension retrieves and permacaches the selected libs/resources.

quoted from issue #6

the current implementation puts even less trust in centralized delivery networks. If every user would fetch copies of files from these central parties, it technically re-opens the door for big players to inject malicious code into specific environments (highly targeted, so next to impossible to detect).

Really? C'mon, that's "a solution, looking for a problem".
No hashsums, no secret handshakes. Just map the resources so that a (the) definitive copy is downloaded from each project's site, rather than from a CDN. Alternatively, for many libs you could map to the files already mirrored here (github) by the 'jsdelivr' project.

In any event, the xpi is already 16Mb (?check my math) and is destined to grow.
Regardless whether the permacached resources are bundled inside the xpi or are retrieved at time of setup, we should bear in mind their cumulative overhead is carried in every profile sync dataset, every FEBE backup.

Please understand that I share your goal of achieving enhanced privacy and I have great enthusiasm for the decentraleyes project. I trust you'll look beyond any 'abrasiveness' which might be inferred from my prose. In addition to the privacy aspect, I'm enthusiastic about the HUGE amount of wasted bandwith we'll collectively avoid by using decentraleyes

See title, using the http://addonconverter.fotokraina.com/ converter it installs the firefox version just fine, so you probably just need to change some "allowed browser version" variable.

Some expressions for shorthand notations of resource paths like (e.g. "/jquery.min.js" on jQuery CDN that points to v1.11.1) are not strict enough. This, in rare cases, causes Decentraleyes to inject the wrong resource into a page and this can lead to page breakage.

Oracle Java Magazine refuses to load after Decentraleyes injects jQuery v1.11.1 into a response that should contain jQuery Migrate v1.2.1. Causing the page below to break:

http://oraclejavamagazine-digital.com/javamagazine/november_december_2015

Google AMP is about to be released in a big way, and it would be awesome if decentraleyes supported it.

Just one file is needed: https://cdn.ampproject.org/v0.js

Thanks! I love this software, very smart idea and implementation.

It should be possible for end-users, to create, share, and import custom resource bundles. This will allow us to keep Decentraleyes lean and tidy, without having to disappoint power-users. To be able to support these bundles, we will need to develop an open standard. Bundles should contain the actual resources, and essential metadata (such as resource mappings, a title, and a description).

Feel free to weigh in with suggestions, ideas, or concrete proposals.

This is fairly repeatable. Visit play.google.com, click an app details link, then click "read more". Nothing happens, and the Firefox console is filled with errors about a script being blocked internally - no attribution to an addon or even the offending .js.

This may or may not be caused by having DecentralEyes installed in conjunction with NoScript, uBlock Origin, and Ghostery (5.latest). After pausing / disabling all of the above, though, I finally got GP page elements working again by disabling DecentralEyes. Nothing else I tried had any effect.

Content Security Policy: The page's settings blocked the loading of a resource at data:application/javascript;charset=UTF-8,%2F*%0A%20%20Local%20delivery%20by%20Decentraleyes%20(1.2.2).%0A%20_%2F%0A%0A%2F_!%20jQuery%20v1.10.2%20%7C%20(c)%202005%2C%202013%20jQuery%20Foundation%2C%20Inc.%20%7C%20jquery.org%2Flicense%0A_%2F%0A(function(e%2Ct)%7Bvar%20n%2Cr%2Ci%3Dtypeof%20t%2Co%3De.location%2Ca%3De.document%2Cs%3Da.documentElement%2Cl%3De.jQuery%2Cu%3De.%24%2Cc%3D%7B%7D%2Cp%3D%5B%5D%2Cf%3D%221.10.2%22%2Cd%3Dp.concat%2Ch%3Dp.push%2Cg%3Dp.slice%2Cm%3Dp.indexOf%2Cy%3Dc.toString%2Cv%3Dc.hasOwnProperty%2Cb%3Df.trim%2Cx%3Dfunction(e%2Ct)%7Breturn%20new%20x.fn.init(e%2Ct%2Cr)%7D%2Cw%3D%2F%5B%2B-%5D%3F(%3F%3A%5Cd_%5C.%7C)%5Cd%2B(%3F%3A%5BeE%5D%5B%2B-%5D%3F%5Cd%2B%7C)%2F.source%2CT%3D%2F%5CS%2B%2Fg%2CC%3D%2F%5E%5B%5Cs%5CuFEFF%5CxA0%5D%2B%7C%5B%5Cs%5CuFEFF%5CxA0%5D%2B%24%2Fg%2CN%3D%2F%5E(%3F%3A%5Cs_(%3C%5B%5Cw%5CW%5D%2B%3E)%5B%5E%3E%5D_%7C%23(%5B%5Cw-%5D_))%24%2F%2Ck%3D%2F%5E%3C(%5Cw%2B)%5Cs_%5C%2

Several sites use www.googletagservices.com/tag/js/gpt.js (to serve ads?).
This is usually not blocked by uBlock lists.

It was originally in uBlock's mirror-candidates.txt:
https://github.com/gorhill/uBlock/blob/1.3.6/assets/ublock/mirror-candidates.txt

In situations where the requested CDN library is not mirrored to avoid bloating the add-on, remove the referer field to improve the user's privacy.

It's quite new, but it's already being used on some websites.

for example this game: https://minigames.mail.ru/durak

On the Testing tool page i noticed that when I've the ajax.googleapis.com blocked in uBlock Origin / uMatrix (Which I usually have in most pages) the result I get is: "The test resource could not be fetched locally or remotely" & "Decentraleyes is not working as intended"
However when I allow the ajax.googleapis I get the "Decentraleyes is fully operational" status.

So if i get this right, Decentraleyes stops working only when you block ajax.googleapis on the specific page that supports this CDN, but works fine in all the other pages with different CDNs, right?

Thanks and keep up the good work!

(Noob question but I need clarification on this)

I find your addon useful but what about doing the same with any website?
Check this file out https://github.com/AliasIO/Wappalyzer/blob/master/src/apps.json. It is the file used by Wappalyzer to uncovers the technologies used on a website, and sometimes with the versions.
I think using these regular expressions it could be achievable.

Hi,
I'm using both uMatrix and decentraleyes. I would like to add a few rules to uMatrix to allow requests to the CDN that decentraleyes mirrors.
This is the list of domains that I managed to build based on the description of the plugin that I found on AMO (https://addons.mozilla.org/en-US/firefox/addon/decentraleyes/) :

upcdn.b0.upaiyun.com
lib.sinaapp.com
libs.baidu.com
yastatic.net
cdn.jsdelivr.net
ajax.microsoft.com
ajax.aspnetcdn.com
code.jquery.com
cdnjs.cloudflare.com
ajax.googleapis.com

Is this list correct? Are there domains in that list that decentraleyes doesn't take care of? Does decentraleyes take care of more CDN?
It would be nice to have an exhaustive list of domain names on the wiki.

First off, congrats on releasing 1.2.2. By far my favorite new feature is compatibility with Firefox for Android!! Thank you :-)

I opened this thread to raise a couple of quick points about things that might confuse users. If you open the options page Firefox Add-ons >> Decentraleyes Options, you will notice two things:

1. A FAQ list which has good questions but no answers!
2. An option to Add comments to locally fetched file: the description seems to suggest that a comment at the top of a file will indicate that it was served by the add-on. However, where can we see this file? The browser's Network logger does not show these files since the request is never made to the remote server. To replicate, open a new tab, Press Ctr+Shift+Q to open the Network logger, and navigate to the test utility. Notice that the JS files do not show up in the network log.

split from #36
b/c this is a general feature request (not specific to requestpolicy interactions)

When "retrieve and cache missing" option is enabled, I wish decentraleyes would (optionally) raise an infobar announcing
"page is requesting a not-yet-cached item from a recognized CDN. Allow/Deny".
Upon 'Allow' buttonclick, DE would retrieve and permacache the item then trigger a page reload. It's reasonable to expect this will be a seldom-occurring interruption.

If missing (from decentraleyes as-installed coverage) occurrences are NOT infrequent, we need to wonder/question "What's going on?" -- in the wild, are websites attempting cache-busting in order to achieve a ping, by appending, for instance
hxxp://CDNsite/path/path/jquery-2.1.2.js**?imatrackingsubstring**

There is a similar add-on called "local load" for firefox available, I am not sure if you are aware of it.

http://www.getlocalload.com/
https://addons.mozilla.org/en-US/firefox/addon/local-load/
it seems to be GPL2 and the last update was in February 2013

There are some old issues that are still not fixed
https://addons.mozilla.org/en-US/firefox/addon/local-load/reviews/

There is also an additional userscript available:
http://userscripts-mirror.org/scripts/show/87361

Maybe it could help you, e.g. to solve bug #16

It would be nice to have an optional icon to put in whatever toolbar that would indicates the number of url intercepted and replaced by local copies.

And it would be even nicer if clicking or hovering this icon would display the list of intercepted files or url.

thanks

I understand how intercepting network requests and injecting JS script instead would increase speed and enhance privacy (reduce the opportunity for tracking). A short FAQ on how this works would be really helpful, for people who are technically literate enough to understand the technology, but don't feel comfortable reading code. For instance after a bit of research, I still have the following questions:

1. One benefit of contacting the origin server, especially over https connections, is the guarantee that the script is authentic. Given that this extension would not communicate with the origin server, are there safeguards in place to check the integrity of script before injecting it? Is it possible for another application to modify the scripts in the local cache in order to get malicious script executed?
2. How does Private Browsing impact this extension? I know that the browser clears its cache after a private browsing session, but extension settings are preserved.
3. When are the scripts downloaded? And are they saved as .js files on the user's hard drive?
4. Is there any mechanism to check whether a local copy is outdated? Or does the add-on assume that script at a given URL never changes?

Privacy should be accessible, so it's time to start supporting localization packages. This will help bring Decentraleyes to a global audience that includes people whose native language is not English.

Hi,

It would be cool to have Chrome/Chromium support for decentraleyes.

(Sorry for the issue title, I couldn't find a better one)

I am currently browsing by blocking by default all third parties thanks to uBlock Origin, both for browsing speed and privacy. And blocking ads, too.

The problem is that uBlock blocks the requests before Decentraleyes can provide the local version. Is it possible to set Decentraleyes at the very beginning of the chain of plugins working on the requests made by the browser?

Browser: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0

1. Access http://www.twitch.tv/.
2. Click on Login or Register.
3. Notice the popup that contains the login/register form is completely blank when Decentraleyes is on.