kevinlekiller / newznab-blacklist Goto Github PK

View Code? Open in Web Editor NEW

19.0 18.0 16.0 693 KB

Blacklist for Newznab.

newznab-blacklist's Introduction

Newznab-Blacklist

Working on a project called nZEDb now : https://github.com/nZEDb/nZEDb

Scripts to clean up your Newznab installation.

BLACKLIST:

There are 2 blacklists, new (recommended), old (the old blacklists, will not be updated).

You can use the blacklists.txt file to manually add blacklists into newznab, or use the SQL file to import them all in 1 go.

All blacklists are disabled by default, enable them in admin - view black list, click the edit button to the right of the blacklist, click enable and save.

IMPORTING THE BLACKLIST:

(Change root in user=root for your own mysql user in the following commands)

Backup your blacklist: from a command line: mysqldump -l --user=root -p newznab binaryblacklist > binaryblacklist.sql

To import the blacklist, from a command where the blacklist is, type: mysql --user=root -p newznab < binaryblacklist.sql to import.

You can also use phpmyadmin to backup/import.

UPDATE PARSING (Temporarily removed, breaches the newznab plus license):

In this folder there are 2 folders, Put in misc/testing and Put in www/lib.

You put the files in those appropriate folders of your newznab installation.

Update_parsing_mod.php will try to fix release names from the "other" categories.

Remove_misc_releases.php is used to remove some releases which update_parsing_mod.php can not fix. (you can run this after you run update_parsing_mod if you want).

Open the files in a text editor and read the options to see how to use them. By default update_parsing will only do 24 hours, I recommend turning this to false for the first run, then switching it back to true.

You must be running php 5.4+ to use update_parsing_mod.php.

Thanks to thezoggy, sy and nivong for help / additions.

Donation link, if you want, thanks!

https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=kevspamail%40gmail%2ecom&lc=CA&item_name=kevin123%20software&currency_code=USD&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHosted

newznab-blacklist's People

Contributors

Stargazers

Watchers

Forkers

thezoggy plague85 dejavu bandite robirizarry jmarcet jjknight notyetrated rkon88 seiniku duduf sfmx mitchmiller forkless stevijo

newznab-blacklist's Issues

Blacklists car racing releases for sql is not working.

The regex needs a trailing ) after xvid

The txt file is correct.

block non-english stuff

if these are suppose to be case iterations of each other, there are a few differences between each string like spelling / things that are in one but not the other..

document / group target of blacklists together?

there are a few major groups of blacklists..

foreign/non-english
junk/virus/spam/passworded
hash/obfuscated/encrypted
stuff targeted for categories that the user doesnt want (console/pc/xxx/whatever)

an example of what it would block and why i would think would be ideal..

hdnectar

looks like hdnectar should be blacklisted
http://nzbindex.nl/search/?q=hdnectar&age=30&max=250&minage=&sort=agedesc&minsize=&maxsize=&dq=&poster=&nfo=&hidespam=0&hidespam=1&more=0

25-50gb uploads... with one at 180gb! thus, creating huge parts tables for people and killing nn indexers

One or multiple blacklists are possibly bad.

One or more of the blacklists might be causing issues with lower than expected releases being added to the site. If anyone has tested, please report back.

Compilation failed: missing...

Just testing out the new blacklists which looks awesome. But on update_binaries.php it will echo this error:
PHP Warning: preg_match(): Compilation failed: missing ) at offset 271 in /var/www/localhost/htdocs/nnplus/www/lib/binaries.php on line 619
PHP Warning: preg_match(): Compilation failed: lookbehind assertion is not fixed length at offset 27 in /var/www/localhost/htdocs/nnplus/www/lib/binaries.php on line 619

Further debugging (inserting echo $blist['ID']."debug\n"; at line 617) will show that problem lies in id's:
100001
100044

best regards

regex - xxx and password

tv shows sometimes have xxx or password in its show name/ep name. so these regex need to be mindful of that...

some examples of valid things it may catch that shouldn't:

Million.Dollar.Password.S02E01.PDTV.XVID-BAJSKORV
Robot.Chicken.S02E10.Password.Swordfish.DVDRip.XviD-SAiNTS
Bellator Fighting Championships S03E11 XXXIII HDTV XviD-QCF

one password format we should catch is [Password],

found for example in:

Guilty Hearts [Bluray - Untouched] [usenet-4all.info] [Password] [powered by ssl-news.info] [18,74 GB] [139/207] "usenet-4all_4044.part137.rar" yEnc
The Four [Bluray - Untouched] [usenet-4all.info] [Password] [powered by ssl-news.info] [23,77 GB] [014/260] "usenet-4all_4043.part012.rar" yEnc

the usnet4all have a whitelist.. so not sure if catching it would be an issue though

important update!

I noticed that it was blocking HDDVD releases (wich are 720p releases) and also BRRIP releases!

I changed the dvd blacklist to:

[\\.-](dvd([-]?r(ip)?)?)[\\.-]

Now it's gone (thanks to CJ!)

Same goes for the xvid stuff

((xvid|divx)(?!hd)|(320|480)p)

Blacklist updated

Optimized the blacklist:
DROP TABLE IF EXISTS binaryblacklist;
CREATE TABLE IF NOT EXISTS binaryblacklist (
ID int(11) unsigned NOT NULL AUTO_INCREMENT,
groupname varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
regex varchar(2000) COLLATE utf8_unicode_ci NOT NULL,
msgcol int(11) unsigned NOT NULL DEFAULT '1',
optype int(11) unsigned NOT NULL DEFAULT '1',
status int(11) unsigned NOT NULL DEFAULT '1',
description varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (ID)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1000041 ;

-- Dumping data for table `binaryblacklist`

INSERT INTO binaryblacklist (ID, groupname, regex, msgcol, optype, status, description) VALUES
(100009, 'alt.binaries.', 'dvd|dvdrip|dvdr|dvd-r|dvd-rip|DVDRip|Password', 1, 1, 0, 'dvd troep'),
(100000, 'alt.binaries.', 'xvid|divx|480p|320p', 1, 1, 0, 'Blocking xvid encodes.'),
(100004, 'alt.binaries.', 'nomytnopyht|lordsofusenet|klaus|ihate|aart|my2cents|ISNOGOOD|Captain Jack|seizoen', 2, 1, 0, 'Virus Posters'),
(100002, 'alt.binaries.', 'TrollHD|Olympics|Serbian|damike99|iMPACT|AVIGUY', 1, 1, 1, ''),
(100003, 'alt.binaries.', 'CowboyUp26|gr4ndstre4m|CowboyUp2012|trein', 1, 1, 1, ''),
(1000040, 'alt.binaries.', '-.[-.]', 1, 1, 1, 'Block Foreign Abbreviations'),
(100008, 'alt.binaries.', 'german|danish|flemish|swedish|swesub|deutsch|DEFA|Knochen|korsub|GERMAN| Keygen|Giro|norwegian|.ita.|.japanese.|.GER.|ita-eng|FLEMISH|iRLS\HYBRiS|FR|NORWEGIAN|Liga|EUFA|Footy|NRL|Snoballkrigen!ATKGalleria|RealCo|[MP4SUX]|Turkish|CytSuNee|NZBRoyalty', 1, 1, 1, ''),
(100012, 'alt.binaries.', 'wii|ps3|xbox', 1, 1, 1, 'Console'),
(1, 'alt.binaries.boneless', 'usenet-4all|u4all|usenet4all', 1, 2, 0, 'only allow u4all posts in boneless'),
(100014, 'alt.binaries.cores', 'xxx', 3, 1, 0, 'Removes XXX.'),
(100016, 'alt.binaries.ghosts', 'xxx', 3, 1, 0, 'removes xxx'),
(100019, 'alt.binaries.mom', 'xxx', 1, 1, 0, 'Removes XXX.'),
(100020, 'alt.binaries.multimedia', '[email protected]|The Phantom|xslidian@hmlt|tim96|GeorgBaer|knt', 2, 1, 1, 'Deletes crap'),
(100021, 'alt.binaries.movies.xvid', 'theknife', 2, 1, 0, NULL),
(100022, 'alt.binaries.', '^[^.^ ]{10,20}$', 1, 1, 0, 'Filters out all hashes between 10 and 20 characters, that haven''t got a space, period or caret in them, a negated character class that starts with a ^, doesn''t have a ^ before every character.'),
(100023, 'alt.binaries.tv', '6UL85|XA7MY|36CLX|Y642D', 1, 1, 0, 'Removes crap.'),
(100024, 'alt.binaries.', 'TheMaker|gilf|lordsofusenet|klaus|ihate|aart|my2cents|ISNOGOOD|Captain Jack', 2, 1, 0, 'VIRUS POSTERS'),
(100025, 'alt.binaries.', '^[a-z0-9]{20}', 1, 1, 0, 'Removes crap.'),
(100026, 'alt.binaries.hdtv.x264', 'zed|ich', 2, 1, 1, 'Removes crap.'),
(100036, 'alt.binaries.', '(^|[./ -]+)(sie|seit|ihn|ihm|haben|besitzen|sein|kriegen|nehmen|welche|jenes|von|auf|gegen|nach|das|sein|der|und|fuer|ersten|leicht|meinem|zum|aus|dem|blitzlicht|alle|grosse|zed|ich|sed|blitzen)([./ -]+|$)', 1, 1, 0, 'Removes many german releases.'),
(100037, 'alt.binaries.', '[aeiouy]{4,80}', 1, 1, 1, 'Removes multiple vowels strung together. See: http://blog.phile.biz/post/39106364519/useful-blacklist-rules'),
(100040, 'alt.binaries.', 'AutoRarPar\d', 1, 1, 1, 'Removes releases with AutoRarPar\d in the name.'),
(100043, 'alt.binaries.', 'CPP-Gebruiker', 2, 1, 0, 'foreign'),
(100044, 'alt.binaries.', 'NoTHing|9NoTHing|EveryFinG', 1, 1, 1, '');

blacklist - 'pw' priv site posts

looks like nzbcave is another one of those sites like nzbroyality but they seem to focus on 1:1 bluray or bluray with hd audio specifically.. thus lots of 15-50gb posts.. heres a 200+gb one i found. overall prob safe to just blacklist due to obfuscated posting/pw/etc that for most its just going to get nuked during cleanup.. so might as well save the parts table and resources nn would waste.

nzbcave.co.uk VIP PW [01/33] ""fliberflaberfluber0904208s06e01fliberflaberfluber.par2"" yEnc (1/1)
collection size: 227.27 GB, parts available: 609304 / 609790 [requires password]

i'm sure there are others out there... not sure if you wanna just toss this in the hdnectar blacklist or what