m-parashar / adbhostgen Goto Github PK

Delete

Delete

Hi,

Thank you for creating a great utility!!! 👍 🥇

I am curious to know if is it a way to add list of domains (talking about 1k+) to the myblacklist/mywhitelist. What I am trying to do is to add domain list of YouTube ads to myblacklist which is around 8k.

To do this manually for each domain will take ages, So is there any way by which we can push the list directly, From what I understood it is not advisable to manually edit the file.

YouTube Ads Domain List.txt

Thank you so much in advance.

VERSION: 20180331
on Netgear R6400 running Linux DD-WRT 4.4.159 #4001 SMP Wed Oct 10 09:28:16 CEST 2018 armv7l DD-WRT (build 37305)

Size of /tmp/mphosts.tmp: 9.9M
Size of /tmp/mpdomains.tmp: 3.6M

Processing blacklist/whitelist files
Processing final mphosts/mpdomains files
Segmentation fault
Removing temporary files
Size of /jffs/dnsmasq/mphosts: 0
Size of /jffs/dnsmasq/mpdomains: 3.1M
Number of ad domains blocked: approx 0

result of free
total used free shared buffers cached
Mem: 254576 51496 203080 0 6292 22312
-/+ buffers/cache: 22892 231684
Swap: 487420 0 487420

For testing purposes, I used -0 to see if a smaller file would work, but, same problem.

I've narrowed it down to this line:
LC_ALL=C cat $tmphosts | sed -r 's/^[[:blank:]]//; s/[[:blank:]]$//; /^$/d; /^\s*$/d' | tr -cd '\000-\177' | cat tmpbl - | grep -Fvwf tmpwl | sort -u | awk -v "IP=$ADHOLEIP" '{sub(/\r$/,""); print IP" "$0}' > $mphosts

if I replace $tmphosts with $tmpdomains it completes, successfully, so, something is going on with the $tmphosts file... I think I'm going to add some code to split the file up, into smaller files, and, see if it can process them, and then concatenate the files back into one file, when complete, or, maybe, narrow it down to some error with one (or, more) of the hosts in the file...

OK, so, I added some clunky code to break the mphosts.tmp file into smaller files, process each one, and add them to the final mphosts file, and, this is what happened, first, I broke it into files with 100000 lines in each, and, a couple of them, segfaulted, so, i dropped it to 50000 lines, each, and, they finished up, without segfaulting, and this seems to have worked, although, I have not yet verified that everything works as expected, yet.

OK, I cleaned up my code, to the best of my abilities, and, it ran it several times, and, it seems to be working fine, I haven't tried changing the number of lines above 50000 to see where the actual limit is, although, from my previous tests, it seems, more than 2 MB worth of lines was about where the segfaults were occurring. If anyone is interested in my code, and, would like to help me make sure it's good enough to make a pull request, just let me know, here.