clean down & parked domains? about hosts HOT 11 CLOSED

Normally I just refresh the source hosts files, whose authors do a pretty good (albeit sometimes spotty) job of it.

Another consideration: just because a donain is "parked" or displays a "parked" home page doesn't mean it doesn't serve ads, or malware, or other things.

Do you sense that I understood your question? If not please clarify.

from hosts.

Ok, thank you!

It is just to clean my own hosts file (ads/tracker/malware/adult) which becomes a little large ~100.000 lines.

This is how i try to do:

@echo off
for /f %%i in (domains.txt) do ping -n 1 %%i >nul && echo UP   %%i&& echo 0.0.0.0 %%i>>hosts-up.txt||echo Down %%i&& echo %%i>>hosts-down.txt

or

#!/bin/bash
for file in domaines.txt; do
    while IFS= read -r host; do
        ping -qc 1 "$host" && echo "$host" >> result.txt;
     echo -e "$(sed '1d' domaines.txt)\n" > domaines.txt
    done < "$file"
done

from hosts.

Thing is, ping uses the ICMP which can be turned off even though the server still responds to http and https requests.

from hosts.

Can you suggest me an other way to clean down host?
Thanks!

from hosts.

I usually clear hosts records when they block services I need.

For example, an earlier version of the consolidated hosts file blocked professional ice hokey feeds. I live in Canada so I fixed that pretty quick :-)

On another occasion several Vimeo servers were blocked.

But there's no harm in blocking obsolete ad servers. I don't worry about that.

from hosts.

Ok, i really appreciate your help. Thank you very much.

Best regards,
Med 😸

from hosts.

Some coding ideas;

1. You can scan port 80 and 443/tcp.
2. Use curl to capture the body, and search this string "Sedo" "parking" "buy this domain"

from hosts.

@Espress0 can you help me to write script?

from hosts.

@smed79 @StevenBlack

can you help me to write script?

Still interested?
Which OS are you using, Windows or Linux? I'll write a code and send it for you(license: WTFPL).

from hosts.

1.You can scan port 80 and 443/tcp.

You can use scan() for port scanning(and if your ISP is not friendly, scan via Tor).
http://www.catonmat.net/blog/tcp-port-scanner-in-bash/

e.g., scan domain.maybe.parked.server 25,80,443,587

1. Use curl to capture the body, and search this string "Sedo" "parking" "buy this domain"

You're using Linux, so curl will help:
curl "http://$HOSTSFQDN/robots.txt"|grep "llow"|wc -l
curl "https://$HOSTSFQDN/robots.txt"|grep "llow"|wc -l

check these 2 commands, and if it is not 0, there's a webpage.

• llow
  to scan "(A|a)llow" or "(D|d)isallow"

from hosts.

@Espress0 Yes still interested, i'am using Linux (Manjaro/Arch).
I'll test and come back.

from hosts.