• 确保上游数据产生的文件自动添加换行符

• adlist
• gfwlist: required to remove intersection with chinalist-root.txt
• chinalist: required to remove intersection with gfwlist-root.txt

ref:
https://github.com/pexcn/daily/blob/b91a75fe425614eb5cf508e34a8a521bc5696544/blacklist/blacklist.conf

EDIT:
like server=/domain.name/

4147efa for https://github.com/pexcn/daily/blob/master/generate.sh#L10-L12 and https://github.com/pexcn/daily/blob/master/generate.sh#L22-L24 ?

以前是有这条的，现在发现没了。

Data source from: @felixonmars/dnsmasq-china-list
And my scripts fetch from this list.

It seems there are more than 700,000 lines in toplist.txt generated from /scripts/_pre_/01-toplist.sh

The following script in /scripts/gfwlist/gfwlist.sh is CPU intensive.

# find intersection set
grep -Fx -f gfwlist.tmp toplist.txt > gfwlist_head.tmp
# find difference set
grep -Fxv -f toplist.txt gfwlist.tmp > gfwlist_tail.tmp
# merge to gfwlist
cat gfwlist_head.tmp gfwlist_tail.tmp > gfwlist.txt

But why does the merged gfwlist.txt only have less than 10,000 lines eventually?

ref: https://github.com/aglent/autoproxy/blob/b36c73d5c6bc2153b39e04526c465de94cef5c2b/chinaiplist.pac#L1-L45

Improve todo list: TODO#L3.
These way can be avoid this type of error: dnsmasq-china-list@f0a2385

# use awk better than sed
cat accelerated-domains.china.conf | awk '{split($0, arr, "/"); print arr[2]}'
#cat accelerated-domains.china.conf | awk '{split($0, arr, "[//]"); print arr[2]}'

# use sed
cat accelerated-domains.china.conf | sed -E 's/.*\/(.*)\/.*/\1/'

• remove safelist and blacklist
• remove unused openwrt scripts
• update readme
• add chinalist-lite
• add google.china.conf ?

• Move TZ to env tag.

For ipset: https://pgl.yoyo.org/adservers/iplist.php

Toooooooooo loooooooooong...

-IP-CIDR,0.0.0.0/32,DIRECT
+IP-CIDR,0.0.0.0/8,DIRECT

and pac?

The list will get intersection elements with @felixonmars/dnsmasq-china-list.

Reference:

• https://docs.aws.amazon.com/AlexaTopSites/latest/index.html
• https://aws.amazon.com/alexa-top-sites/getting-started/

Back to Basics.

https://github.com/felixonmars/dnsmasq-china-list/blob/master/google.china.conf

append those lines to PAC list:

# data source from:
# https://github.com/shadowsocks/luci-app-shadowsocks/blob/master/files/root/usr/bin/ss-rules
192.0.0.0/24
192.31.196.0/24
192.52.193.0/24
192.175.48.0/24
224.0.0.0/3 -> 224.0.0.0/4
240.0.0.0/4
255.255.255.255/32


127.0.0.1/32 ?

update and check shadowrocket config template:
https://github.com/pexcn/daily/blob/master/template/shadowrocket/whitelist.conf#L8
https://github.com/pexcn/daily/blob/master/template/shadowrocket/gfwlist.conf#L8

When I use whitelist pac,I can only visit China's websites. No foreign websites like Goole,Facebook or Twitter are accessible.What should I do? I have reloaded the pac and checked my proxy and when I use gfwlist pac,everything is Ok. Is the problem of whitelist pac?

Excuse me. Could you tell me where to get data like top-1m.csv.zip file in scripts/_pre_/01-toplist.sh? Is it from Alexa? How to get other kind of data?

These domain names have (Commented out has been processed):

# cn
top
xn--55qx5d    # .公司
xn--fiqs8s    # .**
xn--io0a7i    # .网络

http://ipverse.net/ipblocks/data/countries/cn.zone

EDIT:
No update.

seems no longer update.

ipv4: https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone
ipv6: https://www.ipdeny.com/ipv6/ipaddresses/blocks/cn.zone

Or add a version with no (root-domain ?) intersection between gfwlist and chinalist and replace it.

Related: #44

Pending.

For dnsmasq, datasource: https://github.com/pexcn/domain-blacklist

Filter out all domain name suffixes: exclude . in one line.

Reference: felixonmars/dnsmasq-china-list#202

Ref:
https://github.com/pexcn/static-build-ci/blob/fb395e18c79ba9f4d3e455e88673fe6959de9ff9/build.sh#L3-L4
https://github.com/shadowsocks/shadowsocks-android/blob/master/core/gfwlist/parse.py

Origin: https://gist.github.com/chilts/7229605

Data sources

• http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
• http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip (included cdn)
  or
  https://umbrella-static.s3.amazonaws.com/top-1m.csv.zip

EDIT:
The umbrella-static data source seems include subdomains, not recommends.

如果一个脚本的执行失败 (当 curl / wget 失败时)，则这个脚本所生成的文件不要上传。

back to basic.

ref:
https://github.com/openwrt/packages/blob/master/net/adblock/files/adblock.sh#L223

reference:
https://p3terx.com/archives/build-openwrt-with-github-actions.html
https://github.com/simonsmh/openwrt-dist/blob/master/.github/workflows/main.yml
https://github.com/P3TERX/Actions-OpenWrt/blob/master/.github/workflows/build-openwrt.yml

scripts/pac/whitelist.sh#L31

# replace last line `,` to empty
sed -i '$ s/,//g' ${WHITELIST_CONTENT}

# replace last character to empty
sed -i '$ s/.$//g' ${WHITELIST_CONTENT}

是否应该将 adlist, chinalist, gfwlist 等里面的域名按照 alexa 排名来排序，这样性能会不会比较好一点？因为放在头部的都是高频访问的域名。

还有一点需要改进的：
sort | uniq 操作不应该在需要生成的最终要使用的文件里面操作，而应该在它所需要的 源 列表里面操作。

Based on gfwlist.txt:

# 127.0.0.1:5300 is non-polluting DNS port
sed -e 's/^/server=\//' -e 's/$/\/127.0.0.1#5300/' gfwlist.txt > safelist-exp.conf
sed -i "1i#\n# Safe list (experimental)\n#\n# Update: $(date +'%Y-%m-%d %T')\n#\n" safelist-exp.conf

Put it into /etc/dnsmasq.d folder.