mirror of
https://github.com/shadowsocks/ShadowsocksX-NG.git
synced 2026-11-02 08:33:46 +00:00
fix(rules): convert root URL paths to exact domains
This commit is contained in:
Qiu Yuzhou
+1
-1
@@ -15,7 +15,7 @@
|
||||
- **Proxy mode**: The user-facing way traffic is routed through the local proxy: rule mode, global (public targets default through Shadowsocks, fixed local targets stay direct via a minimal ACL), or direct mode through an ACL-backed local SOCKS endpoint. All three project the local SOCKS and HTTP inbounds as the system-proxy target.
|
||||
- **Rule mode**: The ACL-backed routing mode in which built-in (and later custom) rules choose proxy versus direct per target; the unmatched default action is a separate persisted sub-option.
|
||||
- **Rule default action**: The rule-mode sub-option that decides proxy versus direct for targets no rule matches. Factory default is proxy-when-unmatched. _Avoid_: fallback (ambiguous).
|
||||
- **Built-in rule snapshot**: An app-shipped, pinned rule set produced by an explicit maintainer update through the sole Python conversion path; Swift validates and consumes snapshots rather than duplicating upstream conversion; ordinary builds only read local snapshots and never fetch or convert. Three sources ship today: **geolocation-cn** (China domain direct candidates), **china-ipv4** (China mainland IPv4 CIDR direct candidates from gaoyifan/china-operator-ip), and **gfwlist** (proxy candidates from the official Base64 AutoProxy GFWList; domain anchors become suffix matches; URL prefixes without a path, query, fragment or user information and with a literal domain host become exact matches, discarding scheme and any valid port; `@@` exceptions fully covered by a proxy rule are omitted). A snapshot ships only the rules that can decide routing; same-action absorbed entries and opposite-action shadowed exceptions are omitted at conversion time and never re-enter matching when a broader rule is later disabled. Each snapshot carries one source identity in metadata for the whole set, plus digest, license, attribution, and conversion loss report. Schema 2 and converter 2.0.0 ship compact JSON rules containing only action and match; old snapshots and rule counts outside the generator bounds fail loading.
|
||||
- **Built-in rule snapshot**: An app-shipped, pinned rule set produced by an explicit maintainer update through the sole Python conversion path; Swift validates and consumes snapshots rather than duplicating upstream conversion; ordinary builds only read local snapshots and never fetch or convert. Three sources ship today: **geolocation-cn** (China domain direct candidates), **china-ipv4** (China mainland IPv4 CIDR direct candidates from gaoyifan/china-operator-ip), and **gfwlist** (proxy candidates from the official Base64 AutoProxy GFWList; domain anchors become suffix matches; URL prefixes with an empty or root path, no query, fragment or user information, and a literal domain host become exact matches, discarding scheme and any valid port; `@@` exceptions fully covered by a proxy rule are omitted). A snapshot ships only the rules that can decide routing; same-action absorbed entries and opposite-action shadowed exceptions are omitted at conversion time and never re-enter matching when a broader rule is later disabled. Each snapshot carries one source identity in metadata for the whole set, plus digest, license, attribution, and conversion loss report. Schema 2 and converter 2.0.0 ship compact JSON rules containing only action and match; old snapshots and rule counts outside the generator bounds fail loading.
|
||||
- **Custom rule**: A user-owned proxy or direct rule with a stable opaque identity and an exact-domain, domain-suffix, or IPv4/IPv6 address-range match; list order carries no routing priority. Same-action absorption and opposite-action shadowing do not erase saved rules or imply that they determine routing; rules violating the fixed local bypass policy are invalid. _Avoid_: user rule.
|
||||
- **Rule identity**: The normalized match condition together with its routing action, shared by equivalent entries from different sources. It is distinct from the persistent identity of an editable custom entry; an individual IP address is equivalent to its full-length CIDR.
|
||||
- **Rule source membership**: The collection of sources containing a rule identity; browsing merges equivalent entries while retaining every source membership. Removing a custom entry does not remove matching built-in entries.
|
||||
|
||||
@@ -17,7 +17,7 @@ ShadowsocksX-NG 的现代化重写版本。新工程的全部源码与构建配
|
||||
- `Vendor/<name>/manifest.json` — 外部二进制的固定供应链清单（tag + 资产 URL + 归档 SHA-256 + bundle 内位置 + 签名 identifier）；二进制本体与 `.fetched.sha256` 戳是构建缓存，不入库。
|
||||
- `Vendor/rules/geolocation-cn/` — 内置中国域名规则快照（issue #63）：`snapshot.json`（规范化规则 + 元数据 + 损失报告）、`manifest.json`（上游版本与快照 SHA-256）、`NOTICE`（许可证与归属）。普通构建只读本地快照，绝不抓取或转换。
|
||||
- `Vendor/rules/china-ipv4/` — 内置中国 IPv4 CIDR 直连候选快照（issue #64）：同上三件套，来源为 [gaoyifan/china-operator-ip](https://github.com/gaoyifan/china-operator-ip) `china.txt` 固定 commit。规则模式的 `proxy_all` ACL 同时编入中国域名与 IPv4 CIDR 直连候选。
|
||||
- `Vendor/rules/gfwlist/` — 内置 GFWList 代理候选快照（issue #65）：同上三件套，来源为 [gfwlist/gfwlist](https://github.com/gfwlist/gfwlist) 官方 Base64 AutoProxy `gfwlist.txt` 固定 commit。域名锚点转换为后缀匹配，满足条件的无路径 URL 前缀转换为精确域名匹配；「未匹配时直连」的 `bypass_all` ACL 编入这些代理候选。
|
||||
- `Vendor/rules/gfwlist/` — 内置 GFWList 代理候选快照（issue #65）：同上三件套，来源为 [gfwlist/gfwlist](https://github.com/gfwlist/gfwlist) 官方 Base64 AutoProxy `gfwlist.txt` 固定 commit。域名锚点转换为后缀匹配，满足条件的空路径或根路径 URL 前缀转换为精确域名匹配；「未匹配时直连」的 `bypass_all` ACL 编入这些代理候选。
|
||||
- `Scripts/` — 供应链脚本（见下节）与打包门槛断言；另有 `update-geolocation-cn.sh` / `update-china-ipv4.sh` / `update-gfwlist.sh`（显式维护动作，抓取固定版上游并转换）与 `verify-rule-snapshots.sh`（构建前离线校验快照完整性）。
|
||||
|
||||
## 外部二进制供应链
|
||||
@@ -85,7 +85,7 @@ Scripts/packaging-gate.sh \
|
||||
|
||||
- **geolocation-cn**（issue #63）：人工复验上游后运行 `Scripts/update-geolocation-cn.sh`，从固定版 [Loyalsoldier/domain-list-custom](https://github.com/Loyalsoldier/domain-list-custom) `geosite.dat` 解析 typed 条目并生成 `Vendor/rules/geolocation-cn/snapshot.json`。
|
||||
- **china-ipv4**（issue #64）：运行 `Scripts/update-china-ipv4.sh`，从固定版 [gaoyifan/china-operator-ip](https://github.com/gaoyifan/china-operator-ip) `china.txt` 规范化并去重 IPv4 CIDR，生成 `Vendor/rules/china-ipv4/snapshot.json`。
|
||||
- **gfwlist**（issue #65）：运行 `Scripts/update-gfwlist.sh`，从固定版 [gfwlist/gfwlist](https://github.com/gfwlist/gfwlist) `gfwlist.txt` 解码官方 Base64 AutoProxy 列表并生成 `Vendor/rules/gfwlist/snapshot.json`。`||host` / `||host^` 转换为域名后缀匹配；`|scheme://host`（可含合法端口）在无路径、查询、fragment 或用户信息，且 host 为无通配符的合法域名时，提取 host 转换为精确域名匹配，忽略协议和端口。末尾 `/` 也视为路径。`@@` 保留直连动作。其他 URL 前缀、路径、通配符、正则、单标签前缀与 IP 字面量规则逐类计入损失报告。被代理规则完全覆盖的 `@@` 例外不写入无效 ACL 项（sslocal 域名匹配 proxy_list 优先于 bypass_list），保留代理规则，遮蔽数只计入 `absorbedCount`；逐条取证输出到维护脚本 stdout。
|
||||
- **gfwlist**（issue #65）：运行 `Scripts/update-gfwlist.sh`，从固定版 [gfwlist/gfwlist](https://github.com/gfwlist/gfwlist) `gfwlist.txt` 解码官方 Base64 AutoProxy 列表并生成 `Vendor/rules/gfwlist/snapshot.json`。`||host` / `||host^` 转换为域名后缀匹配；`|scheme://host`（可含合法端口）在路径为空或仅为 `/`，无查询、fragment 或用户信息，且 host 为无通配符的合法域名时，提取 host 转换为精确域名匹配，忽略协议和端口。仅含根路径 `/` 可转换，包含具体路径的规则继续跳过。`@@` 保留直连动作。其他 URL 前缀、路径、通配符、正则、单标签前缀与 IP 字面量规则逐类计入损失报告。被代理规则完全覆盖的 `@@` 例外不写入无效 ACL 项（sslocal 域名匹配 proxy_list 优先于 bypass_list），保留代理规则，遮蔽数只计入 `absorbedCount`；逐条取证输出到维护脚本 stdout。
|
||||
|
||||
快照使用 schema 2 / converter 2.0.0 的紧凑 JSON：每条规则只有 `action` 和 `match`，来源只在 `metadata.source`，转换报告只保留计数。被吸收的 `.cn` 条目与被遮蔽的 GFWList 例外不随快照分发，禁用宽规则后不再恢复；详见 [ADR-0024](../docs/adr/0024-snapshot-drops-absorbed-and-forensics.md)。旧版本快照拒载。
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"""Convert the official Base64 GFWList AutoProxy list into an NG2 rule snapshot
|
||||
(issue #65).
|
||||
|
||||
Domain anchors become suffix matches. URL prefixes without a path and with a
|
||||
Domain anchors become suffix matches. URL prefixes with an empty or root path and a
|
||||
literal domain host become exact matches, discarding the scheme and port.
|
||||
Other URL conditions are reported as losses rather than expanded to domains.
|
||||
`@@` exceptions shadowed by a broader proxy rule are kept out of the ACL and
|
||||
@@ -96,7 +96,7 @@ def parse_url_prefix(url: str, original: str, is_exception: bool) -> dict:
|
||||
_ = parsed.port # Validate numeric syntax and range before dropping it.
|
||||
except ValueError:
|
||||
return skipped
|
||||
if parsed.path or "?" in url or "#" in url or parsed.username is not None:
|
||||
if parsed.path not in {"", "/"} or "?" in url or "#" in url or parsed.username is not None:
|
||||
return skipped
|
||||
if (
|
||||
parsed.netloc.endswith(":")
|
||||
|
||||
@@ -57,7 +57,7 @@ def inputs():
|
||||
'china-ipv4': ('\n'.join(f'1.0.{i}.7/24' for i in range(100)) + '\n').encode(),
|
||||
'gfwlist': base64.b64encode(('||example.com\n@@||safe.example.com\n@@||other.example\n'
|
||||
'|https://exact.example.net\n@@|https://direct.example.net\n'
|
||||
'|https://path.example.net/\n'
|
||||
'|https://root.example.net/\n|https://path.example.net/a\n'
|
||||
+ '\n'.join(f'||gfw{i}.example.net' for i in range(100))).encode()),
|
||||
}
|
||||
|
||||
@@ -93,7 +93,7 @@ class ConversionBehaviorTests(unittest.TestCase):
|
||||
self.assertEqual(snapshot['metadata']['inputDigest'], hashlib.sha256(digest_input).hexdigest())
|
||||
self.assertEqual(snapshot['lossReport']['convertedCount'], len(snapshot['rules']))
|
||||
if name == 'gfwlist':
|
||||
for action, host in [('proxy', 'exact.example.net'), ('direct', 'direct.example.net')]:
|
||||
for action, host in [('proxy', 'exact.example.net'), ('direct', 'direct.example.net'), ('proxy', 'root.example.net')]:
|
||||
self.assertIn({'action': action, 'match': {'kind': 'domainExact', 'value': host}}, snapshot['rules'])
|
||||
|
||||
def test_geosite_reads_published_typed_protobuf_and_selects_category(self):
|
||||
@@ -156,7 +156,7 @@ class ConversionBehaviorTests(unittest.TestCase):
|
||||
|
||||
def test_gfw_url_prefix_with_conditions_or_invalid_host_stays_skipped(self):
|
||||
patterns = [
|
||||
'https://example.com/', 'https://example.com/path',
|
||||
'https://example.com/path', 'https://example.com//',
|
||||
'https://*.example.com', 'https://exam*ple.com',
|
||||
'https://1.2.3.4', 'https://999.2.3.4', 'https://[2001:db8::1]',
|
||||
'https://localhost', 'https://bad..example', 'https://-bad.example',
|
||||
@@ -174,6 +174,17 @@ class ConversionBehaviorTests(unittest.TestCase):
|
||||
self.assertEqual([r['match']['value'] for r in rules], ['keep.example'])
|
||||
self.assertEqual(report['skipped'], {'urlPrefix': 1})
|
||||
|
||||
def test_gfw_url_prefix_root_path_converts_and_deduplicates(self):
|
||||
rules, report = GFW.convert_document(
|
||||
'|https://Example.COM/\n|http://example.com\n'
|
||||
'@@|https://safe.example.org/\n@@|http://safe.example.org\n')
|
||||
self.assertEqual(rules, [
|
||||
{'action': 'proxy', 'match': {'kind': 'domainExact', 'value': 'example.com'}},
|
||||
{'action': 'direct', 'match': {'kind': 'domainExact', 'value': 'safe.example.org'}},
|
||||
])
|
||||
self.assertEqual(report['convertedCount'], 2)
|
||||
self.assertEqual(report['skipped'], {})
|
||||
|
||||
def test_gfw_exact_proxy_only_shadows_same_exact_exception(self):
|
||||
lines = ['|https://example.com', '@@|http://example.com',
|
||||
'@@|https://sub.example.com', '@@||example.com', '@@||sub.example.com']
|
||||
|
||||
@@ -52,7 +52,7 @@ class SnapshotConverterTests(unittest.TestCase):
|
||||
output = io.StringIO()
|
||||
with contextlib.redirect_stdout(output):
|
||||
rules, report = gfw.convert_document(
|
||||
"||Example.COM^\n@@||Safe.Example.COM^\n@@||other.example\n|https://path.example/\n")
|
||||
"||Example.COM^\n@@||Safe.Example.COM^\n@@||other.example\n|https://path.example/a\n")
|
||||
snapshot = {"schemaVersion": gfw.SCHEMA_VERSION, "rules": rules, "lossReport": report}
|
||||
self.assert_slim(snapshot)
|
||||
self.assertEqual(report["absorbedCount"], 1)
|
||||
|
||||
@@ -17,7 +17,7 @@ UPSTREAM_URL="https://raw.githubusercontent.com/gfwlist/gfwlist/${UPSTREAM_VERSI
|
||||
UPSTREAM_MIRROR_URL="https://cdn.jsdelivr.net/gh/gfwlist/gfwlist@${UPSTREAM_VERSION}/gfwlist.txt"
|
||||
UPSTREAM_SHA256="750d34564363075fafa12d8ab9739f269e2fbd2bf3c5f4ab8cfd859bfd3441df"
|
||||
LICENSE="LGPL-2.1"
|
||||
ATTRIBUTION="GFWList AutoProxy rules derived from gfwlist/gfwlist gfwlist.txt at commit ${UPSTREAM_VERSION}. Official Base64 AutoProxy 0.2.9 list. Domain anchors become suffix matches. URL prefixes without a path, query, fragment or user information and with a literal domain host become exact matches, discarding scheme and any valid port. Other URL conditions, wildcard hosts, IP literals and regex rules are reported as losses. List is candidate routing data, not a guarantee of reachability or completeness."
|
||||
ATTRIBUTION="GFWList AutoProxy rules derived from gfwlist/gfwlist gfwlist.txt at commit ${UPSTREAM_VERSION}. Official Base64 AutoProxy 0.2.9 list. Domain anchors become suffix matches. URL prefixes with an empty or root path, no query, fragment or user information, and a literal domain host become exact matches, discarding scheme and any valid port. Other URL conditions, wildcard hosts, IP literals and regex rules are reported as losses. List is candidate routing data, not a guarantee of reachability or completeness."
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -14,4 +14,4 @@ This change does not alter the Python algorithms, pinned snapshots, schema or co
|
||||
|
||||
ADR-0024's snapshot slimming contract and explicit-update/offline-build boundary remain in force. The earlier exclusion of converter consolidation applied to that change's scope; this decision handles it separately. A Swift generation CLI or general conversion framework adds build and migration cost without a current product requirement.
|
||||
|
||||
The GFWList path now also extracts literal domain hosts from URL prefixes with no path, query, fragment or user information; a trailing slash counts as a path. It emits exact-domain matches and preserves the proxy/direct action, deliberately discarding scheme and any valid port. Wildcard hosts, IP literals, invalid URLs and remaining URL conditions stay in the `urlPrefix` loss category. Exact proxy matches only shadow equal exact exceptions; suffix proxy matches can cover either match kind. This extends conversion behavior without changing the schema 2 / converter 2.0.0 artifact contract or regenerating the pinned distribution snapshots. Generated consumer fixtures cover the new match kind.
|
||||
The GFWList path now also extracts literal domain hosts from URL prefixes with an empty or root (`/`) path and no query, fragment or user information; other paths remain unsupported. It emits exact-domain matches and preserves the proxy/direct action, deliberately discarding scheme and any valid port. Wildcard hosts, IP literals, invalid URLs and remaining URL conditions stay in the `urlPrefix` loss category. Exact proxy matches only shadow equal exact exceptions; suffix proxy matches can cover either match kind. This extends conversion behavior without changing the schema 2 / converter 2.0.0 artifact contract or regenerating the pinned distribution snapshots. Generated consumer fixtures cover the new match kind.
|
||||
|
||||
Reference in New Issue
Block a user