Python shutil.move 跨文件系统「原子发布」静默失效：os.rename EXDEV 退化成 copyfile 的 O_TRUNC 截断+重写，下游读到 0 字节空文件和半截 JSON

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

PASS=12 FAIL=4
RESULT: BAD

PASS=16 FAIL=0
RESULT: OK

cd scenario/2026-09-26-0000
docker compose up -d --build
docker compose ps
docker compose logs --tail=20

nas-gateway: backing=/var/lib/nas mount=/srv/store expose_nlink=False
nas-mountd: store mounted at /srv/store

docker compose exec publisher python3 /srv/ops/probe_syscalls.py

== 1) 两块盘的 st_dev ==
   /srv/store   st_dev=2097315
   /srv/stage   st_dev=2304
   same_filesystem=False
== 2) 跨盘 os.rename（原子搬运的正确手段？） ==
   rename FAILED errno=18 Invalid cross-device link
== 3) 跨盘 os.replace ==
   replace FAILED errno=18 Invalid cross-device link
== 4) 跨盘 shutil.move（会退化成什么？） ==
   shutil.move 返回 '/srv/store/_probe3.json'（看起来成功）   ← 危险信号
   dst 内容 '{"v": "moved", "items": [1, 2, 3]}'
== 5) 硬链接语义 ==
   store 上文件的 st_nlink=1
   挂载内 os.link FAILED errno=1 Operation not permitted
   跨盘 os.link FAILED errno=18 Invalid cross-device link
== 6) 挂载内 rename 仍应可用（发布器要能靠它做原子替换） ==
   挂载内 rename OK, 读到 'in-fs\n'

docker compose exec publisher bash -c "/srv/ops/seed.sh && bash /srv/ops/verify.sh"

=== A 组：发布的原子性 ===
  rename(local->nas) EXDEV errno=18 Cross-device link
  [PASS] A1 暂存盘与生效目录不同文件系统（rename EXDEV）——跨盘 rename 不可能原子
  旧长度=17002 新长度=121943 期间观测到的长度集合=[0, 17002, 40960, 90112, 121943]
  出现了不属于任何一次发布的长度: [0, 40960, 90112]
  [FAIL] A2 发布过程中 live 出现过 0 或中间长度 —— 下游能读到空文件/半截内容
  commit(): 回读=True 比对=False 不匹配时失败=False
  [FAIL] B4 commit() 只赋值不比对：verified 恒定，等于没有自检
=== C 组：并发发布 + 下游读，是否读到半截 ===
READER reads=976 ok=969 torn=0 empty=7 missing=0 distinct_digests=113 RESULT: BAD
  bad sample: {"bytes": 0, "reason": "empty_read (O_TRUNC 清空窗口)", ...}
  [FAIL] C2 下游读到了半截/空内容：发布不是原子的
  [FAIL] C3 下游继续读到半截/空内容（torn/empty > 0）：发布仍不是原子的

───────────────────────────────────────────
PASS=12 FAIL=4
RESULT: BAD

docker compose exec publisher bash -lc '
python3 /srv/client/reader.py --seconds 10 --interval 0.01 &
python3 /srv/client/publish_burst.py --rounds 200 --pause 0.003
wait'

READER reads=979 ok=964 torn=3 empty=15 missing=0 RESULT: BAD
  bad sample: {"bytes": 0, "reason": "empty_read (O_TRUNC 清空窗口)"}
  bad sample: {"bytes": 1302, "tail": "\u0000\u0000\u0000...", "reason": "json_parse_failed (正被写一半)"}

docker compose exec publisher python3 - <<'PY'
import os, shutil, time, threading, json
live, stage = "/srv/store/current.json", "/srv/stage/demo.next"
old = {"v": "old", "items": [{"id": i, "t": "o"*1000} for i in range(10)]}   # 小
new = {"v": "new", "items": [{"id": i, "t": "n"*4000} for i in range(60)]}   # 大
open(live, "w").write(json.dumps(old)); old_len = os.path.getsize(live)
open(stage, "w").write(json.dumps(new)); new_len = os.path.getsize(stage)
print("old_len=%s new_len=%s" % (old_len, new_len))
obs, stop = [], threading.Event()
def watch():
    while not stop.is_set():
        try: obs.append(os.path.getsize(live))
        except OSError: obs.append(-1)
threading.Thread(target=watch).start()
time.sleep(0.02)
shutil.move(stage, live)        # ← 问题态原句
time.sleep(0.02); stop.set()
u = sorted(set(obs))
print("长度集合(前12) =", u[:12], "共 %d 种" % len(u))
print("出现 0 字节窗口 (O_TRUNC):", 0 in u)
print("出现不属于任何一次发布的中间长度:", [x for x in u if x not in (old_len, new_len)][:8])
PY

old_len=10223 new_len=241273
长度集合(前12) = [0, 8192, 10223, 12288, 16384, 20480, 24576, 28672, 32768, 36864, 40960, 45056] 共 60 种
出现 0 字节窗口 (O_TRUNC): True
出现不属于任何一次发布的中间长度: [0, 8192, 12288, 16384, 20480, 24576, ...]

docker compose exec publisher python3 -c "import inspect, shutil; print(inspect.getsource(shutil.move))"

    try:
        os.rename(src, real_dst)
    except OSError:                       # ← EXDEV 走这里
        ...
        else:
            copy_function(src, real_dst)  # ← 默认 copy2 → copyfile
            os.unlink(src)

docker compose exec publisher python3 -c "import inspect, shutil; print(inspect.getsource(shutil.copyfile))"

        with open(src, 'rb') as fsrc:
            with open(dst, 'wb') as fdst:   # ← O_TRUNC！目标当场变 0 字节
                ...
                copyfileobj(fsrc, fdst)

def atomic_publish(src: str, dst: str, *, tmp_prefix: str = ".publish-") -> None:
    store = os.path.dirname(dst) or "."
    os.makedirs(store, exist_ok=True)
    mode = _target_mode(dst)          # 新建 0644；已有则沿用，避免 0600 收窄

    fd, tmp = tempfile.mkstemp(prefix=tmp_prefix, suffix=".tmp", dir=store)  # ← 关键：dir=store
    try:
        os.fchmod(fd, mode)           # ← 绕开 mkstemp 的 0600
        with os.fdopen(fd, "wb") as out, open(src, "rb") as fh:
            shutil.copyfileobj(fh, out, 1 << 16)
            out.flush()
            os.fsync(out.fileno())    # 数据先落地
        os.replace(tmp, dst)          # ← 同文件系统内换名 = 原子
        tmp = None
    finally:
        if tmp is not None:
            try: os.unlink(tmp)
            except FileNotFoundError: pass
    # 目录项也 fsync 一下，"新名字生效"才是持久的
    ...

    report["readback_matches"] = (
        report["live_sha256"] == report["stage_sha256"]
        and report["live_bytes"] == report["stage_bytes"]
    )
    report["verified"] = bool(report["readback_matches"])
    if not report["verified"]:
        raise PublishError("publish content mismatch: ...")   # ← 大声失败

docker cp app/publisher.py 3day-83-publisher:/srv/app/publisher.py
docker compose exec publisher bash -c \
  "find /srv/app -name __pycache__ -exec rm -rf {} + 2>/dev/null"

docker compose exec publisher bash -c "/srv/ops/seed.sh && bash /srv/ops/verify.sh"

  旧长度=17002 新长度=121943 期间观测到的长度集合=[17002, 121943]
  长度只在两个稳定值之间跳变 —— 下游不会看到中间态
  [PASS] A2 发布过程中 live 的长度只在「旧长度/新长度」间跳变（真原子替换）
  commit(): 回读=True 比对=True 不匹配时失败=True
  [PASS] B4 commit() 会回读并比对 stage/live，不匹配时大声失败
READER reads=977 ok=975 torn=0 empty=0 missing=2 distinct_digests=109 RESULT: OK
  [PASS] C2 并发发布期间下游读到的内容全程完整（torn=0 且 empty=0）
  [PASS] C3 下游全程读到完整内容（torn=0 且 empty=0）：发布已经是原子的

───────────────────────────────────────────
PASS=16 FAIL=0
RESULT: OK

修复前  A2 长度集合 = [0, 17002, 40960, 90112, 121943]   ← 0 与中间值都可见
修复后  A2 长度集合 = [17002, 121943]                    ← 只有两个稳定值

docker compose exec publisher bash -lc '
python3 /srv/client/reader.py --seconds 15 --interval 0.002 --out /srv/state/r_long.json > /srv/state/r_long.log 2>&1 &
sleep 0.3
python3 /srv/client/publish_burst.py --rounds 400 --pause 0.001 > /srv/state/b_long.log 2>&1
wait; cat /srv/state/r_long.log; cat /srv/state/b_long.log'

READER reads=6947 ok=6937 torn=0 empty=0 missing=10 distinct_digests=400 RESULT: OK
BURST rounds=400 updated=400 verified=400
LAST stage_bytes=358 live_bytes=358 sha_match=True

新建 live 的 mode = 0o644 其他用户可读 = True
报告 live_mode = 0o644 atomic = True

正确：抛出 PublishError -> publish content mismatch: stage(sha=ffffffffffffffff bytes=151) != live(sha=ca02a3b1266297b0 bytes=151)

/srv/ops/apply_fix.sh --restore && bash /srv/ops/verify.sh   # PASS=12 FAIL=4 RESULT: BAD
/srv/ops/apply_fix.sh          && bash /srv/ops/verify.sh   # PASS=16 FAIL=0 RESULT: OK

docker compose down -v && docker compose up -d --build && sleep 6
docker compose exec publisher bash -c "/srv/ops/seed.sh && bash /srv/ops/verify.sh 2>&1 | tail -4"
# → PASS=16 FAIL=0 / RESULT: OK