Compare commits
1 Commits
main
...
backup/age
| Author | SHA1 | Date | |
|---|---|---|---|
|
0f3f1cf7f7 |
117
B侧设计文档.md
117
B侧设计文档.md
@ -1,117 +0,0 @@
|
|||||||
# B 侧设计文档（主控端 + 协议 + 网络）
|
|
||||||
|
|
||||||
> 负责人：成员 B ｜ 主责 crate：`viewer/`、`common/` ｜ 配合：A（agent/）
|
|
||||||
> 按里程碑推进，每个里程碑独立 feature 分支，合入即打 tag。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## B1 · 联调基线（1-2 天，和 A 合作完成）
|
|
||||||
|
|
||||||
目标：拿到 H.264 版双机实测数据，作为一切优化的对照基线。
|
|
||||||
|
|
||||||
1. **HUD 扩展**：新增 丢包率(估算)/抖动(jitter)/帧间隔 p95 三项读数
|
|
||||||
- 抖动 = 相邻帧 E2E 差值的绝对值 EMA；帧间隔 p95 用环形缓冲取 95 分位；
|
|
||||||
- 码流异常计数（解码失败次数）也上 HUD，验证关键帧自愈是否生效。
|
|
||||||
2. **测试脚本** `tools/bench.md`：固定三个场景各测 60s —— 静止桌面 / 播放视频 / 拖动窗口，记录 HUD 截图。
|
|
||||||
3. **产出**：基线数据表填入本文件末尾。
|
|
||||||
|
|
||||||
## B2 · 码率自适应（3-4 天）
|
|
||||||
|
|
||||||
目标：弱网自动降码率保流畅，好网缓慢升码率提画质。
|
|
||||||
|
|
||||||
### 协议扩展（common/）
|
|
||||||
|
|
||||||
```rust
|
|
||||||
/// B2 新增控制消息（流 2）
|
|
||||||
RateReport {
|
|
||||||
loss_permille: u16, // 接收端估算丢包率 ‰
|
|
||||||
rtt_ms: u32, // 最近 RTT
|
|
||||||
e2e_ms: u32, // 平滑后端到端延迟
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
viewer 每 2s 汇总一次发送（流 2 复用，无新增握手）。
|
|
||||||
|
|
||||||
### 控制算法（先简后精，AIMD 风格）
|
|
||||||
|
|
||||||
| 网络状态（连续 3 个周期） | 动作 |
|
|
||||||
|---|---|
|
|
||||||
| 丢包 > 8% 或 E2E > 400ms | 码率 ×0.6（下限 1Mbps），同时通知编码器降帧率到 20fps |
|
|
||||||
| 丢包 3-8% | 码率 ×0.9 |
|
|
||||||
| 丢包 < 1% 且 E2E < 150ms 持续 10s | 码率 +500Kbps 步进上探（上限 15Mbps） |
|
|
||||||
|
|
||||||
### agent 侧配合点（给 A 的接口需求）
|
|
||||||
|
|
||||||
- 需要运行时改码率的入口：优先查 openh264 crate 是否暴露 `SetOption(RC_PARAM)`；
|
|
||||||
- 若 crate 不支持 → 方案 B：保留 encoder 实例但按新码率重建（重建成本 ~50ms，降码率不频繁，可接受）；
|
|
||||||
- 降级到极限（1Mbps 仍丢包 >15%）→ 主动降分辨率 1080p→720p（A 实现，协议传 `ResChange` 通知，viewer 重配解码器与窗口尺寸）。
|
|
||||||
|
|
||||||
## B3 · P2P 打洞 + 信令 + 中继（1.5-2 周，核心里程碑）
|
|
||||||
|
|
||||||
目标：跨公网可用。三层结构：
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────┐ ①注册/交换地址(WebSocket) ┌──────────┐
|
|
||||||
│ viewer │ ───────────────────────────→ │ 信令服务器 │
|
|
||||||
└────┬────┘ └──────────┘
|
|
||||||
│ ②拿到对方候选地址(IP:port×N) ↑
|
|
||||||
│ ③a 直连(打洞) ──────────────────── agent 注册
|
|
||||||
│ ③b 失败 → 走 TURN 中继
|
|
||||||
```
|
|
||||||
|
|
||||||
### 信令服务器（B 主建，A 参与评审）
|
|
||||||
|
|
||||||
- 最小实现：单二进制 Rust + WebSocket（tokio-tungstenite），无数据库；
|
|
||||||
- 消息：`Register{agent_id}` / `Connect{agent_id}` / `Answer{candidates, auth_code_hint}`；
|
|
||||||
- agent 开机注册自身 ID（设备码），viewer 输对方 ID 发起；
|
|
||||||
- 中立服务器，不碰媒体数据，带宽成本 ≈ 0。
|
|
||||||
|
|
||||||
### 候选地址与打洞
|
|
||||||
|
|
||||||
1. agent 侧收集候选：本机局域网地址、路由器公网映射（STUN 探测，自建 coturn 的 stun 或公共 stun）；
|
|
||||||
2. 双方经信令交换候选后**同时**向对方发探测包（UDP 打洞，锥形 NAT 成功率高；对称 NAT 基本打不通）；
|
|
||||||
3. 打通 → 直连（QUIC 直接到该地址）；打不通 → TURN 中继兜底。
|
|
||||||
|
|
||||||
### 中继方案（务实路线）
|
|
||||||
|
|
||||||
QUIC 本身没有标准 TURN 集成，两条路：
|
|
||||||
- **路线一（推荐先做）**：自建中继服务 `relay` —— viewer 把 QUIC 包封装转发给 relay，relay 转给 agent（本质是应用层 TURN，复用信令服务器的部署）；
|
|
||||||
- **路线二（二期）**：coturn + 手写 ICE-lite，仅在路线一性能不足时投入。
|
|
||||||
|
|
||||||
### 安全（随 B3 一起做，替代现在的跳过校验）
|
|
||||||
|
|
||||||
- 预共享验证码升级为 **PAKE（SPAKE2）**：验证码不落服务器、不落网络，双方推导会话密钥；
|
|
||||||
- 信令只传地址候选，媒体端到端加密保持 QUIC TLS 不变。
|
|
||||||
|
|
||||||
## B4 · 多屏 + 文件传输（各 2-3 天，穿插做）
|
|
||||||
|
|
||||||
**多屏**：
|
|
||||||
- 协议：`ScreenList{screens: Vec<{id, w, h}>}`（AuthOk 里带）、`SelectScreen{id}`；
|
|
||||||
- agent 枚举 DXGI outputs（A 实现），viewer 加屏幕切换下拉；
|
|
||||||
- 第二期再做"多屏同显"。
|
|
||||||
|
|
||||||
**文件传输**：
|
|
||||||
- 首选走已有 QUIC 连接开独立双向流（不引入 aria2 依赖，部署简单）；
|
|
||||||
- 协议：`FileOffer{name, size}` → `FileAccept` → 原始分块流（1MB 块 + 校验）；
|
|
||||||
- viewer 做拖拽发送 + 保存对话框。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 里程碑与分支规划
|
|
||||||
|
|
||||||
| 里程碑 | 分支 | 预估 | 验收标准 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| B1 联调基线 | feature/hud-baseline | 1-2 天 | 三场景基线数据入库 |
|
|
||||||
| B2 码率自适应 | feature/rate-adapt | 3-4 天 | 模拟弱网（clumsy/丢包工具）下自动降码率，画面不断流 |
|
|
||||||
| B3 打洞+中继 | feature/p2p-relay | 1.5-2 周 | 两台跨公网机器通过 ID 互联；打洞失败自动走中继 |
|
|
||||||
| B4 多屏/文件 | feature/screens-files | 3-5 天 | 多屏切换、拖拽传文件可用 |
|
|
||||||
|
|
||||||
顺序建议：B1 → B2 → B3 → B4（B3 是价值最大项，B4 可穿插）。
|
|
||||||
|
|
||||||
## 基线数据表（B1 完成后填写）
|
|
||||||
|
|
||||||
| 场景 | FPS | RTT | E2E | 码率 | 丢包 | 抖动 |
|
|
||||||
|---|---|---|---|---|---|---|
|
|
||||||
| 静止桌面 | | | | | | |
|
|
||||||
| 视频播放 | | | | | | |
|
|
||||||
| 拖动窗口 | | | | | | |
|
|
||||||
113
Cargo.lock
generated
113
Cargo.lock
generated
@ -6,14 +6,9 @@ version = 4
|
|||||||
name = "agent"
|
name = "agent"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"agent-core",
|
||||||
"anyhow",
|
"anyhow",
|
||||||
"common",
|
"common",
|
||||||
"lz4_flex",
|
|
||||||
"nvenc",
|
|
||||||
"openh264",
|
|
||||||
"quinn",
|
|
||||||
"rcgen",
|
|
||||||
"rustls",
|
|
||||||
"tokio",
|
"tokio",
|
||||||
"windows",
|
"windows",
|
||||||
]
|
]
|
||||||
@ -58,27 +53,6 @@ dependencies = [
|
|||||||
"serde",
|
"serde",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "bitfields"
|
|
||||||
version = "1.0.3"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "ef6e59298da389bc0649c7463856b34c6e17fe542f88939426ede4436c6b1195"
|
|
||||||
dependencies = [
|
|
||||||
"bitfields-impl",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "bitfields-impl"
|
|
||||||
version = "1.0.3"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "f2c044f98f86f15414668d6c8187c7e4fadab1ad2b31680f648703e0fe07c555"
|
|
||||||
dependencies = [
|
|
||||||
"proc-macro2",
|
|
||||||
"quote",
|
|
||||||
"syn 2.0.119",
|
|
||||||
"thiserror",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "bitflags"
|
name = "bitflags"
|
||||||
version = "1.3.2"
|
version = "1.3.2"
|
||||||
@ -625,17 +599,6 @@ version = "0.2.2"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
|
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvenc"
|
|
||||||
version = "0.1.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "3227c3b09b1e042dab7d8d82cea77c0404e7d0c2bbb92e7e87cb319749cd20dc"
|
|
||||||
dependencies = [
|
|
||||||
"bitfields",
|
|
||||||
"libloading",
|
|
||||||
"windows",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "once_cell"
|
name = "once_cell"
|
||||||
version = "1.21.4"
|
version = "1.21.4"
|
||||||
@ -1581,54 +1544,32 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows"
|
name = "windows"
|
||||||
version = "0.62.2"
|
version = "0.58.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580"
|
checksum = "dd04d41d93c4992d421894c18c8b43496aa748dd4c081bac0dc93eb0489272b6"
|
||||||
dependencies = [
|
|
||||||
"windows-collections",
|
|
||||||
"windows-core",
|
|
||||||
"windows-future",
|
|
||||||
"windows-numerics",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "windows-collections"
|
|
||||||
version = "0.3.2"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610"
|
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-core",
|
"windows-core",
|
||||||
|
"windows-targets",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-core"
|
name = "windows-core"
|
||||||
version = "0.62.2"
|
version = "0.58.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
|
checksum = "6ba6d44ec8c2591c134257ce647b7ea6b20335bf6379a27dac5f1641fcf59f99"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-implement",
|
"windows-implement",
|
||||||
"windows-interface",
|
"windows-interface",
|
||||||
"windows-link",
|
|
||||||
"windows-result",
|
"windows-result",
|
||||||
"windows-strings",
|
"windows-strings",
|
||||||
]
|
"windows-targets",
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "windows-future"
|
|
||||||
version = "0.3.2"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb"
|
|
||||||
dependencies = [
|
|
||||||
"windows-core",
|
|
||||||
"windows-link",
|
|
||||||
"windows-threading",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-implement"
|
name = "windows-implement"
|
||||||
version = "0.60.2"
|
version = "0.58.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
|
checksum = "2bbd5b46c938e506ecbce286b6628a02171d56153ba733b6c741fc627ec9579b"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"proc-macro2",
|
"proc-macro2",
|
||||||
"quote",
|
"quote",
|
||||||
@ -1637,9 +1578,9 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-interface"
|
name = "windows-interface"
|
||||||
version = "0.59.3"
|
version = "0.58.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
|
checksum = "053c4c462dc91d3b1504c6fe5a726dd15e216ba718e84a0e46a88fbe5ded3515"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"proc-macro2",
|
"proc-macro2",
|
||||||
"quote",
|
"quote",
|
||||||
@ -1652,32 +1593,23 @@ version = "0.2.1"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "windows-numerics"
|
|
||||||
version = "0.3.1"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26"
|
|
||||||
dependencies = [
|
|
||||||
"windows-core",
|
|
||||||
"windows-link",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-result"
|
name = "windows-result"
|
||||||
version = "0.4.1"
|
version = "0.2.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
|
checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-link",
|
"windows-targets",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-strings"
|
name = "windows-strings"
|
||||||
version = "0.5.1"
|
version = "0.1.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
|
checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-link",
|
"windows-result",
|
||||||
|
"windows-targets",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@ -1714,15 +1646,6 @@ dependencies = [
|
|||||||
"windows_x86_64_msvc",
|
"windows_x86_64_msvc",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "windows-threading"
|
|
||||||
version = "0.2.1"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37"
|
|
||||||
dependencies = [
|
|
||||||
"windows-link",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows_aarch64_gnullvm"
|
name = "windows_aarch64_gnullvm"
|
||||||
version = "0.52.6"
|
version = "0.52.6"
|
||||||
|
|||||||
@ -1,9 +1,8 @@
|
|||||||
[workspace]
|
[workspace]
|
||||||
resolver = "2"
|
resolver = "2"
|
||||||
# agent-core：平台无关会话层 + 软件编码视频管线（当前仅 linux-agent 使用）
|
# agent-core：平台无关的被控端会话层（Windows/Linux 共用）；linux-agent：Linux 被控端
|
||||||
# linux-agent：Linux 被控端（X11 + XTest）
|
|
||||||
members = ["common", "agent-core", "agent", "linux-agent", "viewer"]
|
members = ["common", "agent-core", "agent", "linux-agent", "viewer"]
|
||||||
# 默认成员保持 Windows 三件套，`cargo build --release` 行为与改动前完全一致；
|
# 默认成员保持 Windows 三件套，`cargo build --release` 依旧只出 agent.exe / viewer.exe；
|
||||||
# Linux 被控端显式构建：`cargo build --release -p linux-agent`
|
# Linux 被控端显式构建：`cargo build --release -p linux-agent`
|
||||||
default-members = ["common", "agent", "viewer"]
|
default-members = ["common", "agent", "viewer"]
|
||||||
|
|
||||||
@ -16,5 +15,5 @@ quinn = "0.11"
|
|||||||
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging"] }
|
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging"] }
|
||||||
rcgen = { version = "0.13", default-features = false, features = ["ring", "pem"] }
|
rcgen = { version = "0.13", default-features = false, features = ["ring", "pem"] }
|
||||||
lz4_flex = "0.11"
|
lz4_flex = "0.11"
|
||||||
windows = "0.62"
|
windows = "0.58"
|
||||||
minifb = "0.27"
|
minifb = "0.27"
|
||||||
|
|||||||
@ -291,13 +291,12 @@ where
|
|||||||
|
|
||||||
// 传输参数调优：
|
// 传输参数调优：
|
||||||
// - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口）
|
// - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口）
|
||||||
// - 发送缓冲上限 512KB：QUIC 内部排队 ≈ 3 帧 @5Mbps，E2E 延迟天花板 ~150ms；
|
// - 发送缓冲上限 2MB：QUIC 内部排队有界（约 1 帧多），配合上游 latest-wins 丢帧
|
||||||
// 配合上游 latest-wins 丢帧，超量部分在进 QUIC 前就被丢弃
|
// 把端到端排队延迟压在 ~0.4s 内
|
||||||
// （与 Windows 端 agent/src/main.rs 保持一致，沿用同一次调优结论）
|
|
||||||
let mut transport = quinn::TransportConfig::default();
|
let mut transport = quinn::TransportConfig::default();
|
||||||
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
|
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
|
||||||
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
|
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
|
||||||
transport.send_window(512 * 1024);
|
transport.send_window(2 * 1024 * 1024);
|
||||||
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
|
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
|
||||||
server_config.transport_config(Arc::new(transport));
|
server_config.transport_config(Arc::new(transport));
|
||||||
|
|
||||||
@ -350,10 +349,8 @@ where
|
|||||||
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
|
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
|
||||||
let t0 = Instant::now();
|
let t0 = Instant::now();
|
||||||
|
|
||||||
// 读循环（流 2）：输入事件 + Ping；Pong 附带**视频下行方向**的 QUIC 丢包统计
|
// 读循环（流 2）：输入事件 + Ping
|
||||||
// （conn.stats().path 统计的是本端发送方向，被控端的发送方向正是视频下行）
|
|
||||||
let keyreq_reader = keyreq.clone();
|
let keyreq_reader = keyreq.clone();
|
||||||
let conn_stats = conn.clone();
|
|
||||||
let reader = tokio::spawn(async move {
|
let reader = tokio::spawn(async move {
|
||||||
loop {
|
loop {
|
||||||
match read_control(&mut irecv).await {
|
match read_control(&mut irecv).await {
|
||||||
@ -363,13 +360,7 @@ where
|
|||||||
}
|
}
|
||||||
Ok(ControlMsg::Ping(t)) => {
|
Ok(ControlMsg::Ping(t)) => {
|
||||||
let agent_ts = t0.elapsed().as_millis() as u64;
|
let agent_ts = t0.elapsed().as_millis() as u64;
|
||||||
let p = conn_stats.stats().path;
|
let _ = ctl_tx.send(ControlMsg::Pong { t, agent_ts });
|
||||||
let _ = ctl_tx.send(ControlMsg::Pong {
|
|
||||||
t,
|
|
||||||
agent_ts,
|
|
||||||
lost: p.lost_packets,
|
|
||||||
sent: p.sent_packets,
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
Ok(_) => {}
|
Ok(_) => {}
|
||||||
Err(_) => break,
|
Err(_) => break,
|
||||||
|
|||||||
@ -4,15 +4,11 @@ version = "0.1.0"
|
|||||||
edition = "2021"
|
edition = "2021"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
|
# 会话层（QUIC/TLS/认证/编码/丢帧）在 agent-core，与 linux-agent 共用
|
||||||
|
agent-core = { path = "../agent-core" }
|
||||||
common = { path = "../common" }
|
common = { path = "../common" }
|
||||||
anyhow.workspace = true
|
anyhow.workspace = true
|
||||||
tokio.workspace = true
|
tokio.workspace = true
|
||||||
quinn.workspace = true
|
|
||||||
rustls.workspace = true
|
|
||||||
rcgen.workspace = true
|
|
||||||
lz4_flex.workspace = true
|
|
||||||
openh264 = { version = "0.9.9", features = ["source"] }
|
|
||||||
nvenc = "0.1.0"
|
|
||||||
windows = { workspace = true, features = [
|
windows = { workspace = true, features = [
|
||||||
"Win32_Foundation",
|
"Win32_Foundation",
|
||||||
"Win32_Graphics_Direct3D",
|
"Win32_Graphics_Direct3D",
|
||||||
|
|||||||
@ -1,59 +0,0 @@
|
|||||||
//! 编码基准：1080p 噪声帧（最恶劣压缩工况）对比单线程与 4 线程 openh264。
|
|
||||||
//! 运行：cargo run --release -p agent --bin encbench
|
|
||||||
|
|
||||||
use openh264::encoder::{
|
|
||||||
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
|
|
||||||
};
|
|
||||||
use openh264::formats::{BgraSliceU8, YUVBuffer};
|
|
||||||
use openh264::OpenH264API;
|
|
||||||
use std::time::Instant;
|
|
||||||
|
|
||||||
fn bench(threads: u16, frames: usize) -> f64 {
|
|
||||||
let mut enc = Encoder::with_api_config(
|
|
||||||
OpenH264API::from_source(),
|
|
||||||
EncoderConfig::new()
|
|
||||||
.bitrate(BitRate::from_bps(5_000_000))
|
|
||||||
.max_frame_rate(FrameRate::from_hz(30.0))
|
|
||||||
.rate_control_mode(RateControlMode::Bitrate)
|
|
||||||
.usage_type(UsageType::ScreenContentRealTime)
|
|
||||||
.intra_frame_period(IntraFramePeriod::from_num_frames(300))
|
|
||||||
.num_threads(threads),
|
|
||||||
)
|
|
||||||
.expect("encoder");
|
|
||||||
|
|
||||||
// 预生成 10 张不同噪声帧轮换（保证每帧 P 帧都真实编码，避免运动估计空转）
|
|
||||||
let mut frames_in = Vec::with_capacity(10);
|
|
||||||
for f in 0..10u32 {
|
|
||||||
let mut frame = vec![0u8; 1920 * 1080 * 4];
|
|
||||||
let mut seed = 12345u32.wrapping_add(f * 7919);
|
|
||||||
for px in frame.chunks_exact_mut(4) {
|
|
||||||
seed = seed.wrapping_mul(1103515245).wrapping_add(12345);
|
|
||||||
px.copy_from_slice(&seed.to_le_bytes());
|
|
||||||
}
|
|
||||||
frames_in.push(YUVBuffer::from_bgra8_source(BgraSliceU8::new(&frame, (1920, 1080))));
|
|
||||||
}
|
|
||||||
|
|
||||||
// 预热 5 帧（首帧 IDR、线程池初始化）
|
|
||||||
for (i, y) in frames_in.iter().enumerate().take(5) {
|
|
||||||
let _ = enc.encode(y).unwrap_or_else(|_| panic!("frame {i}"));
|
|
||||||
}
|
|
||||||
let t = Instant::now();
|
|
||||||
for (i, y) in frames_in.iter().cycle().take(frames).enumerate() {
|
|
||||||
let _ = enc.encode(y).unwrap_or_else(|_| panic!("frame {i}"));
|
|
||||||
}
|
|
||||||
t.elapsed().as_millis() as f64 / frames as f64
|
|
||||||
}
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
println!("1080p 噪声帧编码基准（含 BGRA→YUV 转换）");
|
|
||||||
let t = Instant::now();
|
|
||||||
let _ = YUVBuffer::from_bgra8_source(BgraSliceU8::new(
|
|
||||||
&vec![7u8; 1920 * 1080 * 4],
|
|
||||||
(1920, 1080),
|
|
||||||
));
|
|
||||||
println!("BGRA→YUV 转换: {}ms/帧", t.elapsed().as_millis());
|
|
||||||
let s = bench(1, 20);
|
|
||||||
println!("单线程: {s:.1}ms/帧 → {}fps", (1000.0 / s) as u32);
|
|
||||||
let m = bench(4, 20);
|
|
||||||
println!("4 线程: {m:.1}ms/帧 → {}fps", (1000.0 / m) as u32);
|
|
||||||
}
|
|
||||||
@ -1,35 +1,20 @@
|
|||||||
//! DXGI Desktop Duplication 采集（Windows 8+）。
|
//! Windows 屏幕采集：DXGI Desktop Duplication（Windows 8+）。
|
||||||
//! 编码器自动选择：NVENC 硬编（N 卡：GPU 内 BGRA→H.264 零转换零拷贝直通）
|
//!
|
||||||
//! → openh264 软编兜底（无 N 卡/驱动异常时自动启用）。
|
//! 采集循环、H.264 编码、丢帧与下行都在 `agent-core`（两端共用），本模块只负责
|
||||||
//! 支持主控端 KeyFrameReq 强制关键帧（解码异常恢复）。
|
//! DXGI 的「取新帧 → 拷到 staging 纹理 → 映射内存」这三步。
|
||||||
|
//!
|
||||||
|
//! `ScreenCapturer` 的语义与 Linux 端一致：`acquire()` 返回 false 即代表超时且画面未变，
|
||||||
|
//! 由 agent-core 决定低频重发上一帧。
|
||||||
|
|
||||||
|
use agent_core::ScreenCapturer;
|
||||||
use anyhow::{anyhow, Result};
|
use anyhow::{anyhow, Result};
|
||||||
use common::{encode_video_frame, VideoFrameHeader};
|
|
||||||
use nvenc::bitstream::BitStream;
|
|
||||||
use nvenc::encoder::RegisteredResource;
|
|
||||||
use nvenc::session::{InitParams, Session};
|
|
||||||
use nvenc::sys::enums::{
|
|
||||||
NVencBufferFormat, NVencPicStruct, NVencPicType, NVencParamsRcMode, NVencTuningInfo,
|
|
||||||
};
|
|
||||||
use nvenc::sys::guids::{NV_ENC_CODEC_H264_GUID, NV_ENC_PRESET_P3_GUID};
|
|
||||||
use openh264::encoder::{
|
|
||||||
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
|
|
||||||
};
|
|
||||||
use openh264::formats::{BgraSliceU8, YUVBuffer};
|
|
||||||
use openh264::OpenH264API;
|
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
|
||||||
use std::sync::Arc;
|
|
||||||
use tokio::sync::watch::Sender;
|
|
||||||
use windows::core::Interface;
|
use windows::core::Interface;
|
||||||
use windows::Win32::Foundation::HMODULE;
|
use windows::Win32::Foundation::HMODULE;
|
||||||
use windows::Win32::Graphics::Direct3D::{
|
use windows::Win32::Graphics::Direct3D::{D3D_DRIVER_TYPE_HARDWARE, D3D_FEATURE_LEVEL_11_0};
|
||||||
D3D_DRIVER_TYPE_HARDWARE, D3D_FEATURE_LEVEL_11_0,
|
|
||||||
};
|
|
||||||
use windows::Win32::Graphics::Direct3D11::{
|
use windows::Win32::Graphics::Direct3D11::{
|
||||||
D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D,
|
D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D,
|
||||||
D3D11_BIND_RENDER_TARGET, D3D11_BIND_SHADER_RESOURCE, D3D11_CREATE_DEVICE_FLAG,
|
D3D11_CREATE_DEVICE_FLAG, D3D11_CPU_ACCESS_READ, D3D11_MAP_READ,
|
||||||
D3D11_CPU_ACCESS_READ, D3D11_MAP_READ, D3D11_MAPPED_SUBRESOURCE,
|
D3D11_MAPPED_SUBRESOURCE, D3D11_SDK_VERSION, D3D11_TEXTURE2D_DESC, D3D11_USAGE_STAGING,
|
||||||
D3D11_SDK_VERSION, D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,
|
|
||||||
};
|
};
|
||||||
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_SAMPLE_DESC};
|
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_SAMPLE_DESC};
|
||||||
use windows::Win32::Graphics::Dxgi::{
|
use windows::Win32::Graphics::Dxgi::{
|
||||||
@ -37,15 +22,10 @@ use windows::Win32::Graphics::Dxgi::{
|
|||||||
DXGI_OUTDUPL_DESC, DXGI_OUTDUPL_FRAME_INFO,
|
DXGI_OUTDUPL_DESC, DXGI_OUTDUPL_FRAME_INFO,
|
||||||
};
|
};
|
||||||
|
|
||||||
use std::time::{Duration, Instant};
|
|
||||||
|
|
||||||
pub struct Duplication {
|
pub struct Duplication {
|
||||||
pub device: ID3D11Device,
|
_device: ID3D11Device,
|
||||||
context: ID3D11DeviceContext,
|
context: ID3D11DeviceContext,
|
||||||
staging_res: ID3D11Resource,
|
staging_res: ID3D11Resource,
|
||||||
/// GPU 内中间纹理（NVENC 注册目标；DEFAULT usage，BGRA 与采集一致）
|
|
||||||
gpu_res: ID3D11Resource,
|
|
||||||
gpu_tex: ID3D11Texture2D,
|
|
||||||
dup: IDXGIOutputDuplication,
|
dup: IDXGIOutputDuplication,
|
||||||
pub width: u32,
|
pub width: u32,
|
||||||
pub height: u32,
|
pub height: u32,
|
||||||
@ -59,13 +39,13 @@ impl Duplication {
|
|||||||
D3D11CreateDevice(
|
D3D11CreateDevice(
|
||||||
None,
|
None,
|
||||||
D3D_DRIVER_TYPE_HARDWARE,
|
D3D_DRIVER_TYPE_HARDWARE,
|
||||||
HMODULE(std::ptr::null_mut()),
|
HMODULE::default(),
|
||||||
D3D11_CREATE_DEVICE_FLAG(0),
|
D3D11_CREATE_DEVICE_FLAG(0),
|
||||||
Some(&[D3D_FEATURE_LEVEL_11_0]),
|
Some(&[D3D_FEATURE_LEVEL_11_0]),
|
||||||
D3D11_SDK_VERSION,
|
D3D11_SDK_VERSION,
|
||||||
Some(&raw mut device),
|
Some(&mut device),
|
||||||
None,
|
None,
|
||||||
Some(&raw mut context),
|
Some(&mut context),
|
||||||
)
|
)
|
||||||
.map_err(|e| anyhow!("D3D11CreateDevice failed: {e}"))?;
|
.map_err(|e| anyhow!("D3D11CreateDevice failed: {e}"))?;
|
||||||
let device = device.ok_or_else(|| anyhow!("no d3d device"))?;
|
let device = device.ok_or_else(|| anyhow!("no d3d device"))?;
|
||||||
@ -85,7 +65,6 @@ impl Duplication {
|
|||||||
let (width, height) = (desc.ModeDesc.Width, desc.ModeDesc.Height);
|
let (width, height) = (desc.ModeDesc.Width, desc.ModeDesc.Height);
|
||||||
anyhow::ensure!(width > 0 && height > 0, "bad desktop size");
|
anyhow::ensure!(width > 0 && height > 0, "bad desktop size");
|
||||||
|
|
||||||
// staging：软编路径 Map 读内存
|
|
||||||
let sdesc = D3D11_TEXTURE2D_DESC {
|
let sdesc = D3D11_TEXTURE2D_DESC {
|
||||||
Width: width,
|
Width: width,
|
||||||
Height: height,
|
Height: height,
|
||||||
@ -108,36 +87,25 @@ impl Duplication {
|
|||||||
let staging = staging_opt.ok_or_else(|| anyhow!("no staging texture"))?;
|
let staging = staging_opt.ok_or_else(|| anyhow!("no staging texture"))?;
|
||||||
let staging_res: ID3D11Resource = staging.cast()?;
|
let staging_res: ID3D11Resource = staging.cast()?;
|
||||||
|
|
||||||
// GPU 中间纹理：NVENC 路径的注册目标（DEFAULT usage，GPU 可读）
|
|
||||||
let gdesc = D3D11_TEXTURE2D_DESC {
|
|
||||||
Usage: D3D11_USAGE_DEFAULT,
|
|
||||||
BindFlags: (D3D11_BIND_RENDER_TARGET.0 | D3D11_BIND_SHADER_RESOURCE.0) as u32,
|
|
||||||
CPUAccessFlags: 0,
|
|
||||||
..sdesc
|
|
||||||
};
|
|
||||||
let mut gpu_opt: Option<ID3D11Texture2D> = None;
|
|
||||||
device
|
|
||||||
.CreateTexture2D(&gdesc, None, Some(&mut gpu_opt))
|
|
||||||
.map_err(|e| anyhow!("CreateTexture2D(gpu) failed: {e}"))?;
|
|
||||||
let gpu_tex = gpu_opt.ok_or_else(|| anyhow!("no gpu texture"))?;
|
|
||||||
let gpu_res: ID3D11Resource = gpu_tex.cast()?;
|
|
||||||
|
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
device,
|
_device: device,
|
||||||
context,
|
context,
|
||||||
staging_res,
|
staging_res,
|
||||||
gpu_res,
|
|
||||||
gpu_tex,
|
|
||||||
dup,
|
dup,
|
||||||
width,
|
width,
|
||||||
height,
|
height,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// true=拿到新帧并已复制；false=超时无更新。
|
impl ScreenCapturer for Duplication {
|
||||||
/// 同时复制到 staging（软编读内存）与 GPU 中间纹理（NVENC 注册输入）。
|
fn size(&self) -> (u32, u32) {
|
||||||
pub fn acquire(&mut self, timeout_ms: u32) -> Result<bool> {
|
(self.width, self.height)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// true=拿到新帧并已复制到 staging；false=超时无更新
|
||||||
|
fn acquire(&mut self, timeout_ms: u32) -> Result<bool> {
|
||||||
unsafe {
|
unsafe {
|
||||||
let mut info = DXGI_OUTDUPL_FRAME_INFO::default();
|
let mut info = DXGI_OUTDUPL_FRAME_INFO::default();
|
||||||
let mut resource: Option<IDXGIResource> = None;
|
let mut resource: Option<IDXGIResource> = None;
|
||||||
@ -146,7 +114,6 @@ impl Duplication {
|
|||||||
let res = resource.ok_or_else(|| anyhow!("no desktop resource"))?;
|
let res = resource.ok_or_else(|| anyhow!("no desktop resource"))?;
|
||||||
let src: ID3D11Resource = res.cast()?;
|
let src: ID3D11Resource = res.cast()?;
|
||||||
self.context.CopyResource(&self.staging_res, &src);
|
self.context.CopyResource(&self.staging_res, &src);
|
||||||
self.context.CopyResource(&self.gpu_res, &src);
|
|
||||||
let _ = self.dup.ReleaseFrame();
|
let _ = self.dup.ReleaseFrame();
|
||||||
Ok(true)
|
Ok(true)
|
||||||
}
|
}
|
||||||
@ -156,8 +123,8 @@ impl Duplication {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 在 staging 映射窗口内执行回调（软编路径）。回调参数 = (像素数据, 行距)。
|
/// 把 staging 纹理映射到内存并去掉行间距
|
||||||
pub fn with_mapped<R>(&mut self, f: impl FnOnce(&[u8], usize) -> R) -> Result<R> {
|
fn copy_frame(&mut self) -> Result<Vec<u8>> {
|
||||||
unsafe {
|
unsafe {
|
||||||
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
|
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
|
||||||
self.context
|
self.context
|
||||||
@ -166,315 +133,17 @@ impl Duplication {
|
|||||||
let (w, h) = (self.width as usize, self.height as usize);
|
let (w, h) = (self.width as usize, self.height as usize);
|
||||||
let pitch = mapped.RowPitch as usize;
|
let pitch = mapped.RowPitch as usize;
|
||||||
let src = std::slice::from_raw_parts(mapped.pData as *const u8, pitch * h);
|
let src = std::slice::from_raw_parts(mapped.pData as *const u8, pitch * h);
|
||||||
let r = f(src, pitch);
|
let mut out = Vec::with_capacity(w * h * 4);
|
||||||
|
for row in 0..h {
|
||||||
|
out.extend_from_slice(&src[row * pitch..row * pitch + w * 4]);
|
||||||
|
}
|
||||||
self.context.Unmap(&self.staging_res, 0);
|
self.context.Unmap(&self.staging_res, 0);
|
||||||
Ok(r)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// NVENC 硬编封装：H.264 High、CBR 码率控制、UltraLowLatency 调优、无 B 帧。
|
|
||||||
/// 输入 = 注册的 GPU 纹理（BGRA 与 DXGI 采集字节序一致，零转换）。
|
|
||||||
struct NvencEncoder {
|
|
||||||
encoder: nvenc::encoder::Encoder,
|
|
||||||
registered: RegisteredResource,
|
|
||||||
bitstream: BitStream,
|
|
||||||
frame_count: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl NvencEncoder {
|
|
||||||
fn new(dup: &mut Duplication, bitrate_bps: u32) -> Result<Self> {
|
|
||||||
// 打开会话（无 N 卡 / 驱动不含编码库时在此失败）
|
|
||||||
let session: Session<_> = Session::open_dx(&dup.device)
|
|
||||||
.map_err(|e| anyhow!("open_dx: {e:?}"))?;
|
|
||||||
let codecs = session
|
|
||||||
.get_encode_codecs()
|
|
||||||
.map_err(|e| anyhow!("codecs: {e:?}"))?;
|
|
||||||
anyhow::ensure!(codecs.contains(&NV_ENC_CODEC_H264_GUID), "NVENC no H264 codec");
|
|
||||||
let (session, mut config) = session
|
|
||||||
.get_encode_preset_config_ex(
|
|
||||||
NV_ENC_CODEC_H264_GUID,
|
|
||||||
NV_ENC_PRESET_P3_GUID,
|
|
||||||
NVencTuningInfo::UltraLowLatency,
|
|
||||||
)
|
|
||||||
.map_err(|e| anyhow!("preset_config: {e:?}"))?;
|
|
||||||
// CBR 桌面实时档：码率恒定，延迟可预测（对标 UU 3.2Mbps@60fps）
|
|
||||||
config.preset_cfg.rc_params.rate_control_mode = NVencParamsRcMode::CBR;
|
|
||||||
config.preset_cfg.rc_params.average_bit_rate = bitrate_bps;
|
|
||||||
config.preset_cfg.gop_len = 240; // 周期 IDR（60fps 下 4s）：网络异常后的自愈恢复点，
|
|
||||||
// 配合 KeyFrameReq 即时恢复，避免长时间冻结
|
|
||||||
config.preset_cfg.frame_interval_p = 1; // 无 B 帧
|
|
||||||
|
|
||||||
let init = InitParams {
|
|
||||||
encode_guid: NV_ENC_CODEC_H264_GUID,
|
|
||||||
preset_guid: NV_ENC_PRESET_P3_GUID,
|
|
||||||
resolution: [dup.width, dup.height],
|
|
||||||
aspect_ratio: [dup.width, dup.height],
|
|
||||||
frame_rate: [60, 1],
|
|
||||||
tuning_info: NVencTuningInfo::UltraLowLatency,
|
|
||||||
buffer_format: NVencBufferFormat::ARGB,
|
|
||||||
encode_config: &mut config.preset_cfg,
|
|
||||||
enable_ptd: true,
|
|
||||||
max_encoder_resolution: [dup.width, dup.height],
|
|
||||||
};
|
|
||||||
let encoder = session
|
|
||||||
.init_encoder(init)
|
|
||||||
.map_err(|e| anyhow!("init_encoder: {e:?}"))?;
|
|
||||||
let registered = encoder
|
|
||||||
.register_resource_dx11(&dup.gpu_tex, NVencBufferFormat::ARGB, 0)
|
|
||||||
.map_err(|e| anyhow!("register_resource: {e:?}"))?;
|
|
||||||
let bitstream = encoder
|
|
||||||
.create_bitstream_buffer()
|
|
||||||
.map_err(|e| anyhow!("bitstream: {e:?}"))?;
|
|
||||||
Ok(Self {
|
|
||||||
encoder,
|
|
||||||
registered,
|
|
||||||
bitstream,
|
|
||||||
frame_count: 0,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// 编码当前 GPU 纹理内容，返回 Annex B H.264 码流。
|
|
||||||
fn encode(&mut self, force_idr: bool, ts_ms: u64) -> Result<Vec<u8>> {
|
|
||||||
// 首帧强制 IDR：解码端从任意时刻接入，没有 IDR 无法解出后续 P 帧
|
|
||||||
let ty = if force_idr || self.frame_count == 0 {
|
|
||||||
NVencPicType::IDR
|
|
||||||
} else {
|
|
||||||
NVencPicType::P
|
|
||||||
};
|
|
||||||
self.encoder.encode_picture(
|
|
||||||
&self.registered,
|
|
||||||
&self.bitstream,
|
|
||||||
self.frame_count,
|
|
||||||
ts_ms,
|
|
||||||
NVencBufferFormat::ARGB,
|
|
||||||
NVencPicStruct::Frame,
|
|
||||||
ty,
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
.map_err(|e| anyhow!("encode_picture: {e:?}"))?;
|
|
||||||
self.frame_count += 1;
|
|
||||||
// 同步读出码流（NVENC 单帧编码 1-3ms，阻塞等待可接受）
|
|
||||||
let lock = self
|
|
||||||
.bitstream
|
|
||||||
.try_lock(true)
|
|
||||||
.map_err(|e| anyhow!("bitstream lock: {e:?}"))?;
|
|
||||||
let out = lock.as_slice().to_vec();
|
|
||||||
drop(lock);
|
|
||||||
if out.is_empty() {
|
|
||||||
anyhow::bail!("nvenc empty bitstream");
|
|
||||||
}
|
|
||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[allow(dead_code)]
|
|
||||||
pub fn desktop_size() -> Result<(u32, u32)> {
|
|
||||||
let d = Duplication::new()?;
|
|
||||||
Ok((d.width, d.height))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 采集主循环：独立线程运行；连接断开（通道关闭）时退出。
|
/// 采集器工厂：agent-core 在采集后端失效时会重新调用
|
||||||
/// 通道为 watch 单槽（latest-wins）：发送即覆盖，网络慢时不产生积压。
|
pub fn new_capturer() -> Result<Box<dyn ScreenCapturer>> {
|
||||||
/// congestion：发送侧 write 超时置位 → 跳过当前帧（不编码不发送），注入率自适应降速。
|
Ok(Box::new(Duplication::new()?))
|
||||||
pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc<AtomicBool>) {
|
|
||||||
let start = Instant::now();
|
|
||||||
loop {
|
|
||||||
let mut dup = match Duplication::new() {
|
|
||||||
Ok(d) => d,
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] init failed: {e}, retry in 2s");
|
|
||||||
std::thread::sleep(Duration::from_secs(2));
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
eprintln!(
|
|
||||||
"[capture] duplication started {}x{}",
|
|
||||||
dup.width, dup.height
|
|
||||||
);
|
|
||||||
|
|
||||||
// ---- 编码器自动选择：NVENC 硬编优先，失败落回 openh264 软编 ----
|
|
||||||
enum EncoderKind {
|
|
||||||
Nvenc(Box<NvencEncoder>),
|
|
||||||
Soft(Box<openh264::encoder::Encoder>),
|
|
||||||
}
|
|
||||||
let mut encoder: EncoderKind = match NvencEncoder::new(&mut dup, 5_000_000) {
|
|
||||||
Ok(nv) => {
|
|
||||||
eprintln!("[capture] encoder = NVENC hardware (CBR 5Mbps@60fps, ARGB direct)");
|
|
||||||
EncoderKind::Nvenc(Box::new(nv))
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] NVENC unavailable ({e}); fallback to openh264 software");
|
|
||||||
match Encoder::with_api_config(
|
|
||||||
OpenH264API::from_source(),
|
|
||||||
EncoderConfig::new()
|
|
||||||
.bitrate(BitRate::from_bps(5_000_000))
|
|
||||||
.max_frame_rate(FrameRate::from_hz(60.0))
|
|
||||||
.rate_control_mode(RateControlMode::Bitrate)
|
|
||||||
.usage_type(UsageType::ScreenContentRealTime)
|
|
||||||
.intra_frame_period(IntraFramePeriod::from_num_frames(300))
|
|
||||||
.num_threads(4),
|
|
||||||
) {
|
|
||||||
Ok(e) => {
|
|
||||||
eprintln!("[capture] encoder = openh264 software (5Mbps@60fps, 4T)");
|
|
||||||
EncoderKind::Soft(Box::new(e))
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] encoder init failed: {e}");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut last_frame: Option<Vec<u8>> = None; // 软编静止重发
|
|
||||||
let mut last_sent = Instant::now() - Duration::from_secs(10);
|
|
||||||
let min_frame = Duration::from_millis(16); // 60fps 上限
|
|
||||||
let mut scratch: Vec<u8> = Vec::new(); // 软编：行距不连续时的复用缓冲
|
|
||||||
// 编码耗时统计（每 120 帧输出一次均值，ms）
|
|
||||||
let mut stat_frames = 0u64;
|
|
||||||
let mut stat_convert_ms = 0u64;
|
|
||||||
let mut stat_encode_ms = 0u64;
|
|
||||||
let mut stat_skipped = 0u64; // 拥塞跳帧计数
|
|
||||||
|
|
||||||
macro_rules! emit {
|
|
||||||
($payload:expr) => {{
|
|
||||||
let header = VideoFrameHeader {
|
|
||||||
width: dup.width,
|
|
||||||
height: dup.height,
|
|
||||||
ts_ms: start.elapsed().as_millis() as u64,
|
|
||||||
};
|
|
||||||
let frame = encode_video_frame(&header, &$payload);
|
|
||||||
if tx.send(Some(frame.clone())).is_err() {
|
|
||||||
return; // 会话结束
|
|
||||||
}
|
|
||||||
last_frame = Some(frame);
|
|
||||||
last_sent = Instant::now();
|
|
||||||
stat_frames += 1;
|
|
||||||
}};
|
|
||||||
}
|
|
||||||
|
|
||||||
loop {
|
|
||||||
let iter_start = Instant::now();
|
|
||||||
match dup.acquire(80) {
|
|
||||||
Ok(true) => {
|
|
||||||
// 主控端请求关键帧（解码异常恢复）
|
|
||||||
let force_key = keyreq.swap(false, Ordering::Relaxed);
|
|
||||||
// 拥塞反馈：发送侧排队深 → 跳过本帧（NVENC 少输入一帧，参考链连续；
|
|
||||||
// 解码端收到的帧序 = 编码端实际输出序，无花屏风险）
|
|
||||||
if congestion.swap(false, Ordering::Relaxed) {
|
|
||||||
stat_skipped += 1;
|
|
||||||
let el = iter_start.elapsed();
|
|
||||||
if el < min_frame {
|
|
||||||
std::thread::sleep(min_frame - el);
|
|
||||||
}
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
match &mut encoder {
|
|
||||||
EncoderKind::Nvenc(enc) => {
|
|
||||||
let enc_start = Instant::now();
|
|
||||||
let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
|
|
||||||
stat_encode_ms += enc_start.elapsed().as_millis() as u64;
|
|
||||||
match r {
|
|
||||||
Ok(payload) => emit!(payload),
|
|
||||||
Err(e) => eprintln!("[capture] nvenc encode failed: {e}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
EncoderKind::Soft(enc) => {
|
|
||||||
if force_key {
|
|
||||||
enc.force_intra_frame();
|
|
||||||
}
|
|
||||||
// 映射窗口内直接做 BGRA→YUV（行距连续时零拷贝，否则复用 scratch）
|
|
||||||
let conv_start = Instant::now();
|
|
||||||
let dims = (dup.width as usize, dup.height as usize);
|
|
||||||
let conv = dup.with_mapped(|src, pitch| {
|
|
||||||
if pitch == dims.0 * 4 {
|
|
||||||
YUVBuffer::from_bgra8_source(BgraSliceU8::new(src, dims))
|
|
||||||
} else {
|
|
||||||
scratch.resize(dims.0 * dims.1 * 4, 0);
|
|
||||||
for (row, dst) in
|
|
||||||
scratch.chunks_exact_mut(dims.0 * 4).enumerate()
|
|
||||||
{
|
|
||||||
dst.copy_from_slice(
|
|
||||||
&src[row * pitch..row * pitch + dims.0 * 4],
|
|
||||||
);
|
|
||||||
}
|
|
||||||
YUVBuffer::from_bgra8_source(BgraSliceU8::new(&scratch, dims))
|
|
||||||
}
|
|
||||||
});
|
|
||||||
match conv {
|
|
||||||
Ok(yuv) => {
|
|
||||||
stat_convert_ms +=
|
|
||||||
conv_start.elapsed().as_millis() as u64;
|
|
||||||
let enc_start = Instant::now();
|
|
||||||
let r = enc.encode(&yuv);
|
|
||||||
stat_encode_ms += enc_start.elapsed().as_millis() as u64;
|
|
||||||
match r {
|
|
||||||
Ok(ebs) => emit!(ebs.to_vec()),
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] encode failed: {e}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] frame convert failed: {e}");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if stat_frames > 0 && stat_frames % 120 == 0 {
|
|
||||||
eprintln!(
|
|
||||||
"[capture] avg convert {}ms, encode {}ms/frame, skipped {} (last 120)",
|
|
||||||
stat_convert_ms / 120,
|
|
||||||
stat_encode_ms / 120,
|
|
||||||
stat_skipped
|
|
||||||
);
|
|
||||||
stat_convert_ms = 0;
|
|
||||||
stat_encode_ms = 0;
|
|
||||||
stat_skipped = 0;
|
|
||||||
}
|
|
||||||
// 帧间隔节流到 ~60fps
|
|
||||||
let el = iter_start.elapsed();
|
|
||||||
if el < min_frame {
|
|
||||||
std::thread::sleep(min_frame - el);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(false) => {
|
|
||||||
// 桌面静止：低频重发（丢包后恢复 + 会话保活）。
|
|
||||||
// 关键：KeyFrameReq 在桌面静止时也必须响应 —— 此前只在新帧路径
|
|
||||||
// 消费该标志，桌面完全静止时解码端请求 IDR 永远得不到响应 → 黑屏。
|
|
||||||
let force_key = keyreq.swap(false, Ordering::Relaxed);
|
|
||||||
if last_sent.elapsed() > Duration::from_millis(500) {
|
|
||||||
match &mut encoder {
|
|
||||||
EncoderKind::Nvenc(enc) => {
|
|
||||||
// 重编当前 GPU 帧（内容未变，输出极小；force_key 时为 IDR）
|
|
||||||
let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
|
|
||||||
match r {
|
|
||||||
Ok(payload) => emit!(payload),
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] nvenc idle encode failed: {e}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
EncoderKind::Soft(_) => {
|
|
||||||
// 软编：直接重发最后一帧码流副本
|
|
||||||
if let Some(f) = last_frame.as_ref() {
|
|
||||||
let _ = tx.send(Some(f.clone()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
last_sent = Instant::now();
|
|
||||||
} else {
|
|
||||||
std::thread::sleep(Duration::from_millis(8));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[capture] acquire failed: {e}");
|
|
||||||
break; // ACCESS_LOST 等，重建
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
std::thread::sleep(Duration::from_millis(300));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,6 +1,11 @@
|
|||||||
//! 输入注入：SendInput 实现（鼠标归一化绝对坐标 + 扫描码键盘）。
|
//! Windows 输入注入：SendInput（鼠标虚拟桌面绝对坐标 + 扫描码键盘）。
|
||||||
|
//!
|
||||||
|
//! 协议里的坐标/扫描码语义两端一致，只是落地 API 不同：
|
||||||
|
//! - 鼠标：`MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_VIRTUALDESK`，直接吃 0..65535 归一化值；
|
||||||
|
//! - 键盘：`KEYEVENTF_SCANCODE` 直接送 Set-1 扫描码（Linux 端则需换算成 X11 keycode）。
|
||||||
|
|
||||||
use common::{InputEvent, MouseButton};
|
use agent_core::InputInjector;
|
||||||
|
use common::MouseButton;
|
||||||
use windows::Win32::UI::Input::KeyboardAndMouse::{
|
use windows::Win32::UI::Input::KeyboardAndMouse::{
|
||||||
SendInput, INPUT, INPUT_0, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY,
|
SendInput, INPUT, INPUT_0, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY,
|
||||||
KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_MIDDLEDOWN,
|
KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_MIDDLEDOWN,
|
||||||
@ -73,9 +78,23 @@ pub fn virtual_desktop() -> (i32, i32, i32, i32) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn handle(ev: InputEvent) {
|
/// SendInput 注入后端（无状态，直接调 Win32）
|
||||||
match ev {
|
pub struct WinInput;
|
||||||
InputEvent::MouseMove { x, y } => {
|
|
||||||
|
impl WinInput {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for WinInput {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl InputInjector for WinInput {
|
||||||
|
fn mouse_move(&self, x: u16, y: u16) {
|
||||||
// 主控端已按虚拟桌面归一化，这里直接透传 0..65535
|
// 主控端已按虚拟桌面归一化，这里直接透传 0..65535
|
||||||
send_mouse(
|
send_mouse(
|
||||||
MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_VIRTUALDESK.0 | MOUSEEVENTF_MOVE.0,
|
MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_VIRTUALDESK.0 | MOUSEEVENTF_MOVE.0,
|
||||||
@ -84,20 +103,24 @@ pub fn handle(ev: InputEvent) {
|
|||||||
0,
|
0,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
InputEvent::MouseDown(b) => match b {
|
|
||||||
MouseButton::Left => send_mouse(MOUSEEVENTF_LEFTDOWN, 0, 0, 0),
|
fn mouse_button(&self, b: MouseButton, down: bool) {
|
||||||
MouseButton::Right => send_mouse(MOUSEEVENTF_RIGHTDOWN.0, 0, 0, 0),
|
let flags = match (b, down) {
|
||||||
MouseButton::Middle => send_mouse(MOUSEEVENTF_MIDDLEDOWN.0, 0, 0, 0),
|
(MouseButton::Left, true) => MOUSEEVENTF_LEFTDOWN,
|
||||||
},
|
(MouseButton::Left, false) => MOUSEEVENTF_LEFTUP,
|
||||||
InputEvent::MouseUp(b) => match b {
|
(MouseButton::Right, true) => MOUSEEVENTF_RIGHTDOWN.0,
|
||||||
MouseButton::Left => send_mouse(MOUSEEVENTF_LEFTUP, 0, 0, 0),
|
(MouseButton::Right, false) => MOUSEEVENTF_RIGHTUP.0,
|
||||||
MouseButton::Right => send_mouse(MOUSEEVENTF_RIGHTUP.0, 0, 0, 0),
|
(MouseButton::Middle, true) => MOUSEEVENTF_MIDDLEDOWN.0,
|
||||||
MouseButton::Middle => send_mouse(MOUSEEVENTF_MIDDLEUP.0, 0, 0, 0),
|
(MouseButton::Middle, false) => MOUSEEVENTF_MIDDLEUP.0,
|
||||||
},
|
};
|
||||||
InputEvent::Wheel { dy } => {
|
send_mouse(flags, 0, 0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wheel(&self, dy: i32) {
|
||||||
send_mouse(MOUSEEVENTF_WHEEL.0, 0, 0, dy as u32);
|
send_mouse(MOUSEEVENTF_WHEEL.0, 0, 0, dy as u32);
|
||||||
}
|
}
|
||||||
InputEvent::KeyDown { scancode, extended } => send_key(scancode, extended, false),
|
|
||||||
InputEvent::KeyUp { scancode, extended } => send_key(scancode, extended, true),
|
fn key(&self, scancode: u16, extended: bool, down: bool) {
|
||||||
|
send_key(scancode, extended, !down);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,161 +1,28 @@
|
|||||||
//! 被控端 Agent：DXGI 桌面复制采集 → LZ4 压缩 → QUIC 下行视频；上行接收输入事件注入。
|
//! 被控端 Agent（Windows）：DXGI 桌面复制采集 + SendInput 键鼠注入。
|
||||||
//! P0 优化：视频帧经 watch 单槽通道（latest-wins，网络慢时只发最新帧，不积压）；
|
//!
|
||||||
//! 键鼠/控制走独立 QUIC 双向流，不与视频互相阻塞。
|
//! 会话层（QUIC/TLS 握手、验证码认证、H.264 编码下行、latest-wins 丢帧、输入分发）在
|
||||||
|
//! `agent-core`，与 `linux-agent` 共用同一份实现；本 crate 只提供平台相关的采集与注入，
|
||||||
|
//! 避免两端各写一份握手/认证造成协议漂移。
|
||||||
|
|
||||||
mod capture;
|
mod capture;
|
||||||
mod input;
|
mod input;
|
||||||
|
|
||||||
|
use agent_core::{parse_args, serve, EncoderSettings, InputInjector, SessionConfig};
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use common::{read_control, write_control, ControlMsg};
|
|
||||||
use rustls::pki_types::PrivatePkcs8KeyDer;
|
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use tokio::io::AsyncWriteExt;
|
|
||||||
use tokio::sync::{mpsc, watch};
|
|
||||||
|
|
||||||
fn parse_args() -> Result<(u16, String)> {
|
|
||||||
let mut port = 47800u16;
|
|
||||||
let mut code = "123456".to_string();
|
|
||||||
let a: Vec<String> = std::env::args().collect();
|
|
||||||
let mut i = 1;
|
|
||||||
while i < a.len() {
|
|
||||||
match a[i].as_str() {
|
|
||||||
"--port" => port = a[i + 1].parse()?,
|
|
||||||
"--code" => code = a[i + 1].clone(),
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
i += 2;
|
|
||||||
}
|
|
||||||
Ok((port, code))
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
async fn main() -> Result<()> {
|
async fn main() -> Result<()> {
|
||||||
let (port, code) = parse_args()?;
|
let (port, code) = parse_args()?;
|
||||||
rustls::crypto::ring::default_provider()
|
|
||||||
.install_default()
|
|
||||||
.ok();
|
|
||||||
|
|
||||||
// 自签名证书（MVP：客户端跳过校验；正式版改为固定公钥/账号体系）
|
let injector: Arc<dyn InputInjector> = Arc::new(input::WinInput::new());
|
||||||
let ck = rcgen::generate_simple_self_signed(vec!["localhost".into()])?;
|
|
||||||
let cert = ck.cert.der().clone();
|
|
||||||
let key = PrivatePkcs8KeyDer::from(ck.key_pair.serialize_der());
|
|
||||||
|
|
||||||
// 传输参数调优：
|
serve(
|
||||||
// - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口）
|
SessionConfig { port, code },
|
||||||
// - 发送缓冲上限 64KB：QUIC 内部排队硬上限（实测 WiFi ~3Mbps 吞吐 → E2E ≤ ~170ms；
|
EncoderSettings::default(),
|
||||||
// 突发吞吐 = 64KB/7ms ≈ 72Mbps，不限制带宽）。注意此窗口为连接级，流 2 输入/Ping
|
// DXGI 后端失效（ACCESS_LOST 等）时 agent-core 会重新调用本工厂
|
||||||
// 数据量极小不受影响。配合下方拥塞反馈跳帧，把排队压在数帧以内
|
capture::new_capturer,
|
||||||
let mut transport = quinn::TransportConfig::default();
|
injector,
|
||||||
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
|
)
|
||||||
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
|
|
||||||
transport.send_window(64 * 1024);
|
|
||||||
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
|
|
||||||
server_config.transport_config(Arc::new(transport));
|
|
||||||
|
|
||||||
let endpoint = quinn::Endpoint::server(
|
|
||||||
server_config,
|
|
||||||
format!("0.0.0.0:{port}").parse()?,
|
|
||||||
)?;
|
|
||||||
println!("[agent] listening 0.0.0.0:{port}, verify code = {code}");
|
|
||||||
|
|
||||||
let conn = endpoint
|
|
||||||
.accept()
|
|
||||||
.await
|
.await
|
||||||
.ok_or_else(|| anyhow::anyhow!("endpoint closed"))?
|
|
||||||
.await?;
|
|
||||||
println!("[agent] peer connected: {}", conn.remote_address());
|
|
||||||
|
|
||||||
// 流 1：认证 + 视频下行
|
|
||||||
let (mut send, mut recv) = conn.accept_bi().await?;
|
|
||||||
|
|
||||||
// 认证
|
|
||||||
let first = read_control(&mut recv).await?;
|
|
||||||
let code_ok = matches!(&first, ControlMsg::Auth { code: c } if *c == code);
|
|
||||||
if !code_ok {
|
|
||||||
write_control(&mut send, &ControlMsg::AuthFail("bad code".into())).await?;
|
|
||||||
send.finish()?;
|
|
||||||
anyhow::bail!("auth failed");
|
|
||||||
}
|
|
||||||
let (w, h) = capture::desktop_size()?;
|
|
||||||
write_control(&mut send, &ControlMsg::AuthOk { width: w, height: h }).await?;
|
|
||||||
println!("[agent] auth ok, desktop {w}x{h}");
|
|
||||||
|
|
||||||
// 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞）
|
|
||||||
let (mut isend, mut irecv) = conn.accept_bi().await?;
|
|
||||||
|
|
||||||
// 采集线程 → 视频帧（watch 单槽：新帧覆盖旧帧，latest-wins）
|
|
||||||
let (tx, mut frames_rx) = watch::channel::<Option<Vec<u8>>>(None);
|
|
||||||
let keyreq = Arc::new(AtomicBool::new(false));
|
|
||||||
let keyreq_capture = keyreq.clone();
|
|
||||||
// 拥塞反馈标志：发送侧 write 超过单帧预算(20ms) → 置位，采集线程跳帧降注入率
|
|
||||||
let congestion = Arc::new(AtomicBool::new(false));
|
|
||||||
let congestion_capture = congestion.clone();
|
|
||||||
std::thread::spawn(move || capture::run(tx, keyreq_capture, congestion_capture));
|
|
||||||
|
|
||||||
// 上行控制消息（Pong）经通道交给主循环写流 2，避免多任务争抢
|
|
||||||
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
|
|
||||||
let t0 = std::time::Instant::now();
|
|
||||||
|
|
||||||
// 读循环（流 2）：输入事件 + Ping；Pong 附带视频下行方向的 QUIC 丢包统计
|
|
||||||
// （conn.stats().path 统计的是本端发送方向，agent 的发送方向即视频下行）
|
|
||||||
let keyreq_reader = keyreq.clone();
|
|
||||||
let conn_stats = conn.clone();
|
|
||||||
let reader = tokio::spawn(async move {
|
|
||||||
loop {
|
|
||||||
match read_control(&mut irecv).await {
|
|
||||||
Ok(ControlMsg::Input(ev)) => input::handle(ev),
|
|
||||||
Ok(ControlMsg::KeyFrameReq) => {
|
|
||||||
keyreq_reader.store(true, Ordering::Relaxed);
|
|
||||||
}
|
|
||||||
Ok(ControlMsg::Ping(t)) => {
|
|
||||||
let agent_ts = t0.elapsed().as_millis() as u64;
|
|
||||||
let p = conn_stats.stats().path;
|
|
||||||
let _ = ctl_tx.send(ControlMsg::Pong {
|
|
||||||
t,
|
|
||||||
agent_ts,
|
|
||||||
lost: p.lost_packets,
|
|
||||||
sent: p.sent_packets,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
Ok(_) => {}
|
|
||||||
Err(_) => break,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
loop {
|
|
||||||
tokio::select! {
|
|
||||||
changed = frames_rx.changed() => {
|
|
||||||
if changed.is_err() {
|
|
||||||
break; // 采集线程退出
|
|
||||||
}
|
|
||||||
if let Some(data) = frames_rx.borrow().clone() {
|
|
||||||
let wt = std::time::Instant::now();
|
|
||||||
let ok = send.write_all(&data).await.is_ok();
|
|
||||||
// write 耗时 > 单帧预算(16.7ms 的 1.2 倍) → QUIC 窗口排队深，
|
|
||||||
// 通知采集线程跳帧，把注入率降到网络实际吞吐水平
|
|
||||||
if wt.elapsed() > std::time::Duration::from_millis(20) {
|
|
||||||
congestion.store(true, Ordering::Relaxed);
|
|
||||||
}
|
|
||||||
if !ok {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
ctl = ctl_rx.recv() => {
|
|
||||||
match ctl {
|
|
||||||
Some(m) => {
|
|
||||||
if write_control(&mut isend, &m).await.is_err() {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => break,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
reader.abort();
|
|
||||||
println!("[agent] session closed");
|
|
||||||
Ok(())
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -15,8 +15,7 @@ pub enum ControlMsg {
|
|||||||
Ping(u64),
|
Ping(u64),
|
||||||
/// t=主控端发送时刻（主控时钟），agent_ts=被控端收到时刻（被控时钟）
|
/// t=主控端发送时刻（主控时钟），agent_ts=被控端收到时刻（被控时钟）
|
||||||
/// 两者配合可估计时钟偏移，进而用帧头 ts_ms 计算端到端延迟
|
/// 两者配合可估计时钟偏移，进而用帧头 ts_ms 计算端到端延迟
|
||||||
/// lost/sent = agent 发送方向（视频下行）的 QUIC 丢包/发包累计计数（conn.stats().path）
|
Pong { t: u64, agent_ts: u64 },
|
||||||
Pong { t: u64, agent_ts: u64, lost: u64, sent: u64 },
|
|
||||||
Input(InputEvent),
|
Input(InputEvent),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -306,18 +306,12 @@ async fn end_to_end_capture_encode_transport_input() -> Result<()> {
|
|||||||
// 流 2：输入 + 控制
|
// 流 2：输入 + 控制
|
||||||
let (mut isend, mut irecv) = conn.open_bi().await?;
|
let (mut isend, mut irecv) = conn.open_bi().await?;
|
||||||
|
|
||||||
// Ping/Pong（延迟测量通道 + 下行丢包统计）
|
// Ping/Pong（延迟测量通道）
|
||||||
write_control(&mut isend, &ControlMsg::Ping(4242)).await?;
|
write_control(&mut isend, &ControlMsg::Ping(4242)).await?;
|
||||||
match read_control(&mut irecv).await? {
|
match read_control(&mut irecv).await? {
|
||||||
ControlMsg::Pong {
|
ControlMsg::Pong { t, agent_ts } => {
|
||||||
t,
|
|
||||||
agent_ts,
|
|
||||||
lost,
|
|
||||||
sent,
|
|
||||||
} => {
|
|
||||||
assert_eq!(t, 4242, "Pong 回显的时间戳不一致");
|
assert_eq!(t, 4242, "Pong 回显的时间戳不一致");
|
||||||
assert!(sent > 0, "下行发包计数应大于 0（视频流已在发）");
|
println!("[smoke] Pong 收到，agent_ts={agent_ts}");
|
||||||
println!("[smoke] Pong 收到，agent_ts={agent_ts}，下行 sent={sent} lost={lost}");
|
|
||||||
}
|
}
|
||||||
other => bail!("期望 Pong，收到 {other:?}"),
|
other => bail!("期望 Pong，收到 {other:?}"),
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,37 +0,0 @@
|
|||||||
# 双机测试基线脚本（B1）
|
|
||||||
|
|
||||||
每个场景测试 60 秒，记录 HUD 读数填入《B侧设计文档.md》末尾的基线数据表。
|
|
||||||
|
|
||||||
## 准备
|
|
||||||
|
|
||||||
- 两端都用最新 release：`agent.exe` + `viewer.exe`（同目录带 3 个 mingw DLL）
|
|
||||||
- 被控机：`agent.exe --port 47800 --code 123456`
|
|
||||||
- 主控机：`viewer.exe --addr <被控机IP> --port 47800 --code 123456`
|
|
||||||
- 测试前确认：被控机亮屏有桌面；同一网络（记下是有线还是 WiFi 频段）
|
|
||||||
|
|
||||||
## 场景
|
|
||||||
|
|
||||||
| # | 场景 | 被控机操作 | 观察点 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| 1 | 静止桌面 | 打开一个空白文档，不动 60s | FPS 应低（静止检测），E2E/JIT 应绿 |
|
|
||||||
| 2 | 视频播放 | 全屏播放任意 1080p 视频 | 码率峰值、FPS、丢包（LOSS 行） |
|
|
||||||
| 3 | 拖动窗口 | 持续快速拖动一个大窗口 60s | 最恶劣工况：E2E/P95/JIT 是否变红、DECFAIL 是否增长 |
|
|
||||||
|
|
||||||
## 记录方法
|
|
||||||
|
|
||||||
每个场景进行中截一张 HUD 图（Win+Shift+S 框选右上角），或肉眼逐行抄录：
|
|
||||||
|
|
||||||
```
|
|
||||||
FPS= RTT= E2E= JIT= P95= LOSS= DECFAIL= RATE=
|
|
||||||
```
|
|
||||||
|
|
||||||
## 附加记录
|
|
||||||
|
|
||||||
- 网络环境：有线 / WiFi 5GHz / WiFi 2.4GHz，信号强度
|
|
||||||
- 被控机 CPU 占用（任务管理器，agent.exe 一行）
|
|
||||||
- 编码器类型：当前 openh264 软编（A 侧换 NVENC 后需重测一轮对比）
|
|
||||||
|
|
||||||
## 注意
|
|
||||||
|
|
||||||
- LOSS 显示 `--` 表示第一个 Pong 还没回来或 Δsent=0（静止场景发送少，属正常）
|
|
||||||
- DECFAIL 在连接建立初期可能有 1-2 次（等第一个关键帧），后续不应增长；持续增长 = 关键帧自愈有问题，报给 A
|
|
||||||
@ -81,15 +81,8 @@ fn now_ms() -> u64 {
|
|||||||
T0.get().unwrap().elapsed().as_millis() as u64
|
T0.get().unwrap().elapsed().as_millis() as u64
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 帧到达间隔环形缓冲（取 p95 用）
|
|
||||||
struct GapRing {
|
|
||||||
last: Option<Instant>,
|
|
||||||
ring: [u32; 64],
|
|
||||||
idx: usize,
|
|
||||||
filled: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// 会话统计（HUD 数据源，全部原子量，跨任务共享）
|
/// 会话统计（HUD 数据源，全部原子量，跨任务共享）
|
||||||
|
#[derive(Default)]
|
||||||
struct Stats {
|
struct Stats {
|
||||||
/// 最近 RTT ms（0=未知）
|
/// 最近 RTT ms（0=未知）
|
||||||
rtt: AtomicU64,
|
rtt: AtomicU64,
|
||||||
@ -102,80 +95,6 @@ struct Stats {
|
|||||||
/// 累计收帧数 / 字节数
|
/// 累计收帧数 / 字节数
|
||||||
frames: AtomicU64,
|
frames: AtomicU64,
|
||||||
bytes: AtomicU64,
|
bytes: AtomicU64,
|
||||||
/// 抖动：相邻帧 |ΔE2E| 的 EMA（ms×16 定点存储避免浮点原子）
|
|
||||||
jitter_x16: AtomicU64,
|
|
||||||
prev_e2e: AtomicI64,
|
|
||||||
/// agent 发送侧（视频下行）QUIC 丢包/发包累计（最近一次 Pong）
|
|
||||||
agent_lost: AtomicU64,
|
|
||||||
agent_sent: AtomicU64,
|
|
||||||
/// H.264 解码失败计数（关键帧自愈验证指标）
|
|
||||||
dec_fail: AtomicU64,
|
|
||||||
/// 帧到达间隔环形缓冲
|
|
||||||
gaps: std::sync::Mutex<GapRing>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Default for Stats {
|
|
||||||
fn default() -> Self {
|
|
||||||
Self {
|
|
||||||
rtt: AtomicU64::new(0),
|
|
||||||
offset: AtomicI64::new(0),
|
|
||||||
offset_valid: AtomicBool::new(false),
|
|
||||||
e2e: AtomicU64::new(0),
|
|
||||||
e2e_valid: AtomicBool::new(false),
|
|
||||||
frames: AtomicU64::new(0),
|
|
||||||
bytes: AtomicU64::new(0),
|
|
||||||
jitter_x16: AtomicU64::new(0),
|
|
||||||
prev_e2e: AtomicI64::new(-1),
|
|
||||||
agent_lost: AtomicU64::new(0),
|
|
||||||
agent_sent: AtomicU64::new(0),
|
|
||||||
dec_fail: AtomicU64::new(0),
|
|
||||||
gaps: std::sync::Mutex::new(GapRing {
|
|
||||||
last: None,
|
|
||||||
ring: [0; 64],
|
|
||||||
idx: 0,
|
|
||||||
filled: 0,
|
|
||||||
}),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Stats {
|
|
||||||
/// 记录一次帧到达：更新间隔环形缓冲与抖动 EMA
|
|
||||||
fn on_frame_arrival(&self, est: Option<i64>) {
|
|
||||||
let now = Instant::now();
|
|
||||||
if let Ok(mut g) = self.gaps.lock() {
|
|
||||||
if let Some(t) = g.last {
|
|
||||||
let gap = (now - t).as_millis() as u32;
|
|
||||||
let i = g.idx;
|
|
||||||
g.ring[i] = gap;
|
|
||||||
g.idx = (i + 1) % g.ring.len();
|
|
||||||
g.filled = (g.filled + 1).min(g.ring.len());
|
|
||||||
}
|
|
||||||
g.last = Some(now);
|
|
||||||
}
|
|
||||||
// 抖动 = |本帧E2E - 上帧E2E| 的 EMA（RFC3550 风格，÷16 衰减）
|
|
||||||
if let Some(e) = est {
|
|
||||||
let prev = self.prev_e2e.load(Ordering::Relaxed);
|
|
||||||
self.prev_e2e.store(e, Ordering::Relaxed);
|
|
||||||
if prev >= 0 {
|
|
||||||
let d = (e - prev).unsigned_abs() as u64;
|
|
||||||
let old = self.jitter_x16.load(Ordering::Relaxed);
|
|
||||||
let sm = if old == 0 { d * 16 } else { old + d * 16 / 16 - old / 16 };
|
|
||||||
self.jitter_x16.store(sm, Ordering::Relaxed);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// 帧间隔 p95（ms），样本不足返回 None
|
|
||||||
fn gap_p95(&self) -> Option<u32> {
|
|
||||||
let g = self.gaps.lock().ok()?;
|
|
||||||
if g.filled < 8 {
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
let mut v: Vec<u32> = g.ring[..g.filled].to_vec();
|
|
||||||
v.sort_unstable();
|
|
||||||
v.get(v.len() * 95 / 100).copied()
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn main() -> Result<()> {
|
fn main() -> Result<()> {
|
||||||
@ -241,105 +160,48 @@ fn session(
|
|||||||
let (w, h) = (width as usize, height as usize);
|
let (w, h) = (width as usize, height as usize);
|
||||||
let stats = Arc::new(Stats::default());
|
let stats = Arc::new(Stats::default());
|
||||||
|
|
||||||
// 读任务（流 1）：H.264 解码 → RGB → 单槽 latest-wins；同时统计端到端延迟与流量
|
// 读任务（流 1）：H.264 解码 → RGB → mpsc；同时统计端到端延迟与流量
|
||||||
// 帧格式 (src_w, src_h, rgb)；(0,0,_) = 解码失败占位，触发 KeyFrameReq
|
let (frame_tx, frame_rx) = std::sync::mpsc::channel::<Vec<u8>>();
|
||||||
// 单槽而非无界队列：渲染跟不上 60fps 到达率时旧帧直接被覆盖，绝不积压
|
|
||||||
//（无界 mpsc 在 NVENC 60fps 下会无限积压 → 延迟线性增长 + 渲染 drain 时
|
|
||||||
// 长时间阻塞窗口消息泵 → 窗口假死、HUD 停刷新）
|
|
||||||
let latest: Arc<std::sync::Mutex<Option<(u32, u32, Vec<u8>)>>> =
|
|
||||||
Arc::new(std::sync::Mutex::new(None));
|
|
||||||
{
|
{
|
||||||
let stats = stats.clone();
|
let stats = stats.clone();
|
||||||
let latest = latest.clone();
|
|
||||||
let mut decoder = openh264::decoder::Decoder::new()
|
let mut decoder = openh264::decoder::Decoder::new()
|
||||||
.map_err(|e| anyhow::anyhow!("decoder init failed: {e}"))?;
|
.map_err(|e| anyhow::anyhow!("decoder init failed: {e}"))?;
|
||||||
let mut rgb_buf: Vec<u8> = Vec::new();
|
|
||||||
let mut last_kfreq = Instant::now();
|
|
||||||
let mut skip_stale = 0u64;
|
|
||||||
let mut last_arr: Option<Instant> = None;
|
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
let mut size_logged = false;
|
|
||||||
loop {
|
loop {
|
||||||
match common::read_video_frame(&mut recv).await {
|
match common::read_video_frame(&mut recv).await {
|
||||||
Ok((hdr, payload)) => {
|
Ok((hdr, payload)) => {
|
||||||
stats.bytes.fetch_add(payload.len() as u64, Ordering::Relaxed);
|
stats.bytes.fetch_add(payload.len() as u64, Ordering::Relaxed);
|
||||||
stats.frames.fetch_add(1, Ordering::Relaxed);
|
stats.frames.fetch_add(1, Ordering::Relaxed);
|
||||||
// 与上一帧的到达间隔：≤100ms 才算"突发到达"。
|
|
||||||
// 桌面静止时 agent 以 2Hz 空闲重发，帧龄天然 ≥500ms，
|
|
||||||
// 不能按陈旧帧丢弃——否则静止桌面所有帧被跳过 →
|
|
||||||
// 永久黑屏 + KeyFrameReq/IDR 死循环（v0.8 实测教训）
|
|
||||||
let burst = matches!(last_arr, Some(t) if t.elapsed() <= Duration::from_millis(100));
|
|
||||||
last_arr = Some(Instant::now());
|
|
||||||
// 端到端延迟 = 接收时刻 - 时钟偏移 - 采集时刻（EMA 平滑）
|
// 端到端延迟 = 接收时刻 - 时钟偏移 - 采集时刻（EMA 平滑）
|
||||||
let mut est = None;
|
|
||||||
let mut stale = false;
|
|
||||||
if stats.offset_valid.load(Ordering::Relaxed) {
|
if stats.offset_valid.load(Ordering::Relaxed) {
|
||||||
let e =
|
let est =
|
||||||
now_ms() as i64 - stats.offset.load(Ordering::Relaxed)
|
now_ms() as i64 - stats.offset.load(Ordering::Relaxed)
|
||||||
- hdr.ts_ms as i64;
|
- hdr.ts_ms as i64;
|
||||||
if (0..5000).contains(&e) {
|
if (0..5000).contains(&est) {
|
||||||
// 积压判定必须同时满足：突发到达（背靠背）+ 帧龄>250ms。
|
|
||||||
// QUIC 可靠有序，积压须逐帧读出但可不解码——跳过解码
|
|
||||||
// 让队列以网络速度排空
|
|
||||||
stale = burst && e > 250;
|
|
||||||
// 空闲重发帧的帧龄不代表显示延迟，不计入 EMA
|
|
||||||
if burst || e < 250 {
|
|
||||||
let old = stats.e2e.load(Ordering::Relaxed) as i64;
|
let old = stats.e2e.load(Ordering::Relaxed) as i64;
|
||||||
let sm = if stats.e2e_valid.load(Ordering::Relaxed) {
|
let sm = if stats.e2e_valid.load(Ordering::Relaxed) {
|
||||||
(old * 4 + e) / 5
|
(old * 4 + est) / 5
|
||||||
} else {
|
} else {
|
||||||
e
|
est
|
||||||
};
|
};
|
||||||
stats.e2e.store(sm as u64, Ordering::Relaxed);
|
stats.e2e.store(sm as u64, Ordering::Relaxed);
|
||||||
stats.e2e_valid.store(true, Ordering::Relaxed);
|
stats.e2e_valid.store(true, Ordering::Relaxed);
|
||||||
est = Some(sm);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
stats.on_frame_arrival(est);
|
|
||||||
if stale {
|
|
||||||
skip_stale += 1;
|
|
||||||
if skip_stale % 60 == 1 {
|
|
||||||
eprintln!(
|
|
||||||
"[viewer] backlog: stale burst frame skipped (total {skip_stale})"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
// 限频请求关键帧：积压排空后从 IDR 重新衔接画面
|
|
||||||
if last_kfreq.elapsed() >= Duration::from_millis(200) {
|
|
||||||
last_kfreq = Instant::now();
|
|
||||||
if let Ok(mut slot) = latest.lock() {
|
|
||||||
*slot = Some((0, 0, Vec::new()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
match decoder.decode(&payload) {
|
match decoder.decode(&payload) {
|
||||||
Ok(Some(yuv)) => {
|
Ok(Some(yuv)) => {
|
||||||
let (dw, dh) = yuv.dimensions();
|
let (dw, dh) = yuv.dimensions();
|
||||||
let (dw, dh) = (dw as u32, dh as u32);
|
let mut rgb = vec![0u8; dw * dh * 3];
|
||||||
if !size_logged {
|
yuv.write_rgb8(&mut rgb);
|
||||||
size_logged = true;
|
if frame_tx.send(rgb).is_err() {
|
||||||
eprintln!(
|
break;
|
||||||
"[viewer] decoded frame {dw}x{dh} (window {}x{})",
|
|
||||||
hdr.width, hdr.height
|
|
||||||
);
|
|
||||||
}
|
|
||||||
rgb_buf.clear();
|
|
||||||
rgb_buf.resize(dw as usize * dh as usize * 3, 0);
|
|
||||||
yuv.write_rgb8(&mut rgb_buf);
|
|
||||||
// latest-wins：直接覆盖旧帧，无积压
|
|
||||||
if let Ok(mut slot) = latest.lock() {
|
|
||||||
*slot = Some((dw, dh, rgb_buf.clone()));
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(None) => {}
|
Ok(None) => {}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
// 解码异常：请求被控端强制关键帧
|
// 解码异常：请求被控端强制关键帧
|
||||||
stats.dec_fail.fetch_add(1, Ordering::Relaxed);
|
|
||||||
eprintln!("[viewer] decode failed: {e}, request keyframe");
|
eprintln!("[viewer] decode failed: {e}, request keyframe");
|
||||||
if let Ok(mut slot) = latest.lock() {
|
let _ = frame_tx.send(Vec::new()); // 空帧占位唤醒渲染循环
|
||||||
*slot = Some((0, 0, Vec::new()));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -358,11 +220,9 @@ fn session(
|
|||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
loop {
|
loop {
|
||||||
match read_control(&mut irecv).await {
|
match read_control(&mut irecv).await {
|
||||||
Ok(ControlMsg::Pong { t, agent_ts, lost, sent }) => {
|
Ok(ControlMsg::Pong { t, agent_ts }) => {
|
||||||
let r = now_ms();
|
let r = now_ms();
|
||||||
stats.rtt.store(r.saturating_sub(t), Ordering::Relaxed);
|
stats.rtt.store(r.saturating_sub(t), Ordering::Relaxed);
|
||||||
stats.agent_lost.store(lost, Ordering::Relaxed);
|
|
||||||
stats.agent_sent.store(sent, Ordering::Relaxed);
|
|
||||||
// agent_ts ≈ (t+r)/2 时刻的 agent 时钟读数 → 偏移 EMA
|
// agent_ts ≈ (t+r)/2 时刻的 agent 时钟读数 → 偏移 EMA
|
||||||
let est = agent_ts as i64 - ((t + r) / 2) as i64;
|
let est = agent_ts as i64 - ((t + r) / 2) as i64;
|
||||||
let old = stats.offset.load(Ordering::Relaxed);
|
let old = stats.offset.load(Ordering::Relaxed);
|
||||||
@ -375,10 +235,7 @@ fn session(
|
|||||||
stats.offset_valid.store(true, Ordering::Relaxed);
|
stats.offset_valid.store(true, Ordering::Relaxed);
|
||||||
}
|
}
|
||||||
Ok(_) => {}
|
Ok(_) => {}
|
||||||
Err(e) => {
|
Err(_) => break,
|
||||||
eprintln!("[viewer] control stream closed: {e} (RTT/LOSS 将停止更新)");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@ -398,11 +255,9 @@ fn session(
|
|||||||
let push = |tx: &tokio::sync::mpsc::UnboundedSender<Vec<u8>>, m: ControlMsg| {
|
let push = |tx: &tokio::sync::mpsc::UnboundedSender<Vec<u8>>, m: ControlMsg| {
|
||||||
let _ = tx.send(common::encode_control(&m));
|
let _ = tx.send(common::encode_control(&m));
|
||||||
};
|
};
|
||||||
// 入会立即请求 IDR：避免从 GOP 中间接入收到裸 P 帧解码失败导致启动黑屏
|
|
||||||
push(&msg_tx, ControlMsg::KeyFrameReq);
|
|
||||||
|
|
||||||
let mut window = Window::new(
|
let mut window = Window::new(
|
||||||
"mvp-remote v0.8.1-CatchUp (Esc 退出)",
|
"mvp-remote viewer (Esc 退出)",
|
||||||
w,
|
w,
|
||||||
h,
|
h,
|
||||||
WindowOptions {
|
WindowOptions {
|
||||||
@ -423,8 +278,6 @@ fn session(
|
|||||||
let mut hud_bytes = 0u64;
|
let mut hud_bytes = 0u64;
|
||||||
let mut hud_fps = 0u64;
|
let mut hud_fps = 0u64;
|
||||||
let mut hud_mbps = 0f64;
|
let mut hud_mbps = 0f64;
|
||||||
let mut hud_lost = 0u64;
|
|
||||||
let mut hud_sent = 0u64;
|
|
||||||
|
|
||||||
loop {
|
loop {
|
||||||
if !window.is_open() || window.is_key_down(Key::Escape) {
|
if !window.is_open() || window.is_key_down(Key::Escape) {
|
||||||
@ -437,16 +290,19 @@ fn session(
|
|||||||
push(&msg_tx, ControlMsg::Ping(now_ms()));
|
push(&msg_tx, ControlMsg::Ping(now_ms()));
|
||||||
}
|
}
|
||||||
|
|
||||||
// 1) 渲染最新帧（单槽 latest-wins：每迭代只取最新一帧，天然防积压）
|
// 1) 渲染最新帧（只保留最新，丢弃积压）；空帧=解码异常，请求关键帧
|
||||||
if let Ok(mut slot) = latest.lock() {
|
loop {
|
||||||
if let Some((sw, sh, raw)) = slot.take() {
|
match frame_rx.try_recv() {
|
||||||
|
Ok(raw) => {
|
||||||
if raw.is_empty() {
|
if raw.is_empty() {
|
||||||
push(&msg_tx, ControlMsg::KeyFrameReq);
|
push(&msg_tx, ControlMsg::KeyFrameReq);
|
||||||
} else if sw > 0 && sh > 0 {
|
} else if raw.len() == w * h * 3 {
|
||||||
blit(&mut buffer, w, h, &raw, sw as usize, sh as usize);
|
blit(&mut buffer, &raw);
|
||||||
dirty = true;
|
dirty = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
Err(_) => break,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2) 键盘
|
// 2) 键盘
|
||||||
@ -511,9 +367,7 @@ fn session(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 6) HUD：指标每秒刷新一次；每秒至少强制呈现一次（视频冻结/黑屏时 HUD 依然可见）
|
// 6) HUD：每秒刷新一次指标，画在右上角
|
||||||
let mut need_present = dirty;
|
|
||||||
let (hud_loss, hud_jit, hud_p95, hud_dec);
|
|
||||||
if hud_last.elapsed() >= Duration::from_secs(1) {
|
if hud_last.elapsed() >= Duration::from_secs(1) {
|
||||||
let f = stats.frames.load(Ordering::Relaxed);
|
let f = stats.frames.load(Ordering::Relaxed);
|
||||||
let b = stats.bytes.load(Ordering::Relaxed);
|
let b = stats.bytes.load(Ordering::Relaxed);
|
||||||
@ -521,47 +375,12 @@ fn session(
|
|||||||
hud_mbps = (b.saturating_sub(hud_bytes)) as f64 * 8.0 / 1e6;
|
hud_mbps = (b.saturating_sub(hud_bytes)) as f64 * 8.0 / 1e6;
|
||||||
hud_frames = f;
|
hud_frames = f;
|
||||||
hud_bytes = b;
|
hud_bytes = b;
|
||||||
// 丢包率 = Δlost / Δsent（agent 发送方向，随 Pong 同步）
|
|
||||||
let l = stats.agent_lost.load(Ordering::Relaxed);
|
|
||||||
let s = stats.agent_sent.load(Ordering::Relaxed);
|
|
||||||
hud_loss = if s > hud_sent {
|
|
||||||
((l - hud_lost) as f64 / (s - hud_sent) as f64 * 100.0).min(99.9)
|
|
||||||
} else {
|
|
||||||
-1.0
|
|
||||||
};
|
|
||||||
hud_lost = l;
|
|
||||||
hud_sent = s;
|
|
||||||
hud_jit = stats.jitter_x16.load(Ordering::Relaxed) / 16;
|
|
||||||
hud_p95 = stats.gap_p95();
|
|
||||||
hud_dec = stats.dec_fail.load(Ordering::Relaxed);
|
|
||||||
hud_last = Instant::now();
|
hud_last = Instant::now();
|
||||||
need_present = true; // HUD 面板每秒至少重绘一次
|
|
||||||
} else {
|
|
||||||
hud_loss = f64::NAN;
|
|
||||||
hud_jit = 0;
|
|
||||||
hud_p95 = None;
|
|
||||||
hud_dec = 0;
|
|
||||||
}
|
}
|
||||||
if need_present {
|
if dirty {
|
||||||
let rtt = stats.rtt.load(Ordering::Relaxed);
|
let rtt = stats.rtt.load(Ordering::Relaxed);
|
||||||
let e2e = stats.e2e.load(Ordering::Relaxed);
|
let e2e = stats.e2e.load(Ordering::Relaxed);
|
||||||
let e2e_ok = stats.e2e_valid.load(Ordering::Relaxed);
|
let e2e_ok = stats.e2e_valid.load(Ordering::Relaxed);
|
||||||
let loss_str = if hud_loss.is_nan() {
|
|
||||||
"--".to_string()
|
|
||||||
} else if hud_loss < 0.0 {
|
|
||||||
"--".to_string()
|
|
||||||
} else {
|
|
||||||
format!("{hud_loss:.1}%")
|
|
||||||
};
|
|
||||||
let loss_c = if hud_loss.is_nan() || hud_loss < 0.0 {
|
|
||||||
C_GRAY
|
|
||||||
} else {
|
|
||||||
lat_color(hud_loss < 1.0, hud_loss < 5.0)
|
|
||||||
};
|
|
||||||
let jit_c = lat_color(hud_jit < 10, hud_jit < 30);
|
|
||||||
let p95_c = hud_p95
|
|
||||||
.map(|p| lat_color(p < 60, p < 150))
|
|
||||||
.unwrap_or(C_GRAY);
|
|
||||||
draw_hud(
|
draw_hud(
|
||||||
&mut buffer,
|
&mut buffer,
|
||||||
w,
|
w,
|
||||||
@ -573,10 +392,6 @@ fn session(
|
|||||||
&if e2e_ok { format!("{e2e}ms") } else { "--".into() },
|
&if e2e_ok { format!("{e2e}ms") } else { "--".into() },
|
||||||
lat_color(e2e < 100, e2e < 200),
|
lat_color(e2e < 100, e2e < 200),
|
||||||
),
|
),
|
||||||
("JIT", &format!("{hud_jit}ms"), jit_c),
|
|
||||||
("P95", &hud_p95.map(|p| format!("{p}ms")).unwrap_or("--".into()), p95_c),
|
|
||||||
("LOSS", &loss_str, loss_c),
|
|
||||||
("DECFAIL", &format!("{hud_dec}"), if hud_dec == 0 { C_GRAY } else { C_RED }),
|
|
||||||
("RATE", &format!("{:.0}MBP", hud_mbps), C_GRAY),
|
("RATE", &format!("{:.0}MBP", hud_mbps), C_GRAY),
|
||||||
("RES", &format!("{w}x{h}"), C_GRAY),
|
("RES", &format!("{w}x{h}"), C_GRAY),
|
||||||
],
|
],
|
||||||
@ -658,7 +473,6 @@ fn glyph(c: u8) -> [u8; 5] {
|
|||||||
b':' => [0x00, 0x36, 0x36, 0x00, 0x00],
|
b':' => [0x00, 0x36, 0x36, 0x00, 0x00],
|
||||||
b'.' => [0x00, 0x60, 0x60, 0x00, 0x00],
|
b'.' => [0x00, 0x60, 0x60, 0x00, 0x00],
|
||||||
b'-' => [0x08, 0x08, 0x08, 0x08, 0x08],
|
b'-' => [0x08, 0x08, 0x08, 0x08, 0x08],
|
||||||
b'%' => [0x13, 0x18, 0x04, 0x0A, 0x11],
|
|
||||||
b' ' => [0x00; 5],
|
b' ' => [0x00; 5],
|
||||||
_ => [0x00; 5],
|
_ => [0x00; 5],
|
||||||
}
|
}
|
||||||
@ -732,17 +546,9 @@ fn draw_hud(buffer: &mut [u32], w: usize, lines: &[(&str, &str, u32)]) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// RGB888 字节（openh264 解码输出）→ minifb 0RGB u32 缓冲。
|
/// RGB888 字节（openh264 解码输出）→ minifb 0RGB u32 缓冲
|
||||||
/// 支持源尺寸与窗口不同（解码器可能对齐到 16 倍数，如 1080→1088）：
|
fn blit(buffer: &mut [u32], raw: &[u8]) {
|
||||||
/// 逐行拷贝重叠区域，源小于窗口时右侧/下方留黑。
|
for (px, chunk) in buffer.iter_mut().zip(raw.chunks_exact(3)) {
|
||||||
fn blit(buffer: &mut [u32], w: usize, h: usize, raw: &[u8], dw: usize, dh: usize) {
|
|
||||||
let cw = w.min(dw);
|
|
||||||
let ch = h.min(dh);
|
|
||||||
for row in 0..ch {
|
|
||||||
let src = &raw[row * dw * 3..row * dw * 3 + cw * 3];
|
|
||||||
let dst = &mut buffer[row * w..row * w + cw];
|
|
||||||
for (px, chunk) in dst.iter_mut().zip(src.chunks_exact(3)) {
|
|
||||||
*px = ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | chunk[2] as u32;
|
*px = ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | chunk[2] as u32;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|||||||
70
构建运行手册.md
70
构建运行手册.md
@ -1,17 +1,17 @@
|
|||||||
# MVP 远程控制 Demo · 构建运行手册
|
# MVP 远程控制 Demo · 构建运行手册
|
||||||
|
|
||||||
> 项目路径：`D:\code\remote\`
|
> 项目路径：`D:\code\remote\`
|
||||||
> 形态：Rust workspace：`common`（协议）、`agent-core`（平台无关会话层，当前供 linux-agent 使用）、`agent`（Windows 被控端）、`linux-agent`（Linux 被控端）、`viewer`（主控端）
|
> 形态：Rust workspace：`common`（协议）、`agent-core`（平台无关会话层）、`agent`（Windows 被控端）、`linux-agent`（Linux 被控端）、`viewer`（主控端）
|
||||||
|
|
||||||
## 1. 功能范围（量级一 Demo）
|
## 1. 功能范围（量级一 Demo）
|
||||||
|
|
||||||
- ✅ DXGI Desktop Duplication 全屏采集 + NVENC 硬编（无 N 卡时自动回落 openh264 软编）
|
- ✅ DXGI Desktop Duplication 全屏采集（GPU 拷贝，脏区/硬编码留待二期）
|
||||||
- ✅ QUIC 加密传输（自签名证书，传输层已加密）
|
- ✅ LZ4 压缩 + QUIC 加密传输（自签名证书，传输层已加密）
|
||||||
- ✅ 验证码认证（默认 123456）
|
- ✅ 验证码认证（默认 123456）
|
||||||
- ✅ 键鼠远程控制（SendInput 注入，虚拟桌面绝对坐标 + 扫描码）
|
- ✅ 键鼠远程控制（SendInput 注入，虚拟桌面绝对坐标 + 扫描码）
|
||||||
- ✅ 桌面静止时低频重发（保持画面新鲜）
|
- ✅ 桌面静止时低频重发（保持画面新鲜）
|
||||||
- ✅ Linux 被控端（X11 采集 + XTest 注入，见第 8 节）
|
- ✅ Linux 被控端（X11 采集 + XTest 注入，与 Windows 端共用同一套协议与会话层）
|
||||||
- ❌ 暂无：P2P 打洞/中继（当前仅局域网/可直连地址）、音频、文件传输
|
- ❌ 暂无：硬件编码、P2P 打洞/中继（当前仅局域网/可直连地址）、音频、文件传输
|
||||||
|
|
||||||
## 2. 环境要求
|
## 2. 环境要求
|
||||||
|
|
||||||
@ -80,30 +80,6 @@ viewer.exe --addr <被控端IP> --port 47800 --code 888888
|
|||||||
4. 高 DPI/多屏：当前只采集主输出（EnumOutputs(0)）。
|
4. 高 DPI/多屏：当前只采集主输出（EnumOutputs(0)）。
|
||||||
5. 主控端窗口固定为远端分辨率，颜色序 BGRA→0RGB 若异常可在 `blit()` 一键翻转。
|
5. 主控端窗口固定为远端分辨率，颜色序 BGRA→0RGB 若异常可在 `blit()` 一键翻转。
|
||||||
|
|
||||||
### 6.1 待修问题（已定位，尚未修，均在 Windows `agent/`）
|
|
||||||
|
|
||||||
1. **错误验证码时 `AuthFail` 送不达**：`agent/src/main.rs` 写完 `AuthFail` 后立刻
|
|
||||||
`send.finish()` 并返回，进程退出会把 QUIC 缓冲区里尚未发出的数据一起带走，
|
|
||||||
客户端只会看到 `connection lost` 而不是 `AuthFail`。
|
|
||||||
修法：`finish()` 后等对端读到并挂断（`conn.closed()` + 超时）再退出。
|
|
||||||
|
|
||||||
2. **软编路径下静止桌面仍不响应关键帧请求**（`04c02d9` 只修了 NVENC 那一半）：
|
|
||||||
`agent/src/capture.rs` 的 `Ok(false)`（画面未变）分支里，`force_key` 被
|
|
||||||
`keyreq.swap` 消费后，只有 `EncoderKind::Nvenc` 分支拿它去重编 IDR；
|
|
||||||
`EncoderKind::Soft` 分支仍然只是「重发最后一帧码流副本」，标志被直接丢弃。
|
|
||||||
后果：**无 N 卡 / 驱动异常（走软编兜底）的机器上，解码端一旦出错，静止画面下
|
|
||||||
永远拿不到 IDR，表现为黑屏。**
|
|
||||||
修法：软编路径同样保留最近一帧裸像素，`force_key` 时重编一个真关键帧。
|
|
||||||
|
|
||||||
3. **单次 `KeyFrameReq` 可能被静默吞掉**：同一个 `keyreq.swap(false, …)` 写在了
|
|
||||||
`if last_sent.elapsed() > 500ms` 的**外面**，因此请求若落在距上次发送 500ms 之内，
|
|
||||||
标志已被消费但分支不进入，该次请求就丢了，必须等下一次请求才可能生效
|
|
||||||
（主控端目前周期性重发，故表现为延迟而非永久黑屏）。
|
|
||||||
修法：把 `swap` 移进时间闸内，或先在闸外预判、进闸后再消费。
|
|
||||||
|
|
||||||
> 第 1、2 条在 `feature/agent-linux` 分支的 `agent-core` 里已经实现（**Linux 侧生效**），
|
|
||||||
> 但按"不与他线争抢 `agent/`"的约定，**本期没有改动 Windows 代码**。
|
|
||||||
|
|
||||||
## 7. 下一阶段任务队列（Phase 1 剩余）
|
## 7. 下一阶段任务队列（Phase 1 剩余）
|
||||||
|
|
||||||
1. H.264：接 openh264（软编兜底）→ NVENC/QSV（硬编优先）
|
1. H.264：接 openh264（软编兜底）→ NVENC/QSV（硬编优先）
|
||||||
@ -114,25 +90,21 @@ viewer.exe --addr <被控端IP> --port 47800 --code 888888
|
|||||||
|
|
||||||
## 8. Linux 被控端（linux-agent）
|
## 8. Linux 被控端（linux-agent）
|
||||||
|
|
||||||
### 8.1 现状与边界（重要）
|
### 8.1 架构：会话层抽到 agent-core
|
||||||
|
|
||||||
本期**只在新增文件里做，未改动 Windows `agent/`**，原因见下：
|
平台差异只留在采集/注入两个模块里，QUIC 握手、认证、编码、丢帧、输入分发全部共用，
|
||||||
|
**主控端 viewer 不需要任何改动**：
|
||||||
|
|
||||||
| crate / 文件 | 职责 | 当前使用者 |
|
| crate / 文件 | 职责 | 平台 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `agent-core/src/lib.rs` | QUIC/TLS 握手、传输调优、验证码认证、会话/select 循环、latest-wins 单槽、输入分发，以及软件编码视频管线（openh264） | **仅 linux-agent** |
|
| `agent-core/src/lib.rs` | QUIC/TLS 握手、验证码认证、H.264 编码下行、latest-wins 丢帧、输入分发 | 共用 |
|
||||||
|
| `agent/src/capture.rs` | DXGI Desktop Duplication 采集 | Windows |
|
||||||
|
| `agent/src/input.rs` | SendInput 注入 | Windows |
|
||||||
| `linux-agent/src/capture.rs` | X11 `GetImage` 采集 + 整帧变更检测 | Linux |
|
| `linux-agent/src/capture.rs` | X11 `GetImage` 采集 + 整帧变更检测 | Linux |
|
||||||
| `linux-agent/src/input.rs` | XTest 注入 + 扫描码→X11 keycode 映射 | Linux |
|
| `linux-agent/src/input.rs` | XTest 注入 + 扫描码→X11 keycode 映射 | Linux |
|
||||||
| `agent/`（Windows） | 保留原有自持会话循环 + NVENC/软编双路编码，**本期未动** | Windows |
|
|
||||||
|
|
||||||
**为什么 Windows 端没有一并接过来**：`agent/` 的 NVENC 路径是「GPU 内 BGRA→H.264 零转换零拷贝」，
|
两端靠 `ScreenCapturer` / `InputInjector` 两个 trait 注入，编码档位（5Mbps / 30fps / 关键帧周期 300）
|
||||||
编码器必须直接吃 DXGI 的 GPU 纹理；而 `agent-core` 目前的接缝是 `ScreenCapturer`（返回 CPU 上的
|
与线格式完全一致；协议**未做任何改动**。
|
||||||
BGRA 字节），硬接会把零拷贝优势废掉。正确的接缝应是「平台视频管线产出**已编码帧**」
|
|
||||||
（`VideoSource` 语义），待 `agent/` 由 NVENC 那条线稳定后再重切，届时应把 `openh264` 从
|
|
||||||
`agent-core` 移到各平台 crate。
|
|
||||||
|
|
||||||
在此之前 Windows 与 Linux 各持一份会话循环，属于**有意的过渡态**：`common/` 协议改动需要两处同步
|
|
||||||
（例如本期跟着补的 `Pong{lost,sent}`）。
|
|
||||||
|
|
||||||
### 8.2 构建
|
### 8.2 构建
|
||||||
|
|
||||||
@ -143,7 +115,7 @@ export PATH="$USERPROFILE/.cargo/bin:/d/mingw64/bin:$PATH" # Git Bash 写法
|
|||||||
cargo build --release
|
cargo build --release
|
||||||
```
|
```
|
||||||
|
|
||||||
Linux 侧显式指定包（跨平台交叉编译不推荐，用容器最省事）：
|
Linux 侧显式指定包（交叉编译不推荐，用容器最省事）：
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo build --release -p linux-agent
|
cargo build --release -p linux-agent
|
||||||
@ -165,12 +137,8 @@ bash scripts/linux-smoke.sh
|
|||||||
```
|
```
|
||||||
|
|
||||||
脚本自建 Xvfb 虚拟屏后跑端到端测试，覆盖：X11 采集 → openh264 编码 → QUIC 下行解帧、
|
脚本自建 Xvfb 虚拟屏后跑端到端测试，覆盖：X11 采集 → openh264 编码 → QUIC 下行解帧、
|
||||||
验证码认证（含**错误验证码被拒**）、Ping/Pong（含下行丢包统计字段）、静止桌面下的关键帧
|
验证码认证（含**错误验证码被拒**）、Ping/Pong、静止桌面下的关键帧恢复（断言拿到 IDR），
|
||||||
恢复，以及**输入注入实证**——注入鼠标移动后，从另一条 X11 连接读回指针真实坐标。
|
以及**输入注入实证**——注入鼠标移动后，从另一条 X11 连接读回指针真实坐标。
|
||||||
|
|
||||||
> 静止关键帧那条用例做过**变异测试**：把修复退回会失败、恢复后通过，确认断言不是摆设。
|
|
||||||
> 注意扰动量必须小（画 64x64 小方块）——整屏刷白会触发编码器场景切换自动 IDR，
|
|
||||||
> 于是"重发旧帧"和"真重编关键帧"拿到同样的字节，用例就失去区分度。
|
|
||||||
|
|
||||||
### 8.5 键码映射（两端最关键的差异）
|
### 8.5 键码映射（两端最关键的差异）
|
||||||
|
|
||||||
@ -184,7 +152,7 @@ bash scripts/linux-smoke.sh
|
|||||||
### 8.6 Linux 侧已知限制
|
### 8.6 Linux 侧已知限制
|
||||||
|
|
||||||
1. 仅支持 32bpp TrueColor 视觉（绝大多数桌面满足），其他格式启动即明确报错而非静默出错；
|
1. 仅支持 32bpp TrueColor 视觉（绝大多数桌面满足），其他格式启动即明确报错而非静默出错；
|
||||||
2. 采集走核心协议 `GetImage`，每帧经 X socket 全量传输，静止桌面也会持续轮询抓帧；
|
2. 采集走核心协议 `GetImage`，每帧经 X socket 全量传输；**MIT-SHM 共享内存 + XDamage 脏区**
|
||||||
**MIT-SHM 共享内存 + XDamage 脏区**是下一步优化（锚点已写在 `linux-agent/src/capture.rs` 顶部注释）；
|
是下一步优化（锚点已写在 `linux-agent/src/capture.rs` 顶部注释）；
|
||||||
3. 仅覆盖 X11 / XWayland；纯 Wayland（无 XWayland）需另走 PipeWire 门户；
|
3. 仅覆盖 X11 / XWayland；纯 Wayland（无 XWayland）需另走 PipeWire 门户；
|
||||||
4. 键位表按 Xorg evdev 标准 keycode 编写，自定义键盘布局可能需要调整。
|
4. 键位表按 Xorg evdev 标准 keycode 编写，自定义键盘布局可能需要调整。
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user