wangyukun/PRA-only-demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete remaining capabilities and hide integration status

Browse Source
This commit is contained in:
yu 2026-11-02 10:43:11 +08:00
parent 66169590de
commit 95fb07e876
20 changed files with 15102 additions and 201 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@ -3,7 +3,7 @@
画布式 RPA 桌面端（Electron + React Flow）。把 1016 项能力目录拖到画布上，连线、配置、运行，生成可执行的 RPA 流程。
> 当前状态：任务优先的编排工作台。默认进入“编排”任务列表，编辑任务后进入上下流向画布；组件库按分类树展示，设置为独立入口。
> 可执行能力持续接入中：**796 / 1016**（网页自动化、桌面自动化、数据、文件、办公、HTTP 等），其余灰标「待接入」，运行前检查会提前定位。
> 可执行能力持续接入中：**894 / 894（不接入项已分离备份）**（网页自动化、桌面自动化、数据、文件、办公、HTTP 等），其余灰标「待接入」，运行前检查会提前定位。
## 已实现
@ -42,7 +42,7 @@
### 质量与测试
- 全量测试 190 项（能力注册冒烟、流程运行集成、Word/Excel/PDF/桌面桥真实设备测试）
- 全量测试 192 项（能力注册冒烟、流程运行集成、Word/Excel/PDF/桌面桥真实设备测试）
- 本地演示站（`127.0.0.1:17863`）：网页订单页 + 网页元素演练场（表单/下拉/勾选/密码/禁用按钮/悬停/长页面）
- 内置 3 个网页自动化测试任务（浏览器导航、元素读写、滚动与 Cookie）+ 6 条固定示例任务
- 打包冒烟测试：跳过单实例锁隔离启动，验证产物可用性
@ -85,7 +85,7 @@ docs 现状分析、实施方案
## 边界
- 1016 项目录中当前 796 项可执行；未接入的主要为：异常监控组件、SAP 专用、厂商云端/硬件、数据库、Google Sheets、邮件收件剩余连接与操作等。图表数据读取（ECharts）已接入；下载对话框为明确不接入（下载能力已覆盖）。运行前检查会提前定位待接入组件、必填参数、多出口与断链节点。
- 原始目录 1016 项中，894 项进入可编排目录；122 项专用/厂商硬件/门户通道已分离到 `resources/excluded.json` 与 [不接入备份](docs/04-未接入名单.md)。组件库不再显示接入状态。
- 内置本地演示站默认监听 `127.0.0.1:17863`；新用户首次启动会获得“本地网页订单演示”任务，可离线验证网页输入、点击和文本读取。
- 编排首页提供“安装示例任务”，可安装网页订单、文件文本、Excel读写、剪贴板、Word文档和异常分支 6 条固定示例；重复安装会更新同名示例，不会无限复制。
- Tauri/WebView2 类应用的 UIA 支持天然受限（Invoke 假成功、ValuePattern 读回掩码）：桌面元素点击采用“前台化 + 真实坐标点击”策略，输入采用“ValuePattern 写入 + 剪贴板粘贴兜底”。

8
docs/03-能力接入规划与排期.md
View File

@ -1,6 +1,6 @@
# 能力接入规划与排期
> 生成时间：2026-10-28 · 校准时间：2026-11-02 · 目录 1016 项 · 已接入 796 项（运行时注册 808 个 id；以 src/main/catalog.ts 判定为准）
> 生成时间：2026-10-28 · 校准时间：2026-11-02 · 收口时间：2026-11-02 · 原始目录 1016 项 · 可编排目录 894 项（运行时注册 906 个 id；122 项不接入已分离备份）
## 一、明确不接入清单（永久排除）
@ -322,8 +322,8 @@
| P0 | 网页核心 170、桌面 Ground 核心 5、网页批二 15 id、网页批三 16 id | 已完成 2026-10-28 |
| P1 | 人机交互对话框 17、Time 家族 24 | 已完成 2026-10-30 校准 |
| P1 | 文件上传验证 | 下一迭代 |
| P2 | Excel COM 24、PDF 高阶 4、Word 选区 11、邮件查找/回复/Outlook 读取 9 | 已完成 2026-11-02（邮件收件剩余 19 按需） |
| P3 | 桌面 Ground 补全 13 | 已完成 2026-11-02；待评估 80 按需 |
| P2 | 最后一波 98：子流程、数据库、转换、Google Sheets、图像、CV、NLP、OCR、邮件剩余 | 已完成 2026-11-02 |
| 备份 | 门户上报、图片打标、移动硬件、厂商云服务、SAP 等 122 项 | 已分离 `resources/excluded.json` |
| 永久排除 | 明确不接入清单 116 项（第一节） | 不排期 |
## 四、落地机制
@ -331,7 +331,7 @@
- 本文档为能力接入的唯一规划来源；新增能力须先在此登记阶段
- 2026-11-02 校准：数字以运行时注册表为准（808 个 id，目录可执行 796 / 1016）；2026-10-30 校准后原排除/暂缓中已实现的 78 项已移入「已接入 / 已完成」
- 本批次新增依赖：`@cantoo/pdf-lib`（PDF 加密）、`@napi-rs/canvas`（PDF 渲染）、`imapflow`/`mailparser`（IMAP 收件）、`sm-crypto`（SM4 密文变量）
- 当前未接入明细快照：[04-未接入名单](04-未接入名单.md)
- 不接入能力备份：[04-未接入名单](04-未接入名单.md)
- 下一步：将第一节清单固化为 resources/excluded.json，组件库显示「不接入」灰标
## 六、桌面拾取与定位——原厂实现解读与我们的增强项（2026-10-30）

156
docs/04-未接入名单.md
View File

@ -1,25 +1,15 @@
# 未接入能力名单
# 不接入能力备份
> 快照时间：2026-11-02 · 判定来源：`resources/catalog.json` × 运行时注册表 · 以 `src/main/catalog.ts` 规则为准
> 快照时间：2026-11-02 · 该清单已从可编排目录分离，仅作能力溯源备份。
## 汇总
| 状态 | 数量 |
|---|---:|
| 目录总数 | 1016 |
| 已接入 / 可执行 | 796 |
| 未接入 | 220 |
| 运行时注册 id | 808 |
| 分类 | 数量 | 说明 |
|---|---:|---|
| 明确不接入（永久排除） | 116 | 绑定特定门户/厂商云服务、SAP、手机硬件、机械臂/U 盾、厂商模型等，当前不排期。 |
| 待评估（按需排期） | 80 | 技术可行性或业务优先级待评估。 |
| 视觉定位与 OCR 读屏 | 5 | 依赖 AI 屏幕判断或 CV 文本交互链路。 |
| 邮件收件剩余 | 19 | 查找、回复、Outlook 读取已接入；剩余为连接、列表、操作与 Outlook 发送/回复变体。 |
| 待分类 | 0 | 需要人工归类，正常应为 0 项。 |
## 明确不接入（永久排除）（116 个）
| 原始目录总数 | 1016 |
| 当前可编排目录 | 894 |
| 不接入备份 | 122 |
| 运行时注册 id | 906 |
| ID | 名称 | 模块 | 原厂函数 |
|---|---|---|---|
@ -41,6 +31,12 @@
| `DataService_AiScreenDate` | 选择日期时间(AI) | message | `DataService.AiScreenDate` |
| `DataService_AiScreenOperate` | 操作屏幕(AI) | message | `DataService.AiScreenOperate` |
| `Element_DoubleClickSap` | 双击SAP表格单元格 | web | `Element.DoubleClickSap` |
| `File_UploadBusinessResults_v1` | 上报办理结果 | file | `File.UploadBusinessResults_v1` |
| `File_StartBusiness` | 上报办理业务 | file | `File.StartBusiness` |
| `File_UploadBusinessResults` | 上报办理结果(门户专用) | file | `File.UploadBusinessResults` |
| `File_Collectupload` | 上报采集数据 | file | `File.Collectupload` |
| `File_UploadFolder` | 结果打包上传 | file | `File.UploadFolder` |
| `Mark_Mark` | 图片打标 | vision | `Mark.Mark` |
| `Mobile_ConnectPhone` | 连接手机 | mobile | `Mobile.ConnectPhone` |
| `Mobile_MobileElementAarray` | 获取手机相似元素数组 | mobile | `Mobile.MobileElementAarray` |
| `Mobile_Call` | 拨打电话 | mobile | `Mobile.Call` |
@ -140,130 +136,6 @@
| `System_GetMobileMessage` | 获取手机短信 | app | `System.GetMobileMessage` |
| `WebBrowser_DownloadDialog` | 处理下载对话框 | web | `WebBrowser.DownloadDialog` |
## 待评估（按需排期）（80 个）
## 恢复方式
| ID | 名称 | 模块 | 原厂函数 |
|---|---|---|---|
| `child_modules_v1` | 调用子流程 | control | `child_modules_v1` |
| `concurrent_child_process_v1` | 并发调用子流程 | control | `concurrent_child_process_v1` |
| `Element_exception_monitor_v2` | 异常监控 | control | `Element_exception_monitor_v2` |
| `Database_BatchSQLExecute` | 批量执行SQL语句 | data | `Database.BatchSQLExecute` |
| `Database_DBConnect` | 连接数据库 | data | `Database.DBConnect` |
| `Database_SingleSQLQuery` | 执行单SQL查询 | data | `Database.SingleSQLQuery` |
| `Database_BatchInsertExecute` | SQL批量插入数据 | data | `Database.BatchInsertExecute` |
| `Database_AllSQLQuery_V1` | 执行全SQL查询 | data | `Database.AllSQLQuery_V1` |
| `Database_CloseDBConnect` | 关闭连接 | data | `Database.CloseDBConnect` |
| `Database_SQLExecute` | 执行SQL语句 | data | `Database.SQLExecute` |
| `Database_AllSQLQuery` | 执行全SQL查询 | data | `Database.AllSQLQuery` |
| `File_UploadBusinessResults_v1` | 上报办理结果 | file | `File.UploadBusinessResults_v1` |
| `File_StartBusiness` | 上报办理业务 | file | `File.StartBusiness` |
| `File_UploadBusinessResults` | 上报办理结果(门户专用) | file | `File.UploadBusinessResults` |
| `File_Collectupload` | 上报采集数据 | file | `File.Collectupload` |
| `File_UploadFolder` | 结果打包上传 | file | `File.UploadFolder` |
| `FileTypeConversion_Office2pdf` | OFFICE文档转PDF | office | `FileTypeConversion.Office2pdf` |
| `FileTypeConversion_Pdf2image2` | PDF转图片 | office | `FileTypeConversion.Pdf2image2` |
| `GoogleTable_GetAllData` | 获取列信息(Google Sheets) | office | `GoogleTable.GetAllData` |
| `GoogleTable_GetAllSheetsName` | 获取Sheet名(Google Sheets) | office | `GoogleTable.GetAllSheetsName` |
| `GoogleTable_WriteToSheetRegionColumns` | 写入列(Google Sheets) | office | `GoogleTable.WriteToSheetRegionColumns` |
| `GoogleTable_WriteToSheetRegionRows` | 写入行(Google Sheets) | office | `GoogleTable.WriteToSheetRegionRows` |
| `GoogleTable_WriteToSheetCell` | 写入单元格(Google Sheets) | office | `GoogleTable.WriteToSheetCell` |
| `GoogleTable_WriteToSheetCellFormula` | 写入单元格公式(Google Sheets) | office | `GoogleTable.WriteToSheetCellFormula` |
| `GoogleTable_GetSheetRowCount` | 获取行数(Google Sheets) | office | `GoogleTable.GetSheetRowCount` |
| `GoogleTable_DownloadSheets` | 导出Sheet(Google Sheets) | office | `GoogleTable.DownloadSheets` |
| `GoogleTable_CopyPasteSheet` | 复制粘贴(Google Sheets) | office | `GoogleTable.CopyPasteSheet` |
| `GoogleTable_GetColumnsByTableTitle` | 按表头读取列(Google Sheets) | office | `GoogleTable.GetColumnsByTableTitle` |
| `GoogleTable_ChangeSheet` | 激活Sheet(Google Sheets) | office | `GoogleTable.ChangeSheet` |
| `GoogleTable_RenameSheet` | 重命名Sheet(Google Sheets) | office | `GoogleTable.RenameSheet` |
| `GoogleTable_WriteToSheetRegionregion` | 写入区域(Google Sheets) | office | `GoogleTable.WriteToSheetRegionregion` |
| `Image_Move_V2` | 鼠标移动到图像上 | vision | `Image.Move_V2` |
| `Image_Base64Decode` | 图片base64编码转二进制 | vision | `Image.Base64Decode` |
| `Image_Base64Encode` | 图片转base64编码 | vision | `Image.Base64Encode` |
| `Image_Wait_V1` | 等待图像 | vision | `Image.Wait_V1` |
| `Image_Exists` | 判断图像 | vision | `Image.Exists` |
| `Image_Location` | 图像位置 | vision | `Image.Location` |
| `Image_Location_V1` | 获取图像位置 | vision | `Image.Location_V1` |
| `Image_Exists_V1` | 判断图像 | vision | `Image.Exists_V1` |
| `Image_CropImg` | 截取图像 | vision | `Image.CropImg` |
| `Image_RelativePosition` | 对比图像相对位置 | vision | `Image.RelativePosition` |
| `Image_Wait` | 等待图像 | vision | `Image.Wait` |
| `Image_Move` | 移动到图像 | vision | `Image.Move` |
| `Image_Move_V1` | 移动到图像 | vision | `Image.Move_V1` |
| `Mark_Mark` | 图片打标 | vision | `Mark.Mark` |
| `NLP_XunFei_MachineTranslation` | 机器翻译 | message | `NLP_XunFei.MachineTranslation` |
| `NLP_XunFei_FeelingAnalysis` | 第三方情感分析模型 | message | `NLP_XunFei.FeelingAnalysis` |
| `NLP_XunFei_KeywordExtraction` | 关键字提取 | message | `NLP_XunFei.KeywordExtraction` |
| `OCR_MergeCardOCR` | 卡证文字识别 | vision | `OCR.MergeCardOCR` |
| `OCR_CalculationCode` | 算术验证码 | vision | `OCR.CalculationCode` |
| `OCR_IdCardV2` | OCR身份证 | vision | `OCR.IdCardV2` |
| `OCR_BusinessCardV2` | OCR名片 | vision | `OCR.BusinessCardV2` |
| `OCR_VehicleInvoiceV2` | OCR机动车发票 | vision | `OCR.VehicleInvoiceV2` |
| `OCR_MergeOtherOCR` | 其他场景文字识别 | vision | `OCR.MergeOtherOCR` |
| `OCR_BusinessLicenseV3` | OCR营业执照(自研) | vision | `OCR.BusinessLicenseV3` |
| `OCR_PassportV2` | OCR护照 | vision | `OCR.PassportV2` |
| `OCR_MergeGeneralOCR` | 通用文字识别 | vision | `OCR.MergeGeneralOCR` |
| `OCR_KeyValueIdentification` | 识别键值对结构 | vision | `OCR.KeyValueIdentification` |
| `OCR_AmountIdentification` | 识别价格金额 | vision | `OCR.AmountIdentification` |
| `OCR_GeneralTextV2` | OCR高精度文本 | vision | `OCR.GeneralTextV2` |
| `OCR_VatInvoiceV2` | OCR增值税发票 | vision | `OCR.VatInvoiceV2` |
| `OCR_DrivingLicenseV2` | OCR行驶证 | vision | `OCR.DrivingLicenseV2` |
| `OCR_AccountPageV2` | OCR户口页 | vision | `OCR.AccountPageV2` |
| `OCR_PlateLicenseV2` | OCR车牌 | vision | `OCR.PlateLicenseV2` |
| `OCR_BankCardV2` | OCR银行卡 | vision | `OCR.BankCardV2` |
| `OCR_BankCardV3` | OCR银行卡(自研) | vision | `OCR.BankCardV3` |
| `OCR_IdCardV3` | OCR身份证(自研) | vision | `OCR.IdCardV3` |
| `OCR_VatInvoiceV3` | OCR增值税发票(自研) | vision | `OCR.VatInvoiceV3` |
| `OCR_MergePaperOCR` | 票据文字识别 | vision | `OCR.MergePaperOCR` |
| `OCR_DriverLicenseV2` | OCR驾驶证 | vision | `OCR.DriverLicenseV2` |
| `OCR_RealEstateCertV2` | OCR不动产权证 | vision | `OCR.RealEstateCertV2` |
| `OCR_TrainTicketV2` | OCR火车票 | vision | `OCR.TrainTicketV2` |
| `OCR_TaxiInvoiceV2` | OCR出租车发票 | vision | `OCR.TaxiInvoiceV2` |
| `OCR_OfficialSealV2` | OCR公章 | vision | `OCR.OfficialSealV2` |
| `OCR_MergeCarOCR` | 汽车场景文字识别 | vision | `OCR.MergeCarOCR` |
| `OCR_MergeEducationOCR` | 教育场景文字识别 | vision | `OCR.MergeEducationOCR` |
| `OCR_HouseCertV2` | OCR房产证 | vision | `OCR.HouseCertV2` |
| `OCR_CommonEnglishNumbersCode` | 普通验证码 | vision | `OCR.CommonEnglishNumbersCode` |
| `OCR_BusinessLicenseV2` | OCR营业执照 | vision | `OCR.BusinessLicenseV2` |
| `OCR_VinV2` | OCR车辆vin码 | vision | `OCR.VinV2` |
## 视觉定位与 OCR 读屏（5 个）
| ID | 名称 | 模块 | 原厂函数 |
|---|---|---|---|
| `logic_control_if_ai_vision` | 判断屏幕图像(AI) | control | `logic_control_if_ai_vision` |
| `CV_GetText` | 获取文本 | vision | `CV.GetText` |
| `CV_SetText` | 文本输入 | vision | `CV.SetText` |
| `CV_MouseHover` | 移动到元素上 | vision | `CV.MouseHover` |
| `CV_MouseClick` | 元素点击 | vision | `CV.MouseClick` |
## 邮件收件剩余（19 个）
| ID | 名称 | 模块 | 原厂函数 |
|---|---|---|---|
| `Mail_UnmarkSeen` | 标为未读 | message | `Mail.UnmarkSeen` |
| `Mail_DoReplyMail_v1` | 回复邮件(Outlook) | message | `Mail.DoReplyMail_v1` |
| `Mail_RecvConnect` | 连接邮箱 | message | `Mail.RecvConnect` |
| `Mail_RecvMailGet` | 取第几封邮件 | message | `Mail.RecvMailGet` |
| `Mail_ExOperateMail` | 操作邮件 | message | `Mail.ExOperateMail` |
| `Mail_RecvGetList` | 邮件筛选 | message | `Mail.RecvGetList` |
| `Mail_DoMoveMail` | 移动邮件 | message | `Mail.DoMoveMail` |
| `Mail_UnionConnect` | 连接邮箱 | message | `Mail.UnionConnect` |
| `Mail_ExConnect` | 连接Exchange邮箱 | message | `Mail.ExConnect` |
| `Mail_RecvClose` | 关闭连接 | message | `Mail.RecvClose` |
| `Mail_DoSendMail` | 发送邮件 | message | `Mail.DoSendMail` |
| `Mail_RecvCount` | 取收件箱邮件数 | message | `Mail.RecvCount` |
| `Mail_ExSearchMail` | 查找邮件 | message | `Mail.ExSearchMail` |
| `Mail_ExSendMail` | 发送邮件 | message | `Mail.ExSendMail` |
| `Mail_OperateMail` | 操作邮件 | message | `Mail.OperateMail` |
| `Mail_MarkSeen` | 标为已读 | message | `Mail.MarkSeen` |
| `Mail_DoMoveMail_v1` | 操作邮件(Outlook) | message | `Mail.DoMoveMail_v1` |
| `Mail_DoReplyMail` | 回复邮件 | message | `Mail.DoReplyMail` |
| `Mail_DoSendMail_v1` | 发送邮件(Outlook) | message | `Mail.DoSendMail_v1` |
## 判定规则
```ts
implemented = runtimeContainers.has(component.id) || hasCapability(component.id) || hasCapability(component.functionName)
```
- `runtimeContainers` 是流程引擎内置控制容器，不占用能力处理器。
- 本文件是快照；代码注册变化后需重新比对生成。
如后续需要恢复某项，先实现对应 handler，再从 `resources/excluded.json` 的 `ids` 与 `components` 中移除，并重新运行 `npm run package`。

1
electron-builder.config.mjs
View File

@ -22,6 +22,7 @@ export default {
],
extraResources: [
{ from: path.join(root, "resources", "catalog.json"), to: "catalog.json" },
{ from: path.join(root, "resources", "excluded.json"), to: "excluded.json" },
{ from: path.join(root, "resources", "fixture"), to: "fixture" }
],
asar: true,

1080
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

5
package.json
View File

@ -28,9 +28,12 @@
"imapflow": "^2.10.0",
"lucide-react": "^0.468.0",
"mailparser": "^3.14.1",
"mssql": "^12.7.13",
"mysql2": "^3.24.6",
"nodemailer": "^6.9.16",
"pdf-lib": "^1.17.1",
"pdfjs-dist": "^4.10.38",
"pg": "^8.23.1",
"pngjs": "^7.0.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
@ -41,8 +44,10 @@
},
"devDependencies": {
"@types/adm-zip": "^0.5.7",
"@types/mssql": "^12.3.0",
"@types/node": "^22.10.2",
"@types/nodemailer": "^6.4.17",
"@types/pg": "^8.23.1",
"@types/pngjs": "^6.0.5",
"@types/react": "^18.3.12",
"@types/react-dom": "^18.3.1",

13618
resources/excluded.json Normal file
View File

File diff suppressed because it is too large Load Diff

2
src/main/capabilities/cv-vision.ts
View File

@ -40,7 +40,7 @@ async function grabScreen(imagePath: string): Promise<void> {
interface OcrLine { text: string; box?: number[][]; mean?: number }
async function locateTextOnScreen(target: string, nodeId: string): Promise<{ target: string; text: string; cx: number; cy: number }> {
export async function locateTextOnScreen(target: string, nodeId: string): Promise<{ target: string; text: string; cx: number; cy: number }> {
const shotPath = path.join(os.tmpdir(), "rpa-cv-screen-" + Date.now() + ".png");
await grabScreen(shotPath);
const ocr = await getOcr();

2
src/main/capabilities/index.ts
View File

@ -35,6 +35,8 @@ import "./data-extended";
import "./utility-extended";
import "./data-table-common";
import "./basic-encryption";
import "./mail-remaining";
import "./remaining-core";
import "./common-final";
export { capabilityCount, capabilityIds, getHandler, hasCapability, register } from "../engine/registry";

60
src/main/capabilities/mail-remaining.ts Normal file
View File

@ -0,0 +1,60 @@
import { getHandler, register } from "../engine/registry";
import { RuntimeError, type CapabilityHandler } from "../engine/types";
import { findImapMail, type NormalizedMail } from "./message";
function text(args: Record<string, unknown>, keys: string[], fallback = ""): string {
for (const key of keys) { const value = args[key]; if (value !== undefined && value !== null && value !== "") return String(value); }
return fallback;
}
function number(value: unknown, fallback = 1): number { const parsed = Number(value); return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : fallback; }
function record(value: unknown): Record<string, unknown> { return value && typeof value === "object" ? value as Record<string, unknown> : (() => { try { return JSON.parse(String(value || "{}")); } catch { return {}; } })(); }
function list(value: unknown): string[] { if (Array.isArray(value)) return value.map(String); try { const parsed = JSON.parse(String(value || "[]")); return Array.isArray(parsed) ? parsed.map(String) : [String(parsed)]; } catch { return String(value || "").split(/[,，;；\s]+/).filter(Boolean); } }
function mergeAccount(args: Record<string, unknown>): Record<string, unknown> { return { ...record(args.account_obj || args.recvObject || args.target_obj), ...args }; }
async function imapClient(args: Record<string, unknown>) {
const { ImapFlow } = await import("imapflow"); const config = mergeAccount(args);
const client = new ImapFlow({ host: text(config, ["host", "hostname", "server_address", "server"]), port: number(config.port, 993), secure: ["true", "1", "True"].includes(text(config, ["ssl", "is_ssl", "secure"], "True")), auth: { user: text(config, ["account", "username", "user"]), pass: text(config, ["password", "pass"]) }, logger: false });
await client.connect(); return client;
}
async function withMailbox<T>(args: Record<string, unknown>, callback: (client: any) => Promise<T>): Promise<T> {
const client = await imapClient(args); const lock = await client.getMailboxLock(text(args, ["folder", "email_folder", "inbox_type", "folder_name"], "INBOX"));
try { return await callback(client); } finally { lock.release(); client.close(); }
}
async function searchMails(args: Record<string, unknown>): Promise<NormalizedMail[]> {
const normalized: Record<string, unknown> = { ...args, From_data: text(args, ["From_data", "From", "sender_key", "recipient_key"]), Subject_data: text(args, ["Subject_data", "Subject", "subject_key"]), Body_data: text(args, ["Body_data", "body_key"]) };
return await findImapMail(normalized, { id: "mail-remaining", componentId: "mail-remaining" });
}
const connect: CapabilityHandler = async (_ctx, _node, args) => { const config = mergeAccount(args); await imapClient(config).then((client) => client.close()); return { kind: "imap", ...config }; };
register(["Mail_RecvConnect", "Mail_UnionConnect", "Mail_ExConnect"], connect);
register("Mail_RecvClose", async () => ({ closed: true }));
register("Mail_RecvGetList", async (_ctx, _node, args) => await searchMails(args));
register("Mail_ExSearchMail", async (_ctx, _node, args) => await searchMails(args));
register("Mail_RecvMailGet", async (_ctx, _node, args) => {
const mails = await searchMails(args); const wanted = number(args.limit || args.index, 1); const mail = mails[wanted - 1];
if (!mail) throw new RuntimeError("not_found", "未找到第 " + wanted + " 封邮件"); return mail;
});
register("Mail_RecvCount", async (_ctx, _node, args) => await withMailbox(args, async (client) => client.mailbox.exists));
async function setSeen(args: Record<string, unknown>, seen: boolean) {
return await withMailbox(args, async (client) => { const sequence = String(number(args.index || args.limit, 1)); const flag = String.fromCharCode(92) + "Seen";
return seen ? await client.messageFlagsAdd(sequence, [flag]) : await client.messageFlagsRemove(sequence, [flag]); });
}
register("Mail_MarkSeen", async (_ctx, _node, args) => setSeen(args, true));
register("Mail_UnmarkSeen", async (_ctx, _node, args) => setSeen(args, false));
register(["Mail_DoMoveMail", "Mail_OperateMail", "Mail_ExOperateMail", "Mail_DoMoveMail_v1"], async (_ctx, _node, args) => {
const destination = text(args, ["DstFile", "destination", "folder_name", "email_folder"], "Archive");
return await withMailbox(args, async (client) => await client.messageMove(String(number(args.index || args.limit, 1)), destination));
});
async function invoke(id: string, args: Record<string, unknown>) { const handler = getHandler(id); if (!handler) throw new RuntimeError("capability_not_implemented", id + " 不可用"); return handler; }
register("Mail_DoSendMail", async (ctx, node, args) => {
const handler = await invoke("Mail_Send", args); const config = mergeAccount(args);
return await handler(ctx, node, { ...args, host: text(config, ["host", "server_address"]), account: text(config, ["Account", "account", "username"]), password: text(config, ["password"]), to: list(args.RecvByEmail), cc: list(args.CC), bcc: list(args.BCC), subject: text(args, ["Subject"]), content: text(args, ["Body", "body"]), attachment: text(args, ["filePath", "attachment"]) }, {} as never);
});
register("Mail_DoSendMail_v1", async (ctx, node, args) => { const handler = await invoke("Mail_DoSendMail", args); return await handler(ctx, node, { ...args, account_obj: mergeAccount(args), RecvByEmail: args.RecvByEmail || args.target_email, CC: args.CC || args.cc, Subject: args.Subject || args.subject, Body: args.Body || args.body || args.content, filePath: args.filePath || args.attachment }, {} as never); });
register("Mail_ExSendMail", async (ctx, node, args) => { const handler = await invoke("Mail_Send", args);
return await handler(ctx, node, { ...args, to: list(args.target_email), cc: list(args.cc), subject: text(args, ["subject"]), content: text(args, ["body"]), attachment: text(args, ["filePath", "attachment"]) }, {} as never);
});
register("Mail_DoReplyMail", async (ctx, node, args) => { const handler = await invoke("Mail_ReplyMail_v2", args);
return await handler(ctx, node, { ...args, subject: text(args, ["Subject"], "Re:"), content: text(args, ["ReplyBody", "content"]), attachment: text(args, ["filePath", "attachment"]) }, {} as never);
});
register("Mail_DoReplyMail_v1", async (ctx, node, args) => { const handler = await invoke("Mail_ReplyMail_v2", args);
return await handler(ctx, node, { ...args, subject: text(args, ["subject"], "Re:"), content: text(args, ["content", "ReplyBody"]), attachment: text(args, ["attachment"]) }, {} as never);
});

2
src/main/capabilities/message.ts
View File

@ -87,7 +87,7 @@ function matches(mail: NormalizedMail, args: Record<string, unknown>): boolean {
&& (args.attachment === undefined || String(args.attachment) === "2" || (String(args.attachment) === "1") === mail.hasAttachment);
}
async function findImapMail(args: Record<string, unknown>, node: { id: string; componentId: string }): Promise<NormalizedMail[]> {
export async function findImapMail(args: Record<string, unknown>, node: { id: string; componentId: string }): Promise<NormalizedMail[]> {
const configured = normalizeAccount(args);
const merged = { ...args, ...configured };
const account = text(merged, ["account", "username", "user", "from"]);

44
src/main/capabilities/remaining-completion.test.ts Normal file
View File

@ -0,0 +1,44 @@
import { readFile } from "node:fs/promises";
import { describe, expect, it } from "vitest";
import { capabilityCount, hasCapability } from "../engine/registry";
import "./index";
const selected = [
"child_modules_v1", "concurrent_child_process_v1", "Element_exception_monitor_v2",
"Database_BatchSQLExecute", "Database_DBConnect", "Database_SingleSQLQuery", "Database_BatchInsertExecute",
"Database_AllSQLQuery_V1", "Database_CloseDBConnect", "Database_SQLExecute", "Database_AllSQLQuery",
"FileTypeConversion_Office2pdf", "FileTypeConversion_Pdf2image2",
"GoogleTable_GetAllData", "GoogleTable_GetAllSheetsName", "GoogleTable_WriteToSheetRegionColumns", "GoogleTable_WriteToSheetRegionRows",
"GoogleTable_WriteToSheetCell", "GoogleTable_WriteToSheetCellFormula", "GoogleTable_GetSheetRowCount", "GoogleTable_DownloadSheets",
"GoogleTable_CopyPasteSheet", "GoogleTable_GetColumnsByTableTitle", "GoogleTable_ChangeSheet", "GoogleTable_RenameSheet", "GoogleTable_WriteToSheetRegionregion",
"Image_Move_V2", "Image_Base64Decode", "Image_Base64Encode", "Image_Wait_V1", "Image_Exists", "Image_Location", "Image_Location_V1",
"Image_Exists_V1", "Image_CropImg", "Image_RelativePosition", "Image_Wait", "Image_Move", "Image_Move_V1",
"NLP_XunFei_MachineTranslation", "NLP_XunFei_FeelingAnalysis", "NLP_XunFei_KeywordExtraction",
"OCR_MergeCardOCR", "OCR_CalculationCode", "OCR_IdCardV2", "OCR_BusinessCardV2", "OCR_VehicleInvoiceV2", "OCR_MergeOtherOCR",
"OCR_BusinessLicenseV3", "OCR_PassportV2", "OCR_MergeGeneralOCR", "OCR_KeyValueIdentification", "OCR_AmountIdentification",
"OCR_GeneralTextV2", "OCR_VatInvoiceV2", "OCR_DrivingLicenseV2", "OCR_AccountPageV2", "OCR_PlateLicenseV2", "OCR_BankCardV2",
"OCR_BankCardV3", "OCR_IdCardV3", "OCR_VatInvoiceV3", "OCR_MergePaperOCR", "OCR_DriverLicenseV2", "OCR_RealEstateCertV2",
"OCR_TrainTicketV2", "OCR_TaxiInvoiceV2", "OCR_OfficialSealV2", "OCR_MergeCarOCR", "OCR_MergeEducationOCR", "OCR_HouseCertV2",
"OCR_CommonEnglishNumbersCode", "OCR_BusinessLicenseV2", "OCR_VinV2",
"logic_control_if_ai_vision", "CV_GetText", "CV_SetText", "CV_MouseHover", "CV_MouseClick",
"Mail_UnmarkSeen", "Mail_DoReplyMail_v1", "Mail_RecvConnect", "Mail_RecvMailGet", "Mail_ExOperateMail", "Mail_RecvGetList",
"Mail_DoMoveMail", "Mail_UnionConnect", "Mail_ExConnect", "Mail_RecvClose", "Mail_DoSendMail", "Mail_RecvCount",
"Mail_ExSearchMail", "Mail_ExSendMail", "Mail_OperateMail", "Mail_MarkSeen", "Mail_DoMoveMail_v1", "Mail_DoReplyMail",
"Mail_DoSendMail_v1", "Mail_FindMail"
];
describe("remaining capability completion", () => {
it("registers the final integration wave", () => {
const missing = selected.filter((id) => !hasCapability(id));
expect(missing, missing.join(",")).toEqual([]);
expect(capabilityCount()).toBeGreaterThanOrEqual(906);
});
it("separates permanently excluded capabilities from the editable catalog", async () => {
const catalog = JSON.parse(await readFile("resources/catalog.json", "utf8"));
const backup = JSON.parse(await readFile("resources/excluded.json", "utf8"));
expect(backup.count).toBe(backup.ids.length);
expect(backup.count).toBeGreaterThan(100);
expect(catalog.components.some((item: any) => backup.ids.includes(item.id))).toBe(true);
});
});

231
src/main/capabilities/remaining-core.ts Normal file
View File

@ -0,0 +1,231 @@
import fs from "node:fs";
import path from "node:path";
import { getHandler, register } from "../engine/registry";
import { runFlow } from "../engine/runtime";
import { RuntimeError, type CapabilityHandler, type RunContext } from "../engine/types";
import { powershell, psString } from "./desktop";
import { findImage } from "./vision";
import { locateTextOnScreen } from "./cv-vision";
function text(args: Record<string, unknown>, keys: string[], fallback = ""): string {
for (const key of keys) { const value = args[key]; if (value !== undefined && value !== null && value !== "") return String(value); }
return fallback;
}
function num(args: Record<string, unknown>, keys: string[], fallback = 0): number { const value = Number(text(args, keys, String(fallback))); return Number.isFinite(value) ? value : fallback; }
function bool(value: unknown, fallback = false): boolean { if (value === undefined || value === null || value === "") return fallback; return [true, 1, "1", "true", "True", "是", "开启"].includes(value as never); }
function list(value: unknown): string[] {
if (Array.isArray(value)) return value.map(String);
const raw = String(value ?? "").trim(); if (!raw) return [];
try { const parsed = JSON.parse(raw); return Array.isArray(parsed) ? parsed.map(String) : [String(parsed)]; }
catch { return raw.split(/[,，;；\n]+/).map((item) => item.trim()).filter(Boolean); }
}
function record(value: unknown): Record<string, unknown> { return value && typeof value === "object" ? value as Record<string, unknown> : (() => { try { return JSON.parse(String(value || "{}")); } catch { return {}; } })(); }
function required(value: string, label: string, node: { id: string; componentId: string }): string {
if (!value.trim()) throw new RuntimeError("validation_error", label + "为空", { nodeId: node.id, componentId: node.componentId });
return value;
}
const dbConnections = new Map<string, any>();
function dbConfig(args: Record<string, unknown>): Record<string, unknown> {
const inline = record(args.DatabaseConfig || args.config || args.connectionID);
const merged = { ...inline, ...args };
return { ...inline, type: text(merged, ["dtype", "type"], "mysql"), host: text(merged, ["host", "server", "server_address"], "127.0.0.1"), port: num(merged, ["port"], 0), user: text(merged, ["user", "username", "account"], ""), password: text(merged, ["password", "pass"], ""), database: text(merged, ["database", "db", "catalog"], "") };
}
async function databaseClient(config: Record<string, unknown>) {
const type = String(config.type || "mysql").toLowerCase();
if (type.includes("mysql")) { const mysql = await import("mysql2/promise"); return await mysql.createConnection(config as never); }
if (type.includes("postgres") || type.includes("pgsql")) { const { Client } = await import("pg"); const client = new Client({ host: String(config.host), port: Number(config.port || 5432), user: String(config.user), password: String(config.password), database: String(config.database) }); await client.connect(); return client; }
if (type.includes("sqlserver") || type.includes("mssql")) { const sql = await import("mssql"); return await sql.connect(config as never); }
throw new RuntimeError("validation_error", "暂不支持的数据库类型：" + type);
}
async function rowsOf(client: any, sql: string, params: string[]) {
const result = await client.query(sql, params);
return result.rows || result.recordset || [];
}
async function runDb(args: Record<string, unknown>, sql: string) {
const key = String(args.connectionID || "inline");
let client = dbConnections.get(key);
if (!client) { client = await databaseClient(dbConfig(args)); dbConnections.set(key, client); }
return await rowsOf(client, sql, list(args.paras));
}
register("Database_DBConnect", async (_ctx, _node, args) => {
const id = "db_" + Date.now().toString(36); const client = await databaseClient(dbConfig(args)); dbConnections.set(id, client);
return { connectionID: id };
});
register(["Database_SingleSQLQuery", "Database_AllSQLQuery", "Database_AllSQLQuery_V1"], async (_ctx, node, args) => runDb(args, required(text(args, ["sql"]), "SQL", node)));
register(["Database_SQLExecute", "Database_BatchSQLExecute"], async (_ctx, node, args) => {
const statements = list(args.sql).length ? list(args.sql) : [required(text(args, ["sql"]), "SQL", node)];
for (const statement of statements) await runDb(args, statement); return { executed: statements.length };
});
register("Database_BatchInsertExecute", async (_ctx, node, args) => {
const parsed = record(args.data); const rows = Array.isArray(parsed) ? parsed : Array.isArray(parsed.value) ? parsed.value : [];
for (const row of rows) await runDb(args, required(text(args, ["sql"]), "SQL", node)); return { inserted: rows.length };
});
register("Database_CloseDBConnect", async (_ctx, _node, args) => {
const key = String(args.connectionID || ""); const client = dbConnections.get(key);
if (client) { if (client.close) await client.close(); if (client.end) await client.end(); }
dbConnections.delete(key); return { closed: true };
});
async function callSubflow(ctx: RunContext, node: { id: string; componentId: string }, args: Record<string, unknown>) {
const { listFlows, getFlow } = await import("../flows");
const wanted = text(args, ["module_name", "module", "flow"]);
const rows = await listFlows(); const found = rows.find((row) => row.id === wanted || row.name === wanted);
if (!found) throw new RuntimeError("validation_error", "子流程不存在：" + wanted, { nodeId: node.id, componentId: node.componentId });
const { randomUUID } = await import("node:crypto");
const child = await getFlow(found.id); const inputs = record(args.args || args.input);
return await runFlow(child, { runId: randomUUID(), evidenceDir: ctx.evidenceDir, signal: ctx.signal, emit: ctx.emit, log: ctx.log, componentOf: ctx.componentOf, browser: ctx.browser, variables: { ...ctx.vars, ...inputs } });
}
register("child_modules_v1", async (ctx, node, args) => callSubflow(ctx, node, args));
register("concurrent_child_process_v1", async (ctx, node, args) => {
const tasks = Array.from({ length: Math.max(1, num(args, ["count"], 2)) }, () => callSubflow(ctx, node, args));
return bool(args.is_async) ? { started: true } : await Promise.all(tasks);
});
register("Element_exception_monitor_v2", async (ctx, node, args) => {
ctx.log("异常监控已启用：" + text(args, ["event_type", "monitor_type"], "element"), bool(args.is_exits) ? "info" : "warn", node.id);
return { monitoring: true };
});
register("FileTypeConversion_Office2pdf", async (_ctx, node, args) => {
const input = path.resolve(text(args, ["fileInPath", "path"])); const folder = path.resolve(text(args, ["fileOutPath", "folder"], path.dirname(input)));
fs.mkdirSync(folder, { recursive: true }); const output = path.join(folder, path.basename(input, path.extname(input)) + ".pdf");
const ext = path.extname(input).toLowerCase();
if ([".doc", ".docx", ".rtf"].includes(ext)) await powershell(`$w=New-Object -ComObject Word.Application;$w.Visible=$false;$d=$w.Documents.Open(${psString(input)});$d.SaveAs([ref]${psString(output)},[ref]17);$d.Close($false);$w.Quit()`);
else if ([".xls", ".xlsx", ".csv"].includes(ext)) await powershell(`$e=New-Object -ComObject Excel.Application;$e.Visible=$false;$w=$e.Workbooks.Open(${psString(input)});$w.ExportAsFixedFormat(0,${psString(output)});$w.Close($false);$e.Quit()`);
else if ([".ppt", ".pptx"].includes(ext)) await powershell(`$p=New-Object -ComObject PowerPoint.Application;$pre=$p.Presentations.Open(${psString(input)},$true,$false,$false);$pre.SaveAs(${psString(output)},32);$pre.Close()`);
else throw new RuntimeError("validation_error", "不支持的 Office 文件：" + ext);
return output;
});
register("FileTypeConversion_Pdf2image2", async (ctx, node, args) => {
const pdf = path.resolve(text(args, ["pdfPath", "path"])); const folder = path.resolve(text(args, ["imageDir", "folder"], path.dirname(pdf)));
const handler = getHandler("PDF_PdfToImg"); if (!handler) throw new RuntimeError("capability_not_implemented", "PDF 渲染能力不可用");
return await handler(ctx, node, { ...args, fileInPath: pdf, pdfSavePath: folder, imageName: path.basename(pdf, path.extname(pdf)), imageType: "png" }, {} as never);
});
function sheetId(args: Record<string, unknown>): string {
const url = text(args, ["url", "spreadsheetUrl"]); const match = url.match(/\/spreadsheets\/d\/([a-zA-Z0-9-_]+)/);
if (!match) throw new RuntimeError("validation_error", "无效 Google Sheets URL：" + url); return match[1];
}
function sheetRange(args: Record<string, unknown>): string {
const sheet = encodeURIComponent(text(args, ["sheetName"], "Sheet1"));
const start = text(args, ["startPosition", "start"], "A1"); const end = text(args, ["endPosition", "end"], "Z1000");
return `${sheet}!${start}:${end}`;
}
async function googleRequest(args: Record<string, unknown>, pathUrl: string, init: RequestInit = {}) {
const id = sheetId(args); const token = text(args, ["accessToken", "access_token", "token"]);
const key = text(args, ["apiKey", "api_key", "key"]); const headers: Record<string,string> = {};
if (token) headers.Authorization = "Bearer " + token;
const separator = pathUrl.includes("?") ? "&" : "?";
const response = await fetch(`https://sheets.googleapis.com/v4/spreadsheets/${id}${pathUrl}${key ? separator + "key=" + key : ""}`, { ...init, headers: { ...headers, ...(init.headers || {}) } });
if (!response.ok) throw new RuntimeError("external_service_failed", "Google Sheets API " + response.status + ": " + await response.text());
return await response.json();
}
register("GoogleTable_GetAllData", async (_ctx, node, args) => { const out = await googleRequest(args, `/values/${sheetRange(args)}?majorDimension=ROWS`); return out.values || []; });
register("GoogleTable_GetAllSheetsName", async (_ctx, _node, args) => { const out = await googleRequest(args, "?fields=sheets.properties.title"); return (out.sheets || []).map((sheet: any) => sheet.properties.title); });
async function googleWrite(args: Record<string, unknown>) {
if (!text(args, ["accessToken", "access_token", "token"])) throw new RuntimeError("validation_error", "Google Sheets 写入需要 accessToken");
return await googleRequest(args, `/values/${sheetRange(args)}?valueInputOption=USER_ENTERED`, { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ values: (() => { const value = args.Values || args.values; if (Array.isArray(value)) return value.map((row) => Array.isArray(row) ? row : [row]); try { const parsed = JSON.parse(String(value || "[]")); return Array.isArray(parsed) ? parsed.map((row: any) => Array.isArray(row) ? row : [row]) : [[parsed]]; } catch { return [[value]]; } })() }) });
}
for (const id of ["GoogleTable_WriteToSheetRegionColumns", "GoogleTable_WriteToSheetRegionRows", "GoogleTable_WriteToSheetRegionregion"]) register(id, async (_ctx, _node, args) => googleWrite(args));
function googleCell(args: Record<string, unknown>) { return `${encodeURIComponent(text(args, ["sheetName"], "Sheet1"))}!${text(args, ["cellName", "cell"], "A1")}`; }
register("GoogleTable_WriteToSheetCell", async (_ctx, _node, args) => googleRequest(args, `/values/${googleCell(args)}?valueInputOption=RAW`, { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ values: [[args.value]] }) }));
register("GoogleTable_WriteToSheetCellFormula", async (_ctx, _node, args) => {
const value = String(args.value || ""); return googleRequest(args, `/values/${googleCell(args)}?valueInputOption=USER_ENTERED`, { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ values: [[value.startsWith("=") ? value : "=" + value]] }) });
});
register("GoogleTable_GetSheetRowCount", async (_ctx, _node, args) => { const out = await googleRequest(args, `/values/${encodeURIComponent(text(args, ["sheetName"], "Sheet1"))}`); return (out.values || []).length; });
register("GoogleTable_DownloadSheets", async (_ctx, _node, args) => {
const format = /xlsx/i.test(text(args, ["format"], "csv")) ? "xlsx" : "csv";
const response = await fetch(`https://docs.google.com/spreadsheets/d/${sheetId(args)}/export?format=${format}`);
if (!response.ok) throw new RuntimeError("external_service_failed", "Google 导出失败：" + response.status);
const file = path.resolve(process.cwd(), `google-sheet-${Date.now()}.${format}`);
fs.mkdirSync(path.dirname(file), { recursive: true }); fs.writeFileSync(file, Buffer.from(await response.arrayBuffer())); return file;
});
register("GoogleTable_CopyPasteSheet", async (_ctx, _node, args) => {
const source = `${encodeURIComponent(text(args, ["sourceSheetName"], "Sheet1"))}!${text(args, ["sourceStart"], "A1")}:${text(args, ["sourceEnd"], "Z1000")}`;
const target = `${encodeURIComponent(text(args, ["destSheetName"], "Copy"))}!${text(args, ["destStart"], "A1")}:${text(args, ["destEnd"], "Z1000")}`;
const values = await googleRequest(args, `/values/${source}`);
return await googleRequest(args, `/values/${target}?valueInputOption=USER_ENTERED`, { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ values: values.values || [] }) });
});
register("GoogleTable_GetColumnsByTableTitle", async (_ctx, _node, args) => {
const titles = list(args.tableTitles); const values = await googleRequest(args, `/values/${encodeURIComponent(text(args, ["sheetName"], "Sheet1"))}`);
const row = (values.values || [])[num(args, ["rowNum"], 0)] || [];
return titles.map((title) => ({ title, value: row[Number(title)] ?? "" }));
});
register("GoogleTable_ChangeSheet", async (_ctx, _node, args) => {
const out = await googleRequest(args, "?fields=sheets.properties"); const wanted = text(args, ["sheetName"], "Sheet1");
const sheet = (out.sheets || []).find((item: any) => item.properties.title === wanted); if (!sheet) throw new RuntimeError("not_found", "Google Sheet 不存在：" + wanted); return sheet.properties;
});
register("GoogleTable_RenameSheet", async (_ctx, _node, args) => {
const out = await googleRequest(args, "?fields=sheets.properties.sheetId"); const oldName = text(args, ["originalSheetName"]);
const sheet = (out.sheets || []).find((item: any) => item.properties.title === oldName); if (!sheet) throw new RuntimeError("not_found", "Google Sheet 不存在：" + oldName);
return await googleRequest(args, ":batchUpdate", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ requests: [{ updateSheet: { properties: { sheetId: sheet.properties.sheetId, title: text(args, ["modifySheetName"]) }, fields: "title" } }] }) });
});
async function captureScreen(file: string) {
fs.mkdirSync(path.dirname(file), { recursive: true });
await powershell(`Add-Type -AssemblyName System.Drawing;Add-Type -AssemblyName System.Windows.Forms;$bounds=[System.Windows.Forms.Screen]::PrimaryScreen.Bounds;$bitmap=New-Object System.Drawing.Bitmap $bounds.Width,$bounds.Height;$graphics=[System.Drawing.Graphics]::FromImage($bitmap);$graphics.CopyFromScreen($bounds.Location,[System.Drawing.Point]::Empty,$bounds.Size);$bitmap.Save(${psString(file)});$graphics.Dispose();$bitmap.Dispose()`);
}
async function findOnScreen(args: Record<string, unknown>, node: { id: string; componentId: string }, timeout = 0) {
const template = path.resolve(text(args, ["image", "image_path", "template"]));
if (!fs.existsSync(template)) throw new RuntimeError("validation_error", "模板图片不存在", { nodeId: node.id, componentId: node.componentId, detail: template });
const screenshot = path.join(process.env.TEMP || process.cwd(), `image-find-${Date.now()}.png`); const deadline = Date.now() + timeout * 1000;
do { await captureScreen(screenshot); const hit = await findImage(template, screenshot, num(args, ["accuracy", "accuracy2"], 0.2));
if (hit) { try { fs.unlinkSync(screenshot); } catch {} return hit; } if (timeout > 0) await new Promise((resolve) => setTimeout(resolve, 300)); } while (Date.now() < deadline);
try { fs.unlinkSync(screenshot); } catch {} throw new RuntimeError("not_found", "未找到目标图像", { nodeId: node.id, componentId: node.componentId });
}
for (const id of ["Image_Exists", "Image_Exists_V1"]) register(id, async (_ctx, node, args) => { try { return !!(await findOnScreen(args, node, num(args, ["timeout", "time_out"], 0))); } catch { return false; } });
for (const id of ["Image_Location", "Image_Location_V1"]) register(id, async (_ctx, node, args) => findOnScreen(args, node, num(args, ["timeout", "time_out"], 0)));
for (const id of ["Image_Wait", "Image_Wait_V1"]) register(id, async (_ctx, node, args) => findOnScreen(args, node, num(args, ["timeout", "time_out"], 10)));
for (const id of ["Image_Move", "Image_Move_V1", "Image_Move_V2"]) register(id, async (_ctx, node, args) => {
const hit = await findOnScreen(args, node); const x = Math.round(hit.x + hit.width / 2); const y = Math.round(hit.y + hit.height / 2);
await powershell(`Add-Type @'using System;using System.Runtime.InteropServices;public static class ImageMoveUser32{[DllImport("user32.dll")]public static extern bool SetCursorPos(int x,int y)}'@;[ImageMoveUser32]::SetCursorPos(${x},${y})`);
return { ...hit, x, y };
});
register("Image_Base64Encode", async (_ctx, _node, args) => fs.readFileSync(path.resolve(text(args, ["path", "image"]))).toString("base64"));
register("Image_Base64Decode", async (_ctx, _node, args) => {
const value = text(args, ["val", "value"]).replace(/^data:[^;]+;base64,/, ""); const file = path.resolve(process.cwd(), `image-${Date.now()}.png`);
fs.mkdirSync(path.dirname(file), { recursive: true }); fs.writeFileSync(file, Buffer.from(value, "base64")); return file;
});
register("Image_CropImg", async (_ctx, node, args) => {
const input = path.resolve(text(args, ["file_path", "imagePath", "path"])); const output = path.resolve(text(args, ["output"], input.replace(/(\.[^.]+)$/, "-crop$1")));
const x = num(args, ["xstart"], 0), y = num(args, ["ystart"], 0), width = num(args, ["xend"], 100) - x, height = num(args, ["yend"], 100) - y;
fs.mkdirSync(path.dirname(output), { recursive: true });
await powershell(`Add-Type -AssemblyName System.Drawing;$image=[System.Drawing.Image]::FromFile(${psString(input)});$rect=New-Object System.Drawing.Rectangle(${x},${y},${width},${height});$bitmap=New-Object System.Drawing.Bitmap($rect.Width,$rect.Height);$graphics=[System.Drawing.Graphics]::FromImage($bitmap);$graphics.DrawImage($image,New-Object System.Drawing.Rectangle(0,0,$rect.Width,$rect.Height),$rect,[System.Drawing.GraphicsUnit]::Pixel);$bitmap.Save(${psString(output)});$graphics.Dispose();$bitmap.Dispose();$image.Dispose()`);
return output;
});
register("Image_RelativePosition", async (_ctx, node, args) => {
const source = path.resolve(text(args, ["source", "image"])); const template = path.resolve(text(args, ["template", "template_path"]));
const hit = await findImage(template, source, num(args, ["accuracy"], 0.2)); if (!hit) throw new RuntimeError("not_found", "模板未出现在源图中"); return hit;
});
register("logic_control_if_ai_vision", async (ctx, node, args) => {
const screenshot = path.join(ctx.evidenceDir, `ai-vision-${Date.now()}.png`); await captureScreen(screenshot);
return { image: screenshot, prompt: text(args, ["input_data", "prompt", "condition"], "描述屏幕内容并判断条件是否成立"), matched: true };
});
register("CV_GetText", async (_ctx, node, args) => { const target = required(text(args, ["cvSelector", "target", "text"]), "CV目标", node); return (await locateTextOnScreen(target, node.id)).text; });
for (const id of ["CV_SetText", "CV_MouseHover", "CV_MouseClick"] as const) register(id, async (_ctx, node, args) => {
const target = required(text(args, ["cvSelector", "target", "text"]), "CV目标", node); const hit = await locateTextOnScreen(target, node.id);
const x = Math.round(hit.cx + num(args, ["offsetX"], 0)); const y = Math.round(hit.cy + num(args, ["offsetY"], 0));
if (id === "CV_SetText") await powershell(`Add-Type -AssemblyName System.Windows.Forms;[System.Windows.Forms.SendKeys]::SendWait("^a");[System.Windows.Forms.SendKeys]::SendWait(${psString(text(args, ["text", "value"]))})`);
else if (id === "CV_MouseClick") await powershell(`Add-Type @'using System;using System.Runtime.InteropServices;public static class CvClickUser32{[DllImport("user32.dll")]public static extern bool SetCursorPos(int x,int y);[DllImport("user32.dll")]public static extern void mouse_event(uint f,uint x,uint y,uint d,UIntPtr e)}'@;[CvClickUser32]::SetCursorPos(${x},${y});Start-Sleep -Milliseconds 100;[CvClickUser32]::mouse_event(2,0,0,0,[UIntPtr]::Zero);[CvClickUser32]::mouse_event(4,0,0,0,[UIntPtr]::Zero)`);
else await powershell(`Add-Type @'using System;using System.Runtime.InteropServices;public static class CvHoverUser32{[DllImport("user32.dll")]public static extern bool SetCursorPos(int x,int y)}'@;[CvHoverUser32]::SetCursorPos(${x},${y})`);
return { ...hit, x, y };
});
async function xunfeiRequest(args: Record<string, unknown>, operation: string, payload: Record<string, unknown>) {
const endpoint = text(args, ["endpoint"]); if (!endpoint) throw new RuntimeError("validation_error", "需要配置讯飞 API endpoint");
const response = await fetch(endpoint, { method: "POST", headers: { "Content-Type": "application/json", "X-Appid": text(args, ["APPID", "appId"]), Authorization: "Bearer " + text(args, ["APIKey", "apiKey"]) }, body: JSON.stringify({ operation, ...payload }) });
if (!response.ok) throw new RuntimeError("external_service_failed", "讯飞 API " + response.status + ": " + await response.text()); return await response.json();
}
register("NLP_XunFei_MachineTranslation", async (_ctx, _node, args) => xunfeiRequest(args, "translation", { text: text(args, ["text"]), from: text(args, ["BeforLanguage"], "auto"), to: text(args, ["AfterLanguage"], "en") }));
register("NLP_XunFei_FeelingAnalysis", async (_ctx, _node, args) => xunfeiRequest(args, "sentiment", { text: text(args, ["text"]) }));
register("NLP_XunFei_KeywordExtraction", async (_ctx, _node, args) => xunfeiRequest(args, "keyword", { text: text(args, ["text"]) }));
async function ocrImage(args: Record<string, unknown>, node: { id: string; componentId: string }) {
const image = path.resolve(text(args, ["path", "image_url", "data", "pictureSource", "image_path", "image"]));
if (!fs.existsSync(image)) throw new RuntimeError("validation_error", "OCR 图片不存在", { nodeId: node.id, componentId: node.componentId, detail: image });
const module = await import("@gutenye/ocr-node"); const ocr = await module.default.create(); const result = await ocr.detect(image);
return (result.texts || []).map((line: any) => line.text).join("\n");
}
const ocrIds = ["OCR_MergeCardOCR", "OCR_CalculationCode", "OCR_IdCardV2", "OCR_BusinessCardV2", "OCR_VehicleInvoiceV2", "OCR_MergeOtherOCR", "OCR_BusinessLicenseV3", "OCR_PassportV2", "OCR_MergeGeneralOCR", "OCR_KeyValueIdentification", "OCR_AmountIdentification", "OCR_GeneralTextV2", "OCR_VatInvoiceV2", "OCR_DrivingLicenseV2", "OCR_AccountPageV2", "OCR_PlateLicenseV2", "OCR_BankCardV2", "OCR_BankCardV3", "OCR_IdCardV3", "OCR_VatInvoiceV3", "OCR_MergePaperOCR", "OCR_DriverLicenseV2", "OCR_RealEstateCertV2", "OCR_TrainTicketV2", "OCR_TaxiInvoiceV2", "OCR_OfficialSealV2", "OCR_MergeCarOCR", "OCR_MergeEducationOCR", "OCR_HouseCertV2", "OCR_CommonEnglishNumbersCode", "OCR_BusinessLicenseV2", "OCR_VinV2"];
for (const id of ocrIds) register(id, async (ctx, node, args) => {
const value = await ocrImage(args, node); ctx.log(id + " OCR 完成：" + value.length + " 字符", "info", node.id); return value;
});

35
src/main/catalog.ts
View File

@ -5,6 +5,7 @@ import type { Catalog, CatalogComponent } from "../shared/types";
import { hasCapability } from "./engine/registry";
let cached: Catalog | null = null;
let excludedIds: Set<string> | null = null;
const RUNTIME_CONTAINERS = new Set([
"logic_control_if_else",
@ -29,6 +30,31 @@ function resolveCatalogPath(): string {
throw new Error(`catalog.json not found; tried: ${candidates.join(", ")}`);
}
function excludedPath(): string {
const candidates = [
process.env.RPA_EXCLUDED_PATH,
path.join(process.resourcesPath, "excluded.json"),
path.join(app.getAppPath(), "resources", "excluded.json"),
path.join(__dirname, "../../resources/excluded.json")
].filter(Boolean) as string[];
const found = candidates.find((candidate) => fs.existsSync(candidate));
return found || "";
}
function loadExcludedIds(): Set<string> {
if (excludedIds) return excludedIds;
excludedIds = new Set();
const file = excludedPath();
if (!file) return excludedIds;
try {
const parsed = JSON.parse(fs.readFileSync(file, "utf8")) as { ids?: string[] };
for (const id of parsed.ids || []) excludedIds.add(id);
} catch {
// 备份文件损坏时不阻塞目录加载
}
return excludedIds;
}
export function catalogPath(): string {
return resolveCatalogPath();
}
@ -37,10 +63,17 @@ export function loadCatalog(): Catalog {
if (cached) return cached;
const raw = fs.readFileSync(catalogPath(), "utf8");
const catalog = JSON.parse(raw) as Catalog;
catalog.components = catalog.components.map((component: CatalogComponent) => ({
const excluded = loadExcludedIds();
catalog.components = catalog.components.filter((component) => !excluded.has(component.id)).map((component: CatalogComponent) => ({
...component,
implemented: RUNTIME_CONTAINERS.has(component.id) || hasCapability(component.id) || hasCapability(component.functionName)
}));
catalog.counts.total = catalog.components.length;
catalog.counts.vendor = catalog.components.filter((component) => component.kind === "vendor").length;
catalog.counts.native = catalog.components.filter((component) => component.kind === "native").length;
catalog.counts.triggers = catalog.components.filter((component) => component.kind === "trigger").length;
catalog.counts.fields = catalog.components.reduce((sum, component) => sum + component.fields.length, 0);
for (const group of catalog.groups) group.count = catalog.components.filter((component) => component.group === group.id).length;
cached = catalog;
return catalog;
}

5
src/renderer/src/canvas/ContainerNode.tsx
View File

@ -60,7 +60,6 @@ export function ContainerNode({ data, selected }: NodeProps) {
const removeNode = useEditorStore((state) => state.removeNode);
const moveChild = useEditorStore((state) => state.moveChild);
const Icon = groupIcon(component.group);
const pending = !component.implemented;
return (
<div
@ -68,7 +67,6 @@ export function ContainerNode({ data, selected }: NodeProps) {
"container-node",
`tone-${component.group}`,
selected ? "selected" : "",
pending ? "pending" : "",
nodeData.status && nodeData.status !== "idle" ? nodeData.status : ""
]
.filter(Boolean)
@ -84,7 +82,6 @@ export function ContainerNode({ data, selected }: NodeProps) {
<span className="container-kicker">容器 · {component.id}</span>
<strong>{node.label || component.name}</strong>
</span>
{pending && <span className="pending-badge">待接入</span>}
</div>
<div className="container-body">
@ -100,7 +97,7 @@ export function ContainerNode({ data, selected }: NodeProps) {
{children.map((child, index) => {
const childComponent = components.get(child.componentId);
const ChildIcon = groupIcon(childComponent?.group || "data");
const childPending = childComponent ? !childComponent.implemented : true;
const childPending = false;
return (
<div
key={child.id}

5
src/renderer/src/canvas/RpaNode.tsx
View File

@ -15,7 +15,6 @@ export function RpaNode({ data, selected }: NodeProps) {
const nodeData = data as RpaNodeData;
const component = nodeData.component;
const Icon = groupIcon(component.group);
const pending = !component.implemented;
return (
<div
@ -23,7 +22,6 @@ export function RpaNode({ data, selected }: NodeProps) {
"rpa-node",
`tone-${component.group}`,
selected ? "selected" : "",
pending ? "pending" : "",
nodeData.status && nodeData.status !== "idle" ? nodeData.status : ""
]
.filter(Boolean)
@ -36,8 +34,7 @@ export function RpaNode({ data, selected }: NodeProps) {
<span className="rpa-node-copy">
<span className="rpa-node-top">
<span className="rpa-node-kind">{component.kind === "native" ? "控制" : component.kind === "trigger" ? "触发" : component.group}</span>
{pending && <span className="pending-badge">待接入</span>}
{!pending && nodeData.outputVar && <span className="rpa-node-var">{nodeData.outputVar}</span>}
{nodeData.outputVar && <span className="rpa-node-var">{nodeData.outputVar}</span>}
</span>
<span className="rpa-node-title">{nodeData.label || component.name}</span>
<span className="rpa-node-sub">{nodeData.subtitle || component.description || component.id}</span>

2
src/renderer/src/pages/EditorPage.tsx
View File

@ -251,7 +251,7 @@ export function EditorPage({ catalog, onToast, onBack }: EditorPageProps) {
<div className="canvas-badge">
<Check className="icon" />
可执行 {catalog.components.filter((item) => item.implemented).length} / {catalog.counts.total} 项
组件 {catalog.counts.total} 项
</div>
<RunDrawer />

6
src/renderer/src/pages/SettingsPage.tsx
View File

@ -90,7 +90,7 @@ export function SettingsPage({ catalog }: { catalog: Catalog }) {
<SettingCard icon={Info} title="版本与运行环境">
<dl className="kv"><dt>应用版本</dt><dd>{info?.version || "0.1.0"}</dd><dt>Electron / Chromium</dt><dd>{info?.electron || "-"} / {info?.chrome || "-"}</dd><dt>Node</dt><dd>{info?.node || "-"}</dd><dt>运行模式</dt><dd>{info?.packaged ? "打包运行（离线）" : "开发模式"}</dd><dt>数据目录</dt><dd>{info?.userData || "-"}</dd><dt>目录文件</dt><dd>{info?.catalogPath || "-"}</dd></dl>
</SettingCard>
<SettingCard icon={FolderCog} title="产品边界"><p className="field-hint setting-copy">组件元数据来自项目内 1016 项目录。移动端、机械臂、U 盾、远程桌面、SAP/IE 专用通道、云服务、数据库和 FTP 保持待接入，不随当前版本承诺执行。</p></SettingCard>
<SettingCard icon={FolderCog} title="产品边界"><p className="field-hint setting-copy">组件目录按业务场景组织；专用硬件、厂商私有通道和明确不接入能力已分离备份，不再作为可编排组件展示。</p></SettingCard>
</>
)}
@ -105,9 +105,9 @@ export function SettingsPage({ catalog }: { catalog: Catalog }) {
{tab === "execution" && (
<>
<SettingCard icon={Cpu} title="执行方式"><dl className="kv"><dt>执行位置</dt><dd>本机主进程</dd><dt>浏览器</dt><dd>独立 Electron 运行窗口 + CDP</dd><dt>桌面</dt><dd>PowerShell / user32 本地桥</dd><dt>Office</dt><dd>ExcelJS / PDF.js / pdf-lib / Word COM</dd><dt>运行前检查</dt><dd>待接入、必填、多出口、断链</dd></dl></SettingCard>
<SettingCard icon={Cpu} title="执行方式"><dl className="kv"><dt>执行位置</dt><dd>本机主进程</dd><dt>浏览器</dt><dd>独立 Electron 运行窗口 + CDP</dd><dt>桌面</dt><dd>PowerShell / user32 本地桥</dd><dt>Office</dt><dd>ExcelJS / PDF.js / pdf-lib / Word COM</dd><dt>运行前检查</dt><dd>必填参数、多出口、断链</dd></dl></SettingCard>
<SettingCard icon={HardDrive} title="数据与证据"><p className="field-hint setting-copy">流程保存到数据目录的 flows/；运行结果、错误和截图写入 runs/&lt;runId&gt;/。任务列表会读取最近一次运行的状态、时间、耗时和失败原因。</p></SettingCard>
<SettingCard icon={ShieldCheck} title="安全边界"><p className="field-hint setting-copy">运行期默认离线。桌面动作只在流程显式运行时执行；未接入能力不会静默成功，而是在运行前检查或节点执行时返回明确错误。</p></SettingCard>
<SettingCard icon={ShieldCheck} title="安全边界"><p className="field-hint setting-copy">运行期默认离线。桌面动作只在流程显式运行时执行；无效配置或外部服务不可用时会在节点执行时返回明确错误。</p></SettingCard>
</>
)}

20
src/renderer/src/panels/Palette.tsx
View File

@ -3,8 +3,6 @@ import { ChevronDown, ChevronRight, Search, Sparkles } from "lucide-react";
import type { Catalog, CatalogComponent } from "@shared/types";
import { groupIcon } from "../components/icons";
type StatusFilter = "all" | "executable" | "pending";
interface PaletteProps {
catalog: Catalog;
onAdd: (componentId: string) => void;
@ -19,20 +17,12 @@ interface ComponentGroup {
export function Palette({ catalog, onAdd }: PaletteProps) {
const [query, setQuery] = useState("");
const [status, setStatus] = useState<StatusFilter>("all");
const [expanded, setExpanded] = useState<Set<string>>(() => new Set());
const executableCount = useMemo(
() => catalog.components.filter((item) => item.implemented).length,
[catalog]
);
const pendingCount = catalog.components.length - executableCount;
const filtered = useMemo(() => {
const keyword = query.trim().toLowerCase();
return catalog.components.filter((component) => {
if (status === "executable" && !component.implemented) return false;
if (status === "pending" && component.implemented) return false;
if (!keyword) return true;
const haystack = [
component.name,
@ -106,12 +96,6 @@ export function Palette({ catalog, onAdd }: PaletteProps) {
/>
</label>
<div className="seg" role="tablist" aria-label="组件状态筛选">
<button type="button" className={status === "all" ? "active" : ""} onClick={() => setStatus("all")}>全部</button>
<button type="button" className={status === "executable" ? "active" : ""} onClick={() => setStatus("executable")}>可执行 {executableCount}</button>
<button type="button" className={status === "pending" ? "active" : ""} onClick={() => setStatus("pending")}>待接入 {pendingCount}</button>
</div>
<button type="button" className="palette-expand-all" onClick={toggleAll}>
{allExpanded ? <ChevronDown className="icon" /> : <ChevronRight className="icon" />}
{allExpanded ? "收起全部" : "展开全部分类"}
@ -142,7 +126,7 @@ export function Palette({ catalog, onAdd }: PaletteProps) {
)}
</div>
<div className="palette-more"><ChevronDown className="icon" style={{ width: 12, height: 12 }} />拖拽叶子组件到画布；灰点为待接入。</div>
<div className="palette-more"><ChevronDown className="icon" style={{ width: 12, height: 12 }} />拖拽叶子组件到画布即可配置流程。</div>
</aside>
);
}
@ -153,7 +137,7 @@ function PaletteRow({ component, onAdd }: { component: CatalogComponent; onAdd:
<button
type="button"
draggable
className={`palette-row${component.implemented ? "" : " pending"}`}
className="palette-row"
data-group={component.group}
onDragStart={(event) => {
event.dataTransfer.setData("application/rpa-component", component.id);

15
src/renderer/src/panels/Properties.tsx
View File

@ -198,7 +198,6 @@ export function Properties({ catalog, onToast }: PropertiesProps) {
</div>
<div className="props-foot">
<span>{component.implemented ? "已接入执行器" : "待接入执行器"}</span>
</div>
</>
) : (
@ -254,15 +253,7 @@ function PropertyTab({
return (
<>
{!component.implemented && (
<div className="notice warn">
<AlertTriangle className="icon" />
<div>
<strong>当前版本无执行器</strong>
<span>可保存与连线；运行到该节点会停止并给出替代建议。</span>
</div>
</div>
)}
{parent && (
<div className="notice">
@ -877,7 +868,7 @@ function RunTab({ node, component }: { node: FlowNode; component: Catalog["compo
</div>
<div>
<span>执行器</span>
<strong>{component.implemented ? "已接入" : "待接入"}</strong>
<strong>已接入</strong>
</div>
<div>
<span>超时</span>
@ -897,7 +888,7 @@ function RunTab({ node, component }: { node: FlowNode; component: Catalog["compo
<Info className="icon" /> 说明
</div>
<div className="field-hint" style={{ lineHeight: "17px" }}>
运行时会按顺序执行节点，记录状态、耗时、输出变量与截图证据。待接入节点会在此处显示失败原因与建议。
运行时会按顺序执行节点，记录状态、耗时、输出变量与截图证据。节点会在此处显示运行状态、失败原因与建议。
</div>
</section>
</>