Compare commits

...

10 Commits

Author SHA1 Message Date
nan
fb78a8068b chore: Update BlendJob wheel to version 0.1.6 and adjust related documentation
Some checks failed
构建扩展 / build (push) Failing after 11m45s
2026-08-17 16:45:29 +08:00
nan
1c98b21235 Add model catalog with specifications for DA3 and MoGe-2 models
- Introduced ModelSpec dataclass to define model attributes.
- Added multiple models including DA3 Small, Base, Giant 1.1, Nested Giant Large 1.1, and MoGe-2 variants.
- Implemented functions to retrieve model details and metadata.
- Included BEN2 model specifications for background removal.
2026-08-17 14:53:25 +08:00
nan
c5b77c5c85 Add ONNX model inference implementations for MoGe-2, BEN2, and DA3
- Introduced `onnx_moge2.py` for MoGe-2 ONNX inference with NumPy post-processing.
- Added `onnx_ben2.py` for BEN2 ONNX inference with Pillow and NumPy preprocessing.
- Created `onnx_da3.py` for Depth Anything 3 inference with ONNX Runtime and NumPy.
- Implemented shared ONNX Runtime session helpers in `onnx_runtime.py`.
- Developed unit tests for server functionalities in `test_server.py`, including model caching and job management.
- Enhanced job server to handle frame extraction without requiring models.
2026-08-17 10:37:55 +08:00
nan
b3a07cde32 Add reusable local HTTP job service for Blender add-ons
- Implement JobClient for handling HTTP requests to the JobServer.
- Create ModalJobOperatorBase for managing Blender job lifecycle.
- Introduce JobContext and JobServer for job management and state handling.
- Add ServiceConnection and ServiceController for managing service processes.
- Implement job submission, cancellation, and resource management endpoints in the server.
- Ensure proper error handling and job state updates throughout the service.
2026-08-16 21:04:10 +08:00
nan
0c8d3a8988 Bump version to 0.11.2 and update uv version to 0.11.3
- Updated version in pyproject.toml and blender_manifest.toml to 0.11.2.
- Enhanced uv executable path documentation in client.py.
- Added inference_mode parameter to reconstruct_request function in client.py.
- Implemented uv download functionality in install_env.py with improved error handling and checksum verification.
- Modified install_uv function to download and extract uv binary instead of using bundled version.
- Updated job progress handling in operators.py to track progress stages.
- Improved download progress reporting in worker.py and adjusted related tests.
- Refactored tests to accommodate changes in status reporting and model downloading.
- Removed vendor.lock.toml as part of the cleanup process.
2026-08-14 23:43:01 +08:00
nan
102e1bfa9b feat: Update documentation and improve depth processing logic; bump version to 0.11.1 2026-08-14 21:57:08 +08:00
nan
5255eb4e96 feat: Bump version to 0.11.0 and update dependencies
- Updated ONNX Runtime to 1.24.4 and replaced OpenCV with PyAV 17.0.0 in the installation environment.
- Enhanced model downloading logic to prioritize Hugging Face and fallback to OmooLab R2, with improved error handling and status reporting.
- Introduced a new environment revision file for shared metadata.
- Refactored video frame extraction to utilize the PyAV library instead of OpenCV.
- Improved installation process to handle retries across specified PyPI indexes.
- Added tests for download progress reporting and model downloading behavior.
2026-08-14 14:47:40 +08:00
nan
2ad30bf4a3 feat: Implement ONNX model inference for BEN2, Depth Anything 3, and MoGe
- Added `ben2_onnx.py` for BEN2 ONNX inference with preprocessing.
- Introduced `da3_onnx.py` for Depth Anything 3 inference, including camera projection calculations.
- Created `moge_onnx.py` for MoGe-2 ONNX inference and post-processing.
- Implemented shared ONNX Runtime session helpers in `onnx_runtime.py`.
- Added model synchronization script `sync_models.py` for downloading and verifying ONNX models.
- Developed unit tests for BEN2, Depth Anything 3, and MoGe ONNX models to ensure functionality.
- Updated `vendor.lock.toml` to remove unused artifacts.
2026-08-14 10:27:09 +08:00
nan
13fab9ce88 Add comprehensive documentation for SnapForm extension
- Created development.md for development setup and licensing information.
- Added features.md detailing functionalities and usage instructions for developers and users.
- Introduced index.md to serve as a navigation hub for the documentation.
- Developed outputs.md to explain output formats and their specifications.
- Established service.md to describe the AI Service architecture and environment setup.
- Created surface.md to outline the process of converting Image Empty to perspective meshes.
2026-08-13 23:57:09 +08:00
nan
caeb6d706c feat: Add MoGe-2 surface generation and export functionality
- Introduced new enum items for depth plane and surface models with caching.
- Added MOGE2_MESH_JOB to operators for converting images to surfaces.
- Implemented surface mesh building and exporting in RunJob operator.
- Enhanced UI to include export format options for MoGe-2.
- Created validation tools for MoGe-2 surface assets and depth planes.
- Updated geometry nodes to support new surface generation features.
- Refactored existing tests to accommodate new functionality and ensure stability.
2026-08-13 22:07:36 +08:00
72 changed files with 7736 additions and 13326 deletions

View File

@@ -24,9 +24,6 @@ jobs:
- name: 安装测试依赖
run: python -m pip install numpy pytest
- name: 准备 Vendor
run: python prepare_vendor.py
- name: 运行测试
run: python -m pytest

3
.gitignore vendored
View File

@@ -7,4 +7,5 @@ build/
*.zip
.pytest_cache/
/vendor/
/models/
wheels/

View File

@@ -1,32 +1,81 @@
## 总规范
# SnapForm 开发约定
- 说明性内容(例如提交说明、文档、注释、mermaid),以中文为主,专有名词、约定俗称可用英文
## 总则
## Markdown 规范
- 说明性内容(提交说明、文档、注释、Mermaid 等)以中文为主,专有名词和约定俗成的表达可使用英文
- 动手前先读 `docs/index.md`;程序边界、调用流程和模块职责见 `docs/development.md#程序设计`
- 修改代码、依赖或执行方式后,同步更新 `README.md` 与对应主题文档,保持文档、测试和实现一致
- 优先选择依赖少、直接、易验证的实现;保留工作区中与当前任务无关的已有改动
- 不使用`---`分割器
- `# Heading` 一级标题仅用于开头
- mermaid 节点名用英文字符
## 项目结构
```text
.
├── README.md # 用户入口与快速上手
├── docs/ # 按功能拆分的设计与开发文档
├── pack.py # 构建各平台 Blender Extension ZIP
├── sync_models.py # 准备并同步 ONNX 模型
├── src/snapform/
│ ├── __init__.py # Blender 注册入口
│ ├── operators.py # Job Operator 与结果导入
│ ├── panel.py # 3D View 面板
│ ├── preferences.py # Extension Preferences
│ ├── properties.py # Scene 设置与动态枚举
│ ├── job_runtime.py # BlendJob Runtime 配置
│ ├── blender/ # Depth Plane、Surface 的 Blender 侧实现
│ ├── server/ # Job、模型、ONNX 推理与输出实现
│ ├── assets/O_SnapForm.blend # Geometry Nodes 与材质资产
│ └── wheels/ # 随扩展分发的 BlendJob wheel
├── tools/ # Blender 资产构建与验证脚本
└── tests/ # Python 自动化测试
```
文档按主题定位:功能与使用 `docs/features.md`、Depth Plane `docs/depth-plane.md`、Surface `docs/surface.md`、输出格式 `docs/outputs.md`、Job Server `docs/server.md`、程序设计与开发 `docs/development.md`。
## 常见命令
优先在项目根目录使用 Git Bash;Windows 路径在 Bash 命令中使用正斜杠。
```bash
# 安装或同步开发依赖
uv sync
# 运行全部测试
uv run pytest
# 运行插件注册、Manifest 与打包相关的快速测试
uv run pytest tests/test_addon.py tests/test_manifest.py tests/test_pack.py -v
# 构建 Windows、macOS、Linux 扩展包到 dist/
uv run python pack.py
# 本地 BlendJob wheel 内容变化后刷新锁文件
uv lock --refresh
```
资产脚本必须由 Blender 执行;具体入口与验证范围见 `docs/development.md#资产构建与验证`,不要用普通 Python 解释器运行。
## 命名规范
- 变量、函数、Docstring 使用英文;同一概念在代码和文档中使用统一表达
- Python 类使用大驼峰,不添加项目名前缀。Operator 使用动宾结构,如 `ConvertToDepthPlane`、`RemoveImageBackground`;Menu、Panel、AddonPreferences、PropertyGroup 分别使用实际类型后缀,如 `SnapFormImageMenu`、`ServerPanel`、`SnapFormPreferences`、`SnapFormSettings`
- `bl_idname` 与挂载到公共 Blender RNA 类型的 Property 使用 `snapform` 项目前缀:Operator 如 `snapform.convert_to_depth_plane`,Menu 如 `SNAPFORM_MT_image_context`,Panel 如 `SNAPFORM_PT_server`,公共 Property 如 `Scene.snapform_settings`
- Operator 自身 Property 不加项目前缀,使用业务名称,如 `input_path`、`process_res`、`resolution_level`
- 扩展包名使用 Manifest 中的真实名称、版本和平台,例如 `SnapForm.v0.11.2.windows-x64.zip`;节点组名称沿用资产中的 `O DA3 Depth Plane`、`O MoGe2 Surface` 等既有名称
## 代码规范
- 避免嵌套结构
- 特殊的、非常规的需要写注释
- 变量、函数名、Docstring 用全英文
- 同一事物用统一表达
- 使用的依赖越少越好
- 选用简单直接的方式实现
- 代码、执行方式改变,调整已有的文档、计划,而非新建
- 避免不必要的嵌套,优先使用提前返回和小而明确的函数
- 只为特殊、非常规或存在兼容性原因的实现写注释,不复述代码本身
- 删除功能时同时删除无效参数、分支、测试和文档,不保留没有实际消费者的兼容层
- Blender 侧不载入 AI 依赖:UI、Operator 和结果导入放在 `src/snapform/` 或 `src/snapform/blender/`,模型下载、ONNX 推理和输出放在 `src/snapform/server/`
- Job 参数使用可序列化的普通字典;Blender 数据只能在主线程的 Operator 响应阶段读写
- 新增或修改 Blender 类型时同步检查 `src/snapform/__init__.py` 的注册与逆序注销,并补充对应测试
- 依赖版本、wheel 或 Manifest 改变时同步检查 `pyproject.toml`、`uv.lock`、`blender_manifest.toml` 和打包测试
## Shell 规范
## 文档与 Markdown 规范
- 优先使用、提供 Git Bash 执行所有 Shell 命令
- Windows 路径在 Bash 命令中优先使用正斜杠形式
## Blender 开发规范
- 插件、资产库压缩包名用大驼峰,如果需要,用`b45`来表示适用blender版本。比如`OmooLab.v0.1.0.b45.zip`
- Operator、Menu、Panel、AddonPreferences... Class 用大驼峰命名,不加任何前缀。Operator 动宾结构,比如`RenderImage`。Menu、Panel 以它们本身为后缀,比如`RenderPanel`
- 根据项目名提炼项目缩写,作为 bl_idname 和 自定义 Property 的前缀以区分其他功能和属性,以`Omoo`为例
- bl_idname 用下划线小写命名,Operator 的 bl_idname 以`omoo.*`为前缀,比如`omoo.render_panel`;Menu 的以`OMOO_MT_*`为前缀;Panel 的以`OMOO_PT_*`为前缀。比如`OMOO_PT_render_panel`
- 自定义 Property 用下划线小写命名,以`omoo_*`为前缀。比如`omoo_progress_factor`
- `README.md` 面向用户;实现原理放入 `docs/` 对应主题,不在 README 堆叠内部细节
- 不使用 `---` 分隔线;一级标题只在文档开头使用一次
- Mermaid 节点 ID 使用英文字符,节点显示文本可以使用中文
- 文档描述当前已实现行为;限制、默认值、命令和文件名必须能从代码或测试中验证

View File

@@ -16,12 +16,18 @@ SnapForm 是一个 Blender 扩展,可以把一张照片、一段视频或一
2. 打开 Blender,进入 `Preferences > Extensions`
3. 点击 `Install from Disk`,选择下载的压缩包
4. 在扩展偏好设置中确认 Storage Root 与 Device(保持默认即可);Device 默认取本机
最快的 CUDA/MPS,否则 CPU。下载超时会自动重试国内镜像,无需手动配置
最快的 DirectML/CoreML provider,否则 CPU。下载超时会自动重试国内镜像,无需手动配置
5. 打开 3D 视图右侧的 `SnapForm` 面板,点击 `Install Environment`
DA3、MoGe-2、utils3d、BEN2 源码包和 uv 已内置在扩展里,安装时直接从本地解包,不需要访问 GitHub。首次使用仍会自动
下载运行环境依赖(PyTorch 等),随后自动下载 `DA3 Small` 和 `BEN2 Base`。其他 AI 模型可在
扩展不内置 uv 或 AI 运行库。首次使用会先下载并校验固定版本的 uv,随后自动安装轻量的 ONNX Runtime 环境,再下载 `MoGe-2 ViT-S Normal` 和
`BEN2 Base`;不安装 PyTorch,也不下载各模型的 Python 源码包。其他 AI 模型可在
`Models` panel 中按需下载,请保持网络畅通,首次安装需要一些时间。
安装时 Environment、MoGe-2 和 BEN2 各自使用独立的 0–100% 进度;Environment 这一轮内部依次完成 uv、Python 环境和 packages。进入后续模型时进度归零,但不显示 `2/2` 等阶段编号。模型源发生瞬时错误时会切换到另一来源继续下载。
任一必需模型在两个来源都下载失败时,`Install Environment` 会明确报错;Environment 本身仍会保留。
选中 Image Empty 后,右键 `SnapForm` 子菜单会在缺少 `MoGe-2 ViT-S Normal` 或 `BEN2 Base`
时显示 `Download Required Models`,一次补齐缺失项,不会重复下载已经就绪的模型。
必需模型补齐前,菜单中的 `Convert to Depth Plane` 会保持禁用。
这些 UI 状态直接检查 Storage Root 的 `models/` 文件,不会仅因打开菜单或面板而启动 Job Server。
## 快速上手
@@ -44,40 +50,52 @@ DA3、MoGe-2、utils3d、BEN2 源码包和 uv 已内置在扩展里,安装时
### 转换为透视网格
选中单张图片的 Image Empty,右键打开 `SnapForm` 子菜单并选择 `Convert to Surface`。SnapForm 会按 DA3
相机内参与深度逐像素反投影,并把生成网格匹配到原 Empty 的可见尺寸、Offset 与 Transform。DA3
confidence 与边拉伸结果会保存为 Surface attributes,再由一个可复用的 Geometry Nodes
选中单张图片的 Image Empty,右键打开 `SnapForm` 子菜单并选择 `Convert to Surface`。可选择 MoGe-2
ViT-S、ViT-B、ViT-L 或 DA3 Small、Base,其中 MoGe-2 ViT-S 为默认模型。SnapForm 会按模型输出的
相机内参与三维点建立透视网格,并匹配原 Empty 的可见尺寸、Offset 与 Transform。MoGe-2 使用内置 NumPy
网格构建与深度断层 mask 生成干净拓扑;DA3 保留 confidence 与边拉伸剔除。两者再由独立的 Geometry Nodes
修改器完成剔面、快速重拓扑、Relax 与 UV 重投。开启 Remove Background 时,背景三角面会在转换期间
直接删除,不再交给 Geometry Nodes。匹配相机默认不创建;开启后会作为 Surface 的子级。
MoGe-2 的像素级 normal 会作为 Object Space normal 贴图接入 `O MoGe2 Surface Layer`,并与原图
Bump 合并;网格本身不写入 custom normal,因此重拓扑降低顶点数后仍能保留细腻的法线细节。
当前此功能仅支持单张图片。
转换对话框提供模型与 `Resolution Scale`、`Create Matching Camera`、`Remove Background`、`Unshaded`,
MoGe-2 ViT-S、ViT-B、ViT-L、DA3 Small、DA3 Base 与 BEN2 全部使用 ONNX。模型执行、相机与深度恢复、
点云重投影和网格生成均不依赖 PyTorch。Windows 使用 DirectML 加速,provider 不可用时会回退到 ONNX Runtime CPU,
并在任务状态中显示实际 provider。
转换对话框提供模型与 `Create Matching Camera`、`Remove Background`、`Unshaded`;DA3 显示
`Resolution Scale`,MoGe-2 显示 `Resolution Level`,
以及分割线下方的 `Keep Original`。
界面会显示计算所得的实际 Process Res。剔除和重拓扑参数直接在
Geometry Nodes 修改器中调整;`Unshaded` 可使用与 Depth Plane 相同的 Emission 材质。普通材质的
Bump Scale 会在材质组内乘以 `0.001`,并按透视表面的深度空间缩放。Surface attributes 统一使用 `o_` 前缀;Service 准备过程显示
Bump Scale 会在材质组内乘以 `0.001`,并按透视表面的深度空间缩放。Surface attributes 统一使用 `o_` 前缀;Server 准备过程显示
`Preparing surface data n/6`,可看到当前阶段与总步骤。
### 视频与序列帧
SnapForm 支持视频(mp4、mkv、mov 等)和序列帧,会逐帧处理并生成逐帧对应的立体面片。处理范围默认
跟随 Empty 上设置的 `Image Frames` 与 `Offset`,不需要额外配置。
跟随 Empty 上设置的 `Image Frames` 与 `Offset`,不需要额外配置。DA3 的转换任务复用同一个 ONNX
Session 逐帧推理,显存峰值不随视频帧数平方增长;Debug 多帧输出仍使用多视图联合推理。
## 任务结果保存在哪里
任务结果保存在 `~/.snapform/jobs/` 下,每个任务一个文件夹,名称统一为
`动作-对象_日期-时间-微秒`。Packed Image 和序列帧的输入副本会直接归档到对应任务根目录,
不再使用公共 `inputs/` 文件夹;导出产物也尽量平铺。视频或序列帧生成的面片会引用任务文件夹里的
序列帧,请保留对应文件夹,不要随意删除。
任务结果保存在 Runtime Storage Root 的 `jobs/` 下,默认是 `~/.snapform/jobs/`。Job Server 接受任务时自动生成 opaque Job ID,并创建同名文件夹;Blender 不预先申请 ID 或目录。Packed Image 和序列帧的临时输入由 Blender 在任务结束后清理,Server 产物直接写入该 Job 目录。视频或序列帧生成的面片会引用其中的序列帧,请保留对应文件夹,不要随意删除。
## 常见问题
- 首次使用需要下载模型,如果下载慢或失败,稍后在 `Models` panel 里重试即可
- 安装 Environment 时提示下载超时:新版会自动探测并切换到镜像、失败时自动重试一次;如果仍然
超时,在扩展偏好设置的 `PyPI Index` 中填入镜像地址(如清华)后重新安装即可
- 从 0.12.0–0.12.2 升级到 0.12.3 后,请重新执行一次 `Install Environment`,让 Storage Root 中的 Environment 更新 BlendJob 运行依赖
- 首次使用需要下载模型,如果下载慢或失败,可在 Image Empty 的 `SnapForm` 右键菜单点击
`Download Required Models`,也可在 `Models` panel 里逐项重试
- 安装 Environment 时 uv 下载失败会自动尝试 OmooLab 镜像;Python packages 需要指定镜像时设置
`UV_INDEX_URL` 后重新安装
- 默认模型下载会在 Hugging Face 与 OmooLab 镜像之间自动回退;某一来源暂时断线不会阻止后续模型尝试另一来源
- 模型直链建连超过 10 秒,或连接后连续 60 秒没有收到数据,会切换到下一来源;不限制总下载时长
- 找不到右键入口时,请确认选中了 Empty,且显示方式为 Image
- 动画面片不动:检查 Empty 的 `Image Frames` 是否大于 1,并确认播放场景帧
## 技术文档
面向开发者的技术内容见 [docs/technical.md](docs/technical.md)。
Blender 侧只负责界面、任务提交与结果导入,AI 与文件输出由独立 Job Server 执行。详细功能见
[文档索引](docs/index.md),模块边界、调用流程、项目结构与开发命令见
[程序设计、开发与许可证](docs/development.md)。

67
docs/depth-plane.md Normal file
View File

@@ -0,0 +1,67 @@
# Depth Plane
## Image Empty 转深度面片
1. 在 3D View 选中 `Object Type = Empty`、`Display As = Image` 的对象。
2. 右键打开 `SnapForm` 子菜单并选择 `Convert to Depth Plane`,通过 `Depth from` 选择深度来源:
- `DA3`:显示 Model 和 Resolution Scale。ONNX 模型使用 camera decoder 恢复相机。Process Res 先以原图长边乘以
Resolution Scale,再钳制到 `252–1008`,最后对齐到最接近的 `14` 倍数。Resolution Scale 以百分比
显示,范围为 `1–100%`,默认 `100%`;界面只在标签中实时显示最终 Process Res。
转换导出 `mini_npz`,使用前需要在
`Models` panel 下载所选模型,并固定使用 `process_res_method="upper_bound_resize"`;设备和参考视图策略
继续使用场景设置。Model 下拉菜单只列出已经下载的 DA3 模型;若没有可用模型,对话框会引导到
`Models` panel 下载。
- `MoGe-2`:使用所选模型输出 metric depth、normal、validity mask 与 normalized intrinsics。Depth Plane 不保存体积较大的 points;Surface 会在 AI 进程内用内置 NumPy 网格构建生成 `mesh.npz`。Debug 始终保留原始 `results.npz`,并按 Output Format 单独生成 Maps、纹理 GLB 或彩色 PLY。
- `Original`:直接把原图作为深度贴图,不显示估算参数。仅在开启 `Remove Background` 时运行 BEN2
取得前景 Alpha;视频或序列帧按需同时抽取原始帧与前景序列,把原始帧作为 `.depth` 序列。
`MoGe-2 ViT-S` 是默认深度来源。选择 `DA3` 时,其 Model 与 Resolution Scale
参数会集中显示在独立边框中。`Remove Background` 与 `Unshaded` 直接显示,不再使用 Options 折叠;
`Keep Original` 默认关闭,并在分割线下方保持为最后一项。仅当开启 `Remove Background` 时才需要
`BEN2 Base`,并把 `.foreground` 的 Alpha 作为独立
Mask,不在 Server 中改写深度值。保存 `.foreground` 时保留 BEN2 Alpha,但 RGB 使用未经修改的原始输入;
因此材质直接用 `.foreground` 同时提供颜色和 Mask,不再额外加载原图 Image Texture。
Image Empty 引用视频或序列帧时,DA3 与 MoGe-2 会按 Empty 的 Image Frames 数值逐帧估计
深度(未设置时为完整帧数,显式设为 1 时只处理 1 帧),起始帧跟随 Empty 的 Offset,与背景蒙版范围一致;
DA3 会复用同一个 ONNX Session 对每帧独立推理,再合并为原有序列输出,避免多帧 Attention 的平方级显存占用。
该行为只用于 Depth Plane 转换;Surface 仅支持单张图片,Debug 多帧输出仍联合推理。
Offset 超出视频长度时同样把窗口收进视频内。材质节点与置换贴图的 Start 保持与原 Empty
一致;Original 模式的视频或序列帧使用 `frame.0001.jpg` 等原图序列作为颜色与深度,
`foreground.0001.png` 只提供前景 Alpha。
DA3 的深度仍以 EXR 序列帧(`depth.0001.exr` 起)保存在 job 目录根下;MoGe-2 分别使用 `moge_depth`、`moge_mask` 与 `moge_normal` EXR 序列。
`Unshaded`、`Remove Background` 与 `Keep Original` 均默认关闭,因此默认删除原对象。
3. DA3 或 MoGe-2 完成后会自动创建面片。DA3 面片几何先由 `mini_npz` 的相机内参与深度确定,再通过对象
Transform 匹配原 Image Empty 的可见尺寸、位置、旋转和 Offset。Original 模式直接使用源 Empty
的可见尺寸、Offset 与 Transform。DA3 会把用于匹配画面尺寸的缩放预乘到 `.depth` 数值,
XY 缩放烘焙到 Mesh,Z 不参与平面匹配;最终对象 Scale 为 `(1, 1, 1)`,视觉深度与旧逻辑一致。
关闭 `Keep Original` 时,成功生成后删除源 Empty。生成面片 Object 使用源 Empty 的原名;开启
`Keep Original` 时不处理
同名冲突,由 Blender 自动为生成面片追加编号。
4. Blender 只创建保留图像纵横比的最低密度基础网格,再把 `O DA3 Depth Plane` 作为 Geometry Nodes
修改器添加到对象。节点组与 `O DA3 Surface` 一同保存在 `O_SnapForm.blend`,关闭
`show_group_selector`,不为每个对象复制节点组。MoGe-2 使用单独的 `O MoGe2 Depth Plane`,只保留
网格细分、metric depth 位移、mask 删点和材质设置;材质使用 `O MoGe2 Depth Plane Layer`,直接接入
模型 normal,不携带 DA3 的材质置换与背景深度分支。DA3 修改器公开以下参数:
- `Displacement`:位于 Geometry 之后的第一项,菜单包含 `Geometry`(默认)与 `Material`。`Geometry`
在节点组内沿法线修改网格,`Material` 保持网格平面并启用材质置换;
- `Subdivision`:位于第二项,默认 `5`。节点读取基础平面的 Bounding Box,短边分段数为 `2^Subdivision`,
长边按宽高比计算并四舍五入;例如 16:9 图片在等级 `5` 时生成 `57×32` 个面;
- `Background Depth`:默认 `-1`,允许正负值,Structure Type 为 `Single`,直接显示且不再放在 Panel 中。前景 Mask 外的结构或
材质置换使用该值;是否存在背景只取决于转换所得色彩贴图的 `Alpha > 0.5` 二值结果;
- `Depth Scale`:默认 `1`,允许正负值,Structure Type 为 `Single`,同时控制 Geometry 与 Material
两条置换路径。
Grid 重建后通过隐藏的 Material 输入重新赋予原图材质,并对所有面执行 Shade Smooth。
材质使用显式指向 `UVMap` 的 UV Map 节点连接颜色、前景 Mask 与深度 Image Texture,不依赖活动 UV。
前景深度先与 `Background Depth` 取 Maximum,再按色彩贴图 Alpha 与背景深度混合,
深度分支在混合前使用 `Alpha > 0.5` 转为二值 Mask,避免半透明前景被背景深度拉扯;
材质表面 Alpha 同样使用二值 Mask。材质置换使用相同的二值和钳制顺序。
节点组还会把 `o_material_displacement` 与 `o_background_depth` 写入点域,材质通过
`Geometry` 类型的 Attribute 节点读取。点域会为顶点保存重复值,但能在应用修改器后继续保留参数;
相比自适应网格本身,其占用很小。Depth Image 与 Foreground Image 是修改器隐藏数据输入,
由导入流程自动设置。
视频与序列帧还会为 Depth 与 Foreground 共同写入隐藏的 `Frame`、`Frame Start`、`Frames` 与 `Cyclic`。
节点组内部不再使用 Scene Time。只有原 Image Empty 是视频或序列时,导入流程才直接给修改器的 `Frame`
socket 添加 Scripted Expression Driver `frame`;静态图保持默认值 `1`,不创建 Driver。导入内容已经
是截取后的帧,因此 Offset 在节点组内恒定为 `0`。
节点组根据 `Frame` 计算从 `1` 开始的 Image Texture Frame:场景帧与 Start 都为 `1` 时读取图片第 `1` 帧;非循环输入
钳制到有效范围;循环输入使用 Floored Modulo,让 `Frame Start` 前后的负、正相对帧都映射到
`1…Frames`,因此开始帧之前也会循环播放。Geometry 置换会像材质 Image Texture 一样随场景帧刷新。

115
docs/development.md Normal file
View File

@@ -0,0 +1,115 @@
# 程序设计、开发与许可证
## 程序设计
SnapForm 分成 Blender 主进程与独立 Job Server 两侧。Blender 侧只负责界面、任务参数和 `.blend` 数据修改;模型下载、媒体准备、ONNX 推理和文件输出都在 Python 3.12 Environment 中运行。两侧由随扩展分发的 BlendJob wheel 连接,避免把 FastAPI、ONNX Runtime 等计算依赖载入 Blender Python。
```mermaid
flowchart LR
User["Image Empty / Debug Panel"] --> Operator["Blender Operator"]
Operator --> Runtime["JobRuntime"]
Runtime --> Server["JobServer FIFO Queue"]
Server --> Handler["SnapForm Job Handler"]
Handler --> Model["ModelManager / ONNX"]
Handler --> JobFiles["Job Directory"]
JobFiles --> Response["Operator response"]
Response --> Scene["Blender Scene"]
```
核心边界如下:
- `src/snapform/__init__.py` 是注册入口,集中注册 Blender 类型、`Scene.snapform_settings`、右键菜单与 Runtime
- `panel.py`、`preferences.py`、`properties.py` 只描述 UI 和 Blender 属性,不直接导入 Server 模型实现
- `operators.py` 将 Blender 状态转换为普通字典参数;Job 完成后在主线程导入图片、网格、材质和相机
- `blender/depth_plane.py` 与 `blender/surface.py` 实现 Image Empty 输入检查、几何构建和资产挂载
- `job_runtime.py` 以入口字符串声明独立 Server,并配置 Environment、Storage Root 与首次安装后的推荐模型下载
- `server/__init__.py` 创建 `JobServer`、注册公开 Job,并把 `ModelManager` 注册为 Resource
- `server/media.py`、`background.py`、`da3.py`、`moge2.py` 负责业务流程;`onnx_*.py` 只处理模型输入、推理和输出恢复
- 根目录 `model_catalog.py` 是 Blender UI 与独立 Server 共用的纯数据声明;`server/model_download.py`、`server/model_manager.py` 分别负责下载和 Session 生命周期
- `server/geometry.py` 生成 NPZ、预览、点云和纹理网格;所有任务产物写入 Runtime 分配的 opaque-ID Job 目录
功能细节继续按主题维护:[Depth Plane](depth-plane.md)、[Surface](surface.md)、[输出格式](outputs.md) 和 [Job Server](server.md)。
## 项目结构
```text
.
├── AGENTS.md
├── README.md
├── pack.py
├── sync_models.py
├── docs/
├── src/
│ └── snapform/
│ ├── __init__.py
│ ├── blender_manifest.toml
│ ├── job_runtime.py
│ ├── operators.py
│ ├── panel.py
│ ├── preferences.py
│ ├── properties.py
│ ├── assets/
│ │ └── O_SnapForm.blend
│ ├── blender/
│ │ ├── depth_plane.py
│ │ └── surface.py
│ ├── model_catalog.py
│ ├── server/
│ │ ├── __init__.py
│ │ ├── background.py
│ │ ├── da3.py
│ │ ├── geometry.py
│ │ ├── media.py
│ │ ├── model_download.py
│ │ ├── model_manager.py
│ │ ├── moge2.py
│ │ ├── onnx_ben2.py
│ │ ├── onnx_da3.py
│ │ ├── onnx_moge2.py
│ │ └── onnx_runtime.py
│ └── wheels/
│ └── blendjob-0.1.6-py3-none-any.whl
├── tests/
└── tools/
```
## 常见命令
在项目根目录使用 Git Bash:
```bash
uv sync
uv run pytest
uv run pytest tests/test_addon.py tests/test_manifest.py tests/test_pack.py -v
uv run python pack.py
```
`pack.py` 根据 `blender_manifest.toml` 分别生成 Windows、macOS 和 Linux 扩展包,例如 `dist/SnapForm.v0.12.3.windows-x64.zip`。从 PyPI 下载并更换 `src/snapform/wheels/` 中随扩展分发的 wheel 后,先刷新锁文件再测试:
```bash
uv lock --refresh
uv run pytest
```
## 资产构建与验证
`tools/` 脚本依赖 `bpy`,必须由 Blender 执行。设置本机 Blender 可执行文件后,可以重建或验证 `O_SnapForm.blend`:
```bash
BLENDER_BIN="/c/Program Files/Blender Foundation/Blender 4.5/blender.exe"
"$BLENDER_BIN" --background --python tools/build_geometry_nodes_asset.py
"$BLENDER_BIN" --background --python tools/validate_depth_plane_asset.py
"$BLENDER_BIN" --background --python tools/validate_geometry_nodes_asset.py
"$BLENDER_BIN" --background --python tools/validate_moge_depth_plane.py
"$BLENDER_BIN" --background --python tools/validate_moge_surface.py
```
只改 Python、文档或打包逻辑时不需要重建 `.blend` 资产;修改节点组、材质组或其接口时,应运行相应资产验证脚本,并再次运行完整测试。
## 上游与许可证
- Depth Anything 3 代码采用 Apache 2.0
- BEN2 ONNX 模型采用其仓库声明的 Apache 2.0 License
- SnapForm Extension 代码采用 GPL-3.0-or-later
- 模型权重许可证由所选 Hugging Face 模型仓库决定,Extension 下载前会在界面显示

110
docs/features.md Normal file
View File

@@ -0,0 +1,110 @@
# 功能与使用
本文档面向开发者和需要了解内部实现细节的用户,从原 README 迁移而来。普通用户请阅读项目根目录的
`README.md`。
`SnapForm` 是一套从图片快速构建空间结构的 Blender Extension。目前以
[Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3)
作为首个 AI 能力,在独立 Environment 中运行 DA3,可将官方导出结果保存在独立 job 目录,
也可把 Image Empty 转换成带几何与材质置换的立体面片。MoGe-2 作为独立深度来源,保留其 metric depth、normal、mask 与 intrinsics 输出,不复用 DA3 Prediction 后处理。Debug 面板还可运行
[BEN2](https://github.com/PramaLLC/BEN2) 背景剔除功能测试。
## 当前功能
- 使用 Blender Extension manifest,可通过 `Install from Disk` 安装
- 在独立 Environment 中通过常驻 Server 运行 ONNX Runtime;MoGe-2、DA3 与 BEN2 的模型执行及后处理完全不使用 PyTorch,避免污染 Blender Python 进程
- Server 的深度模型缓存只保留 DA3 或 MoGe-2 其中一个,切换时主动释放前一个;BEN2 使用独立缓存,
可与当前深度模型共存
- Blender 使用内置 `urllib.request` 访问仅监听 `127.0.0.1` 的 Server;连接由随机端口与 Instance ID 归属当前 Runtime
- 常驻 Server 复用 ONNX Runtime session、BEN2 与最近使用的深度模型,连续执行时不再重复加载
- 每个 Blender 实例持有自己的 Job Server;`Job Server` 面板显示状态与模型缓存,并提供安装、启动、清理、重启和停止控制
- Environment、模型和任务结果默认保存在 `~/.snapform`,可由多个 Blender 版本共享
- 当前仅启用 FP32 DA3 Small 和 Base;它们不随首次安装下载,Giant 1.1 与 Nested Giant Large 1.1 暂停使用
- 支持 BEN2 Base 单图背景剔除测试,输出透明 PNG 与推理耗时
- 支持单张 JPG、JPEG、PNG、WEBP 图片或常见视频文件输入
- 视频按顺序截取前若干帧,可设置参与重建的最大帧数
- `Models` panel 默认折叠,展开后可逐项查看和下载所有可用模型
- Environment 已安装时不额外占用面板空间;模型列表每个模型只占一行
- `Debug` 默认折叠,内部的 `DA3`、`MoGe-2` 与 `BEN2` 也分别默认折叠
- 使用一个输出类型下拉菜单和一个 `Generate Output` 按钮
- DA3 与 MoGe-2 Debug 都先选择单一 Output Format,只显示当前格式的专属参数;GLB 与 PLY 不会在一次测试中同时生成
- Debug 的模型选项按格式过滤;PLY 只提供两个带 Gaussian head 的 Giant 模型
- Debug 与 Convert to Depth Plane 的模型选项直接显示下载状态;未下载模型带
`(Missing)`,已下载模型保持原名
- 分别调用 DA3 的 `glb`、`gs_ply`、`depth_vis` 和 `mini_npz` exporter
- `Textured Surface (GLB)` 直接使用 DA3 Prediction 中的深度、置信度和相机参数,把相邻像素
组成三角面并写入 `mesh.glb`;每个输入帧生成一张独立纹理网格,不做多视图融合
- Image Empty 右键菜单可直接运行 DA3,根据 `mini_npz` 深度生成面片
- Image Empty 右键菜单可直接运行 BEN2,保留原图并换用新建的透明背景图片
- 缺少必需的 `MoGe-2 ViT-S Normal` 或 `BEN2 Base` 时,Image Empty 的 `SnapForm`
右键子菜单显示 `Download Required Models`;单次任务只下载缺失项,并在补齐前禁用
`Convert to Depth Plane`
- 菜单、模型面板和转换对话框直接检查 Storage Root 的 `models/` 必需文件,显示模型状态时不会启动 Job Server
- Image Empty 支持视频与序列帧输入:`Convert to Depth Plane` 逐帧估计深度,
`Remove Background` 逐帧抠图,入口保持不变。DA3 转换时每帧独立执行 ONNX 推理并复用模型 Session;
Debug 多帧任务继续联合推理,以保留跨视图相机与深度关系
- 视频生成的序列帧以外部文件引用保留在 job 目录(Blender 无法整体 Pack 序列帧);
Geometry Nodes 与材质 Image Texture 均可按场景帧播放序列帧
- 使用浮点深度贴图、`O DA3 Depth Plane` Geometry Nodes 和原图 Bump 构建立体面片
- 转换对话框选择推理模式、处理分辨率与背景移除;细分和置换方式在 Geometry Nodes 修改器中调整
- Debug 的 DA3 输出或 BEN2 背景剔除完成后只打开对应 job 文件夹,不修改当前场景
- 完整保留每次任务生成的所有官方文件
- 安装和模型下载使用后台子进程;推理提交给常驻 Server,状态栏实时显示阶段、百分比进度条与取消按钮
- 后台任务运行时,Install、Download 和 Export 按钮保持原位置并变灰,任务完成后恢复
## 使用
1. 运行 `uv run python pack.py`,分别构建 `windows-x64`、`macos-arm64`、
`linux-x64` 三个平台的 Extension ZIP(如 `SnapForm.v0.12.3.macos-arm64.zip`)。
安装包不携带 uv 或 AI 项目源码;首次安装 Environment 时下载固定版本 uv,官方源建连超过 3 秒后回退 OmooLab 镜像,并校验固定 SHA-256。
2. 在 Blender 的 `Preferences > Extensions` 中选择 `Install from Disk`。
3. 在 Extension Preferences 中确认 Storage Root 与 Device。Device 是正式功能
(Image Empty 右键菜单)使用的推理设备,默认取本机最快的 DirectML/CoreML provider,否则 CPU,且只列出
本机支持的设备。
4. 在 `Job Server` 点击 `Install Environment`。首次安装会下载 Python、ONNX Runtime、FastAPI、Uvicorn;
uv 由 Job Runtime 的 Environment 安装器下载并校验,DA3、BEN2 与 MoGe 不安装源码包,
并在 Environment 安装后自动下载 `MoGe-2 ViT-S Normal` 和 FP16 `BEN2 Base` 推荐模型。首次安装
依次显示 `Environment`、`MoGe-2 ViT-S Normal` 与 `BEN2 Base`;每项都独立从 0 走到 100%,
进入下一项时进度条归零,模型 post-install 不显示 `2/2` 等阶段编号。uv、Python Environment 与 packages 合并在第一轮,
两个默认模型由普通的 `post_install(runtime)` 发起正式 Server Job。
安装完成后直接写入 Environment Manifest,不再额外导入全部组件进行验证;`DA3 Base` 等其他模型可稍后在 `Models` 面板按需下载。
依赖安装显示当前安装输出;模型下载显示速度,例如
`Downloading DA3 Small 14.2 MB/s`;镜像来源显示为
`Downloading DA3 Small (mirror) 14.2 MB/s`。下载按字节连续推进,重试或切换镜像时进度不会倒退。
Environment 安装阶段与 `Models` 面板单独下载使用相同的模型名称格式,不会改写为通用的 `Downloading packages`。
Environment 安装后即使模型下载中断也仍会保留,但 `Install Environment` 会以失败结束并显示模型下载错误。
可稍后在 Image Empty 的 `SnapForm` 右键菜单用 `Download Required Models` 一次补齐缺失的
MoGe-2 ViT-S 与 BEN2,也可在 `Models` 面板单独补下。
安装不做启动网络预检;每个模型都先尝试 Hugging Face,失败后尝试 OmooLab R2,两者都失败时当前 Job 失败。
下载源不写入持久状态;需要手动指定依赖镜像时可通过
`UV_INDEX_URL` 环境变量覆盖。
5. 在 3D View 按 `N`,打开 `SnapForm` 页签。
6. 如需其他模型,展开 `Models` panel,找到模型并点击该行的 `Download`。
7. 展开默认收起的 `Debug`,选择 `Output Type` 和本次执行使用的模型。
8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数,然后点击 `Generate Output`。
9. 测试背景剔除时,确认 `Models` panel 中的 `BEN2 Base` 已就绪,然后在 `Debug` 的
`BEN2` 区域选择单张图片并点击 `Remove Background`。
完成后会打开 job 文件夹,其中的 `foreground.png` 带透明 Alpha;状态栏显示
BEN2 推理、模型加载与任务总耗时。当前测试固定关闭较慢的 foreground refinement。
## Image Empty 去背景
1. 确认 `Models` panel 中的 `BEN2 Base` 已就绪;首次安装 Environment 时会默认下载。
2. 在 3D View 选中 Image Empty,右键打开 `SnapForm` 子菜单并选择 `Remove Background`。
3. 完成后以 Fake User 保留原 Image datablock,新建并 Pack 一张透明 PNG 到当前 `.blend`,然后只把
当前 Empty 的 `data` 换成新图;Empty 的 Transform、显示尺寸、Offset 及其他设置保持不变。
4. 新 Image 使用原图去掉格式后再追加 `.foreground` 的名称。例如 `img.png` 生成
`img.foreground`,`img.png.001` 生成 `img.001.foreground`。已有同名 datablock 时由
Blender 自动追加编号。
原图已 Pack 时会先写出临时 PNG 交给 BEN2,任务结束后删除临时输入;原 Image 仍保持 Pack
状态和原始内容。BEN2 的 `foreground.png` 也会保留在对应 job 目录中。
Image Empty 引用视频或序列帧时,Remove Background 直接执行,默认取 Empty 的 Image Frames
数值(未设置时为完整帧数,显式设为 1 时只处理 1 帧),起始帧跟随 Empty 的 Offset;若 Offset 加 Frames 超出视频长度,
会把窗口整体收进视频内(保证实际处理 Frames 帧数),Offset 过小时从第 1 帧开始。输出
`foreground.0001.png` 等透明序列帧(帧号从 0001 开始,与 Blender 场景帧一一对应),并把
源帧以 `frame.0001.jpg` 等命名复制到 job 目录根下供材质颜色导入。替换 Empty 的 Image 为
加载这些帧的 SEQUENCE Image,从原 Empty 的 Start 位置开始播放(Start 原样保留),
Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因此保持外部引用 job 目录;
请保留对应 job 目录,否则序列会失效。

10
docs/index.md Normal file
View File

@@ -0,0 +1,10 @@
# SnapForm 文档
技术内容已按主题拆分:
- [功能与使用](features.md):能力概览、安装使用和 Image Empty 去背景
- [Depth Plane](depth-plane.md):深度面片的数据、节点与材质流程
- [Surface](surface.md):透视网格、Geometry Nodes 后处理与材质法线
- [输出格式](outputs.md):Debug 输出类型、参数和任务产物
- [Job Server](server.md):Environment、模型缓存、任务目录与运行机制
- [程序设计、开发与许可证](development.md):模块边界、调用流程、项目结构、常用命令与许可证

29
docs/outputs.md Normal file
View File

@@ -0,0 +1,29 @@
# 输出格式
## DA3 ONNX
`Output Type` 提供以下结果:
- Point Cloud (GLB):由 depth、confidence、相机参数和原图生成彩色点云
- Depth Preview (Images):保存深度着色预览
- Depth Data (NPZ):保存 depth、confidence、extrinsics、intrinsics 与处理后的图像
- Textured Surface (GLB):SnapForm 根据 Prediction 构建纹理网格
DA3 Small 与 Base 的 ONNX 图不包含 Gaussian head,因此不提供 Gaussian Splats。默认推理参数为:
- `process_res=504`
- `process_res_method="upper_bound_resize"`
- `ref_view_strategy="saddle_balanced"`
- `conf_thresh_percentile=40.0`
- `num_max_points=1_000_000`
- `show_cameras=True`
相机外参与内参由模型的 camera decoder 输出恢复,不运行 ray-pose 求解器。
## MoGe-2 ONNX
MoGe-2 Debug 可保存原始 NPZ、depth/normal/mask maps、纹理 GLB 或彩色 PLY。Surface 输出由模型 points、mask 与 depth edge 在内部构建三角网格。
## BEN2 ONNX
BEN2 输出 `foreground.png`,RGB 保留原始输入,Alpha 来自 ONNX mask 并恢复到输入分辨率。

123
docs/server.md Normal file
View File

@@ -0,0 +1,123 @@
# Job Server
## 运行架构
SnapForm 在独立的 Python 3.12 Environment 中运行 HTTP Job Server,Blender 只通过标准库 HTTP Client 提交和轮询任务,不把计算依赖载入 Blender Python。Server 只监听 `127.0.0.1`,由外置 BlendJob wheel 提供的 `ServerController` 保存随机端口与 Instance ID;专属本地进程协议不使用额外 Token 鉴权。
每个 Blender 实例启动并持有一个 SnapForm 专属 Server,不发现或复用其它 Blender、其它插件启动的本地 Server。多个 Blender 可以对应多个进程,但模型、Environment 与任务产物仍可共享同一个 Storage Root。
Environment、模型和任务结果默认保存在 `~/.snapform`:
```text
~/.snapform/
├── .venv/
├── environment.json
├── models/
├── jobs/
├── manifest.json
├── install.log
├── install-status.json # 仅在安装期间存在
└── server.log
```
不同 Blender 版本可共享此目录。也可以在 Extension Preferences 的 `Storage Root` 中修改位置。连接端口与 Instance ID 只保存在启动它们的 Blender 进程内存中。
Blender 侧的通用 HTTP Runtime 由 `src/snapform/wheels/` 中随扩展分发的 BlendJob package 提供;独立 Environment 由 Runtime 从 PyPI 自动安装相同版本。`JobServer` 包装 FastAPI 并提供 Job、FIFO Queue 与 Resource API,`JobClient` 是无第三方依赖的 Blender 侧访问器。Blender 侧只把 `server/__init__.py:server` 入口字符串交给 Runtime,不导入 `server` package;独立进程启动后才取得其中的 `server` 实例、注册全部公开 Job,并创建 `model_manager` Resource。
SnapForm 根 package 不再有 `client.py`、`environment.py`、`storage.py` 或 `models.py`。这些文件会制造并不存在的业务边界:
- Storage Root、`.venv`、Manifest、安装状态与日志路径由 `JobRuntime` 唯一管理;
- `InstallEnvironment` 由 `runtime.operator_classes()` 提供,Python 与 Job 自身的额外 packages 直接声明在传给 Runtime 的 `environment` 字典中;BlendJob 不需要重复声明;
- 模型 catalog 由 `model_catalog.py` 统一声明;下载和 ONNX Session 属于 Server,分别位于 `model_download.py` 与 `model_manager.py`;
- Blender UI 复用同一 catalog,直接检查 Storage Root 下的模型必需文件,不为显示下载状态启动 Server;已加载 Session 状态仍读取现有 Server Snapshot。
BlendJob 的 `JobClient` 只是无依赖 HTTP transport,不是 SnapForm 业务抽象,也不向 Operator 暴露 `JobRequest`。
SnapForm 的业务 Operator 直接提交参数字典,Job Type 一一对应:
- `download_model` → `download-model`
- `download_required_models` → `download-required-models`
- `remove_background` → `remove-background`
- `extract_frames` → `extract-frames`
- `generate_da3_depth` → `generate-da3-depth`
- `generate_moge2_depth` → `generate-moge2-depth`
- `generate_da3_surface` → `generate-da3-surface`
- `generate_moge2_surface` → `generate-moge2-surface`
- `debug_da3` → `debug-da3`
- `debug_moge2` → `debug-moge2`
- `debug_ben2` → `debug-ben2`
DA3 与 MoGe-2 的生产任务不再合并成带 Backend 参数的通用请求。每个 Operator 的 `request(context)` 直接构造自己的有效参数,Debug 任务也与生产任务完全分开。
Server 端没有 `execute_*`、`*_result` 或 CLI worker 中转。全部 `@server.job(...)` 都集中在 `server/__init__.py`:简单任务直接实现,复杂任务调用按领域拆分的 `media.py`、`background.py`、`moge2.py` 或 `da3.py`;共享的纯几何计算位于 `geometry.py`。模型下载由 `ModelManager` 单向调用 `model_download.py`,不存在循环 import。
## ONNX-only Environment
扩展不内置 `uv` 可执行文件或 AI 运行库。`Install Environment` 首先下载固定版本的 uv,校验 SHA-256 后解压到共享 Storage Root,再用它创建环境并安装:
- ONNX Runtime 1.24.4;Windows 使用 `onnxruntime-directml`
- NumPy 2.4.2、Pillow 12.1.1、PyAV 17.0.0、trimesh 4.11.2
- BlendJob 0.1.6、FastAPI 0.139.2、Uvicorn 0.51.0、huggingface-hub 1.4.1
运行环境不安装 PyTorch、TorchVision、xFormers,也不安装 DA3、MoGe、BEN2 或 utils3d 的 Python 源码包。模型推理和 SnapForm 所需的相机、点云、网格后处理均由 ONNX Runtime、NumPy 与内部实现完成。
Windows 当前发行环境提供 DirectML 与 CPU;`Auto` 优先使用 `DmlExecutionProvider`。显式指定不可用设备时会安全回退到 CPU。其他平台安装标准 ONNX Runtime,并按可用 provider 选择 CoreML、OpenVINO 或 CPU。
`environment.json` 保存规范化后的声明,`manifest.json` 保存该声明、BlendJob 版本与基础 Server packages 的 Hash。Python、packages、平台 packages、BlendJob 版本或 Environment Schema 改变后需要重新执行 `Install Environment`,不再维护手工 Manifest Revision。
## 模型
所有正式功能都使用 ONNX:
- MoGe-2 ViT-S、ViT-B、ViT-L Normal
- DA3 Small、DA3 Base
- BEN2 Base
首次安装在 `post_install(runtime)` 中自动下载 `MoGe-2 ViT-S Normal` 与 FP16 `BEN2 Base`。它们和 `Models` panel 的后续下载一样,都是 `download-model` Job,与推理共用队列、HTTP 进度和取消协议。模型按声明的目录名保存在 `~/.snapform/models/`,并保留仓库内部结构。声明了文件清单与校验值的模型先走 Hugging Face `resolve` 直链流式下载,失败后尝试 `https://models.omoolab.xyz/<repository-name>/<relative-path>`;两者都失败时当前 Job 直接失败,不记录或猜测后续下载源。模型直链的 TCP、代理 CONNECT 与 TLS 建连最长等待 10 秒;连接后连续 60 秒没有收到响应数据则视为当前来源失败并切换下一来源。该空闲超时不限制模型总下载时长。R2 文件会校验大小与 SHA-256。未声明清单的仓库仍使用 `huggingface_hub` 快照下载。可用 `HF_ENDPOINT` 指定 Hugging Face 镜像。
Image Empty 的右键菜单会读取 `model_manager` 状态;缺少 MoGe-2 ViT-S 或 BEN2 时显示
`Download Required Models`。对应的 `download-required-models` Job 在 Server 端再次检查就绪状态,
只下载缺失项,并把多个下载的进度合并为单次 0–100%。必需模型补齐前,右键菜单中的
`Convert to Depth Plane` 保持禁用。
`model_manager` Resource 在 Server 内持有模型 catalog、磁盘就绪状态和内存 Session。Blender UI 使用 catalog 声明的目录名与 `ready_patterns` 直接检查 `<storage-root>/models/`,因此展开模型面板、打开转换对话框或 Image Empty 右键菜单都不会仅为状态检测启动 Server。真正下载或推理时,Server 仍通过 `ModelManager.ready()` 重新验证文件。`ModelSessionCache` 保留 BEN2,并在 DA3 与 MoGe-2 之间只保留最近使用的一种 Depth Session。`GET /resources/model_manager` 继续返回 catalog、下载状态和当前加载状态;`POST /resources/model_manager/clear` 只允许在没有 Active 或 Queued Job 时执行。
DA3 ONNX 使用模型的 camera decoder 恢复外参与内参,不提供 ray-pose 参数。MoGe-2 的 metric depth、normal、mask、normalized intrinsics,以及 BEN2 的 Alpha,均在 NumPy 后处理中恢复。
## 安装网络与镜像
安装器直接开始工作,不做阻塞式网络预检。packages 由 uv 安装,可用 `UV_INDEX_URL` 显式指定镜像。Windows 系统代理会注入子进程;同时支持 `HTTP_PROXY`、`HTTPS_PROXY` 与 `ALL_PROXY`。
uv 默认从 Astral 的 GitHub Release 下载,失败后回退到 `https://mirror.omoolab.xyz/uv/<version>/`。两个来源使用相同的连续字节进度并共用固定 SHA-256 校验。下载压缩包只作临时文件,提取完成后删除。模型下载不经过 GitHub。
首次安装依次显示 Environment、MoGe-2 下载与 BEN2 下载,每项分别使用 0–100%。Environment 由 Job Runtime 依次完成 uv 下载、Python Environment 创建与 packages 安装,uv 不再占用独立一轮进度。两个模型下载由 SnapForm 的 `post_install(runtime)` 通过 `runtime.request()` 顺序发起;切换模型时进度归零,但不显示 `2/2` 等阶段编号。
任一 post-install 模型失败时,Environment 文件仍然保留,但安装 Operator 返回失败并显示聚合后的模型错误,
不再把带 `post_install_error` 的终态报告为成功。
## 生命周期
扩展启用后不启动 Server;第一次执行 Job 或点击 `Start` 时才为当前 Blender 启动专属实例。正常关闭或禁用扩展时直接停止自己启动的进程;Server 也监视 Blender 父进程 PID,父进程异常退出后会自行停止。
AI、帧提取和 Models panel 下载的 Job 进度保存在 Server 内存中。Handler 只使用 `context.progress(value, message)` 发布进度,终态由 JobServer 内部管理;SnapForm 不再维护独立的 `server/progress.py` 或 Job 状态 JSON。Blender 每 0.25 秒轮询 `GET /jobs/{job_id}`,任务目录不写 `status.json`。Operator 直接提交参数字典;Job 被接受时由 `JobServer` 自动生成 ID 并创建 `jobs/<opaque-job-id>/`,提交响应同时返回 `job_id` 与 `directory`。路径通过 `JobContext.directory` 传给 Handler,SnapForm Server 强制将输出写入该目录。
`job_runtime.py` 创建唯一的 `runtime = JobRuntime(...)`,以自身目录为 `entrypoint_root`,传入 `server:server` 相对入口字符串与显式 `snapform` namespace,并导出 `runtime.JobOperatorBase`。Runtime 独占 Storage Root、Environment 路径、安装进程、Server 控制和 Status Bar。Operator 未覆盖 `request(context)` 时会提交全部 Property;覆盖后返回值纯替换默认参数。可选的 `response(context, result)` 在 Blender 主线程直接完成结果导入。SnapForm 不再提供 Client 请求层、Job Mixin、Status Bar 或 Build 中转 Operator。
Environment 安装是 Server 启动前唯一的 Bootstrap 流程:`environment.json` 记录期望安装的环境声明,`install-status.json` 在安装期间回传进度并在任务收尾后删除,`install.log` 持久记录安装输出。Bootstrap 不创建 Job 目录。Environment 成功后执行普通的 `post_install(runtime)`;默认模型下载此时已经是正式 Server Job,会创建自己的 Job 目录,并使用与其它任务相同的进度和取消协议。
Server 使用标准库单 Worker FIFO Queue,全局同时只执行一个 Job,包括模型下载。Server 忙碌时新 Job 进入 `queued`,不再返回 409;排队任务可以在执行前取消。取消运行中任务不会重启 Server;流式模型下载会在分块检查点结束,ONNX Runtime 当前调用则在返回后的最近检查点结束。JobServer 最多保留 100 个已完成 Job 的内存状态,后续提交时清理更早记录。
Server 由 Environment 通过 `python -m blendjob.runner` 启动,避免 BlendJob 内部模块名遮蔽 Python 标准库。启动与崩溃信息写入 `~/.snapform/server.log`。
## 已知限制
DA3 ONNX 的注意力是朴素实现(`MatMul → Softmax → MatMul`),峰值显存与 `(帧数 × 每帧 patch 数)²` 成正比。迁移到 ONNX 前的 PyTorch 版使用 `scaled_dot_product_attention`(`XFORMERS_DISABLED=1`),按块计算注意力分数、不落地完整注意力矩阵,显存近似线性,因此视频能处理几十上百帧;ONNX 导出把 SDPA 回退成朴素注意力,多帧在中高分辨率下会显存爆炸(约 12 帧 @ 1008 即需 ~90 GB)。
Depth Plane 的视频、序列帧转换会复用同一个 ONNX Session 逐帧独立推理,再把结果合并到现有序列输出,因此峰值显存只取决于单帧分辨率。逐帧结果不保留 DA3 的跨视图尺度与相机位姿一致性,适用于当前逐帧动画结构。Surface 只支持单张图片;Debug 多帧输出仍使用联合推理,以保留多视图关系。
联合推理时,`onnx_da3.infer` 会在执行前按 `views × patches` 预估注意力张量,超过 4 GiB 直接报错并提示减少帧数或 Resolution Scale,避免 onnxruntime 报出难以定位的分配失败或 `UnicodeDecodeError`(DirectML 在中文 Windows 下报错信息为 GBK 编码,被 UTF-8 解码掩盖)。
待解决方向:
- 重新导出 DA3 ONNX,使用 ORT 的 `Attention` / `MultiHeadAttention` 贡献算子,或注入内存高效的注意力算子;
- 若未来需要统一坐标系的视频重建,可按小窗口分批推理,再对齐相机位姿与深度尺度。

190
docs/surface.md Normal file
View File

@@ -0,0 +1,190 @@
# Surface
## Image Empty 转透视网格
1. 在 3D View 选中仅包含一帧的 Image Empty,右键打开 `SnapForm` 子菜单并选择
`Convert to Surface`。该入口独立于
`Convert to Depth Plane`,当前不处理视频或序列帧。
模型列表包含三个 MoGe-2 Normal 模型和两个 DA3 模型,默认使用 MoGe-2 ViT-S Normal。DA3 的
Resolution Scale 与 Depth Plane 使用相同设计;MoGe-2 改为显示共享的 Resolution Level。
2. Server 请求 DA3 的 `depth`、`conf` 与 `intrinsics`。仅在开启 `Remove Background` 时运行 BEN2 取得
前景 Alpha,并在输出前删除包含背景顶点的三角面;关闭时不加载 BEN2。Depth 不做 smoothing,Server
反投影深度网格,建立源三角面,计算 confidence percentile 与 3×3 相对深度不连续度,并写入
基于原图像素坐标的源 `UVMap`。这些步骤只准备修改器的输入网格,不执行 Relax、拓扑修复、平滑、
小岛清理或最终 UV 重投影。运行时显示的六个 `Preparing surface data` 步骤均指这段 Server 端源数据准备。
3. Blender 把勾选 `Usage > Modifier` 的 `O DA3 Surface` 节点组直接作为 Geometry Nodes 修改器
添加到 Surface,不另设修改器名称。节点组内没有子节点组,依次完成:
- 修改器顶层依次显示 `Resolution`、`Relax Iterations`、`Smooth`;默认展开的 `Culling` Panel
依次包含 `Depth Continuity` 与 `Confidence`。前者直接读取面域 3×3 相对深度差,并使用指数映射:
默认 `50%` 对应官方 `rtol=0.04`,数值越大,内部阈值越小,剔除的断层面越多;
- 每轮 Remesh Iteration 依次执行 Connected Merge、循环拓扑修复与一次 Smooth,使新拓扑在进入下一轮
Merge 前先稳定;拓扑修复删除非流形面、单面拓扑和小面积 mesh islands;
- 默认 `Resolution` 为原始有效三角边长中位数乘以 `sqrt(10) * 2`。`sqrt(10)` 来自原先 90% 面数缩减
的二维边距估计,再乘以 2 为迭代合并留出更平滑、稳定的默认尺度;节点内部 Connected Merge 的
Distance 为 `Resolution * 0.5`;
- Smooth 采用 `O Smooth` 的边界检测和模糊法线权重,固定开放边界与薄面,只保留一个
`Smooth` 程度系数;节点组不再额外执行 Flip Faces;
- 从修改前的原始表面以 nearest-surface 重新采样 `UVMap`,最终明确写为 Corner 域 `2D Vector`。
MoGe-2 使用独立的 `O MoGe2 Surface`。Server 由内置 NumPy 网格构建函数
生成有限深度范围内的紧凑三角网格,并保留模型 mask、逐点 depth 及相邻点相对深度差。Geometry Nodes
按官方 `mask & ~depth_map_edge(depth, rtol=0.04)` 的含义,先由 `Mask` 删除 mask 外的面,再由
`Depth Continuity` 按同一份 3×3 邻域相对深度差删除断层面。DA3 与 MoGe-2 使用相同的属性、
名称、指数滑块和 FACE 删除逻辑,不再额外删除孤立点。之后才执行拓扑修复、Relax、孤岛清理与 UV 重投影。
节点组不提供 Confidence 或 Remesh 开关;Remesh 仍是节点处理流程的一部分。
4. 节点组随插件放在 `O_SnapForm.blend`。Blender 4.5 首次 Append 后复用同一 datablock;
Blender 5.0 以 linked ID 载入后调用 Pack,后续 Surface 继续复用。插件不会为每个 Surface 新建节点组,
也不会在添加后重命名 Geometry Nodes 修改器。节点组只启用 `Usage > Modifier`,不 Mark as Asset;
修改器关闭 `show_group_selector`。`Fix Topology` 位于默认折叠的 `Options` Panel。
节点图按 Culling、Remesh、Topology Fix、Island Cleanup 与 UV 分区摆放;各区就近使用独立的
Group Input,并隐藏没有连线的接口,避免跨越整个节点图的输入连线。
`O DA3 Surface` 与 `O MoGe2 Surface` 的 `Resolution` 默认值均为 `0.1`,所有输入 socket 的
Structure Type 均固定为 `Single`。
5. Server 准备过程固定显示 6 个编号阶段:验证 DA3 数据、反投影、准备 DA3 与可选 BEN2 数据、构建原始
源三角面、源 UV/attributes、写出网格。状态栏显示 `Preparing surface data n/6: ...`。
6. Server 在 job 根目录写出 `mesh.npz` 与 `texture.png`。NPZ 保存 camera-space `vertices`、
`faces`、`uv`、`intrinsics`、`image_size` 与有效深度中位数 `reference_depth`。Blender 读取这些
数据后直接创建 Surface 与原图材质,不需要 Open3D 或 OpenCV 进入 Blender Python 环境。
DA3 Surface 只保留 Geometry Nodes 后处理实际使用的 attributes:点域 `o_depth`、面域
`o_face_confidence` 与面域 `o_depth_discontinuity`。Remove Background 已在 Server
建面前直接删除背景三角面,不写入任何 background attribute。MoGe-2 的导入网格保留点域 `o_depth`,
以及面域 `o_mask`、`o_depth_discontinuity`,不伪造 confidence、stretch 或 background 属性,
也不写入 custom normal。完整分辨率的 `normal_map` 保存在 NPZ 中并转换为材质贴图。
所有 MoGe-2 Image datablock 都以源图片名而非对象名为基名:depth、normal 与 mask 分别使用
`图片名.depth`、`图片名.normal`、`图片名.mask`,不附加 `.moge2`。
7. 网格以 `reference_depth` 截面匹配 Image Empty 的显示矩形:该截面的投影四角与 Empty 的可见
边界重合,主点同时匹配 Empty Offset。网格对象直接沿用源 Empty 的 `matrix_world`,因此位置、旋转、
缩放及父级关系产生的世界变换保持一致。更近的顶点沿 Empty 局部 `+Z` 方向凸出,三角面正面朝向
观察相机,启用 Backface Culling 时可从正面看到。
8. 转换对话框不再提供后处理参数,只保留模型与 Resolution Scale 设置、`Create Matching Camera`、
`Remove Background`、`Unshaded`,以及分割线下方的 `Keep Original`。背景移除默认关闭;开启后 Server
使用 BEN2,并在写出 Surface 前直接删除背景三角面,不再由 Geometry Nodes 的 Background 开关处理。
该预删除同时适用于 DA3 与 MoGe-2;因此 MoGe-2 转换后即使关闭 `Mask`,BEN2 已删除的背景也不会恢复。
`Unshaded` 使用与 Depth Plane 相同的 Emission/Transparent 材质路径;普通材质的
Bump Scale 按 `abs(object.scale.z) × camera scale_z` 设置,并在材质组内乘以固定基准 `0.001`,
与透视表面实际深度尺度一致;它允许负数以反转凹凸方向。
相机默认关闭;开启后会在 Empty 局部 `+Z` 侧创建一台朝向局部 `-Z` 的透视相机,并成为生成 Surface
的子级,使二者移动时保持投影关系。关闭 `Keep Original` 时,网格成功创建后删除源 Empty。
表面、材质与相机会归入同一个 `Convert to Surface` Undo 步骤。
Process Res 的设计依据 DA3 官方实现:API 与 CLI 默认使用 `process_res=504` 和
`process_res_method="upper_bound_resize"`;输入预处理的 `PATCH_SIZE=14`,会把宽高分别调整到最接近的
`14` 倍数。DA3 论文以 `504×504` 为 base resolution,公开的训练尺寸长边为 `504`、`672`、`756` 或
`896`。官方 CLI 虽给出 `1024` 示例,但没有声明更高分辨率会提升质量,也没有规定 `1000` 为硬上限。
因此转换入口使用相对原图的 Resolution Scale,同时通过上下限避免 Process Res 过低或过高,并把训练范围
之外的值明确视为实验设置。参考
[DA3 API](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)、
[输入预处理实现](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/src/depth_anything_3/utils/io/input_processor.py)
与[技术报告](https://depth-anything-3.github.io/assets/da3_tech_report_2025.pdf)。
Debug 的 Textured Surface GLB 会把 depth、confidence 与 confidence percentile 写入自定义顶点
attributes。其他 DA3 Debug 三维输出会同时写出未压缩的 `prediction.attributes.npz`,保存完整
depth、confidence、intrinsics、extrinsics 与 confidence percentile。当前 trimesh 的 glTF 点图元
导出器只写 Position/Color,无法可靠把任意 attribute 嵌入点云 GLB,因此点云先以 companion NPZ
保留完整结构;后续若改为 SnapForm 自有 glTF writer,可再把这些字段直接嵌入点图元。
所有任务使用统一的状态栏进度显示。变化中的阶段说明显示在右侧的进度条内部,取消按钮
位于进度条右侧。状态字段支持阶段、当前操作与预计剩余秒数,界面统一组合为紧凑文案并保证
进度单调递增。任务进程与 Server 请求在后台启动,按钮点击后立即进入进度状态。Image Depth 的阶段包括 Environment 启动、输入准备、模型加载、深度估算、缓存
写出和面片构建。进度更新会强制刷新状态栏与窗口,无需移动鼠标即可实时显示。安装中断后残留的
半成品 Environment 会在下次安装前自动清理,不会因为中断而无法再次安装。
导入过程精确保留 NPZ 内深度值之间的差值。例如深度 `2.0` 和 `5.5` 在
`Displacement Strength = -1` 时,沿局部 Z 轴相差 `3.5 Blender Unit`。它不会把数值压缩到
`0..1`,也不再提供视觉归一化模式。
面片尺寸以有效深度的中位数 `Zref` 作为参考截面。先用 `mini_npz` 中的相机内参计算原始
投影面积,再恢复源图纵横比:
```text
RawWidth = depth_width × Zref × abs(Displacement Strength) / fx
RawHeight = depth_height × Zref × abs(Displacement Strength) / fy
Area = RawWidth × RawHeight
Aspect = source_image_width / source_image_height
Width = sqrt(Area × Aspect)
Height = sqrt(Area / Aspect)
```
这样即使 DA3 估计出的 `fx/fy` 不一致,也不会把正方形原图拉成长方形。主点 `cx/cy` 仍用于
确定原始相机面片,随后对象 XY Transform 将它匹配到源 Image Empty。生成对象会记录
`snapform_reference_depth`、`snapform_camera_plane_width` 和
`snapform_camera_plane_height`,便于检查实际采用的
参考深度和尺寸。
材质始终导入深度 Image Texture,并连接到 Displacement 节点和 Material Output;材质 Surface
Displacement 设置为 `Displacement Only`,DA3 与 Original 都使用 `Midlevel = 0`、`Scale = 1`。MoGe-2 的位移只在专用 Geometry Nodes 中处理。
`o_material_displacement` 为 `False` 时,材质把 Height 乘以零,置换只在几何节点中生效;为 `True` 时,
几何节点关闭结构置换,材质 Height 恢复。色彩贴图的 Alpha 二值结果同时用于选择原始深度或有符号的
`Background Depth`,使两种置换路径保持一致;未移除背景的普通不透明图片因此全部视为前景。
视频与序列帧的颜色、深度与前景 Image Texture
通过 image_user 的 Frames / Offset / Start 把第 i 帧映射到场景 `Start+i` 帧,并启用 Auto Refresh。
颜色贴图到 Principled BSDF 之间使用独立的 Shader Node Group。DA3 Surface 使用 `O DA3 Surface Layer`,
MoGe-2 Surface 使用独立的 `O MoGe2 Surface Layer`;它额外接收模型输出的 `Normal` 贴图,使用
`Normal Map` 节点按 `Object Space` 解码后连接到 Bump 的 `Normal` 输入,使像素级模型法线与原图
Bump 合并,不依赖重拓扑后的顶点密度;`Normal Strength` 默认值为 `1.0`,只调节模型 normal map,
原图 Bump 仍由独立的 `Bump Scale` 控制;
Depth Plane 使用 `O DA3 Depth Plane Layer` 材质组,并把背景剔除、背景深度与材质置换 Attribute
读取全部收进组内。两者都接收 `Color`、`Alpha` 与 `Bump Scale`,只有 `O DA3 Depth Plane Layer`
额外接收 Float 类型的 `Depth`。
`O DA3 Surface Layer` 只输出 `Color`、`Roughness`、`Alpha` 与 `Normal`,以原图 Color 保留 Bump 效果,
不创建 Displacement 节点或输出;`O DA3 Depth Plane Layer` 另外输出 `Displacement`。外层材质只保留贴图、
材质组、Principled BSDF 与 Material Output,方便继续调整。
Depth Plane 的 Bump Height 直接使用材质组的 `Color` 输入,只受独立的 `Bump Scale` 影响,不使用深度贴图,
也不经过 `Displacement` 模式开关、`Depth Scale` 或 `Background Depth`;材质 Displacement 仍使用这些参数处理后的深度。
无论选择哪种置换方式,原始彩色贴图都会连接
Principled BSDF 的 `Base Color`,并通过
`Bump.Height → Bump.Normal → Principled.Normal` 提供表面细节。Bump 的基础 Distance 为
`0.001`,会乘以面片 Transform 的统一缩放倍率。原图 Color 还会经过 `Float Curve` 连接到
Principled BSDF 的 `Roughness`,曲线两端输出分别为 `0.5` 和 `0.1`;材质 IOR 保持
Principled BSDF 默认值 `1.5`。
开启 `Unshaded` 后,色彩贴图 Color 连接 Emission,二值 Alpha 作为 Mix Shader 系数,在
Transparent BSDF 与 Emission 之间混合后连接 Material Output;不创建 Principled BSDF、Bump 或
Float Curve。开启 `Remove Background` 时色彩贴图使用 BEN2 输出;关闭时使用原图。
这里的“原始距离”是 DA3 模型输出的尺度。普通 Small、Base、Large 和 Giant 模型的深度不应
被解释为现实世界的米;需要现实尺度时应选择带 metric depth 的
`DA3 Nested Giant Large 1.1`。此外,当前结果仍是固定 XY 平面上的深度位移面,不是使用
相机内参逐像素反投影得到的完整 camera-space 网格;中位深度截面为 Geometry Nodes
深度面片提供正确的整体高宽尺度。
生成完成后会直接创建面片,不提供缓存重建入口或单独的导入参数。重新选择
`Convert to Depth Plane` 会按当前 Depth from 再次转换。
面片、深度 Image、前景 Image、Material 与 Geometry Nodes 修改器会在同一个 Blender Undo 步骤中创建;转换完成后执行
`Ctrl+Z` 会完整撤销本次转换,不会越过转换去回退更早的 Image Empty 操作。
Image Empty 使用 Packed Image 时,SnapForm 会通过 Blender 写出临时 PNG 交给 DA3,
不会 Unpack 或修改源 Image;任务结束后自动删除临时文件。普通图片仍直接读取源文件。DA3 的原始
`results.npz` 保留在对应 job 文件夹根下,生成的浮点深度 Image 会 Pack 到
当前 `.blend`。Depth Image 与 Displace Texture 根据源 Image datablock 命名。例如 Empty Object
名为 `obj_img`、引用的 Image 名为 `img.png` 时,Depth Image 与 Texture 都命名为 `img.depth`;
生成的 Object、Mesh 与 Material 仍命名为 `obj_img`。Pack 使用 OpenEXR,因而不会把大于 `1.0`
的原始深度截断。已有同名 datablock 时由 Blender 自动追加 `.001` 等编号。
视频输入时 Blender 无法整体 Pack 序列帧,因此 `.depth` 与颜色序列改为引用 job 目录根下的
`depth.0001.exr`、`frame.0001.jpg`(`.frame`)或 `foreground.0001.png`,并在每帧保留原始
浮点深度;保存 `.blend` 后再次打开仍需对应 job 目录存在。
选择 PLY 后,模型菜单只会提供 `DA3 Giant 1.1` 和
`DA3 Nested Giant Large 1.1`。Small 和 Base 没有 Gaussian head,无法由
DA3 生成 `gs_ply`。
每个 Server Job 被接受时,Job Runtime 在 Runtime Storage Root 的 `jobs/` 下创建一个独立文件夹。文件夹名是 32 位 opaque Job ID,不包含任务类型或对象名称。执行逻辑从 `JobContext.directory` 获取路径,输出产物直接写入该目录。
Blender 为 Packed Image 或序列帧生成的输入副本先保存在系统临时目录,Server 读取完成后由 Blender 清理,不归档到 Job 目录。普通磁盘文件和视频仍直接读取原文件。Server 日志统一写入 `server.log`,进度只保存在 Server 内存中。Environment Bootstrap 使用 Storage Root 下的 `install.log` 与临时的 `install-status.json`,不创建 Job 目录。
与几何输出相关的 Job Type 为 `generate-da3-depth`、`generate-moge2-depth`、`generate-da3-surface`、`generate-moge2-surface`、`debug-da3` 与 `debug-moge2`。每种 Job 只接收对应模型与输出所需的参数。
DA3 ONNX exporter 产生的 `depth_vis/`、`exports/mini_npz/` 会在完成后自动平铺,
不会保留只含一层产物的子目录:
```text
<opaque-job-id>/
├── scene.glb、*.ply、*.jpg 或 results.npz
└── 其它对应 Debug 格式的产物
<opaque-job-id>/
├── foreground.png 单图结果
├── foreground.0001.png… 视频/序列帧结果
├── frame.0001.jpg… 视频/序列帧的源帧副本
└── depth.0001.exr… AI 深度序列帧
```
建议从 `DA3 Base` 开始。它使用 Apache 2.0 模型权重。

View File

@@ -1,534 +0,0 @@
# SnapForm 技术文档
本文档面向开发者和需要了解内部实现细节的用户,从原 README 迁移而来。普通用户请阅读项目根目录的
`README.md`。
`SnapForm` 是一套从图片快速构建空间结构的 Blender Extension。目前以
[Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3)
作为首个 AI 能力,在独立 Environment 中运行 DA3,可将官方导出结果保存在独立 job 目录,
也可把 Image Empty 转换成带几何与材质置换的立体面片。MoGe-2 作为独立深度来源,保留其 metric depth、normal、mask 与 intrinsics 输出,不复用 DA3 Prediction 后处理。Debug 面板还可运行
[BEN2](https://github.com/PramaLLC/BEN2) 背景剔除功能测试。
## 当前功能
- 使用 Blender Extension manifest,可通过 `Install from Disk` 安装
- 在独立 Environment 中通过常驻 Service 运行 Torch、DA3、MoGe-2 与 BEN2,避免污染 Blender Python 进程
- Blender 使用内置 `urllib.request` 访问仅监听 `127.0.0.1` 的 Service;请求带随机 Token
- 常驻 Service 复用 Torch、CUDA、BEN2 与最近使用的 DA3 模型,连续执行时不再重复加载
- 多个 Blender 实例自动连接同一个共享 Service;`AI Service` 面板显示状态、端口、客户端与模型缓存,并提供安装、启动、清理、重启和停止控制
- Environment、模型和任务结果默认保存在 `~/.snapform`,可由多个 Blender 版本共享
- 当前启用 DA3 Small 和 Base;Large 1.1、Giant 1.1 与 Nested Giant Large 1.1 暂停使用
- 支持 BEN2 Base 单图背景剔除测试,输出透明 PNG 与推理耗时
- 支持单张 JPG、JPEG、PNG、WEBP 图片或常见视频文件输入
- 视频按顺序截取前若干帧,可设置参与重建的最大帧数
- `Models` panel 默认折叠,展开后可逐项查看和下载所有可用模型
- Environment 已安装时不额外占用面板空间;模型列表每个模型只占一行
- `Debug` 默认折叠,内部的 `DA3 Output Test`、`MoGe-2 API Test` 与 `BEN2 Removal Test` 也分别默认折叠
- 使用一个输出类型下拉菜单和一个 `Generate Output` 按钮
- 只显示当前格式的专属参数;GLB 与 PLY 设置不会同时出现
- Debug 的模型选项按格式过滤;PLY 只提供两个带 Gaussian head 的 Giant 模型
- Debug 与 Convert to Depth Plane 的模型选项直接显示下载状态;未下载模型带
`(Missing)`,已下载模型保持原名
- 分别调用 DA3 的 `glb`、`gs_ply`、`depth_vis` 和 `mini_npz` exporter
- `Textured Surface (GLB)` 直接使用 DA3 Prediction 中的深度、置信度和相机参数,把相邻像素
组成三角面并写入 `mesh.glb`;每个输入帧生成一张独立纹理网格,不做多视图融合
- Image Empty 右键菜单可直接运行 DA3,根据 `mini_npz` 深度生成面片
- Image Empty 右键菜单可直接运行 BEN2,保留原图并换用新建的透明背景图片
- Image Empty 支持视频与序列帧输入:`Convert to Depth Plane` 逐帧估计深度,
`Remove Background` 逐帧抠图,入口保持不变
- 视频生成的序列帧以外部文件引用保留在 job 目录(Blender 无法整体 Pack 序列帧);
Geometry Nodes 与材质 Image Texture 均可按场景帧播放序列帧
- 使用浮点深度贴图、`O DA3 Depth Plane` Geometry Nodes 和原图 Bump 构建立体面片
- 转换对话框选择推理模式、处理分辨率与背景移除;细分和置换方式在 Geometry Nodes 修改器中调整
- Debug 的 DA3 输出或 BEN2 背景剔除完成后只打开对应 job 文件夹,不修改当前场景
- 完整保留每次任务生成的所有官方文件
- 安装和模型下载使用后台子进程;推理提交给常驻 Service,状态栏实时显示阶段、百分比进度条与取消按钮
- 后台任务运行时,Install、Download 和 Export 按钮保持原位置并变灰,任务完成后恢复
## 使用
1. 先运行 `uv run python prepare_vendor.py`,按 `vendor.lock.toml` 下载并校验固定版本的
第三方构建输入;再运行 `uv run python pack.py`,分别构建 `windows-x64`、`macos-arm64`、
`linux-x64` 三个平台的 Extension ZIP(如 `SnapForm.v0.10.11.macos-arm64.zip`)。
每个包只含对应平台的 uv 二进制,并内置 DA3、BEN2 源码包。
2. 在 Blender 的 `Preferences > Extensions` 中选择 `Install from Disk`。
3. 在 Extension Preferences 中确认 Storage Root 与 Device。Device 是正式功能
(Image Empty 右键菜单)使用的推理设备,默认取本机最快的 CUDA/MPS,否则 CPU,且只列出
本机支持的设备。
4. 在 `AI Service` 点击 `Install Environment`。首次安装会下载 Python、PyTorch、FastAPI、Uvicorn;
DA3、BEN2 源码包与 uv 已内置在扩展中,安装时从 安装任务的临时目录中使用,安装结束后自动清理,
并在 Environment 安装后自动下载 `DA3 Small` 和 `BEN2 Base` 推荐模型。Environment 安装
固定显示 `Environment`、`PyTorch`、`AI components` 三个用户阶段;xFormers(macOS 跳过)、
FastAPI 等内部安装步骤合并在 `AI components` 中。耗时最长的 PyTorch 与 AI 组件占主要进度
区间。安装完成后直接写入 Environment Manifest,不再额外导入全部组件进行验证。推荐模型属于安装
后的自动步骤,不计入阶段编号;`DA3 Base` 等其他模型可稍后在 `Models` 面板按需下载。
状态文字使用 `阶段/总数 阶段名 | 当前操作 | 预计剩余时间`,依赖安装会结合预设耗时与实际
进度修正预期;模型下载根据字节数和速度计算剩余时间,例如
`Downloading DA3 Small | 14.2 MB/s | About 2 min left`。重试或切换镜像时进度不会倒退。
Environment 安装后即使模型下载中断也不影响 Service 使用,可稍后在 `Models` 面板单独补下。
安装不做启动网络预检:pip 安装失败会自动依次重试清华、阿里镜像,模型下载由 worker
自行探测回退到 HF 镜像;需要手动指定镜像时可通过 `PIP_INDEX_URL` 环境变量覆盖。
5. 在 3D View 按 `N`,打开 `SnapForm` 页签。
6. 如需其他模型,展开 `Models` panel,找到模型并点击该行的 `Download`。
7. 展开默认收起的 `Debug`,选择 `Output Type` 和本次执行使用的模型。
8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数,然后点击 `Generate Output`。
9. 测试背景剔除时,确认 `Models` panel 中的 `BEN2 Base` 已就绪,然后在 `Debug` 的
`BEN2 Removal Test` 区域选择单张图片并点击 `Remove Background`。
完成后会打开 job 文件夹,其中的 `foreground.png` 带透明 Alpha;状态栏显示
BEN2 推理、模型加载与任务总耗时。当前测试固定关闭较慢的 foreground refinement。
## Image Empty 去背景
1. 确认 `Models` panel 中的 `BEN2 Base` 已就绪;首次安装 Environment 时会默认下载。
2. 在 3D View 选中 Image Empty,右键打开 `SnapForm` 子菜单并选择 `Remove Background`。
3. 完成后以 Fake User 保留原 Image datablock,新建并 Pack 一张透明 PNG 到当前 `.blend`,然后只把
当前 Empty 的 `data` 换成新图;Empty 的 Transform、显示尺寸、Offset 及其他设置保持不变。
4. 新 Image 使用原图去掉格式后再追加 `.foreground` 的名称。例如 `img.png` 生成
`img.foreground`,`img.png.001` 生成 `img.001.foreground`。已有同名 datablock 时由
Blender 自动追加编号。
原图已 Pack 时会先写出临时 PNG 交给 BEN2,任务结束后删除临时输入;原 Image 仍保持 Pack
状态和原始内容。BEN2 的 `foreground.png` 也会保留在对应 job 目录中。
Image Empty 引用视频或序列帧时,Remove Background 直接执行,默认取 Empty 的 Image Frames
数值(未设置时为完整帧数,显式设为 1 时只处理 1 帧),起始帧跟随 Empty 的 Offset;若 Offset 加 Frames 超出视频长度,
会把窗口整体收进视频内(保证实际处理 Frames 帧数),Offset 过小时从第 1 帧开始。输出
`foreground.0001.png` 等透明序列帧(帧号从 0001 开始,与 Blender 场景帧一一对应),并把
源帧以 `frame.0001.jpg` 等命名复制到 job 目录根下供材质颜色导入。替换 Empty 的 Image 为
加载这些帧的 SEQUENCE Image,从原 Empty 的 Start 位置开始播放(Start 原样保留),
Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因此保持外部引用 job 目录;
请保留对应 job 目录,否则序列会失效。
## Image Empty 转深度面片
1. 在 3D View 选中 `Object Type = Empty`、`Display As = Image` 的对象。
2. 右键打开 `SnapForm` 子菜单并选择 `Convert to Depth Plane`,通过 `Depth from` 选择深度来源:
- `DA3`:显示 Model、Use Ray Pose 和 Resolution Scale。Process Res 先以原图长边乘以
Resolution Scale,再钳制到 `252–1008`,最后对齐到最接近的 `14` 倍数。Resolution Scale 以百分比
显示,范围为 `1–100%`,默认 `100%`;界面只在标签中实时显示最终 Process Res。
转换导出 `mini_npz`,使用前需要在
`Models` panel 下载所选模型,并固定使用 `process_res_method="upper_bound_resize"`;设备和参考视图策略
继续使用场景设置。Model 下拉菜单只列出已经下载的 DA3 模型;若没有可用模型,对话框会引导到
`Models` panel 下载。
- `MoGe-2`:使用 Base Normal 模型输出 metric depth、normal、validity mask 与 normalized intrinsics。正式转换的 `results.npz` 不保存体积较大的 points;Debug 测试会保留 points,并另外写出 depth、normal、mask 预览。
- `Original`:直接把原图作为深度贴图,不显示估算参数。仅在开启 `Remove Background` 时运行 BEN2
取得前景 Alpha;视频或序列帧按需同时抽取原始帧与前景序列,把原始帧作为 `.depth` 序列。
`Original` 是默认深度来源。选择 `DA3` 时,其 Model、Use Ray Pose 与 Resolution Scale
参数会集中显示在独立边框中。`Remove Background` 与 `Unshaded` 直接显示,不再使用 Options 折叠;
`Keep Original` 默认关闭,并在分割线下方保持为最后一项。仅当开启 `Remove Background` 时才需要
`BEN2 Base`,并把 `.foreground` 的 Alpha 作为独立
Mask,不在 Service 中改写深度值。保存 `.foreground` 时保留 BEN2 Alpha,但 RGB 使用未经修改的原始输入;
因此材质直接用 `.foreground` 同时提供颜色和 Mask,不再额外加载原图 Image Texture。
Image Empty 引用视频或序列帧时,DA3 与 MoGe-2 会按 Empty 的 Image Frames 数值逐帧估计
深度(未设置时为完整帧数,显式设为 1 时只处理 1 帧),起始帧跟随 Empty 的 Offset,与背景蒙版范围一致;
Offset 超出视频长度时同样把窗口收进视频内。材质节点与置换贴图的 Start 保持与原 Empty
一致;Original 模式的视频或序列帧使用 `frame.0001.jpg` 等原图序列作为颜色与深度,
`foreground.0001.png` 只提供前景 Alpha。
DA3 的深度仍以 EXR 序列帧(`depth.0001.exr` 起)保存在 job 目录根下;MoGe-2 分别使用 `moge_depth`、`moge_mask` 与 `moge_normal` EXR 序列。
`Unshaded`、`Remove Background` 与 `Keep Original` 均默认关闭,因此默认删除原对象。
3. DA3 或 MoGe-2 完成后会自动创建面片。DA3 面片几何先由 `mini_npz` 的相机内参与深度确定,再通过对象
Transform 匹配原 Image Empty 的可见尺寸、位置、旋转和 Offset。Original 模式直接使用源 Empty
的可见尺寸、Offset 与 Transform。DA3 会把用于匹配画面尺寸的缩放预乘到 `.depth` 数值,
XY 缩放烘焙到 Mesh,Z 不参与平面匹配;最终对象 Scale 为 `(1, 1, 1)`,视觉深度与旧逻辑一致。
关闭 `Keep Original` 时,成功生成后删除源 Empty。生成面片 Object 使用源 Empty 的原名;开启
`Keep Original` 时不处理
同名冲突,由 Blender 自动为生成面片追加编号。
4. Blender 只创建保留图像纵横比的最低密度基础网格,再把 `O DA3 Depth Plane` 作为 Geometry Nodes
修改器添加到对象。节点组与 `O DA3 Surface` 一同保存在 `O_SnapForm.blend`,关闭
`show_group_selector`,不为每个对象复制节点组。MoGe-2 使用单独的 `O MoGe2 Depth Plane`,只保留
网格细分、metric depth 位移、mask 删点和材质设置;材质使用 `O MoGe2 Depth Plane Layer`,直接接入
模型 normal,不携带 DA3 的材质置换与背景深度分支。DA3 修改器公开以下参数:
- `Displacement`:位于 Geometry 之后的第一项,菜单包含 `Geometry`(默认)与 `Material`。`Geometry`
在节点组内沿法线修改网格,`Material` 保持网格平面并启用材质置换;
- `Subdivision`:位于第二项,默认 `5`。节点读取基础平面的 Bounding Box,短边分段数为 `2^Subdivision`,
长边按宽高比计算并四舍五入;例如 16:9 图片在等级 `5` 时生成 `57×32` 个面;
- `Background Depth`:默认 `-1`,允许正负值,Structure Type 为 `Single`,直接显示且不再放在 Panel 中。前景 Mask 外的结构或
材质置换使用该值;是否存在背景只取决于转换所得色彩贴图的 `Alpha > 0.5` 二值结果;
- `Depth Scale`:默认 `1`,允许正负值,Structure Type 为 `Single`,同时控制 Geometry 与 Material
两条置换路径。
Grid 重建后通过隐藏的 Material 输入重新赋予原图材质,并对所有面执行 Shade Smooth。
材质使用显式指向 `UVMap` 的 UV Map 节点连接颜色、前景 Mask 与深度 Image Texture,不依赖活动 UV。
前景深度先与 `Background Depth` 取 Maximum,再按色彩贴图 Alpha 与背景深度混合,
深度分支在混合前使用 `Alpha > 0.5` 转为二值 Mask,避免半透明前景被背景深度拉扯;
材质表面 Alpha 同样使用二值 Mask。材质置换使用相同的二值和钳制顺序。
节点组还会把 `o_material_displacement` 与 `o_background_depth` 写入点域,材质通过
`Geometry` 类型的 Attribute 节点读取。点域会为顶点保存重复值,但能在应用修改器后继续保留参数;
相比自适应网格本身,其占用很小。Depth Image 与 Foreground Image 是修改器隐藏数据输入,
由导入流程自动设置。
视频与序列帧还会为 Depth 与 Foreground 共同写入隐藏的 `Frame`、`Frame Start`、`Frames` 与 `Cyclic`。
节点组内部不再使用 Scene Time。只有原 Image Empty 是视频或序列时,导入流程才直接给修改器的 `Frame`
socket 添加 Scripted Expression Driver `frame`;静态图保持默认值 `1`,不创建 Driver。导入内容已经
是截取后的帧,因此 Offset 在节点组内恒定为 `0`。
节点组根据 `Frame` 计算从 `1` 开始的 Image Texture Frame:场景帧与 Start 都为 `1` 时读取图片第 `1` 帧;非循环输入
钳制到有效范围;循环输入使用 Floored Modulo,让 `Frame Start` 前后的负、正相对帧都映射到
`1…Frames`,因此开始帧之前也会循环播放。Geometry 置换会像材质 Image Texture 一样随场景帧刷新。
## Image Empty 转透视网格
1. 在 3D View 选中仅包含一帧的 Image Empty,右键打开 `SnapForm` 子菜单并选择
`Convert to Surface`。该入口独立于
`Convert to Depth Plane`,当前不处理视频或序列帧。
对话框中的 Resolution Scale 与 Depth Plane 使用相同设计:原图长边乘以 `1–100%` 的比例后,
将实际 Process Res 钳制在 `252–1008` 并按 `14` 对齐。
2. Service 请求 DA3 的 `depth`、`conf` 与 `intrinsics`。仅在开启 `Remove Background` 时运行 BEN2 取得
前景 Alpha,并在输出前删除包含背景顶点的三角面;关闭时不加载 BEN2。Depth 不做 smoothing,Service
反投影深度网格,建立源三角面,计算 confidence percentile 与 camera-space edge stretch,并写入
基于原图像素坐标的源 `UVMap`。这些步骤只准备修改器的输入网格,不执行 Relax、拓扑修复、平滑、
小岛清理或最终 UV 重投影。运行时显示的六个 `Preparing surface data` 步骤均指这段 Service 端源数据准备。
3. Blender 把勾选 `Usage > Modifier` 的 `O DA3 Surface` 节点组直接作为 Geometry Nodes 修改器
添加到 Surface,不另设修改器名称。节点组内没有子节点组,依次完成:
- 修改器顶层依次显示 `Resolution`、`Relax Iterations`、`Smooth`;默认展开的 `Culling` Panel
只包含 `Confidence` 与 `Stretch`。两个百分比输入都使用 `0–100` 的节点值,并在组内
转换为 `0–1` 阈值;`Stretch = 100%` 明确选择全部面,剔面后同时删除
Face Count 为零的孤立顶点;
- 每轮 Remesh Iteration 依次执行 Connected Merge、循环拓扑修复与一次 Smooth,使新拓扑在进入下一轮
Merge 前先稳定;拓扑修复删除非流形面、单面拓扑和小面积 mesh islands;
- 默认 `Resolution` 为原始有效三角边长中位数乘以 `sqrt(10) * 2`。`sqrt(10)` 来自原先 90% 面数缩减
的二维边距估计,再乘以 2 为迭代合并留出更平滑、稳定的默认尺度;节点内部 Connected Merge 的
Distance 为 `Resolution * 0.5`;
- Smooth 采用 `O Smooth` 的边界检测和模糊法线权重,固定开放边界与薄面,只保留一个
`Smooth` 程度系数;节点组不再额外执行 Flip Faces;
- 从修改前的原始表面以 nearest-surface 重新采样 `UVMap`,最终明确写为 Corner 域 `2D Vector`。
4. 节点组随插件放在 `O_SnapForm.blend`。Blender 4.5 首次 Append 后复用同一 datablock;
Blender 5.0 以 linked ID 载入后调用 Pack,后续 Surface 继续复用。插件不会为每个 Surface 新建节点组,
也不会在添加后重命名 Geometry Nodes 修改器。节点组只启用 `Usage > Modifier`,不 Mark as Asset;
修改器关闭 `show_group_selector`。`Fix Topology` 位于默认折叠的 `Options` Panel。
节点图按 Culling、Remesh、Topology Fix、Island Cleanup 与 UV 分区摆放;各区就近使用独立的
Group Input,并隐藏没有连线的接口,避免跨越整个节点图的输入连线。
5. Service 准备过程固定显示 6 个编号阶段:验证 DA3 数据、反投影、准备 DA3 与可选 BEN2 数据、构建原始
源三角面、源 UV/attributes、写出网格。状态栏显示 `Preparing surface data n/6: ...`。
6. Service 在 job 根目录写出 `mesh.npz` 与 `texture.png`。NPZ 保存 camera-space `vertices`、
`faces`、`uv`、`intrinsics`、`image_size` 与有效深度中位数 `reference_depth`。Blender 读取这些
数据后直接创建 Surface 与原图材质,不需要 Open3D 或 OpenCV 进入 Blender Python 环境。
Surface 保留下列结构化 attributes:
- 点域:`o_depth`、`o_confidence`、`o_confidence_percentile`、`o_background`;
- 面域:`o_face_confidence`、`o_edge_stretch`、`o_background_face`。
7. 网格以 `reference_depth` 截面匹配 Image Empty 的显示矩形:该截面的投影四角与 Empty 的可见
边界重合,主点同时匹配 Empty Offset。网格对象直接沿用源 Empty 的 `matrix_world`,因此位置、旋转、
缩放及父级关系产生的世界变换保持一致。更近的顶点沿 Empty 局部 `+Z` 方向凸出,三角面正面朝向
观察相机,启用 Backface Culling 时可从正面看到。
8. 转换对话框不再提供后处理参数,只保留模型与 Resolution Scale 设置、`Create Matching Camera`、
`Remove Background`、`Unshaded`,以及分割线下方的 `Keep Original`。背景移除默认关闭;开启后 Service
使用 BEN2,并在写出 Surface 前直接删除背景三角面,不再由 Geometry Nodes 的 Background 开关处理。
`Unshaded` 使用与 Depth Plane 相同的 Emission/Transparent 材质路径;普通材质的
Bump Scale 按 `abs(object.scale.z) × camera scale_z` 设置,并在材质组内乘以固定基准 `0.001`,
与透视表面实际深度尺度一致;它允许负数以反转凹凸方向。
相机默认关闭;开启后会在 Empty 局部 `+Z` 侧创建一台朝向局部 `-Z` 的透视相机,并成为生成 Surface
的子级,使二者移动时保持投影关系。关闭 `Keep Original` 时,网格成功创建后删除源 Empty。
表面、材质与相机会归入同一个 `Convert to Surface` Undo 步骤。
Process Res 的设计依据 DA3 官方实现:API 与 CLI 默认使用 `process_res=504` 和
`process_res_method="upper_bound_resize"`;输入预处理的 `PATCH_SIZE=14`,会把宽高分别调整到最接近的
`14` 倍数。DA3 论文以 `504×504` 为 base resolution,公开的训练尺寸长边为 `504`、`672`、`756` 或
`896`。官方 CLI 虽给出 `1024` 示例,但没有声明更高分辨率会提升质量,也没有规定 `1000` 为硬上限。
因此转换入口使用相对原图的 Resolution Scale,同时通过上下限避免 Process Res 过低或过高,并把训练范围
之外的值明确视为实验设置。参考
[DA3 API](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)、
[输入预处理实现](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/src/depth_anything_3/utils/io/input_processor.py)
与[技术报告](https://depth-anything-3.github.io/assets/da3_tech_report_2025.pdf)。
Debug 的 Textured Surface GLB 会把 depth、confidence 与 confidence percentile 写入自定义顶点
attributes。其他 DA3 Debug 三维输出会同时写出未压缩的 `prediction.attributes.npz`,保存完整
depth、confidence、intrinsics、extrinsics 与 confidence percentile。当前 trimesh 的 glTF 点图元
导出器只写 Position/Color,无法可靠把任意 attribute 嵌入点云 GLB,因此点云先以 companion NPZ
保留完整结构;后续若改为 SnapForm 自有 glTF writer,可再把这些字段直接嵌入点图元。
所有任务使用统一的状态栏进度显示。变化中的阶段说明显示在右侧的进度条内部,取消按钮
位于进度条右侧。状态字段支持阶段、当前操作与预计剩余秒数,界面统一组合为紧凑文案并保证
进度单调递增。任务进程与 Service 请求在后台启动,按钮点击后立即进入进度状态。Image Depth 的阶段包括 Environment 启动、输入准备、模型加载、深度估算、缓存
写出和面片构建。进度更新会强制刷新状态栏与窗口,无需移动鼠标即可实时显示。安装中断后残留的
半成品 Environment 会在下次安装前自动清理,不会因为中断而无法再次安装。
导入过程精确保留 NPZ 内深度值之间的差值。例如深度 `2.0` 和 `5.5` 在
`Displacement Strength = -1` 时,沿局部 Z 轴相差 `3.5 Blender Unit`。它不会把数值压缩到
`0..1`,也不再提供视觉归一化模式。
面片尺寸以有效深度的中位数 `Zref` 作为参考截面。先用 `mini_npz` 中的相机内参计算原始
投影面积,再恢复源图纵横比:
```text
RawWidth = depth_width × Zref × abs(Displacement Strength) / fx
RawHeight = depth_height × Zref × abs(Displacement Strength) / fy
Area = RawWidth × RawHeight
Aspect = source_image_width / source_image_height
Width = sqrt(Area × Aspect)
Height = sqrt(Area / Aspect)
```
这样即使 DA3 估计出的 `fx/fy` 不一致,也不会把正方形原图拉成长方形。主点 `cx/cy` 仍用于
确定原始相机面片,随后对象 XY Transform 将它匹配到源 Image Empty。生成对象会记录
`snapform_reference_depth`、`snapform_camera_plane_width` 和
`snapform_camera_plane_height`,便于检查实际采用的
参考深度和尺寸。
材质始终导入深度 Image Texture,并连接到 Displacement 节点和 Material Output;材质 Surface
Displacement 设置为 `Displacement Only`,DA3 与 Original 都使用 `Midlevel = 0`、`Scale = 1`。MoGe-2 的位移只在专用 Geometry Nodes 中处理。
`o_material_displacement` 为 `False` 时,材质把 Height 乘以零,置换只在几何节点中生效;为 `True` 时,
几何节点关闭结构置换,材质 Height 恢复。色彩贴图的 Alpha 二值结果同时用于选择原始深度或有符号的
`Background Depth`,使两种置换路径保持一致;未移除背景的普通不透明图片因此全部视为前景。
视频与序列帧的颜色、深度与前景 Image Texture
通过 image_user 的 Frames / Offset / Start 把第 i 帧映射到场景 `Start+i` 帧,并启用 Auto Refresh。
颜色贴图到 Principled BSDF 之间使用独立的 Shader Node Group。Surface 使用 `O DA3 Surface Layer`;
Depth Plane 使用 `O DA3 Depth Plane Layer` 材质组,并把背景剔除、背景深度与材质置换 Attribute
读取全部收进组内。两者都接收 `Color`、`Alpha` 与 `Bump Scale`,只有 `O DA3 Depth Plane Layer`
额外接收 Float 类型的 `Depth`。
`O DA3 Surface Layer` 只输出 `Color`、`Roughness`、`Alpha` 与 `Normal`,以原图 Color 保留 Bump 效果,
不创建 Displacement 节点或输出;`O DA3 Depth Plane Layer` 另外输出 `Displacement`。外层材质只保留贴图、
材质组、Principled BSDF 与 Material Output,方便继续调整。
Depth Plane 的 Bump Height 直接使用材质组的 `Color` 输入,只受独立的 `Bump Scale` 影响,不使用深度贴图,
也不经过 `Displacement` 模式开关、`Depth Scale` 或 `Background Depth`;材质 Displacement 仍使用这些参数处理后的深度。
无论选择哪种置换方式,原始彩色贴图都会连接
Principled BSDF 的 `Base Color`,并通过
`Bump.Height → Bump.Normal → Principled.Normal` 提供表面细节。Bump 的基础 Distance 为
`0.001`,会乘以面片 Transform 的统一缩放倍率。原图 Color 还会经过 `Float Curve` 连接到
Principled BSDF 的 `Roughness`,曲线两端输出分别为 `0.5` 和 `0.1`;材质 IOR 保持
Principled BSDF 默认值 `1.5`。
开启 `Unshaded` 后,色彩贴图 Color 连接 Emission,二值 Alpha 作为 Mix Shader 系数,在
Transparent BSDF 与 Emission 之间混合后连接 Material Output;不创建 Principled BSDF、Bump 或
Float Curve。开启 `Remove Background` 时色彩贴图使用 BEN2 输出;关闭时使用原图。
这里的“原始距离”是 DA3 模型输出的尺度。普通 Small、Base、Large 和 Giant 模型的深度不应
被解释为现实世界的米;需要现实尺度时应选择带 metric depth 的
`DA3 Nested Giant Large 1.1`。此外,当前结果仍是固定 XY 平面上的深度位移面,不是使用
相机内参逐像素反投影得到的完整 camera-space 网格;中位深度截面为 Geometry Nodes
深度面片提供正确的整体高宽尺度。
生成完成后会直接创建面片,不提供缓存重建入口或单独的导入参数。重新选择
`Convert to Depth Plane` 会按当前 Depth from 再次转换。
面片、深度 Image、前景 Image、Material 与 Geometry Nodes 修改器会在同一个 Blender Undo 步骤中创建;转换完成后执行
`Ctrl+Z` 会完整撤销本次转换,不会越过转换去回退更早的 Image Empty 操作。
Image Empty 使用 Packed Image 时,SnapForm 会通过 Blender 写出临时 PNG 交给 DA3,
不会 Unpack 或修改源 Image;任务结束后自动删除临时文件。普通图片仍直接读取源文件。DA3 的原始
`results.npz` 保留在对应 job 文件夹根下,生成的浮点深度 Image 会 Pack 到
当前 `.blend`。Depth Image 与 Displace Texture 根据源 Image datablock 命名。例如 Empty Object
名为 `obj_img`、引用的 Image 名为 `img.png` 时,Depth Image 与 Texture 都命名为 `img.depth`;
生成的 Object、Mesh 与 Material 仍命名为 `obj_img`。Pack 使用 OpenEXR,因而不会把大于 `1.0`
的原始深度截断。已有同名 datablock 时由 Blender 自动追加 `.001` 等编号。
视频输入时 Blender 无法整体 Pack 序列帧,因此 `.depth` 与颜色序列改为引用 job 目录根下的
`depth.0001.exr`、`frame.0001.jpg`(`.frame`)或 `foreground.0001.png`,并在每帧保留原始
浮点深度;保存 `.blend` 后再次打开仍需对应 job 目录存在。
选择 PLY 后,模型菜单只会提供 `DA3 Giant 1.1` 和
`DA3 Nested Giant Large 1.1`。Small、Base 和 Large 没有 Gaussian head,无法由
DA3 生成 `gs_ply`。
每个任务在 `~/.snapform/jobs/` 下用独立文件夹包裹。文件夹统一采用
`动作-对象_日期-时间-微秒`,每种任务都有唯一名称,例如
`generate-depth-plane_20260807-153000-123456`。微秒用于避免同一秒启动的任务共用目录。
Blender 为 Packed Image 或序列帧生成的输入副本也归档到对应 job 根目录:单图命名为
`input.png`,序列帧命名为 `frame.0001.*`。它们以前暂存在 `jobs/inputs/`,现在不再创建这个
公共中转文件夹。普通磁盘文件和视频仍直接读取原文件,不额外复制。
日志、运行中的状态文件、归档输入与输出产物均直接放在 job 根目录。各类任务名称为:
- `install-environment`
- `download-model`
- `export-reconstruction`
- `generate-depth-plane`
- `generate-image-mesh`
- `remove-background`
- `remove-image-background`
- `generate-original-depth-plane`
DA3 官方 exporter 产生的 `gs_ply/`、`depth_vis/`、`exports/mini_npz/` 会在完成后自动平铺,
不会保留只含一层产物的子目录:
```text
export-reconstruction_20260807-153000-123456/
├── job.log
├── status.json
└── scene.glb
export-reconstruction_20260807-153000-123456/
├── job.log
├── status.json
└── *.ply
export-reconstruction_20260807-153000-123456/
├── job.log
├── status.json
└── *.jpg
export-reconstruction_20260807-153000-123456/
├── job.log
├── status.json
└── results.npz
remove-background_20260807-153000-123456/
├── job.log
├── status.json
├── foreground.png 单图结果
├── foreground.0001.png… 视频/序列帧结果
├── frame.0001.jpg… 视频/序列帧的源帧副本
└── depth.0001.exr… AI 深度序列帧
```
建议从 `DA3 Base` 开始。它使用 Apache 2.0 模型权重;`DA3 Large 1.1` 权重采用
CC BY-NC 4.0,仅限非商业用途。
## 输出格式说明
`Output Type` 提供以下结果:
- Point Cloud (GLB):DA3 `glb`
- Gaussian Splats (PLY):DA3 `gs_ply`,同时传入 `infer_gs=True`
- Depth Preview (Images):DA3 `depth_vis`
- Depth Data (NPZ):DA3 `mini_npz`
- Textured Surface (GLB):SnapForm 根据 DA3 Prediction 构建并输出 `mesh.glb`
Point Cloud 专属设置仅在对应类型下显示,`gs_views_interval` 仅用于 Gaussian Splats。
Textured Surface 使用 0–100% 的 Confidence Threshold 删除置信度最低的一定比例:
- `process_res=504`
- `process_res_method="upper_bound_resize"`
- `use_ray_pose=False`
- `ref_view_strategy="saddle_balanced"`
- `conf_thresh_percentile=40.0`
- `num_max_points=1_000_000`
- `show_cameras=True`
- `export_kwargs["gs_ply"]["gs_views_interval"]=1`
DA3 官方 `glb` 是彩色点云与可选相机线框;`gs_ply` 是带 Gaussian 属性的 PLY;
`depth_vis` 是原图与着色深度图的并排预览;`mini_npz` 保存深度、置信度和可用的相机参数。
Point Cloud GLB 任务会关闭其默认附带的深度预览,使五种输出产物保持分离。
官方格式与限制可参阅
[DA3 Python API 文档](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)。
## AI Service
Environment 指磁盘上已安装的 Python、依赖及模型执行组件;Service 是由 Environment 启动、接收 Blender 任务的
常驻 HTTP 进程。`manifest.json` 记录当前 Environment 的安装信息;Service 的 `/health` 返回进程状态和
实例标识。项目发布前不维护安装清单或任务协议的兼容版本号。
Extension 本身不包含 Torch。`Install Environment` 使用 `uv` 创建独立 Python 3.12
环境,并安装。机器上即使没有安装 `uv` 或 Python 也没关系:安装程序由 Blender
自带的 Python 启动,缺 `uv` 时会解包扩展内置的对应平台二进制,再用它下载
Python 3.12 与全部依赖。`uv` 会缓存在扩展存储目录下,后续安装直接复用。
安装不做网络预检,直接开始;环境变量仍可切换镜像与代理:`PIP_INDEX_URL`(PyPI 镜像)、
`SNAPFORM_TORCH_INDEX`(PyTorch 索引)、`HF_ENDPOINT`(Hugging Face 镜像)、
`HTTP(S)_PROXY`/`ALL_PROXY`(代理)。未指定 PyPI 镜像时,
pip 安装失败会自动依次重试镜像(清华、阿里云);模型下载不预探测,官方 Hugging Face
失败后自动回退 OmooLab R2(`models.omoolab.xyz`),并校验文件大小和 SHA-256。
GitHub 不再参与安装:DA3、BEN2、uv 已内置,缺少内置文件会直接报错提示重新安装,不再下载兜底。
单独下载模型时遵循同样的顺序:Hugging Face(可用 `HF_ENDPOINT` 指定其他入口)失败后自动切换到
OmooLab R2,不再使用 ModelScope。
Windows 上代理若只在系统设置里而未写入环境变量,安装程序会尝试自动从系统代理注入子进程。
- PyTorch 2.9.1
- TorchVision 0.24.1
- xFormers 0.0.33.post2(与 PyTorch 2.9.1 对应,仅 Windows/Linux;macOS 无 wheel 自动跳过)
- 固定到已验证 commit 的 Depth Anything 3
- 固定到已验证 commit 的 BEN2
- FastAPI 0.139.2 与 Uvicorn 0.51.0
- DA3 与 BEN2 源码包随扩展打包,安装时复制到临时目录后由 uv 本地安装,完成后立即清理
(无需访问 GitHub);SnapForm 直接调用 DA3
底层 export 接口,`gs_ply` 导出不会触发 `gs_video` 渲染,因此不需要 gsplat
与 C++ 编译工具链
- `vendor/` 是不进入 Git 的本地缓存;`vendor.lock.toml` 固定下载 URL、commit/version 与
SHA-256,`patches/<包名>/` 进入 Git 并存放 SnapForm 对源码的覆盖文件(移除 xformers
硬依赖、export 入口改懒加载、
射线位姿的 3×3 QR 分解在 MPS 上改走 CPU),
`pack.py` 打包时自动套入官方包,扩展 zip 内才是修补后的版本;更新上游只需替换官方包
并核对覆盖文件是否仍适用
- `addict` 2.4.0(补充 DA3 上游包元数据暂未声明的运行依赖)
安装 Environment 时会自动选择 PyTorch 后端:检测到 NVIDIA 驱动则安装 CUDA 12.8 版本,否则安装
CPU 版;macOS 始终安装 CPU 版 PyTorch(MPS 包含其中)。CPU 可以用于验证,但实际推理通常很慢。
Extension Preferences 的 `Device` 决定正式功能的推理设备,默认取本机最快的 CUDA/MPS,否则
CPU;Debug 面板的 `Debug Device` 只影响 Debug 任务。两处设备选项都只列出本机支持的设备:
Windows 不显示 MPS,机器上没有可用 NVIDIA 驱动时不显示 CUDA。Environment 安装完成时会检测真实
可用的设备并写入 `manifest.json`,设备列表以该检测结果为准。
CUDA 12.8 的 PyTorch wheel 自带 CUDA 运行库,无需安装 CUDA Toolkit;只需要
NVIDIA 驱动 ≥ 570(数据中心显卡可用更老驱动)。驱动不满足或没有 NVIDIA 显卡时,
设备会自动回退到 CPU(macOS 为 MPS),只有显式选择 CUDA 才会报错。
macOS(Apple Silicon)会自动从 PyPI 安装 CPU 版 PyTorch,推理设备默认使用 MPS
(Apple Metal)加速,也可以在设备选项里切换 CPU。PyTorch 自 2.3 起不再提供 Intel
Mac(x86_64)轮子,Intel Mac 暂不支持。DA3 射线位姿估计中的 3×3 QR 分解在 MPS
上尚未实现,SnapForm 的覆盖文件会单独把该分解放到 CPU 执行,其余推理仍走 MPS。
Extension 启用后只会查找已有的共享 Service,不会主动启动新进程。第一次执行 AI 操作或在侧边栏
手动点击 `Start` 时才会启动 Service。Service 只绑定回环地址,并让操作系统分配动态端口;第一个实例把
端口、进程和随机 Token 写入 `service.json`,其余 Blender 通过标准库
`urllib.request` 连接同一个 Service。`service.lock` 保证同一 Storage Root 同时只有一个 Service,Token
则阻止同机其他进程直接调用接口。Service 全局同时只执行一个 AI 任务。
每个 Blender 会注册独立客户端并每 5 秒发送心跳。正常关闭 Blender 时先注销自身;如果当前 Service
就是由这个 Blender 启动的,会直接终止它,无需等待。异常退出的客户端会在 10 秒后过期;Service
在最后一个客户端离开后仍会保留模型缓存等待 20 秒再退出,作为 Service 由其他进程启动等场景的兜底。
取消任务不会重启 Service,并会保留已经加载的模型;
如果取消发生在一次不可中断的 Torch 调用中,任务会在该调用返回后的最近检查点结束。
侧边栏 `AI Service` 面板以 `Running (Clients: n)` 表示空闲服务,以 `Running (Busy)` 表示正在执行任务,
并显示缓存模型和活动任务。Blender 完成结果导入时会立即释放当前任务对应的本地 Busy 快照,无需等待
下一次 Service 轮询。空闲且只有当前 Blender 连接时,可以清理模型缓存、重启或停止 Service;
多客户端或任务执行期间不会允许破坏共享服务。
BEN2 在首次使用后保持加载;DA3 保留最近使用的一个模型。切换 DA3 模型时会释放旧模型;加载时如果
显存不足,会先释放另一类缓存模型并重试。
Environment Manifest Revision 变化时,旧 Environment 需要重新执行 `Install Environment`。
Service 的启动和崩溃日志写入 `~/.snapform/service.log`。
Environment、模型和任务结果默认保存在用户主目录的 `~/.snapform`:
```text
~/.snapform/
├── .venv/
├── models/
├── jobs/
├── service.log
├── service.json # Service 运行时存在
└── service.lock # Service 运行时持有
```
`models/` 下每个模型一个文件夹,命名统一带前缀:DA3 为 `da3_*`(如 `da3_small`、`da3_base`),
BEN2 为 `ben2_*`(如 `ben2_base`)。旧版无前缀的 DA3 文件夹会在首次使用时自动改名为新名称。
不同 Blender 版本安装相同 Extension 后会共用这个目录。也可以在 Extension
Preferences 的 `Storage Root` 中修改共享位置。不要把模型写入 Extension 安装目录,
因为该目录可能只读,并且升级时会被替换。不要同时从多个 Blender 实例执行
`Install Environment`;Service、任务队列和模型读取可以共享。
## 开发
```bash
uv run python prepare_vendor.py
uv run pytest tests -v
uv run python pack.py
```
项目结构:
```text
.
├── vendor.lock.toml 第三方构建输入的版本、URL 与 SHA-256
├── prepare_vendor.py 下载并校验第三方构建输入
├── pack.py
├── patches/ 进入 Git 的 DA3 源码覆盖文件
├── vendor/ 不进入 Git 的下载缓存;本地调试也直接读取这里
├── src/
│ └── snapform/
│ ├── blender_manifest.toml
│ ├── service.py
│ ├── install_env.py
│ ├── models.py
│ ├── image_depth.py
│ ├── operators.py
│ ├── panel.py
│ ├── preferences.py
│ ├── properties.py
│ ├── client.py
│ └── worker.py
└── tests/
```
## 上游与许可证
- Depth Anything 3 代码采用 Apache 2.0。
- BEN2 代码与模型权重采用 MIT License。
- 本 Extension 代码采用 GPL-3.0-or-later。
- 模型权重许可证由所选 Hugging Face 模型仓库决定,Extension 下载前会在界面显示。

124
pack.py
View File

@@ -1,6 +1,3 @@
import hashlib
import io
import tarfile
import tomllib
from pathlib import Path
from zipfile import ZIP_DEFLATED, ZipFile
@@ -8,93 +5,8 @@ from zipfile import ZIP_DEFLATED, ZipFile
PROJECT_ROOT = Path(__file__).resolve().parent
SOURCE_DIR = PROJECT_ROOT / "src" / "snapform"
VENDOR_DIR = PROJECT_ROOT / "vendor"
DIST_DIR = PROJECT_ROOT / "dist"
PATCHES_DIR = PROJECT_ROOT / "patches"
LOCK_PATH = PROJECT_ROOT / "vendor.lock.toml"
def load_vendor_lock():
"""Return the locked third-party artifact metadata."""
with LOCK_PATH.open("rb") as stream:
lock = tomllib.load(stream)
if lock.get("schema_version") != 1:
raise RuntimeError("Unsupported vendor.lock.toml schema")
return lock
VENDOR_LOCK = load_vendor_lock()
PLATFORMS = {
platform: artifact["filename"]
for platform, artifact in VENDOR_LOCK["platforms"].items()
}
BUNDLED_ARCHIVES = {
f"vendor/{artifact['filename']}": artifact["filename"]
for artifact in VENDOR_LOCK["artifacts"].values()
}
OVERLAYS = {
artifact["filename"]: artifact["patches"]
for artifact in VENDOR_LOCK["artifacts"].values()
if "patches" in artifact
}
def file_sha256(path):
"""Return the SHA-256 digest for a file."""
digest = hashlib.sha256()
with Path(path).open("rb") as stream:
while chunk := stream.read(1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def validate_artifact(path, expected_sha256):
"""Require an artifact to exist and match its locked digest."""
if not path.is_file():
raise RuntimeError(
f"Missing vendor artifact: {path}. "
"Run: uv run python prepare_vendor.py"
)
actual = file_sha256(path)
if actual != expected_sha256:
raise RuntimeError(
f"Vendor artifact hash mismatch: {path}. "
"Run: uv run python prepare_vendor.py"
)
def bundled_artifact_bytes(filename):
"""Return a bundled artifact with its configured overlay applied."""
archive_path = VENDOR_DIR / filename
overlay_dir = PATCHES_DIR / OVERLAYS.get(filename, filename)
if not overlay_dir.is_dir():
return archive_path.read_bytes()
if not filename.endswith(".tar.gz"):
raise ValueError(f"Unsupported overlay archive: {filename}")
return _apply_tar_overlay(archive_path, overlay_dir)
def _apply_tar_overlay(archive_path, overlay_dir):
with tarfile.open(archive_path, "r:gz") as source:
members = source.getmembers()
top_level = members[0].name.split("/", 1)[0]
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w:gz") as output:
for member in members:
if member.isdir() or member.islnk() or member.issym():
output.addfile(member)
continue
relative = member.name[len(top_level) + 1:]
overlay_file = overlay_dir / relative
if overlay_file.is_file():
info = tarfile.TarInfo(member.name)
info.size = overlay_file.stat().st_size
info.mode = member.mode
output.addfile(info, overlay_file.open("rb"))
else:
with source.extractfile(member) as extracted:
output.addfile(member, extracted)
return buffer.getvalue()
PLATFORMS = ("windows-x64", "macos-arm64", "linux-x64")
def manifest_for_platform(platform):
@@ -119,31 +31,6 @@ def manifest_for_platform(platform):
return "\n".join(lines) + "\n"
def bundled_vendor_files(platform):
entries = {
name: VENDOR_DIR / filename
for name, filename in BUNDLED_ARCHIVES.items()
}
entries[f"vendor/{PLATFORMS[platform]}"] = (
VENDOR_DIR / PLATFORMS[platform]
)
return entries
def validate_vendor_files(platform):
"""Validate common archives and the selected uv archive."""
for artifact in VENDOR_LOCK["artifacts"].values():
validate_artifact(
VENDOR_DIR / artifact["filename"],
artifact["sha256"],
)
uv_artifact = VENDOR_LOCK["platforms"][platform]
validate_artifact(
VENDOR_DIR / uv_artifact["filename"],
uv_artifact["sha256"],
)
def build_extension(platform):
if platform not in PLATFORMS:
raise ValueError(f"Unsupported platform: {platform}")
@@ -152,8 +39,6 @@ def build_extension(platform):
f"{manifest['name']}.v{manifest['version']}.{platform}.zip"
)
DIST_DIR.mkdir(exist_ok=True)
vendor_files = bundled_vendor_files(platform)
validate_vendor_files(platform)
with ZipFile(output_path, "w", ZIP_DEFLATED) as archive:
for source_path in sorted(SOURCE_DIR.rglob("*")):
if not source_path.is_file():
@@ -161,8 +46,6 @@ def build_extension(platform):
if "__pycache__" in source_path.parts:
continue
relative_path = source_path.relative_to(SOURCE_DIR)
if relative_path.parts[0] == "vendor":
continue
if source_path.name == "blender_manifest.toml":
continue
archive.write(source_path, relative_path)
@@ -170,11 +53,6 @@ def build_extension(platform):
"blender_manifest.toml",
manifest_for_platform(platform),
)
for archive_name, source_path in vendor_files.items():
archive.writestr(
archive_name,
bundled_artifact_bytes(source_path.name),
)
return output_path

View File

@@ -1,93 +0,0 @@
[build-system]
requires = ["hatchling>=1.25", "hatch-vcs>=0.4"]
build-backend = "hatchling.build"
[project]
name = "depth-anything-3"
version = "0.0.0"
description = "Depth Anything 3"
readme = "README.md"
requires-python = ">=3.9, <=3.13"
license = { text = "Apache-2.0" }
authors = [{ name = "Your Name" }]
dependencies = [
"pre-commit",
"trimesh",
"torch>=2",
"torchvision",
"einops",
"huggingface_hub",
"imageio",
"numpy<2",
"opencv-python",
"open3d",
"fastapi",
"uvicorn",
"requests",
"typer",
"pillow",
"omegaconf",
"evo",
"e3nn",
"moviepy",
"plyfile",
"pillow_heif",
"safetensors",
"uvicorn",
"moviepy==1.0.3",
"typer>=0.9.0",
"pycolmap",
]
[project.optional-dependencies]
app = ["gradio>=5", "pillow>=9.0"] # requires that python3>=3.10
gs = ["gsplat @ git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70"]
all = ["depth-anything-3[app,gs]"]
[project.scripts]
da3 = "depth_anything_3.cli:app"
[project.urls]
Homepage = "https://github.com/ByteDance-Seed/Depth-Anything-3"
[tool.hatch.version]
source = "vcs"
[tool.hatch.build.targets.wheel]
packages = ["src/depth_anything_3"]
[tool.hatch.build.targets.sdist]
include = [
"/README.md",
"/pyproject.toml",
"/src/depth_anything_3",
]
[tool.hatch.metadata]
allow-direct-references = true
[tool.mypy]
plugins = ["jaxtyping.mypy_plugin"]
[tool.black]
line-length = 99
target-version = ['py37', 'py38', 'py39', 'py310', 'py311']
include = '\.pyi?$'
exclude = '''
/(
| \.git
)/
'''
[tool.isort]
profile = "black"
multi_line_output = 3
include_trailing_comma = true
known_third_party = ["bson","cruise","cv2","dataloader","diffusers","omegaconf","tensorflow","torch","torchvision","transformers","gsplat"]
known_first_party = ["common", "data", "models", "projects"]
sections = ["FUTURE","STDLIB","THIRDPARTY","FIRSTPARTY","LOCALFOLDER"]
skip_gitignore = true
line_length = 99
no_lines_before="THIRDPARTY"

View File

@@ -1,23 +0,0 @@
pre-commit
trimesh
torch>=2
torchvision
einops
huggingface_hub
imageio
numpy<2
opencv-python
open3d
fastapi
uvicorn
requests
typer
pillow
omegaconf
evo
e3nn
moviepy
plyfile
pillow_heif
safetensors
pycolmap

View File

@@ -1,84 +0,0 @@
# Copyright (c) 2025 ByteDance Ltd. and/or its affiliates
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from depth_anything_3.specs import Prediction
# Canonical set of export formats this dispatcher understands. Treat this as the
# server-trusted enum for any request-facing validation (e.g. the HTTP backend) instead
# of letting a client-supplied export_format string reach the dispatcher unchecked.
SUPPORTED_EXPORT_FORMATS = frozenset(
{
"glb",
"mini_npz",
"npz",
"feat_vis",
"depth_vis",
"gs_ply",
"gs_video",
"colmap",
}
)
def export(
prediction: Prediction,
export_format: str,
export_dir: str,
**kwargs,
):
if "-" in export_format:
export_formats = export_format.split("-")
for export_format in export_formats:
export(prediction, export_format, export_dir, **kwargs)
return # Prevent falling through to single-format handling
if export_format == "glb":
from .glb import export_to_glb
export_to_glb(prediction, export_dir, **kwargs.get(export_format, {}))
elif export_format == "mini_npz":
from .npz import export_to_mini_npz
export_to_mini_npz(prediction, export_dir)
elif export_format == "npz":
from .npz import export_to_npz
export_to_npz(prediction, export_dir)
elif export_format == "feat_vis":
from .feat_vis import export_to_feat_vis
export_to_feat_vis(prediction, export_dir, **kwargs.get(export_format, {}))
elif export_format == "depth_vis":
from .depth_vis import export_to_depth_vis
export_to_depth_vis(prediction, export_dir)
elif export_format == "gs_ply":
from .gs import export_to_gs_ply
export_to_gs_ply(prediction, export_dir, **kwargs.get(export_format, {}))
elif export_format == "gs_video":
from .gs import export_to_gs_video
export_to_gs_video(prediction, export_dir, **kwargs.get(export_format, {}))
elif export_format == "colmap":
from .colmap import export_to_colmap
export_to_colmap(prediction, export_dir, **kwargs.get(export_format, {}))
else:
raise ValueError(f"Unsupported export format: {export_format}")
__all__ = [
export,
]

View File

@@ -1,531 +0,0 @@
# Copyright (c) 2025 ByteDance Ltd. and/or its affiliates
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import torch
from einops import repeat
from .geometry import unproject_depth
def compute_optimal_rotation_intrinsics_batch(
rays_origin, rays_target, z_threshold=1e-4, reproj_threshold=0.2, weights=None,
n_sample = None,
n_iter=100,
num_sample_for_ransac=8,
rand_sample_iters_idx=None,
):
"""
Args:
rays_origin (torch.Tensor): (B, N, 3)
rays_target (torch.Tensor): (B, N, 3)
z_threshold (float): Threshold for z value to be considered valid.
Returns:
R (torch.tensor): (3, 3)
focal_length (torch.tensor): (2,)
principal_point (torch.tensor): (2,)
"""
device = rays_origin.device
B, N, _ = rays_origin.shape
z_mask = torch.logical_and(
torch.abs(rays_target[:, :, 2]) > z_threshold, torch.abs(rays_origin[:, :, 2]) > z_threshold
) # (B, N, 1)
rays_origin = rays_origin.clone()
rays_target = rays_target.clone()
rays_origin[:, :, 0][z_mask] /= rays_origin[:, :, 2][z_mask]
rays_origin[:, :, 1][z_mask] /= rays_origin[:, :, 2][z_mask]
rays_target[:, :, 0][z_mask] /= rays_target[:, :, 2][z_mask]
rays_target[:, :, 1][z_mask] /= rays_target[:, :, 2][z_mask]
rays_origin = rays_origin[:, :, :2]
rays_target = rays_target[:, :, :2]
assert weights is not None, "weights must be provided"
weights[~z_mask] = 0
A_list = []
max_chunk_size = 2
for i in range(0, rays_origin.shape[0], max_chunk_size):
A = ransac_find_homography_weighted_fast_batch(
rays_origin[i:i+max_chunk_size],
rays_target[i:i+max_chunk_size],
weights[i:i+max_chunk_size],
n_iter=n_iter,
n_sample = n_sample,
num_sample_for_ransac=num_sample_for_ransac,
reproj_threshold=reproj_threshold,
rand_sample_iters_idx=rand_sample_iters_idx,
max_inlier_num=8000,
)
A = A.to(device)
A_need_inv_mask = torch.linalg.det(A) < 0
A[A_need_inv_mask] = -A[A_need_inv_mask]
A_list.append(A)
A = torch.cat(A_list, dim=0)
R_list = []
f_list = []
pp_list = []
for i in range(A.shape[0]):
R, L = ql_decomposition(A[i])
L = L / L[2][2]
f = torch.stack((L[0][0], L[1][1]))
pp = torch.stack((L[2][0], L[2][1]))
R_list.append(R)
f_list.append(f)
pp_list.append(pp)
R = torch.stack(R_list)
f = torch.stack(f_list)
pp = torch.stack(pp_list)
return R, f, pp
# https://www.reddit.com/r/learnmath/comments/v1crd7/linear_algebra_qr_to_ql_decomposition/
def ql_decomposition(A):
device = A.device
P = torch.tensor([[0, 0, 1], [0, 1, 0], [1, 0, 0]], device=device).float()
A_tilde = torch.matmul(A, P)
# MPS 尚未实现 torch.linalg.qr,仅把 3x3 分解放到 CPU 执行后移回
use_cpu = device.type == "mps"
if use_cpu:
A_tilde = A_tilde.cpu()
Q_tilde, R_tilde = torch.linalg.qr(A_tilde)
if use_cpu:
Q_tilde = Q_tilde.to(device=device)
R_tilde = R_tilde.to(device=device)
Q = torch.matmul(Q_tilde, P)
L = torch.matmul(torch.matmul(P, R_tilde), P)
d = torch.diag(L)
Q[:, 0] *= torch.sign(d[0])
Q[:, 1] *= torch.sign(d[1])
Q[:, 2] *= torch.sign(d[2])
L[0] *= torch.sign(d[0])
L[1] *= torch.sign(d[1])
L[2] *= torch.sign(d[2])
return Q, L
def find_homography_least_squares_weighted_torch(src_pts, dst_pts, confident_weight):
"""
src_pts: (N,2) source points (torch.Tensor, float32/float64)
dst_pts: (N,2) target points (torch.Tensor, float32/float64)
confident_weight: (N,) weights (torch.Tensor)
Returns: (3,3) homography matrix H (torch.Tensor)
"""
assert src_pts.shape == dst_pts.shape
N = src_pts.shape[0]
if N < 4:
raise ValueError("At least 4 points are required to compute homography.")
assert confident_weight.shape == (N,)
w = confident_weight.sqrt().unsqueeze(1) # (N,1)
x = src_pts[:, 0:1] # (N,1)
y = src_pts[:, 1:2] # (N,1)
u = dst_pts[:, 0:1]
v = dst_pts[:, 1:2]
zeros = torch.zeros_like(x)
# Construct A matrix (2N, 9)
A1 = torch.cat([-x * w, -y * w, -w, zeros, zeros, zeros, x * u * w, y * u * w, u * w], dim=1)
A2 = torch.cat([zeros, zeros, zeros, -x * w, -y * w, -w, x * v * w, y * v * w, v * w], dim=1)
A = torch.cat([A1, A2], dim=0) # (2N, 9)
# SVD
# Note: torch.linalg.svd returns U, S, Vh, where Vh is the transpose of V
_, _, Vh = torch.linalg.svd(A)
H = Vh[-1].reshape(3, 3)
H = H / H[-1, -1]
return H
def ransac_find_homography_weighted(
src_pts,
dst_pts,
confident_weight,
n_iter=100,
sample_ratio=0.2,
reproj_threshold=3.0,
num_sample_for_ransac=16,
random_seed=None,
):
"""
RANSAC version of weighted Homography estimation.
Sample 4 points from the top 50% weighted points each time.
reproj_threshold: points with reprojection error less than this value are inliers
Returns: best_H
"""
if random_seed is not None:
torch.manual_seed(random_seed)
N = src_pts.shape[0]
assert N >= 4
# 1. Select top 50% weighted points
sorted_idx = torch.argsort(confident_weight, descending=True)
n_sample = max(num_sample_for_ransac, int(N * sample_ratio))
candidate_idx = sorted_idx[:n_sample]
best_inlier_mask = None
best_score = 0
for _ in range(n_iter):
# 2. Randomly sample 4 points
idx = candidate_idx[torch.randperm(n_sample)[:num_sample_for_ransac]]
# 3. Compute Homography
try:
H = find_homography_least_squares_weighted_torch(
src_pts[idx], dst_pts[idx], confident_weight[idx]
)
except Exception:
H = torch.eye(3, dtype=src_pts.dtype, device=src_pts.device)
# 4. Compute reprojection error for all points
src_homo = torch.cat(
[src_pts, torch.ones(N, 1, dtype=src_pts.dtype, device=src_pts.device)], dim=1
)
proj = (H @ src_homo.T).T
proj = proj[:, :2] / proj[:, 2:3]
error = ((proj - dst_pts) ** 2).sum(dim=1).sqrt() # Euclidean distance
inlier_mask = error < reproj_threshold
total_score = (inlier_mask * confident_weight).sum().item()
n_inlier = inlier_mask.sum().item()
if n_inlier < 4:
continue # At least 4 inliers required for fitting
if total_score > best_score:
best_score = total_score
best_inlier_mask = inlier_mask
# 5. Refit Homography using inliers
H_inlier = find_homography_least_squares_weighted_torch(
src_pts[best_inlier_mask], dst_pts[best_inlier_mask], confident_weight[best_inlier_mask]
)
return H_inlier
def find_homography_least_squares_weighted_torch_batch(
src_pts_batch, dst_pts_batch, confident_weight_batch
):
"""
Batch version of weighted least squares Homography
src_pts_batch: (B, K, 2)
dst_pts_batch: (B, K, 2)
confident_weight_batch: (B, K)
Returns: (B, 3, 3)
"""
B, K, _ = src_pts_batch.shape
w = confident_weight_batch.sqrt().unsqueeze(2) # (B,K,1)
x = src_pts_batch[:, :, 0:1]
y = src_pts_batch[:, :, 1:2]
u = dst_pts_batch[:, :, 0:1]
v = dst_pts_batch[:, :, 1:2]
zeros = torch.zeros_like(x)
A1 = torch.cat([-x * w, -y * w, -w, zeros, zeros, zeros, x * u * w, y * u * w, u * w], dim=2)
A2 = torch.cat([zeros, zeros, zeros, -x * w, -y * w, -w, x * v * w, y * v * w, v * w], dim=2)
A = torch.cat([A1, A2], dim=1) # (B, 2K, 9)
# SVD: torch.linalg.svd supports batch
_, _, Vh = torch.linalg.svd(A)
H = Vh[:, -1].reshape(B, 3, 3)
H = H / H[:, 2:3, 2:3]
return H
def ransac_find_homography_weighted_fast(
src_pts,
dst_pts,
confident_weight,
n_sample,
n_iter=100,
reproj_threshold=3.0,
num_sample_for_ransac=8,
random_seed=None,
rand_sample_iters_idx=None,
):
"""
Batch version of RANSAC weighted Homography estimation.
Returns: H_inlier
"""
if random_seed is not None:
torch.manual_seed(random_seed)
N = src_pts.shape[0]
device = src_pts.device
assert N >= 4
# 1. Select top weighted points by sample_ratio
sorted_idx = torch.argsort(confident_weight, descending=True)
candidate_idx = sorted_idx[:n_sample] # (n_sample,)
if rand_sample_iters_idx is None:
rand_sample_iters_idx = torch.stack(
[torch.randperm(n_sample, device=device)[:num_sample_for_ransac] for _ in range(n_iter)],
dim=0,
) # (n_iter, num_sample_for_ransac)
# 2. Generate all sampling groups at once
# shape: (n_iter, num_sample_for_ransac)
rand_idx = candidate_idx[rand_sample_iters_idx] # (n_iter, num_sample_for_ransac)
# 3. Construct batch input
src_pts_batch = src_pts[rand_idx] # (n_iter, num_sample_for_ransac, 2)
dst_pts_batch = dst_pts[rand_idx] # (n_iter, num_sample_for_ransac, 2)
confident_weight_batch = confident_weight[rand_idx] # (n_iter, num_sample_for_ransac)
# 4. Batch fit Homography
H_batch = find_homography_least_squares_weighted_torch_batch(
src_pts_batch, dst_pts_batch, confident_weight_batch
) # (n_iter, 3, 3)
# 5. Batch evaluate inliers for all H
src_homo = torch.cat(
[src_pts, torch.ones(N, 1, dtype=src_pts.dtype, device=src_pts.device)], dim=1
) # (N,3)
src_homo_expand = src_homo.unsqueeze(0).expand(n_iter, N, 3) # (n_iter, N, 3)
dst_pts_expand = dst_pts.unsqueeze(0).expand(n_iter, N, 2) # (n_iter, N, 2)
confident_weight_expand = confident_weight.unsqueeze(0).expand(n_iter, N) # (n_iter, N)
# H_batch: (n_iter, 3, 3)
proj = torch.bmm(src_homo_expand, H_batch.transpose(1, 2)) # (n_iter, N, 3)
proj_xy = proj[:, :, :2] / proj[:, :, 2:3] # (n_iter, N, 2)
error = ((proj_xy - dst_pts_expand) ** 2).sum(dim=2).sqrt() # (n_iter, N)
inlier_mask = error < reproj_threshold # (n_iter, N)
total_score = (inlier_mask * confident_weight_expand).sum(dim=1) # (n_iter,)
# 6. Select the sampling group with the highest score
best_idx = torch.argmax(total_score)
best_inlier_mask = inlier_mask[best_idx] # (N,)
inlier_src_pts = src_pts[best_inlier_mask]
inlier_dst_pts = dst_pts[best_inlier_mask]
inlier_confident_weight = confident_weight[best_inlier_mask]
max_inlier_num = 10000
sorted_idx = torch.argsort(inlier_confident_weight, descending=True)
# method 1: sort according to confident_weight, and only keep max_inlier_num pts
# sorted_idx = sorted_idx[:max_inlier_num]
# method 2: random choose max_inlier_num pts
sorted_idx = sorted_idx[torch.randperm(len(sorted_idx))[:max_inlier_num]]
inlier_src_pts = inlier_src_pts[sorted_idx]
inlier_dst_pts = inlier_dst_pts[sorted_idx]
inlier_confident_weight = inlier_confident_weight[sorted_idx]
# 7. Refit Homography using inliers
H_inlier = find_homography_least_squares_weighted_torch(
inlier_src_pts, inlier_dst_pts, inlier_confident_weight
)
return H_inlier
def ransac_find_homography_weighted_fast_batch(
src_pts, # (B, N, 3)
dst_pts, # (B, N, 2)
confident_weight, # (B, N)
n_sample,
n_iter=100,
reproj_threshold=3.0,
num_sample_for_ransac=8,
max_inlier_num=10000,
random_seed=None,
rand_sample_iters_idx=None,
):
"""
Batch version of RANSAC weighted Homography estimation (supports batch).
Input:
src_pts: (B, N, 2)
dst_pts: (B, N, 2)
confident_weight: (B, N)
Returns:
H_inlier: (B, 3, 3)
"""
if random_seed is not None:
torch.manual_seed(random_seed)
B, N, _ = src_pts.shape
assert N >= 4
device = src_pts.device
# 1. Select top weighted points by sample_ratio
sorted_idx = torch.argsort(confident_weight, descending=True, dim=1) # (B, N)
candidate_idx = sorted_idx[:, :n_sample] # (B, n_sample)
# 2. Generate all sampling groups at once
# rand_idx: (B, n_iter, num_sample_for_ransac)
if rand_sample_iters_idx is None:
rand_sample_iters_idx = torch.stack(
[torch.randperm(n_sample, device=device)[:num_sample_for_ransac] for _ in range(n_iter)],
dim=0,
) # (n_iter, num_sample_for_ransac)
rand_idx = candidate_idx[:, rand_sample_iters_idx] # (B, n_iter, num_sample_for_ransac)
# 3. Construct batch input
# Indexing method below: (B, n_iter, num_sample_for_ransac, ...)
b_idx = torch.arange(B, device=device).view(B, 1, 1).expand(B, n_iter, num_sample_for_ransac)
src_pts_batch = src_pts[b_idx, rand_idx] # (B, n_iter, num_sample_for_ransac, 2)
dst_pts_batch = dst_pts[b_idx, rand_idx] # (B, n_iter, num_sample_for_ransac, 2)
confident_weight_batch = confident_weight[b_idx, rand_idx] # (B, n_iter, num_sample_for_ransac)
# 4. Batch fit Homography
# Need to implement batch version that supports (B, n_iter, num_sample_for_ransac, ...) input
# Output H_batch: (B, n_iter, 3, 3)
cB, cN = src_pts_batch.shape[:2]
H_batch = find_homography_least_squares_weighted_torch_batch(
src_pts_batch.flatten(0, 1), dst_pts_batch.flatten(0, 1), confident_weight_batch.flatten(0, 1)
) # (B, n_iter, 3, 3)
H_batch = H_batch.unflatten(0, (cB, cN))
# 5. Batch evaluate inliers for all H
src_homo = torch.cat(
[src_pts, torch.ones(B, N, 1, dtype=src_pts.dtype, device=src_pts.device)], dim=2
) # (B, N, 3)
src_homo_expand = src_homo.unsqueeze(1).expand(B, n_iter, N, 3) # (B, n_iter, N, 3)
dst_pts_expand = dst_pts.unsqueeze(1).expand(B, n_iter, N, 2) # (B, n_iter, N, 2)
confident_weight_expand = confident_weight.unsqueeze(1).expand(B, n_iter, N) # (B, n_iter, N)
# H_batch: (B, n_iter, 3, 3)
# Need to reshape H_batch to (B*n_iter, 3, 3), src_homo_expand to (B*n_iter, N, 3)
H_batch_flat = H_batch.reshape(-1, 3, 3)
src_homo_expand_flat = src_homo_expand.reshape(-1, N, 3)
proj = torch.bmm(src_homo_expand_flat, H_batch_flat.transpose(1, 2)) # (B*n_iter, N, 3)
proj_xy = proj[:, :, :2] / proj[:, :, 2:3] # (B*n_iter, N, 2)
proj_xy = proj_xy.reshape(B, n_iter, N, 2)
error = ((proj_xy - dst_pts_expand) ** 2).sum(dim=3).sqrt() # (B, n_iter, N)
inlier_mask = error < reproj_threshold # (B, n_iter, N)
total_score = (inlier_mask * confident_weight_expand).sum(dim=2) # (B, n_iter)
# 6. Select the sampling group with the highest score
best_idx = torch.argmax(total_score, dim=1) # (B,)
best_inlier_mask = inlier_mask[torch.arange(B, device=device), best_idx] # (B, N)
# 7. Refit Homography using inliers
H_inlier_list = []
for b in range(B):
mask = best_inlier_mask[b]
inlier_src_pts = src_pts[b][mask] # (?, 3)
inlier_dst_pts = dst_pts[b][mask] # (?, 2)
inlier_confident_weight = confident_weight[b][mask] # (?)
sorted_idx = torch.argsort(inlier_confident_weight, descending=True)
# # method 1: sort according to confident_weight, and only keep max_inlier_num pts
# sorted_idx = sorted_idx[:max_inlier_num]
# method 2: random choose max_inlier_num pts
if len(sorted_idx) > max_inlier_num:
# random choose from first 95% confident pts
keep_len = max(int(len(sorted_idx) * 0.95), max_inlier_num)
sorted_idx = sorted_idx[:keep_len]
perm = torch.randperm(len(sorted_idx), device=device)[:max_inlier_num]
sorted_idx = sorted_idx[perm]
inlier_src_pts = inlier_src_pts[sorted_idx]
inlier_dst_pts = inlier_dst_pts[sorted_idx]
inlier_confident_weight = inlier_confident_weight[sorted_idx]
H_inlier = find_homography_least_squares_weighted_torch(
inlier_src_pts, inlier_dst_pts, inlier_confident_weight
) # (3, 3)
H_inlier_list.append(H_inlier)
H_inlier = torch.stack(H_inlier_list, dim=0) # (B, 3, 3)
return H_inlier
def get_params_for_ransac(N, device):
n_iter=100
sample_ratio=0.3
num_sample_for_ransac=8
n_sample = max(num_sample_for_ransac, int(N * sample_ratio))
rand_sample_iters_idx = torch.stack(
[torch.randperm(n_sample, device=device)[:num_sample_for_ransac] for _ in range(n_iter)],
dim=0,
) # (n_iter, num_sample_for_ransac)
return n_iter, num_sample_for_ransac, n_sample, rand_sample_iters_idx
def camray_to_caminfo(camray, confidence=None, reproj_threshold=0.2, training=False):
"""
Args:
camray: (B, S, num_patches_y, num_patches_x, 6)
confidence: (B, S, num_patches_y, num_patches_x)
Returns:
R: (B, S, 3, 3)
T: (B, S, 3)
focal_lengths: (B, S, 2)
principal_points: (B, S, 2)
"""
if confidence is None:
confidence = torch.ones_like(camray[:, :, :, :, 0])
B, S, num_patches_y, num_patches_x, _ = camray.shape
# identity K, assume imw=imh=2.0
I_K = torch.eye(3, dtype=camray.dtype, device=camray.device)
I_K[0, 2] = 1.0
I_K[1, 2] = 1.0
# repeat I_K to match camray
I_K = I_K.unsqueeze(0).unsqueeze(0).expand(B, S, -1, -1)
cam_plane_depth = torch.ones(
B, S, num_patches_y, num_patches_x, 1, dtype=camray.dtype, device=camray.device
)
I_cam_plane_unproj = unproject_depth(
cam_plane_depth,
I_K,
c2w=None,
ixt_normalized=True,
num_patches_x=num_patches_x,
num_patches_y=num_patches_y,
) # (B, S, num_patches_y, num_patches_x, 3)
camray = camray.flatten(0, 1).flatten(1, 2) # (B*S, num_patches_y*num_patches_x, 6)
I_cam_plane_unproj = I_cam_plane_unproj.flatten(0, 1).flatten(
1, 2
) # (B*S, num_patches_y*num_patches_x, 3)
confidence = confidence.flatten(0, 1).flatten(1, 2) # (B*S, num_patches_y*num_patches_x)
# Compute optimal rotation to align rays
N = camray.shape[-2]
device = camray.device
n_iter, num_sample_for_ransac, n_sample, rand_sample_iters_idx = get_params_for_ransac(N, device)
# Use batch processing (confidence is guaranteed to be not None at this point)
if training:
camray = camray.clone().detach()
I_cam_plane_unproj = I_cam_plane_unproj.clone().detach()
confidence = confidence.clone().detach()
R, focal_lengths, principal_points = compute_optimal_rotation_intrinsics_batch(
I_cam_plane_unproj,
camray[:, :, :3],
reproj_threshold=reproj_threshold,
weights=confidence,
n_sample = n_sample,
n_iter=n_iter,
num_sample_for_ransac=num_sample_for_ransac,
rand_sample_iters_idx=rand_sample_iters_idx,
)
T = torch.sum(camray[:, :, 3:] * confidence.unsqueeze(-1), dim=1) / torch.sum(
confidence, dim=-1, keepdim=True
)
R = R.reshape(B, S, 3, 3)
T = T.reshape(B, S, 3)
focal_lengths = focal_lengths.reshape(B, S, 2)
principal_points = principal_points.reshape(B, S, 2)
return R, T, 1.0 / focal_lengths, principal_points + 1.0
def get_extrinsic_from_camray(camray, conf, patch_size_y, patch_size_x, training=False):
pred_R, pred_T, pred_focal_lengths, pred_principal_points = camray_to_caminfo(
camray, confidence=conf.squeeze(-1), training=training
)
pred_extrinsic = torch.cat(
[
torch.cat([pred_R, pred_T.unsqueeze(-1)], dim=-1),
repeat(
torch.tensor([0, 0, 0, 1], dtype=pred_R.dtype, device=pred_R.device),
"c -> b s 1 c",
b=pred_R.shape[0],
s=pred_R.shape[1],
),
],
dim=-2,
) # B, S, 4, 4
return pred_extrinsic, pred_focal_lengths, pred_principal_points

View File

@@ -1,22 +0,0 @@
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "moge"
version = "2.0.0"
description = "MoGe monocular geometry estimation"
readme = "README.md"
license = {text = "MIT"}
dependencies = [
"huggingface_hub",
"numpy",
"pillow",
"torch>=2.0.0",
"utils3d",
]
requires-python = ">=3.9"
[tool.setuptools.packages.find]
where = ["."]
include = ["moge*"]

View File

@@ -1,16 +0,0 @@
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "utils3d"
version = "1.7"
description = "Utilities for 3D computer vision"
readme = "README.md"
license = {text = "MIT"}
dependencies = ["numpy", "scipy"]
requires-python = ">=3.8"
[tool.setuptools.packages.find]
where = ["."]
include = ["utils3d*"]

View File

@@ -1,111 +0,0 @@
import argparse
import hashlib
import sys
import tomllib
import urllib.request
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent
LOCK_PATH = PROJECT_ROOT / "vendor.lock.toml"
VENDOR_DIR = PROJECT_ROOT / "vendor"
CHUNK_SIZE = 1024 * 1024
def load_lock(path=LOCK_PATH):
"""Load and validate the vendor lock file."""
with Path(path).open("rb") as stream:
lock = tomllib.load(stream)
if lock.get("schema_version") != 1:
raise RuntimeError("Unsupported vendor.lock.toml schema")
return lock
def file_sha256(path):
"""Return the SHA-256 digest for a file."""
digest = hashlib.sha256()
with Path(path).open("rb") as stream:
while chunk := stream.read(CHUNK_SIZE):
digest.update(chunk)
return digest.hexdigest()
def download_artifact(artifact, destination_dir=VENDOR_DIR):
"""Download one locked artifact and verify its content."""
destination_dir = Path(destination_dir)
destination = destination_dir / artifact["filename"]
expected = artifact["sha256"]
if destination.is_file() and file_sha256(destination) == expected:
print(f"Ready: {destination.name}")
return destination
destination_dir.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(f"{destination.name}.part")
temporary.unlink(missing_ok=True)
print(f"Downloading: {destination.name}")
request = urllib.request.Request(
artifact["url"],
headers={"User-Agent": "SnapForm vendor preparation"},
)
try:
with urllib.request.urlopen(request) as response, temporary.open("wb") as output:
while chunk := response.read(CHUNK_SIZE):
output.write(chunk)
actual = file_sha256(temporary)
if actual != expected:
raise RuntimeError(
f"SHA-256 mismatch for {destination.name}: "
f"expected {expected}, got {actual}"
)
temporary.replace(destination)
except Exception:
temporary.unlink(missing_ok=True)
raise
return destination
def prepare(platforms=None, lock_path=LOCK_PATH, destination_dir=VENDOR_DIR):
"""Prepare common third-party archives for selected platforms."""
lock = load_lock(lock_path)
available = lock["platforms"]
selected = list(available) if platforms is None else platforms
unknown = sorted(set(selected) - set(available))
if unknown:
raise ValueError(f"Unsupported platform: {', '.join(unknown)}")
artifacts = list(lock["artifacts"].values())
artifacts.extend(available[platform] for platform in selected)
return [
download_artifact(artifact, destination_dir)
for artifact in artifacts
]
def parse_args(argv=None):
"""Parse command-line options."""
parser = argparse.ArgumentParser(
description="Download and verify third-party files for SnapForm.",
)
parser.add_argument(
"--platform",
action="append",
dest="platforms",
help="Prepare one platform; repeat as needed. Default: all platforms.",
)
return parser.parse_args(argv)
def main(argv=None):
"""Run vendor preparation from the command line."""
args = parse_args(argv)
try:
paths = prepare(args.platforms)
except (OSError, RuntimeError, ValueError) as error:
print(f"Preparation failed: {error}", file=sys.stderr)
return 1
print(f"Prepared {len(paths)} files")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -1,14 +1,17 @@
[project]
name = "snapform"
version = "0.10.11"
version = "0.12.3"
description = "Fast image-guided structure building tools for Blender"
readme = "README.md"
requires-python = ">=3.11"
license = "GPL-3.0-or-later"
dependencies = []
dependencies = ["blendjob"]
[dependency-groups]
dev = ["numpy", "pytest"]
dev = ["numpy", "pillow", "pytest"]
[tool.uv]
package = false
[tool.uv.sources]
blendjob = { path = "src/snapform/wheels/blendjob-0.1.6-py3-none-any.whl" }

View File

@@ -1,57 +1,43 @@
import bpy
from . import client
from .image_depth import (
ApplyRemovedBackground,
BuildDepthPlane,
BuildOriginalDepthPlane,
BuildMoge2DepthPlane,
ConvertImageToDepthPlane,
from .blender import depth_plane as image_depth
from .blender import surface as image_mesh
from .blender.depth_plane import (
ConvertToDepthPlane,
RemoveImageBackground,
SnapFormImageMenu,
draw_image_context_menu,
)
from .image_mesh import BuildImageMesh, ConvertImageToMesh
from .blender.surface import ConvertToSurface
from .operators import (
CancelJob,
ControlService,
RunJob,
cancel_active_job,
close_active_job,
JOB_OPERATOR_CLASSES,
)
from .job_runtime import runtime
from .panel import (
Ben2RemovalDebugPanel,
ServerPanel,
Da3OutputDebugPanel,
DebugPanel,
AiServicePanel,
ModelsPanel,
Moge2ApiDebugPanel,
)
from .preferences import SnapFormPreferences
from .properties import SnapFormSettings
from .status_bar import draw_status_bar
CLASSES = (
SnapFormPreferences,
SnapFormSettings,
RunJob,
CancelJob,
ControlService,
*JOB_OPERATOR_CLASSES,
SnapFormImageMenu,
ApplyRemovedBackground,
RemoveImageBackground,
BuildDepthPlane,
BuildOriginalDepthPlane,
BuildMoge2DepthPlane,
ConvertImageToDepthPlane,
BuildImageMesh,
ConvertImageToMesh,
AiServicePanel,
ConvertToDepthPlane,
ConvertToSurface,
ServerPanel,
ModelsPanel,
DebugPanel,
Da3OutputDebugPanel,
Moge2ApiDebugPanel,
Da3OutputDebugPanel,
Ben2RemovalDebugPanel,
)
@@ -62,20 +48,15 @@ def register():
bpy.types.Scene.snapform_settings = bpy.props.PointerProperty(
type=SnapFormSettings,
)
bpy.types.STATUSBAR_HT_header.append(draw_status_bar)
bpy.types.VIEW3D_MT_object_context_menu.append(draw_image_context_menu)
bpy.types.OUTLINER_MT_object.append(draw_image_context_menu)
client.schedule_service_start()
runtime.register()
def unregister():
cancel_active_job()
close_active_job()
client.unschedule_service()
client.detach_service_client()
runtime.unregister()
bpy.types.VIEW3D_MT_object_context_menu.remove(draw_image_context_menu)
bpy.types.OUTLINER_MT_object.remove(draw_image_context_menu)
bpy.types.STATUSBAR_HT_header.remove(draw_status_bar)
if hasattr(bpy.types.Scene, "snapform_settings"):
del bpy.types.Scene.snapform_settings
for class_type in reversed(CLASSES):

Binary file not shown.

View File

@@ -0,0 +1 @@
"""Blender-process geometry construction and import logic."""

View File

@@ -1,5 +1,6 @@
import re
import shutil
import tempfile
from pathlib import Path
import bpy
@@ -19,6 +20,7 @@ MATERIAL_DISPLACEMENT_ATTRIBUTE = "o_material_displacement"
BACKGROUND_DEPTH_ATTRIBUTE = "o_background_depth"
DEPTH_SCALE_ATTRIBUTE = "o_depth_scale"
SURFACE_MATERIAL_NODE_GROUP_NAME = "O DA3 Surface Layer"
MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME = "O MoGe2 Surface Layer"
DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME = "O DA3 Depth Plane Layer"
MOGE_MATERIAL_NODE_GROUP_NAME = "O MoGe2 Depth Plane Layer"
DA3_PATCH_SIZE = 14
@@ -52,14 +54,11 @@ VIDEO_NAME_SUFFIXES = (
def image_depth_model_items(owner, context):
from .models import depth_plane_model_enum_items
from ..properties import depth_plane_model_enum_items
return depth_plane_model_enum_items(owner, context)
_last_depth_plane_model = "MOGE2_VITB_NORMAL"
def is_image_empty(candidate):
return bool(
candidate
@@ -169,6 +168,14 @@ def depth_data_name(source_object):
return f"{image_base_name(source_object.data)}.depth"
def normal_data_name(source_object):
return f"{image_base_name(source_object.data)}.normal"
def mask_data_name(source_object):
return f"{image_base_name(source_object.data)}.mask"
def copy_depth_image(source_object):
"""Return a renamed copy of the source image for use as the depth map."""
depth_image = source_object.data.copy()
@@ -212,19 +219,17 @@ def process_resolution_scale_property():
def draw_process_resolution(layout, owner, image):
process_res = scaled_process_resolution(
image,
owner.snapform_process_res_factor,
owner.process_res_factor,
)
layout.prop(
owner,
"snapform_process_res_factor",
"process_res_factor",
text=f"Resolution Scale ({process_res} px)",
)
def save_packed_image(image, scene):
from .client import temporary_input_path
output_path = temporary_input_path()
output_path = Path(tempfile.mkdtemp(prefix="snapform-input-")) / "input.png"
image_settings = scene.render.image_settings
original_file_format = image_settings.file_format
try:
@@ -280,9 +285,7 @@ def prepare_image_input(image, scene):
return source_path, False
sequence_frames = collect_sequence_frames(image)
if sequence_frames:
from .client import temporary_input_directory
temporary_directory = Path(temporary_input_directory())
temporary_directory = Path(tempfile.mkdtemp(prefix="snapform-input-"))
try:
for index, frame_path in enumerate(sequence_frames):
suffix = frame_path.suffix.lower() or ".jpg"
@@ -820,11 +823,12 @@ def create_moge_mask_image(name, mask_values):
return image
def create_moge_normal_image(name, normal_values):
def create_moge_normal_image(name, normal_values, convert_coordinates=True):
import numpy as np
converted = normal_values.astype(np.float32, copy=True)
converted *= np.array([1.0, -1.0, -1.0], dtype=np.float32)
if convert_coordinates:
converted *= np.array([1.0, -1.0, -1.0], dtype=np.float32)
converted = np.clip(converted * 0.5 + 0.5, 0.0, 1.0)
height, width = converted.shape[:2]
image = bpy.data.images.new(
@@ -1006,10 +1010,13 @@ def create_moge_material(
uv.location = (-900.0, 200.0)
color = nodes.new("ShaderNodeTexImage")
color.image = color_image
color.interpolation = "Linear"
color.extension = "EXTEND"
color.location = (-650.0, 300.0)
depth = nodes.new("ShaderNodeTexImage")
depth.image = depth_image
depth.interpolation = "Linear"
depth.extension = "EXTEND"
depth.location = (-650.0, -100.0)
if frame_count > 1:
configure_image_user(
@@ -1127,11 +1134,11 @@ def create_plane_mesh(source_object, bounds=None, subdivision_level=5):
return mesh
def material_node_group(depth_plane=False):
def material_node_group(depth_plane=False, surface_group_name=None):
group_name = (
DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME
if depth_plane
else SURFACE_MATERIAL_NODE_GROUP_NAME
else surface_group_name or SURFACE_MATERIAL_NODE_GROUP_NAME
)
existing = bpy.data.node_groups.get(group_name)
if existing is not None:
@@ -1143,6 +1150,9 @@ def material_node_group(depth_plane=False):
("Alpha", "NodeSocketFloat"),
("Bump Scale", "NodeSocketFloat"),
]
if group_name == MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME:
input_sockets.insert(2, ("Normal", "NodeSocketColor"))
input_sockets.insert(3, ("Normal Strength", "NodeSocketFloat"))
if depth_plane:
input_sockets.insert(2, ("Depth", "NodeSocketFloat"))
for name, socket_type in input_sockets:
@@ -1160,6 +1170,10 @@ def material_node_group(depth_plane=False):
socket.default_value = 1.0
socket.min_value = -10000.0
socket.max_value = 10000.0
elif name == "Normal Strength":
socket.default_value = 1.0
socket.min_value = 0.0
socket.max_value = 10.0
output_sockets = [
("Color", "NodeSocketColor"),
("Roughness", "NodeSocketFloat"),
@@ -1190,6 +1204,16 @@ def material_node_group(depth_plane=False):
roughness_curve.location = (-250.0, 350.0)
bump = nodes.new("ShaderNodeBump")
bump.location = (-250.0, 0.0)
if group_name == MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME:
normal_map = nodes.new("ShaderNodeNormalMap")
normal_map.space = "OBJECT"
normal_map.location = (-500.0, -150.0)
links.new(group_input.outputs["Normal"], normal_map.inputs["Color"])
links.new(
group_input.outputs["Normal Strength"],
normal_map.inputs["Strength"],
)
links.new(normal_map.outputs["Normal"], bump.inputs["Normal"])
displacement = None
processed_alpha = group_input.outputs["Alpha"]
@@ -1269,7 +1293,7 @@ def material_node_group(depth_plane=False):
group.description = (
"Prepare DA3 depth-plane material outputs."
if depth_plane
else "Prepare DA3 surface material outputs."
else f"Prepare {group_name} material outputs."
)
group.use_fake_user = True
return group
@@ -1293,6 +1317,8 @@ def create_material(
depth_frame_offset=0,
depth_frame_duration=None,
use_cyclic=None,
surface_group_name=None,
normal_image=None,
):
material = bpy.data.materials.new(name)
material.use_nodes = True
@@ -1347,13 +1373,24 @@ def create_material(
alpha = color_texture.outputs["Alpha"]
material_inputs = nodes.new("ShaderNodeGroup")
is_depth_plane = displacement_image is not None
material_inputs.node_tree = material_node_group(is_depth_plane)
material_inputs.node_tree = material_node_group(
is_depth_plane,
surface_group_name=surface_group_name,
)
material_inputs.name = material_inputs.node_tree.name
material_inputs.label = material_inputs.name
material_inputs.location = (-300.0, 350.0)
material_inputs.inputs["Bump Scale"].default_value = depth_scale
links.new(color_texture.outputs["Color"], material_inputs.inputs["Color"])
links.new(alpha, material_inputs.inputs["Alpha"])
if normal_image is not None and "Normal" in material_inputs.inputs:
normal_texture = nodes.new("ShaderNodeTexImage")
normal_texture.image = normal_image
normal_texture.interpolation = "Linear"
normal_texture.extension = "EXTEND"
normal_texture.location = (-1000.0, -50.0)
links.new(uv_map.outputs["UV"], normal_texture.inputs["Vector"])
links.new(normal_texture.outputs["Color"], material_inputs.inputs["Normal"])
if unshaded:
emission = nodes.new("ShaderNodeEmission")
transparent = nodes.new("ShaderNodeBsdfTransparent")
@@ -1447,7 +1484,7 @@ def set_displacement_only(material):
def node_asset_path():
return Path(__file__).with_name("assets") / NODE_ASSET_NAME
return Path(__file__).parent.parent / "assets" / NODE_ASSET_NAME
def load_depth_plane_node_group(group_name=DEPTH_PLANE_NODE_GROUP_NAME):
@@ -1498,6 +1535,53 @@ def modifier_input_identifier(node_group, name):
raise RuntimeError(f"Geometry Nodes input is missing: {name}")
def modifier_input_slots(modifier):
"""Return the runtime input collection, or None for the legacy API."""
properties = getattr(modifier, "properties", None)
if properties is None:
return None
return getattr(properties, "inputs", None)
def set_modifier_input(modifier, identifier, value):
"""Assign a value to a Geometry Nodes modifier input socket."""
inputs = modifier_input_slots(modifier)
if inputs is not None:
# Blender 5.2+ exposes inputs as runtime RNA keyed by socket identifier.
getattr(inputs, identifier).value = value
else:
# Blender 4.x exposes inputs as modifier ID properties.
modifier[identifier] = value
def get_modifier_input(modifier, identifier):
"""Read a Geometry Nodes modifier input socket value."""
inputs = modifier_input_slots(modifier)
if inputs is not None:
return getattr(inputs, identifier).value
return modifier[identifier]
def modifier_input_driver_path(modifier, identifier):
"""Return the driver data path for a modifier input socket."""
if modifier_input_slots(modifier) is not None:
return (
f'modifiers["{modifier.name}"]'
f".properties.inputs.{identifier}.value"
)
return f'modifiers["{modifier.name}"]["{identifier}"]'
def drive_modifier_input(owner_object, modifier, identifier, expression="frame"):
"""Add a scripted driver to a modifier input so it follows the scene frame."""
driver = owner_object.driver_add(
modifier_input_driver_path(modifier, identifier)
)
driver.driver.type = "SCRIPTED"
driver.driver.expression = expression
return driver
def add_depth_plane_processing(
plane_object,
depth_image,
@@ -1525,12 +1609,10 @@ def add_depth_plane_processing(
("Cyclic", use_cyclic),
):
identifier = modifier_input_identifier(node_group, name)
modifier[identifier] = value
set_modifier_input(modifier, identifier, value)
if animate_frame:
frame_identifier = modifier_input_identifier(node_group, "Frame")
driver = modifier.driver_add(f'["{frame_identifier}"]')
driver.driver.type = "SCRIPTED"
driver.driver.expression = "frame"
drive_modifier_input(plane_object, modifier, frame_identifier)
plane_object.update_tag(refresh={"DATA"})
return modifier
@@ -1561,12 +1643,11 @@ def add_moge_depth_plane_processing(
("Frames", max(int(frame_duration or 1), 1)),
("Cyclic", use_cyclic),
):
modifier[modifier_input_identifier(node_group, name)] = value
identifier = modifier_input_identifier(node_group, name)
set_modifier_input(modifier, identifier, value)
if animate_frame:
frame_identifier = modifier_input_identifier(node_group, "Frame")
driver = modifier.driver_add(f'["{frame_identifier}"]')
driver.driver.type = "SCRIPTED"
driver.driver.expression = "frame"
drive_modifier_input(plane_object, modifier, frame_identifier)
plane_object.update_tag(refresh={"DATA"})
return modifier
@@ -1784,88 +1865,6 @@ def build_depth_plane(
)
class BuildDepthPlane(bpy.types.Operator):
bl_idname = "snapform.build_depth_plane"
bl_label = "Build Depth Plane"
bl_options = {"INTERNAL"}
snapform_source_object_name: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_npz_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_depth_data_name: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_background_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_color_frames_dir: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_depth_frames_dir: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_delete_source: bpy.props.BoolProperty(
default=True,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_unshaded: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
def execute(self, context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
if not is_image_empty(source_object):
self.report(
{"ERROR"},
"The source image empty is no longer available",
)
return {"CANCELLED"}
npz_path = Path(self.snapform_npz_path)
if not npz_path.is_file():
self.report(
{"ERROR"},
f"Depth result does not exist: {npz_path}",
)
return {"CANCELLED"}
try:
foreground_image = None
if self.snapform_background_path:
foreground_image = create_removed_background_image(
source_object,
self.snapform_background_path,
assign_to_source=False,
)
color_image = None
if self.snapform_color_frames_dir:
color_image = create_sequence_image(
f"{image_base_name(source_object.data)}.frame",
self.snapform_color_frames_dir,
)
build_depth_plane(
context,
source_object,
npz_path,
self.snapform_delete_source,
unshaded=self.snapform_unshaded,
depth_name=self.snapform_depth_data_name,
color_image=color_image,
foreground_image=foreground_image,
depth_frames_directory=(
self.snapform_depth_frames_dir or None
),
)
except (OSError, RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
return {"FINISHED"}
def build_moge2_depth_plane(
context,
source_object,
@@ -1918,13 +1917,13 @@ def build_moge2_depth_plane(
depth_scale = apply_plane_transform(plane_object, source_object, bounds)
apply_object_scale_to_mesh(plane_object)
depth_image = create_depth_image(
f"{depth_data_name(source_object)}.moge2",
depth_data_name(source_object),
displacement[0] * depth_scale,
)
if len(depth) > 1 and data_frames_directory is not None:
bpy.data.images.remove(depth_image, do_unlink=True)
depth_image = create_moge_sequence_image(
f"{depth_data_name(source_object)}.moge2",
depth_data_name(source_object),
[frame * depth_scale for frame in displacement],
context.scene,
data_frames_directory,
@@ -1962,173 +1961,6 @@ def build_moge2_depth_plane(
)
class BuildMoge2DepthPlane(bpy.types.Operator):
bl_idname = "snapform.build_moge2_depth_plane"
bl_label = "Build MoGe-2 Depth Plane"
bl_options = {"INTERNAL"}
snapform_source_object_name: bpy.props.StringProperty(options={"HIDDEN"})
snapform_npz_path: bpy.props.StringProperty(options={"HIDDEN"})
snapform_background_path: bpy.props.StringProperty(options={"HIDDEN"})
snapform_color_frames_dir: bpy.props.StringProperty(options={"HIDDEN"})
snapform_data_frames_dir: bpy.props.StringProperty(options={"HIDDEN"})
snapform_delete_source: bpy.props.BoolProperty(default=True, options={"HIDDEN"})
snapform_unshaded: bpy.props.BoolProperty(default=False, options={"HIDDEN"})
def execute(self, context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
if not is_image_empty(source_object):
self.report({"ERROR"}, "The source image empty is no longer available")
return {"CANCELLED"}
try:
foreground_image = None
if self.snapform_background_path:
foreground_image = create_removed_background_image(
source_object,
self.snapform_background_path,
assign_to_source=False,
)
color_image = None
if self.snapform_color_frames_dir:
color_image = create_sequence_image(
f"{image_base_name(source_object.data)}.frame",
self.snapform_color_frames_dir,
)
build_moge2_depth_plane(
context,
source_object,
Path(self.snapform_npz_path),
self.snapform_delete_source,
unshaded=self.snapform_unshaded,
color_image=color_image,
foreground_image=foreground_image,
data_frames_directory=(
self.snapform_data_frames_dir or None
),
)
except (OSError, RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
return {"FINISHED"}
class BuildOriginalDepthPlane(bpy.types.Operator):
bl_idname = "snapform.build_original_depth_plane"
bl_label = "Build Original Depth Plane"
bl_options = {"INTERNAL"}
snapform_source_object_name: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_background_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_color_frames_dir: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_depth_frames_dir: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_depth_npz_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_delete_source: bpy.props.BoolProperty(
default=True,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_unshaded: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
def execute(self, context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
if not is_image_empty(source_object):
self.report(
{"ERROR"},
"The source image empty is no longer available",
)
return {"CANCELLED"}
try:
foreground_image = None
if self.snapform_background_path:
foreground_image = create_removed_background_image(
source_object,
self.snapform_background_path,
assign_to_source=False,
)
color_image = None
if self.snapform_color_frames_dir:
color_image = create_sequence_image(
f"{image_base_name(source_object.data)}.frame",
self.snapform_color_frames_dir,
)
depth_image = None
if self.snapform_depth_npz_path:
depth_arrays = extract_depth_arrays(
self.snapform_depth_npz_path
)
if len(depth_arrays) == 1:
depth_image = create_depth_image(
depth_data_name(source_object),
depth_arrays[0],
)
else:
depth_image = create_depth_sequence_image(
depth_data_name(source_object),
depth_arrays,
context.scene,
self.snapform_depth_frames_dir,
)
build_image_depth_plane(
context,
source_object,
self.snapform_delete_source,
unshaded=self.snapform_unshaded,
color_image=color_image,
foreground_image=foreground_image,
depth_frames_directory=(
self.snapform_depth_frames_dir or None
),
depth_image=depth_image,
)
except (OSError, RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
return {"FINISHED"}
class ApplyRemovedBackground(bpy.types.Operator):
bl_idname = "snapform.apply_removed_background"
bl_label = "Apply Removed Background"
bl_options = {"INTERNAL"}
snapform_source_object_name: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_output_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
def execute(self, _context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
if not is_image_empty(source_object):
self.report(
{"ERROR"},
"The source image empty is no longer available",
)
return {"CANCELLED"}
try:
create_removed_background_image(
source_object,
self.snapform_output_path,
)
except RuntimeError as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
return {"FINISHED"}
class RemoveImageBackground(bpy.types.Operator):
bl_idname = "snapform.remove_image_background"
bl_label = "Remove Background"
@@ -2137,20 +1969,16 @@ class RemoveImageBackground(bpy.types.Operator):
@classmethod
def poll(cls, context):
from .operators import active_job
from .client import service_busy, model_ready
from .models import BEN2_MODEL
from ..job_runtime import runtime
from ..properties import BEN2_MODEL_KEY, model_ready
return (
is_image_empty(context.object)
and active_job() is None
and not service_busy()
and model_ready(BEN2_MODEL.key)
and not runtime.server_busy()
and model_ready(BEN2_MODEL_KEY)
)
def execute(self, context):
from .operators import IMAGE_BACKGROUND_JOB
source_object = context.object
total_frames = video_total_frames(source_object.data)
if total_frames > 1:
@@ -2166,11 +1994,10 @@ class RemoveImageBackground(bpy.types.Operator):
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=IMAGE_BACKGROUND_JOB,
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_delete_input=delete_input,
result = bpy.ops.snapform.process_image_background(
input_path=str(source_path),
source_object_name=source_object.name,
delete_input=delete_input,
)
if "CANCELLED" not in result:
return {"FINISHED"}
@@ -2179,42 +2006,37 @@ class RemoveImageBackground(bpy.types.Operator):
return {"CANCELLED"}
class ConvertImageToDepthPlane(bpy.types.Operator):
bl_idname = "snapform.convert_image_to_depth_plane"
class ConvertToDepthPlane(bpy.types.Operator):
bl_idname = "snapform.convert_to_depth_plane"
bl_label = "Convert to Depth Plane"
bl_description = "Convert an image to a displaced depth plane"
bl_options = {"REGISTER"}
snapform_unshaded: bpy.props.BoolProperty(
unshaded: bpy.props.BoolProperty(
name="Unshaded",
description="Display the source color through an Emission shader",
default=False,
)
snapform_remove_background: bpy.props.BoolProperty(
remove_background: bpy.props.BoolProperty(
name="Remove Background",
description="Remove the background during conversion with BEN2",
default=False,
)
snapform_model: bpy.props.EnumProperty(
name="Model",
model: bpy.props.EnumProperty(
name="Depth From",
description="Model or original image used to build depth",
items=image_depth_model_items,
default=1,
default=0,
)
snapform_use_ray_pose: bpy.props.BoolProperty(
name="Use Ray Pose",
description="Use ray-based camera pose estimation",
default=True,
)
snapform_process_res_factor: process_resolution_scale_property()
snapform_moge_resolution_level: bpy.props.IntProperty(
process_res_factor: process_resolution_scale_property()
resolution_level: bpy.props.IntProperty(
name="Resolution Level",
description="MoGe-2 inference resolution level; higher is finer and slower",
default=5,
min=0,
max=9,
)
snapform_keep_original: bpy.props.BoolProperty(
keep_original: bpy.props.BoolProperty(
name="Keep Original",
description="Keep the source Image Empty after conversion",
default=False,
@@ -2222,64 +2044,35 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
@classmethod
def poll(cls, context):
from .operators import active_job
from .client import service_busy
from ..job_runtime import runtime
return (
is_image_empty(context.object)
and active_job() is None
and not service_busy()
and not runtime.server_busy()
)
def invoke(self, context, _event):
global _last_depth_plane_model
if not self.properties.is_property_set("snapform_model"):
available_keys = [
item[0] for item in image_depth_model_items(self, context)
]
if available_keys:
self.snapform_model = (
_last_depth_plane_model
if _last_depth_plane_model in available_keys
else available_keys[0]
)
return context.window_manager.invoke_props_dialog(self, width=340)
def draw(self, context):
layout = self.layout
layout.prop(self, "snapform_model")
if self.snapform_model in {"SMALL", "BASE"}:
estimate_box = layout.box()
estimate_box.prop(self, "snapform_use_ray_pose")
draw_process_resolution(estimate_box, self, context.object.data)
elif self.snapform_model.startswith("MOGE2_"):
from .client import model_ready
moge_box = layout.box()
if not model_ready(self.snapform_model):
moge_box.label(text="MoGe-2 model is not installed", icon="ERROR")
moge_box.label(text="Download it in Models")
moge_box.prop(self, "snapform_moge_resolution_level")
layout.prop(self, "snapform_remove_background")
layout.prop(self, "snapform_unshaded")
layout.prop(self, "model")
if self.model in {"SMALL", "BASE"}:
draw_process_resolution(layout, self, context.object.data)
elif self.model.startswith("MOGE2_"):
layout.prop(self, "resolution_level")
layout.prop(self, "remove_background")
layout.prop(self, "unshaded")
layout.separator(type="LINE")
layout.prop(self, "snapform_keep_original")
layout.prop(self, "keep_original")
def execute(self, context):
from .operators import (
IMAGE_DEPTH_JOB,
MOGE2_DEPTH_JOB,
ORIGINAL_BACKGROUND_JOB,
ORIGINAL_FRAMES_JOB,
)
source_object = context.object
if video_total_frames(source_object.data) > 1:
context.scene.snapform_settings.video_frames = (
default_extract_frames(source_object)
)
if self.snapform_model == "ORIGINAL":
if self.model == "ORIGINAL":
try:
source_path, delete_input = prepare_image_input(
source_object.data,
@@ -2288,17 +2081,17 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
except RuntimeError as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=(
ORIGINAL_BACKGROUND_JOB
if self.snapform_remove_background
else ORIGINAL_FRAMES_JOB
),
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_unshaded=self.snapform_unshaded,
run_original = (
bpy.ops.snapform.generate_original_background
if self.remove_background
else bpy.ops.snapform.extract_original_frames
)
result = run_original(
input_path=str(source_path),
source_object_name=source_object.name,
delete_source=not self.keep_original,
delete_input=delete_input,
unshaded=self.unshaded,
)
if "CANCELLED" not in result:
return {"FINISHED"}
@@ -2306,7 +2099,7 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
remove_temporary_input(source_path)
return {"CANCELLED"}
if self.snapform_model.startswith("MOGE2_"):
if self.model.startswith("MOGE2_"):
try:
source_path, delete_input = prepare_image_input(
source_object.data,
@@ -2315,18 +2108,15 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
except RuntimeError as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=MOGE2_DEPTH_JOB,
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_moge_resolution_level=(
self.snapform_moge_resolution_level
),
snapform_model_key=self.snapform_model,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_unshaded=self.snapform_unshaded,
snapform_remove_background=self.snapform_remove_background,
result = bpy.ops.snapform.generate_moge2_depth(
input_path=str(source_path),
source_object_name=source_object.name,
resolution_level=self.resolution_level,
model=self.model,
delete_source=not self.keep_original,
delete_input=delete_input,
unshaded=self.unshaded,
remove_background=self.remove_background,
)
if "CANCELLED" not in result:
return {"FINISHED"}
@@ -2338,7 +2128,7 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
try:
process_res = scaled_process_resolution(
image,
self.snapform_process_res_factor,
self.process_res_factor,
)
except ValueError as error:
self.report({"ERROR"}, str(error))
@@ -2352,25 +2142,21 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=IMAGE_DEPTH_JOB,
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_depth_data_name=depth_data_name(source_object),
snapform_model_key=self.snapform_model,
snapform_use_ray_pose=self.snapform_use_ray_pose,
snapform_process_res=process_res,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_unshaded=self.snapform_unshaded,
snapform_remove_background=self.snapform_remove_background,
result = bpy.ops.snapform.generate_da3_depth(
input_path=str(source_path),
source_object_name=source_object.name,
depth_data_name=depth_data_name(source_object),
model=self.model,
process_res=process_res,
delete_source=not self.keep_original,
delete_input=delete_input,
unshaded=self.unshaded,
remove_background=self.remove_background,
)
if "CANCELLED" in result:
if delete_input:
remove_temporary_input(source_path)
return {"CANCELLED"}
global _last_depth_plane_model
_last_depth_plane_model = self.snapform_model
return {"FINISHED"}
@@ -2379,25 +2165,35 @@ class SnapFormImageMenu(bpy.types.Menu):
bl_label = "SnapForm"
def draw(self, _context):
from .client import env_installed
from ..job_runtime import runtime
from ..properties import required_models_ready
layout = self.layout
layout.operator_context = "INVOKE_DEFAULT"
if not env_installed():
operator = layout.operator(
"snapform.run_job",
if not runtime.environment_ready():
layout.operator(
"snapform.install_environment",
text="Install Environment",
icon="IMPORT",
)
operator.job_type = "INSTALL_ENV"
return
layout.operator(
"snapform.convert_image_to_depth_plane",
models_ready = required_models_ready()
if not models_ready:
layout.operator(
"snapform.download_required_models",
text="Download Required Models",
icon="IMPORT",
)
layout.separator()
depth_plane_row = layout.row()
depth_plane_row.enabled = models_ready
depth_plane_row.operator(
"snapform.convert_to_depth_plane",
text="Convert to Depth Plane",
icon="MOD_DISPLACE",
)
layout.operator(
"snapform.convert_image_to_mesh",
"snapform.convert_to_surface",
text="Convert to Surface",
icon="MESH_GRID",
)

View File

@@ -2,49 +2,52 @@ from pathlib import Path
import bpy
from .image_depth import (
from .depth_plane import (
MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME,
create_material,
draw_process_resolution,
image_depth_model_items,
is_image_empty,
normal_data_name,
plane_bounds,
prepare_image_input,
process_resolution_scale_property,
remove_temporary_input,
scaled_process_resolution,
set_modifier_input,
video_total_frames,
)
_last_mesh_model = "BASE"
POINT_ATTRIBUTES = {
"o_depth": ("FLOAT", "point_depth"),
"o_confidence": ("FLOAT", "point_confidence"),
"o_confidence_percentile": (
"FLOAT",
"point_confidence_percentile",
),
"o_background": ("BOOLEAN", "point_background"),
}
FACE_ATTRIBUTES = {
"o_face_confidence": (
"FLOAT",
"face_confidence_percentile",
),
"o_edge_stretch": ("FLOAT", "face_edge_stretch"),
"o_background_face": ("BOOLEAN", "face_background"),
}
OPTIONAL_FACE_ATTRIBUTES = {
"o_mask": ("BOOLEAN", "face_mask"),
"o_depth_discontinuity": ("FLOAT", "face_depth_discontinuity"),
}
NODE_GROUP_NAME = "O DA3 Surface"
MOGE_NODE_GROUP_NAME = "O MoGe2 Surface"
NODE_ASSET_NAME = "O_SnapForm.blend"
def load_image_mesh_data(npz_path):
def image_surface_model_items(owner, context):
from ..properties import surface_model_enum_items
return surface_model_enum_items(owner, context)
def load_image_mesh_data(npz_path, moge2=False):
import numpy as np
try:
with np.load(npz_path, allow_pickle=False) as archive:
required = (
required = [
"vertices",
"faces",
"uv",
@@ -52,13 +55,13 @@ def load_image_mesh_data(npz_path):
"image_size",
"reference_depth",
"point_depth",
"point_confidence",
"point_confidence_percentile",
"point_background",
"face_confidence_percentile",
"face_edge_stretch",
"face_background",
)
]
if moge2:
required.extend(("face_mask", "face_depth_discontinuity", "normal_map"))
else:
required.extend(
("face_confidence_percentile", "face_depth_discontinuity")
)
missing = [key for key in required if key not in archive.files]
if missing:
raise ValueError(
@@ -81,7 +84,16 @@ def load_image_mesh_data(npz_path):
**POINT_ATTRIBUTES,
**FACE_ATTRIBUTES,
}.items()
if key in archive.files
}
for _name, (_data_type, key) in OPTIONAL_FACE_ATTRIBUTES.items():
if key in archive.files:
attributes[key] = np.asarray(archive[key])
if "normal_map" in archive.files:
attributes["normal_map"] = np.asarray(
archive["normal_map"],
dtype=np.float32,
)
except OSError as error:
raise ValueError(f"Unable to read surface data: {npz_path}") from error
if vertices.ndim != 2 or vertices.shape[1] != 3:
@@ -96,11 +108,22 @@ def load_image_mesh_data(npz_path):
raise ValueError(f"Unsupported image size: {image_size}")
if not reference_depth > 0.0:
raise ValueError("Surface reference depth must be positive")
if moge2 and attributes["normal_map"].shape != (
image_size[1],
image_size[0],
3,
):
raise ValueError("Surface normal map and image sizes do not match")
if moge2 and not np.isfinite(attributes["normal_map"]).all():
raise ValueError("Surface normal map contains invalid values")
for _name, (_data_type, key) in POINT_ATTRIBUTES.items():
if len(attributes[key]) != len(vertices):
if key in attributes and len(attributes[key]) != len(vertices):
raise ValueError(f"Point attribute has the wrong length: {key}")
for _name, (_data_type, key) in FACE_ATTRIBUTES.items():
if len(attributes[key]) != len(faces):
if key in attributes and len(attributes[key]) != len(faces):
raise ValueError(f"Face attribute has the wrong length: {key}")
for _name, (_data_type, key) in OPTIONAL_FACE_ATTRIBUTES.items():
if key in attributes and len(attributes[key]) != len(faces):
raise ValueError(f"Face attribute has the wrong length: {key}")
return (
vertices,
@@ -185,7 +208,18 @@ def assign_mesh_attribute(mesh, name, data_type, domain, values):
attribute.data.foreach_set("value", array.reshape(-1))
def assign_snapform_attributes(mesh, attributes):
def assign_snapform_attributes(mesh, attributes, moge2=False):
if moge2:
assign_mesh_attribute(mesh, "o_depth", "FLOAT", "POINT", attributes["point_depth"])
assign_mesh_attribute(mesh, "o_mask", "BOOLEAN", "FACE", attributes["face_mask"])
assign_mesh_attribute(
mesh,
"o_depth_discontinuity",
"FLOAT",
"FACE",
attributes["face_depth_discontinuity"],
)
return
for name, (data_type, key) in POINT_ATTRIBUTES.items():
assign_mesh_attribute(
mesh,
@@ -202,38 +236,104 @@ def assign_snapform_attributes(mesh, attributes):
"FACE",
attributes[key],
)
def remove_surface_background(vertices, faces, uv, attributes):
"""Remove background faces and compact matching point/face data."""
import numpy as np
background_faces = np.asarray(attributes["face_background"], dtype=bool)
kept_faces = faces[~background_faces]
if not len(kept_faces):
raise ValueError("Background removal deleted every surface face")
used_vertices = np.unique(kept_faces)
index_map = np.full(len(vertices), -1, dtype=np.int32)
index_map[used_vertices] = np.arange(len(used_vertices), dtype=np.int32)
filtered_attributes = {}
for _name, (_data_type, key) in POINT_ATTRIBUTES.items():
filtered_attributes[key] = attributes[key][used_vertices]
for _name, (_data_type, key) in FACE_ATTRIBUTES.items():
filtered_attributes[key] = attributes[key][~background_faces]
return (
vertices[used_vertices],
index_map[kept_faces],
uv[used_vertices],
filtered_attributes,
for name, (data_type, key) in OPTIONAL_FACE_ATTRIBUTES.items():
if key in attributes:
assign_mesh_attribute(
mesh,
name,
data_type,
"FACE",
attributes[key],
)
assign_mesh_attribute(
mesh,
"o_depth_discontinuity",
"FLOAT",
"FACE",
attributes["face_depth_discontinuity"],
)
def prepare_moge_surface_archive(results_path, output_path):
import numpy as np
with np.load(results_path, allow_pickle=False) as archive:
required = {"depth", "mask", "normal", "points", "intrinsics"}
missing = required.difference(archive.files)
if missing:
raise ValueError(f"MoGe-2 surface data is missing: {', '.join(sorted(missing))}")
depth = np.asarray(archive["depth"], dtype=np.float32)
mask = np.asarray(archive["mask"], dtype=bool)
points = np.asarray(archive["points"], dtype=np.float32)
intrinsics = np.asarray(archive["intrinsics"], dtype=np.float32)
normal = np.asarray(archive["normal"], dtype=np.float32)
if depth.shape[0] != 1 or points.shape[:3] != (*depth.shape,):
raise ValueError("Convert to Surface requires one matching MoGe-2 point grid")
raw_depth = depth[0]
finite = np.isfinite(raw_depth) & (raw_depth > 0.0)
finite &= np.isfinite(points[0]).all(axis=-1)
height, width = raw_depth.shape
indices = np.arange(height * width, dtype=np.int32).reshape(height, width)
quads = np.stack(
(indices[:-1, :-1], indices[1:, :-1], indices[1:, 1:], indices[:-1, 1:]),
axis=-1,
).reshape(-1, 4)
faces = np.concatenate((quads[:, (0, 1, 2)], quads[:, (0, 2, 3)]), axis=0)
faces = faces[finite.reshape(-1)[faces].all(axis=1)]
if not len(faces):
raise ValueError("MoGe-2 output does not contain a valid surface")
flat_points = points[0].reshape(-1, 3)
face_mask = mask[0].reshape(-1)[faces].all(axis=1)
depth_edge_ratio = depth_discontinuity_ratio(raw_depth)
face_depth_discontinuity = depth_edge_ratio.reshape(-1)[faces].max(axis=1)
used = np.unique(faces)
index_map = np.full(height * width, -1, dtype=np.int32)
index_map[used] = np.arange(len(used), dtype=np.int32)
u = np.linspace(0.0, 1.0, width, dtype=np.float32)
v = np.linspace(1.0, 0.0, height, dtype=np.float32)
uv = np.stack(np.meshgrid(u, v), axis=-1).reshape(-1, 2)[used]
camera_intrinsics = intrinsics[0].copy()
principal_x = float(camera_intrinsics[0, 2])
principal_y = float(camera_intrinsics[1, 2])
if 0.0 <= principal_x <= 1.0 and 0.0 <= principal_y <= 1.0:
camera_intrinsics[0] *= width
camera_intrinsics[1] *= height
camera_intrinsics[2] = (0.0, 0.0, 1.0)
point_depth = raw_depth.reshape(-1)[used]
masked_depth = raw_depth[finite & mask[0]]
reference_depth = float(np.median(masked_depth if len(masked_depth) else point_depth))
np.savez(
output_path,
vertices=flat_points[used],
faces=index_map[faces],
uv=uv,
intrinsics=camera_intrinsics,
image_size=np.array((width, height), dtype=np.int32),
reference_depth=np.float32(reference_depth),
point_depth=point_depth,
normal_map=normal[0],
face_mask=face_mask,
face_depth_discontinuity=face_depth_discontinuity,
)
return output_path
def depth_discontinuity_ratio(depth):
import numpy as np
padded = np.pad(depth, 1, mode="edge")
windows = np.lib.stride_tricks.sliding_window_view(padded, (3, 3))
difference = np.nanmax(windows, axis=(-2, -1))
difference -= np.nanmin(windows, axis=(-2, -1))
return difference / np.maximum(depth, 1e-6)
def node_asset_path():
return Path(__file__).with_name("assets") / NODE_ASSET_NAME
return Path(__file__).parent.parent / "assets" / NODE_ASSET_NAME
def load_geometry_node_group():
existing = bpy.data.node_groups.get(NODE_GROUP_NAME)
def load_geometry_node_group(group_name=NODE_GROUP_NAME):
existing = bpy.data.node_groups.get(group_name)
if existing is not None:
return existing
asset_path = node_asset_path()
@@ -243,15 +343,15 @@ def load_geometry_node_group():
if not use_pack:
directory = str(asset_path) + "/NodeTree/"
result = bpy.ops.wm.append(
filepath=directory + NODE_GROUP_NAME,
filepath=directory + group_name,
directory=directory,
filename=NODE_GROUP_NAME,
filename=group_name,
link=False,
do_reuse_local_id=True,
)
if "FINISHED" not in result:
raise RuntimeError("Unable to append the Geometry Nodes asset")
node_group = bpy.data.node_groups.get(NODE_GROUP_NAME)
node_group = bpy.data.node_groups.get(group_name)
if node_group is None:
raise RuntimeError("Appended Geometry Nodes asset is unavailable")
node_group.use_fake_user = True
@@ -260,11 +360,11 @@ def load_geometry_node_group():
source,
target,
):
if NODE_GROUP_NAME not in source.node_groups:
if group_name not in source.node_groups:
raise RuntimeError(
f"Geometry Nodes asset is missing {NODE_GROUP_NAME}"
f"Geometry Nodes asset is missing {group_name}"
)
target.node_groups = [NODE_GROUP_NAME]
target.node_groups = [group_name]
node_group = target.node_groups[0]
if use_pack:
linked_group = node_group
@@ -307,13 +407,22 @@ def material_depth_scale(mesh_object, camera_parameters):
)
def add_geometry_nodes_processing(mesh_object, vertices, faces):
node_group = load_geometry_node_group()
def add_geometry_nodes_processing(
mesh_object,
vertices,
faces,
group_name=NODE_GROUP_NAME,
):
node_group = load_geometry_node_group(group_name)
modifier = mesh_object.modifiers.new(name=node_group.name, type="NODES")
modifier.node_group = node_group
modifier.show_group_selector = False
identifier = modifier_input_identifier(node_group, "Resolution")
modifier[identifier] = default_remesh_resolution(vertices, faces)
set_modifier_input(
modifier,
identifier,
default_remesh_resolution(vertices, faces),
)
def create_matching_camera(
@@ -359,8 +468,10 @@ def build_image_mesh(
delete_source,
create_camera,
unshaded,
remove_background=False,
moge2=False,
):
import numpy as np
(
camera_vertices,
faces,
@@ -369,14 +480,7 @@ def build_image_mesh(
image_size,
reference_depth,
attributes,
) = load_image_mesh_data(npz_path)
if remove_background:
camera_vertices, faces, uv, attributes = remove_surface_background(
camera_vertices,
faces,
uv,
attributes,
)
) = load_image_mesh_data(npz_path, moge2=moge2)
vertices, camera_parameters = empty_anchored_vertices(
source_object,
camera_vertices,
@@ -388,7 +492,7 @@ def build_image_mesh(
mesh = bpy.data.meshes.new(source_name)
mesh.from_pydata(vertices.tolist(), [], faces.tolist())
assign_vertex_uv(mesh, uv)
assign_snapform_attributes(mesh, attributes)
assign_snapform_attributes(mesh, attributes, moge2=moge2)
mesh.update()
for polygon in mesh.polygons:
polygon.use_smooth = True
@@ -396,16 +500,40 @@ def build_image_mesh(
context.collection.objects.link(mesh_object)
mesh_object.matrix_world = source_object.matrix_world.copy()
depth_scale = material_depth_scale(mesh_object, camera_parameters)
normal_image = None
if moge2 and attributes.get("normal_map", np.empty(0)).size:
from .depth_plane import create_moge_normal_image
normal_map = attributes["normal_map"].copy()
normal_map[..., 0] /= camera_parameters["scale_x"]
normal_map[..., 1] /= -camera_parameters["scale_y"]
normal_map[..., 2] /= -camera_parameters["scale_z"]
lengths = np.linalg.norm(normal_map, axis=-1, keepdims=True)
normal_map /= np.maximum(lengths, 1e-6)
normal_image = create_moge_normal_image(
normal_data_name(source_object),
normal_map,
convert_coordinates=False,
)
material = create_material(
source_name,
source_object.data,
depth_scale,
unshaded=unshaded,
surface_group_name=(
MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME if moge2 else None
),
normal_image=normal_image,
)
mesh.materials.append(material)
mesh_object["snapform_reference_depth"] = reference_depth
mesh_object["snapform_source_type"] = "IMAGE_MESH"
add_geometry_nodes_processing(mesh_object, vertices, faces)
add_geometry_nodes_processing(
mesh_object,
vertices,
faces,
MOGE_NODE_GROUP_NAME if moge2 else NODE_GROUP_NAME,
)
if create_camera:
create_matching_camera(
context,
@@ -426,88 +554,42 @@ def build_image_mesh(
return mesh_object
class BuildImageMesh(bpy.types.Operator):
bl_idname = "snapform.build_image_mesh"
bl_label = "Build Image Surface"
bl_options = {"INTERNAL"}
snapform_source_object_name: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_npz_path: bpy.props.StringProperty(
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_delete_source: bpy.props.BoolProperty(
default=True,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_create_camera: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_unshaded: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_remove_background: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
def execute(self, context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
if not is_image_empty(source_object):
self.report({"ERROR"}, "The source Image Empty no longer exists")
return {"CANCELLED"}
try:
build_image_mesh(
context,
source_object,
self.snapform_npz_path,
self.snapform_delete_source,
self.snapform_create_camera,
self.snapform_unshaded,
self.snapform_remove_background,
)
except (OSError, RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
return {"FINISHED"}
class ConvertImageToMesh(bpy.types.Operator):
bl_idname = "snapform.convert_image_to_mesh"
class ConvertToSurface(bpy.types.Operator):
bl_idname = "snapform.convert_to_surface"
bl_label = "Convert to Surface"
bl_description = "Convert an Image Empty into a perspective textured surface"
bl_options = {"REGISTER"}
snapform_model: bpy.props.EnumProperty(
name="Model",
model: bpy.props.EnumProperty(
name="Depth From",
description="Model used to estimate image depth",
items=image_depth_model_items,
items=image_surface_model_items,
default=0,
)
snapform_use_ray_pose: bpy.props.BoolProperty(
name="Use Ray Pose",
description="Use ray-based camera pose estimation",
default=True,
process_res_factor: process_resolution_scale_property()
resolution_level: bpy.props.IntProperty(
name="Resolution Level",
description="MoGe-2 inference resolution level; higher is finer and slower",
default=5,
min=0,
max=9,
)
snapform_process_res_factor: process_resolution_scale_property()
snapform_create_camera: bpy.props.BoolProperty(
create_camera: bpy.props.BoolProperty(
name="Create Matching Camera",
description="Create a perspective camera matching the source projection",
default=False,
)
snapform_unshaded: bpy.props.BoolProperty(
unshaded: bpy.props.BoolProperty(
name="Unshaded",
description="Display the source color through an Emission shader",
default=False,
)
snapform_remove_background: bpy.props.BoolProperty(
remove_background: bpy.props.BoolProperty(
name="Remove Background",
description="Remove the background during conversion with BEN2",
default=False,
)
snapform_keep_original: bpy.props.BoolProperty(
keep_original: bpy.props.BoolProperty(
name="Keep Original",
description="Keep the source Image Empty after conversion",
default=False,
@@ -515,56 +597,49 @@ class ConvertImageToMesh(bpy.types.Operator):
@classmethod
def poll(cls, context):
from .models import MODELS
from .operators import active_job
from .client import service_busy, model_ready
from ..job_runtime import runtime
from ..properties import model_catalog
return (
is_image_empty(context.object)
and video_total_frames(context.object.data) == 1
and active_job() is None
and not service_busy()
and any(model_ready(model_key) for model_key in MODELS)
and not runtime.server_busy()
and any(
model.get("ready")
and model.get("enabled")
and model.get("family") in {"moge2", "da3"}
for model in model_catalog()
)
)
def invoke(self, context, _event):
global _last_mesh_model
if not self.properties.is_property_set("snapform_model"):
available_keys = [
item[0] for item in image_depth_model_items(self, context)
]
if available_keys:
self.snapform_model = (
_last_mesh_model
if _last_mesh_model in available_keys
else available_keys[0]
)
return context.window_manager.invoke_props_dialog(self, width=360)
def draw(self, context):
layout = self.layout
layout.prop(self, "snapform_model")
layout.prop(self, "snapform_use_ray_pose")
draw_process_resolution(layout, self, context.object.data)
layout.prop(self, "snapform_create_camera")
layout.prop(self, "snapform_remove_background")
layout.prop(self, "snapform_unshaded")
layout.prop(self, "model")
if self.model.startswith("MOGE2_"):
layout.prop(self, "resolution_level")
else:
draw_process_resolution(layout, self, context.object.data)
layout.prop(self, "create_camera")
layout.prop(self, "remove_background")
layout.prop(self, "unshaded")
layout.separator(type="LINE")
layout.prop(self, "snapform_keep_original")
layout.prop(self, "keep_original")
def execute(self, context):
from .operators import IMAGE_MESH_JOB
source_object = context.object
if video_total_frames(source_object.data) != 1:
self.report({"ERROR"}, "Convert to Surface currently supports still images")
return {"CANCELLED"}
try:
process_res = scaled_process_resolution(
source_object.data,
self.snapform_process_res_factor,
)
process_res = 1
if not self.model.startswith("MOGE2_"):
process_res = scaled_process_resolution(
source_object.data,
self.process_res_factor,
)
source_path, delete_input = prepare_image_input(
source_object.data,
context.scene,
@@ -572,23 +647,28 @@ class ConvertImageToMesh(bpy.types.Operator):
except (RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=IMAGE_MESH_JOB,
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_model_key=self.snapform_model,
snapform_use_ray_pose=self.snapform_use_ray_pose,
snapform_process_res=process_res,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_mesh_create_camera=self.snapform_create_camera,
snapform_unshaded=self.snapform_unshaded,
snapform_remove_background=self.snapform_remove_background,
)
common = {
"input_path": str(source_path),
"source_object_name": source_object.name,
"model": self.model,
"delete_source": not self.keep_original,
"delete_input": delete_input,
"create_camera": self.create_camera,
"unshaded": self.unshaded,
"remove_background": self.remove_background,
}
if self.model.startswith("MOGE2_"):
result = bpy.ops.snapform.generate_moge2_surface(
**common,
resolution_level=self.resolution_level,
)
else:
result = bpy.ops.snapform.generate_da3_surface(
**common,
process_res=process_res,
)
if "CANCELLED" in result:
if delete_input:
remove_temporary_input(source_path)
return {"CANCELLED"}
global _last_mesh_model
_last_mesh_model = self.snapform_model
return {"FINISHED"}

View File

@@ -1,7 +1,7 @@
schema_version = "1.0.0"
id = "snapform"
version = "0.10.11"
version = "0.12.3"
name = "SnapForm"
tagline = "Build spatial structures from images, fast"
maintainer = "OmooLab"
@@ -10,9 +10,10 @@ type = "add-on"
blender_version_min = "4.5.0"
platforms = ["windows-x64", "macos-arm64", "linux-x64"]
license = ["SPDX:GPL-3.0-or-later"]
wheels = ["./wheels/blendjob-0.1.6-py3-none-any.whl"]
[permissions]
network = "Access the local Service and download environment dependencies and model weights"
network = "Access the local Job Server and download environment dependencies and model weights"
files = "Read source images and store environment data"
[build]

View File

@@ -1,960 +0,0 @@
import json
import os
import platform
import secrets
import shutil
import socket
import subprocess
import sys
import tempfile
import threading
import time
import uuid
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import ProxyHandler, Request, build_opener
import bpy
from .models import (
BEN2_MODEL,
get_downloadable_model,
model_directory_name,
selected_model_key,
)
DA3_COMMIT = "3d835ec1a5802d64a8b8b15f817a1ab54809bfe4"
BEN2_COMMIT = "2c99a5da477b5523585bfa5c893888a6e818a8f6"
MANIFEST_FILENAME = "manifest.json"
MANIFEST_REVISION = 5
SERVICE_ADDRESS = "127.0.0.1"
DEFAULT_STORAGE_ROOT = Path.home() / ".snapform"
JOB_FOLDER_NAMES = {
"INSTALL_ENV": "install-environment",
"DOWNLOAD_MODEL": "download-model",
"RECONSTRUCT": "export-reconstruction",
"IMAGE_DEPTH": "generate-depth-plane",
"MOGE2_DEPTH": "generate-moge2-depth-plane",
"MOGE2_DEBUG": "test-moge2-api",
"IMAGE_MESH": "generate-image-mesh",
"REMOVE_BACKGROUND": "remove-background",
"IMAGE_BACKGROUND": "remove-image-background",
"ORIGINAL_BACKGROUND": "generate-original-depth-plane",
"ORIGINAL_FRAMES": "extract-original-frames",
}
@dataclass(frozen=True)
class JobRequest:
payload: dict
@dataclass
class Connection:
port: int
token: str
pid: int
instance_id: str
@dataclass
class ServiceProcess:
process: subprocess.Popen
log_file: object
_service_connection = None
_launched_service = None
_registered_instance_id = None
_client_id = uuid.uuid4().hex
_service_auto_start = False
_service_snapshot = {"state": "STOPPED", "message": "Stopped"}
_local_opener = build_opener(ProxyHandler({}))
_client_label = f"Blender {getattr(bpy.app, 'version_string', '')}".strip()
def preferences():
addon = bpy.context.preferences.addons.get(__package__)
return addon.preferences if addon is not None else None
def storage_root():
addon_preferences = preferences()
configured = getattr(addon_preferences, "storage_root", "")
root = (
Path(bpy.path.abspath(configured)).expanduser()
if configured
else DEFAULT_STORAGE_ROOT
)
root.mkdir(parents=True, exist_ok=True)
return root
def venv_directory():
return storage_root() / ".venv"
def venv_python():
directory = venv_directory()
if os.name == "nt":
return directory / "Scripts" / "python.exe"
return directory / "bin" / "python"
def env_installed():
ready_path = storage_root() / MANIFEST_FILENAME
if not venv_python().is_file() or not ready_path.is_file():
return False
try:
manifest = json.loads(ready_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
return False
return (
isinstance(manifest, dict)
and manifest.get("manifest_revision") == MANIFEST_REVISION
)
def models_directory():
path = storage_root() / "models"
path.mkdir(parents=True, exist_ok=True)
return path
def model_directory(model_key):
root = models_directory()
directory = root / model_directory_name(model_key)
old_moge_directory = root / "moge2_base_normal"
if (
model_key == "MOGE2_VITB_NORMAL"
and not directory.is_dir()
and old_moge_directory.is_dir()
):
old_moge_directory.rename(directory)
legacy_directory = root / model_key.lower()
if (
not directory.is_dir()
and legacy_directory.is_dir()
and legacy_directory != directory
):
legacy_directory.rename(directory)
return directory
def model_ready(model_key):
model = get_downloadable_model(model_key)
path = model_directory(model_key)
return all(
any(path.glob(pattern))
for pattern in model.ready_patterns
)
def env_devices():
"""Devices detected by the installed environment, or None when unknown."""
ready_path = storage_root() / MANIFEST_FILENAME
if not ready_path.is_file():
return None
try:
payload = json.loads(ready_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
return None
devices = payload.get("devices")
if not isinstance(devices, list) or not devices:
return None
return [str(name).lower() for name in devices]
def machine_devices():
"""Devices the machine can run before environment installation."""
devices = ["cpu"]
if shutil.which("nvidia-smi"):
devices.insert(0, "cuda")
elif sys.platform == "darwin" and platform.machine().lower() in {
"arm64",
"aarch64",
}:
devices.insert(0, "mps")
return devices
def supported_devices():
"""Inference devices available on this machine, fastest first."""
return env_devices() or machine_devices()
DEVICE_LABELS = {
"cuda": ("CUDA", "NVIDIA CUDA"),
"mps": ("MPS", "Apple Metal (MPS)"),
"cpu": ("CPU", "CPU"),
}
def device_enum_items(_owner, _context):
"""Blender EnumProperty items listing only devices supported on this machine."""
return [
(name.upper(), *DEVICE_LABELS[name], index)
for index, name in enumerate(supported_devices())
]
def production_device():
"""Device configured for production tasks, falling back to Auto."""
addon_preferences = preferences()
configured = getattr(addon_preferences, "device", "")
return configured or "AUTO"
def jobs_directory():
path = storage_root() / "jobs"
path.mkdir(parents=True, exist_ok=True)
return path
def temporary_input_path():
directory = Path(tempfile.mkdtemp(prefix="snapform-input-"))
return directory / "input.png"
def temporary_input_directory():
return Path(tempfile.mkdtemp(prefix="snapform-input-"))
def move_input_into_job(input_path, job_directory):
"""Move a Blender-generated input copy into the flat job directory."""
input_path = Path(input_path)
job_directory = Path(job_directory)
if input_path.is_file():
destination = job_directory / "input.png"
input_path.replace(destination)
try:
input_path.parent.rmdir()
except OSError:
pass
return destination
if not input_path.is_dir():
raise ValueError(f"Temporary input does not exist: {input_path}")
for source in input_path.iterdir():
if not source.is_file():
raise ValueError(f"Temporary input contains a directory: {source}")
destination = job_directory / source.name
if destination.exists():
raise ValueError(f"Job input filename already exists: {source.name}")
source.replace(destination)
input_path.rmdir()
return job_directory
def job_folder_name(job_type):
"""Return the folder name segment for a job type."""
return JOB_FOLDER_NAMES.get(job_type, job_type.lower())
def new_job_paths(job_type):
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S-%f")
stem = f"{job_folder_name(job_type)}_{timestamp}"
directory = jobs_directory() / stem
directory.mkdir(parents=True, exist_ok=True)
return {
"log": directory / "job.log",
"status": directory / "status.json",
"result": directory,
}
def job_id_from_status(status_path):
return Path(status_path).parent.name
def uv_executable():
"""Return the bundled uv path inside the shared storage."""
return str(uv_cache_path())
def uv_cache_path():
if os.name == "nt":
return storage_root() / "tools" / "uv.exe"
return storage_root() / "tools" / "uv"
def install_env_command(status_path):
executable = uv_executable()
torch_variant = "cu128" if "cuda" in machine_devices() else "cpu"
script = Path(__file__).with_name("install_env.py")
command = [
sys.executable,
str(script),
"--uv",
executable,
"--venv-dir",
str(venv_directory()),
"--manifest",
str(storage_root() / MANIFEST_FILENAME),
"--models-dir",
str(models_directory()),
"--torch-variant",
torch_variant,
"--da3-commit",
DA3_COMMIT,
"--ben2-commit",
BEN2_COMMIT,
"--status",
str(status_path),
]
return command
def service_log_path():
return storage_root() / "service.log"
def service_discovery_path():
return storage_root() / "service.json"
def service_lock_path():
return storage_root() / "service.lock"
def _available_service_port():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
listener.bind((SERVICE_ADDRESS, 0))
return listener.getsockname()[1]
def _service_url(connection, path):
return f"http://{SERVICE_ADDRESS}:{connection.port}{path}"
def _service_http_request(connection, method, path, payload=None, timeout=2.0):
data = None
headers = {"X-SnapForm-Token": connection.token}
if payload is not None:
data = json.dumps(payload).encode("utf-8")
headers["Content-Type"] = "application/json"
request = Request(
_service_url(connection, path),
data=data,
headers=headers,
method=method,
)
try:
with _local_opener.open(request, timeout=timeout) as response:
return json.loads(response.read().decode("utf-8"))
except HTTPError as error:
try:
detail = json.loads(error.read().decode("utf-8")).get("detail")
except (json.JSONDecodeError, UnicodeDecodeError):
detail = None
raise RuntimeError(
detail or f"Service returned HTTP {error.code}"
)
except (OSError, URLError) as error:
raise RuntimeError(f"Unable to reach Service: {error}")
def _read_service_discovery():
try:
payload = json.loads(service_discovery_path().read_text(encoding="utf-8"))
return Connection(
port=int(payload["port"]),
token=str(payload["token"]),
pid=int(payload["pid"]),
instance_id=str(payload["instance_id"]),
)
except (KeyError, TypeError, ValueError, OSError, json.JSONDecodeError):
return None
def _check_service(connection, timeout=0.3):
health = _service_http_request(
connection,
"GET",
"/health",
timeout=timeout,
)
if health.get("instance_id") != connection.instance_id:
raise RuntimeError("Service discovery is stale")
return health
def _discover_live_service(timeout=0.3):
connection = _read_service_discovery()
if connection is None:
return None, None
try:
return connection, _check_service(connection, timeout=timeout)
except RuntimeError:
return None, None
def start_service():
global _launched_service
if _launched_service is not None and _launched_service.process.poll() is None:
return
_close_launched_service()
port = _available_service_port()
token = secrets.token_urlsafe(32)
instance_id = uuid.uuid4().hex
log_file = service_log_path().open("a", encoding="utf-8")
command = [
str(venv_python()),
"-u",
str(Path(__file__).with_name("service.py")),
"--host",
SERVICE_ADDRESS,
"--port",
str(port),
"--token",
token,
"--instance-id",
instance_id,
"--discovery",
str(service_discovery_path()),
"--lock",
str(service_lock_path()),
]
try:
process = subprocess.Popen(
command,
stdout=log_file,
stderr=subprocess.STDOUT,
cwd=storage_root(),
creationflags=creation_flags(),
)
except OSError:
log_file.close()
raise
_launched_service = ServiceProcess(process, log_file)
def ensure_service(timeout=60.0):
global _service_auto_start, _service_connection, _service_snapshot
_service_auto_start = True
deadline = time.monotonic() + timeout
last_launch = 0.0
starting = False
while time.monotonic() < deadline:
connection, health = _discover_live_service(timeout=0.5)
if connection is not None:
_service_connection = connection
_register_service_client(connection)
_service_snapshot = _format_service_snapshot(connection, health)
_redraw_service_ui()
return connection
if not starting:
_service_snapshot = {"state": "STARTING", "message": "Starting"}
_redraw_service_ui()
starting = True
_close_finished_launched_service()
if (
_launched_service is None
and time.monotonic() - last_launch >= 1.0
):
start_service()
last_launch = time.monotonic()
time.sleep(0.05)
raise RuntimeError(
f"Service did not become ready; log: {service_log_path()}"
)
def submit_service_job(request):
connection = _service_connection or ensure_service()
payload = dict(request.payload)
payload["client_id"] = _client_id
_service_http_request(
connection,
"POST",
"/jobs",
payload=payload,
timeout=5.0,
)
return connection
def stop_service():
global _service_auto_start, _service_connection, _service_snapshot
global _registered_instance_id
connection = _service_connection
if connection is None:
connection, _health = _discover_live_service(timeout=0.5)
if connection is None:
_close_launched_service(terminate=True)
_service_auto_start = False
_service_snapshot = {"state": "STOPPED", "message": "Stopped"}
return
_register_service_client(connection)
_service_http_request(
connection,
"POST",
"/shutdown",
payload={"client_id": _client_id},
timeout=2.0,
)
_service_auto_start = False
deadline = time.monotonic() + 10.0
while time.monotonic() < deadline:
try:
_check_service(connection, timeout=0.2)
except RuntimeError:
break
time.sleep(0.05)
else:
raise RuntimeError("Service did not stop")
_service_connection = None
_registered_instance_id = None
_close_finished_launched_service()
_service_snapshot = {"state": "STOPPED", "message": "Stopped"}
def restart_service():
global _service_auto_start, _service_connection, _registered_instance_id
global _service_snapshot
connection = ensure_service()
_service_http_request(
connection,
"POST",
"/shutdown",
payload={"client_id": _client_id},
timeout=2.0,
)
_service_auto_start = True
_service_connection = None
_registered_instance_id = None
_service_snapshot = {"state": "STARTING", "message": "Restarting"}
def enable_service():
global _service_auto_start, _service_connection, _service_snapshot
_service_auto_start = True
connection, health = _discover_live_service(timeout=0.2)
if connection is not None:
_service_connection = connection
_register_service_client(connection)
_service_snapshot = _format_service_snapshot(connection, health)
_redraw_service_ui()
return
start_service()
_service_snapshot = {"state": "STARTING", "message": "Starting"}
_redraw_service_ui()
def clear_service_models():
connection = ensure_service()
return _service_http_request(
connection,
"POST",
"/models/clear",
timeout=5.0,
)
def cancel_service_job(job_id):
connection = ensure_service()
return _service_http_request(
connection,
"DELETE",
f"/jobs/{job_id}",
payload={"client_id": _client_id},
timeout=2.0,
)
def detach_service_client():
global _service_connection, _registered_instance_id
connection = _service_connection
if connection is not None and _registered_instance_id is not None:
try:
_service_http_request(
connection,
"DELETE",
f"/clients/{_client_id}",
timeout=0.5,
)
except RuntimeError:
pass
_service_connection = None
_registered_instance_id = None
# 当前 Blender 启动的 Service 随 Blender 关闭,不等待空闲超时。
_close_launched_service(terminate=True)
def schedule_service_start():
timers = getattr(bpy.app, "timers", None)
if timers is None:
return
if timers.is_registered(_service_timer):
return
timers.register(_service_timer, first_interval=1.0, persistent=True)
def unschedule_service():
timers = getattr(bpy.app, "timers", None)
if timers is None:
return
if timers.is_registered(_service_timer):
timers.unregister(_service_timer)
def service_status():
return dict(_service_snapshot)
def service_busy():
return _service_snapshot.get("state") == "BUSY"
def mark_service_job_complete(job_id):
"""Release a stale local busy snapshot after its terminal status is observed."""
global _service_snapshot
active = _service_snapshot.get("active_job")
if active and active.get("job_id") != job_id:
return
if _service_snapshot.get("state") != "BUSY":
return
_service_snapshot = {
**_service_snapshot,
"busy": False,
"active_job": None,
"state": "READY",
"message": "Ready",
}
def _service_timer():
global _service_connection, _service_snapshot, _registered_instance_id
if not env_installed():
_service_snapshot = {
"state": "UNAVAILABLE",
"message": "Environment is not installed",
}
return 2.0
connection, health = _discover_live_service(timeout=0.2)
if connection is None:
_service_connection = None
_registered_instance_id = None
_close_finished_launched_service()
if not _service_auto_start:
if _service_snapshot.get("state") != "STOPPED":
_service_snapshot = {"state": "STOPPED", "message": "Stopped"}
_redraw_service_ui()
return 2.0
try:
start_service()
except OSError as error:
_service_snapshot = {"state": "ERROR", "message": str(error)}
return 2.0
_service_snapshot = {"state": "STARTING", "message": "Starting"}
_redraw_service_ui()
return 1.0
try:
_service_connection = connection
if _registered_instance_id == connection.instance_id:
try:
_service_http_request(
connection,
"POST",
f"/clients/{_client_id}/heartbeat",
timeout=0.5,
)
except RuntimeError:
_registered_instance_id = None
_register_service_client(connection)
health = _check_service(connection, timeout=0.5)
_service_snapshot = _format_service_snapshot(connection, health)
except RuntimeError as error:
_service_connection = None
_registered_instance_id = None
_service_snapshot = {"state": "ERROR", "message": str(error)}
_redraw_service_ui()
return 5.0
def _register_service_client(connection):
global _registered_instance_id
if _registered_instance_id == connection.instance_id:
return
_service_http_request(
connection,
"POST",
"/clients",
payload={"client_id": _client_id, "label": _client_label},
timeout=2.0,
)
_registered_instance_id = connection.instance_id
def _format_service_snapshot(connection, health):
state = "BUSY" if health.get("busy") else "READY"
return {
**health,
"state": state,
"message": "Busy" if state == "BUSY" else "Ready",
"port": connection.port,
}
def _redraw_service_ui():
if threading.current_thread() is not threading.main_thread():
return
try:
from .status_bar import redraw_ui
redraw_ui(bpy.context, force=True)
except (AttributeError, RuntimeError):
pass
def _close_finished_launched_service():
if _launched_service is None or _launched_service.process.poll() is None:
return
_close_launched_service()
def _close_launched_service(terminate=False):
global _launched_service
launched = _launched_service
_launched_service = None
if launched is None:
return
if terminate and launched.process.poll() is None:
launched.process.terminate()
try:
launched.process.wait(timeout=3.0)
except subprocess.TimeoutExpired:
launched.process.kill()
launched.process.wait(timeout=3.0)
launched.log_file.close()
def download_model_command(model_key, status_path):
model = get_downloadable_model(model_key)
command = [
str(venv_python()),
str(Path(__file__).with_name("worker.py")),
"download",
"--repository",
model.repository,
"--model-dir",
str(model_directory(model_key)),
"--status",
str(status_path),
"--label",
model.label,
]
for pattern in model.download_patterns:
command.extend(["--allow-pattern", pattern])
for pattern in model.ready_patterns:
command.extend(["--required-pattern", pattern])
if model.r2_directory:
command.extend(["--r2-directory", model.r2_directory])
for filename, size, checksum in model.r2_files:
command.extend(
["--r2-file", f"{filename}|{size}|{checksum}"]
)
return command
def remove_background_request(
settings,
input_path,
status_path,
result_directory,
model_key,
video_frames=None,
start_frame=1,
device=None,
):
return JobRequest(
{
"job_id": job_id_from_status(status_path),
"command": "remove-background",
"parameters": {
"input": str(input_path),
"model_dir": str(model_directory(model_key)),
"output_dir": str(result_directory),
"status": str(status_path),
"device": (device or settings.device).lower(),
"video_frames": (
settings.video_frames
if video_frames is None
else video_frames
),
"start_frame": start_frame,
},
}
)
def extract_frames_request(
settings,
input_path,
status_path,
result_directory,
video_frames=None,
start_frame=1,
):
return JobRequest(
{
"job_id": job_id_from_status(status_path),
"command": "extract-frames",
"parameters": {
"input": str(input_path),
"output_dir": str(result_directory),
"status": str(status_path),
"video_frames": (
settings.video_frames
if video_frames is None
else video_frames
),
"start_frame": start_frame,
},
}
)
def reconstruct_request(
settings,
export_format,
status_path,
result_directory,
input_path=None,
model_key=None,
process_res=None,
process_res_method=None,
use_ray_pose=None,
remove_background=False,
mesh_filter_edges=None,
start_frame=1,
device=None,
):
snapform_selected_key = model_key or selected_model_key(settings, export_format)
snapform_selected_process_res = (
settings.process_res if process_res is None else process_res
)
snapform_selected_process_res_method = (
settings.process_res_method
if process_res_method is None
else process_res_method
)
snapform_selected_ray_pose = (
settings.use_ray_pose if use_ray_pose is None else use_ray_pose
)
parameters = {
"input": str(
Path(input_path)
if input_path is not None
else Path(bpy.path.abspath(settings.input_path))
),
"model_dir": str(model_directory(snapform_selected_key)),
"output_dir": str(result_directory),
"export_format": export_format,
"status": str(status_path),
"process_res": snapform_selected_process_res,
"process_res_method": snapform_selected_process_res_method,
"conf_thresh_percentile": settings.conf_thresh_percentile,
"num_max_points": settings.num_max_points,
"show_cameras": 1 if settings.show_cameras else 0,
"gs_views_interval": settings.gs_views_interval,
"video_frames": settings.video_frames,
"start_frame": start_frame,
"device": (device or settings.device).lower(),
"use_ray_pose": 1 if snapform_selected_ray_pose else 0,
"ref_view_strategy": settings.ref_view_strategy,
"remove_background_model_dir": "",
"mesh_filter_edges": bool(
getattr(settings, "mesh_filter_edges", True)
if mesh_filter_edges is None
else mesh_filter_edges
),
"mesh_max_edge_length": float(
getattr(settings, "mesh_max_edge_length", 0.1)
),
}
if remove_background:
parameters["remove_background_model_dir"] = str(
model_directory(BEN2_MODEL.key)
)
return JobRequest(
{
"job_id": job_id_from_status(status_path),
"command": "reconstruct",
"parameters": parameters,
}
)
def moge_request(
settings,
status_path,
result_directory,
input_path=None,
resolution_level=None,
include_points=False,
remove_background=False,
start_frame=1,
device=None,
model_key=None,
):
from .models import BEN2_MODEL, MOGE2_MODEL
selected_model_key = model_key or MOGE2_MODEL.key
parameters = {
"input": str(
Path(input_path)
if input_path is not None
else Path(bpy.path.abspath(settings.input_path))
),
"model_dir": str(model_directory(selected_model_key)),
"output_dir": str(result_directory),
"status": str(status_path),
"resolution_level": int(
settings.moge_resolution_level
if resolution_level is None
else resolution_level
),
"include_points": bool(include_points),
"video_frames": settings.video_frames,
"start_frame": start_frame,
"device": (device or settings.device).lower(),
"remove_background_model_dir": (
str(model_directory(BEN2_MODEL.key))
if remove_background
else ""
),
}
return JobRequest(
{
"job_id": job_id_from_status(status_path),
"command": "moge",
"parameters": parameters,
}
)
def creation_flags():
if os.name != "nt":
return 0
return subprocess.CREATE_NO_WINDOW
def read_status(path):
try:
return json.loads(Path(path).read_text(encoding="utf-8"))
except (FileNotFoundError, json.JSONDecodeError, OSError):
return None

View File

@@ -1,979 +0,0 @@
import argparse
import filecmp
import json
import os
import platform
import queue
import shutil
import subprocess
import sys
import tarfile
import tempfile
import threading
import time
import urllib.request
import zipfile
from pathlib import Path
SCRIPT_DIRECTORY = Path(__file__).resolve().parent
if str(SCRIPT_DIRECTORY) not in sys.path:
sys.path.insert(0, str(SCRIPT_DIRECTORY))
from models import BEN2_MODEL, DOWNLOADABLE_MODELS, MOGE2_MODEL
PYPI_MIRRORS = (
"https://pypi.tuna.tsinghua.edu.cn/simple",
"https://mirrors.aliyun.com/pypi/simple/",
)
PYPI_DEFAULT_INDEX = "https://pypi.org/simple/"
TORCH_VERSION = "2.9.1"
TORCHVISION_VERSION = "0.24.1"
XFORMERS_VERSION = "0.0.33.post2"
ADDICT_VERSION = "2.4.0"
FASTAPI_VERSION = "0.139.2"
UVICORN_VERSION = "0.51.0"
MANIFEST_REVISION = 5
TORCH_INDEXES = {
"cu128": "https://download.pytorch.org/whl/cu128",
"cpu": "https://download.pytorch.org/whl/cpu",
}
_resolved_pypi_index = None
DEFAULT_MODEL_KEYS = (MOGE2_MODEL.key, BEN2_MODEL.key)
RUNTIME_STAGE_COUNT = 3
ENVIRONMENT_SLICE = (0.02, 0.06)
PYTORCH_SLICE = (0.06, 0.48)
XFORMERS_SLICE = (0.48, 0.62)
AI_COMPONENTS_SLICE = (0.62, 0.85)
MODELS_SLICE_START = 0.85
MODELS_SLICE_END = 1.0
RAMP_RATE = 0.02
CREEP_CAP = 0.60
INSTALL_ESTIMATES = {
"PyTorch": 8 * 60,
"AI components": 8 * 60,
}
INSTALL_ESTIMATE_RANGES = {
"PyTorch": PYTORCH_SLICE,
"AI components": (XFORMERS_SLICE[0], AI_COMPONENTS_SLICE[1]),
}
CHECK_CODE = (
"import json; import torch; "
"from depth_anything_3.api import DepthAnything3; "
"from ben2 import BEN_Base; "
"from moge.model.v2 import MoGeModel; "
"import fastapi, uvicorn; "
"mps = getattr(torch.backends, 'mps', None); "
"print('SNAPFORM_DEVICES=' + json.dumps({"
"'torch': torch.__version__, "
"'cuda': torch.cuda.is_available(), "
"'mps': bool(mps is not None and mps.is_available())}))"
)
def configure_standard_streams():
"""Windows 图形进程可能继承 GBK stdout,安装日志统一写为 UTF-8。"""
for stream in (sys.stdout, sys.stderr):
reconfigure = getattr(stream, "reconfigure", None)
if callable(reconfigure):
reconfigure(encoding="utf-8", errors="backslashreplace")
def write_status(
path,
progress,
message,
error=None,
stage=None,
stages=None,
stage_label=None,
eta_seconds=None,
):
payload = {"progress": progress, "message": message}
if error:
payload["error"] = error
if stage is not None:
payload["stage"] = stage
payload["stages"] = stages
payload["stage_label"] = stage_label
if eta_seconds is not None:
payload["eta_seconds"] = max(int(eta_seconds), 0)
Path(path).write_text(
json.dumps(payload, ensure_ascii=False),
encoding="utf-8",
)
def child_environment(environment=None):
"""Inherit the current environment and inject system/registry proxy settings into child processes (uv only reads proxy env variables)."""
process_environment = os.environ.copy()
process_environment["PYTHONIOENCODING"] = "utf-8:backslashreplace"
process_environment["PYTHONUTF8"] = "1"
for key, value in urllib.request.getproxies().items():
env_key = f"{key.upper()}_PROXY"
if env_key not in process_environment:
process_environment[env_key] = value
if _resolved_pypi_index:
process_environment["PIP_INDEX_URL"] = _resolved_pypi_index
if environment:
process_environment.update(environment)
return process_environment
def run(command, environment=None):
print("Running:", subprocess.list2cmdline([str(value) for value in command]))
subprocess.check_call(
[str(value) for value in command],
env=child_environment(environment),
)
def format_elapsed(seconds):
seconds = int(seconds)
if seconds < 60:
return f"{seconds}s"
return f"{seconds // 60}m {seconds % 60:02d}s"
def parse_package_count(line):
try:
return int(line.split()[1])
except (IndexError, ValueError):
return 0
def estimated_remaining(label, progress, elapsed=0.0):
expected_seconds = INSTALL_ESTIMATES.get(label)
progress_range = INSTALL_ESTIMATE_RANGES.get(label)
if expected_seconds is None or progress_range is None:
return None
start, end = progress_range
if end <= start:
return 0.0
remaining_fraction = 1.0 - min(
max((progress - start) / (end - start), 0.0),
1.0,
)
return max(expected_seconds * remaining_fraction - elapsed, 0.0)
def effective_torch_index(torch_variant):
return os.environ.get("SNAPFORM_TORCH_INDEX") or TORCH_INDEXES.get(torch_variant)
def resolve_indexes(args):
"""???????????????????????????????????"""
global _resolved_pypi_index
_resolved_pypi_index = (
getattr(args, "pypi_index", None)
or os.environ.get("PIP_INDEX_URL")
)
def _run_install_once(
status_path,
stage,
stages,
label,
command,
environment,
start,
end,
clock,
):
"""Run a uv install command once and map package-count events onto the stage progress range."""
print("Running:", subprocess.list2cmdline([str(value) for value in command]))
process_environment = child_environment(environment)
process_environment["UV_LOG"] = "debug"
process = subprocess.Popen(
[str(value) for value in command],
env=process_environment,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
)
output_queue = queue.Queue()
def read_output():
for raw_line in process.stdout:
output_queue.put(raw_line)
output_queue.put(None)
threading.Thread(target=read_output, daemon=True).start()
def eased(fraction):
return start + min(max(fraction, 0.0), 1.0) * (end - start)
installed = []
total = 0
displayed = start
target = start
started = clock()
last_tick = started
last_heartbeat = 0.0
def detail(message):
write_status(
status_path,
displayed,
message,
stage=stage,
stages=stages,
stage_label=label,
eta_seconds=estimated_remaining(
label,
displayed,
clock() - started,
),
)
while True:
try:
raw_line = output_queue.get(timeout=0.2)
empty = False
except queue.Empty:
raw_line = None
empty = True
now = clock()
elapsed = now - started
dt = now - last_tick
last_tick = now
if total:
effective = max(target, eased(CREEP_CAP))
if effective > displayed:
displayed = min(effective, displayed + dt * RAMP_RATE)
if raw_line is None and not empty:
break
if empty:
if elapsed - last_heartbeat >= 5.0:
last_heartbeat = elapsed
detail("Downloading packages")
continue
line = raw_line.rstrip()
print(line)
if line.startswith("Resolved ") and "package" in line:
total = parse_package_count(line)
resolved_progress = eased(0.02)
displayed = max(displayed, resolved_progress)
target = max(target, resolved_progress)
detail(f"Resolving {total} packages")
elif line.startswith("Prepared ") and "package" in line:
done = parse_package_count(line)
if total:
target = max(target, eased(0.10 + 0.60 * done / total))
detail(f"Downloading packages {done}/{total}")
elif line.startswith("Installed ") and "package" in line:
done = parse_package_count(line)
if total:
target = max(target, eased(0.75 + 0.25 * done / total))
detail(f"Installing packages {done}/{total}")
elif line.startswith("+"):
entry = line[1:].strip()
installed.append(entry)
return_code = process.wait()
if return_code != 0:
raise subprocess.CalledProcessError(return_code, command)
return installed
def run_install(
status_path,
stage,
stages,
label,
command,
environment=None,
start=0.0,
end=1.0,
clock=time.monotonic,
):
"""Run a uv install command, retrying with mirrors when the default index fails."""
attempts = [environment]
if _resolved_pypi_index in (None, PYPI_DEFAULT_INDEX):
attempts.extend(
{"PIP_INDEX_URL": mirror} for mirror in PYPI_MIRRORS
)
for index, attempt_environment in enumerate(attempts):
try:
return _run_install_once(
status_path,
stage,
stages,
label,
command,
attempt_environment,
start,
end,
clock,
)
except subprocess.CalledProcessError:
if index + 1 == len(attempts):
raise
print(f"Retrying with PyPI mirror: {PYPI_MIRRORS[0]}")
def venv_python(venv_dir):
if sys.platform == "win32":
return Path(venv_dir) / "Scripts" / "python.exe"
return Path(venv_dir) / "bin" / "python"
def install_stages(args, venv_dir, python, uv, vendor_dir=None):
# macOS 的 PyTorch wheel 只在 PyPI 上,download.pytorch.org 没有 macosx 构建
torch_index = (
None
if sys.platform == "darwin"
else effective_torch_index(args.torch_variant)
)
vendor_dir = Path(vendor_dir or Path(venv_dir).parent / "vendor")
def archive_requirement(project_name, filename):
local = Path(vendor_dir) / filename
if not local.is_file():
raise RuntimeError(
"Bundled vendor artifact is missing; reinstall the extension: "
f"{local}"
)
return f"{project_name} @ {local.as_uri()}"
da3_requirement = archive_requirement(
"depth-anything-3",
"depth-anything-3.tar.gz",
)
ben2_requirement = archive_requirement(
"ben2",
"ben2.tar.gz",
)
utils3d_requirement = archive_requirement(
"utils3d",
"utils3d.tar.gz",
)
moge_requirement = archive_requirement(
"moge",
"moge.tar.gz",
)
torch_command = [
uv,
"pip",
"install",
"-v",
"--python",
python,
f"torch=={TORCH_VERSION}",
f"torchvision=={TORCHVISION_VERSION}",
]
if torch_index is not None:
torch_command.extend(["--index-url", torch_index])
stages = [
(
"Environment",
[uv, "venv", "--clear", "--python", "3.12", venv_dir],
None,
"plain",
),
(
"PyTorch",
torch_command,
None,
"install",
),
(
"AI components",
[
uv,
"pip",
"install",
"-v",
"--python",
python,
da3_requirement,
f"addict=={ADDICT_VERSION}",
ben2_requirement,
utils3d_requirement,
moge_requirement,
f"fastapi=={FASTAPI_VERSION}",
f"uvicorn=={UVICORN_VERSION}",
"setuptools",
"wheel",
],
None,
"install",
),
]
if sys.platform != "darwin":
xformers_command = [
uv,
"pip",
"install",
"-v",
"--python",
python,
f"xformers=={XFORMERS_VERSION}",
]
if args.torch_variant == "cu128":
xformers_command.extend(["--index-url", torch_index])
else:
xformers_command.append("--no-deps")
stages.insert(
2,
("AI components", xformers_command, None, "install"),
)
models = DOWNLOADABLE_MODELS
worker_script = Path(__file__).with_name("worker.py")
models_dir = Path(args.models_dir)
for model_key in DEFAULT_MODEL_KEYS:
model = models[model_key]
download_command = [
python,
worker_script,
"download",
"--repository",
model.repository,
"--model-dir",
models_dir / model.directory_name,
"--status",
args.status,
"--label",
model.label,
]
for pattern in model.download_patterns:
download_command.extend(["--allow-pattern", pattern])
for pattern in model.ready_patterns:
download_command.extend(["--required-pattern", pattern])
if model.r2_directory:
download_command.extend(
["--r2-directory", model.r2_directory]
)
for filename, size, checksum in model.r2_files:
download_command.extend(
["--r2-file", f"{filename}|{size}|{checksum}"]
)
stages.append(
(
model.label,
download_command,
None,
"download",
)
)
return stages
def env_installed(venv_dir, manifest_path=None):
ready_path = Path(manifest_path or Path(venv_dir).parent / "manifest.json")
if not venv_python(venv_dir).is_file() or not ready_path.is_file():
return False
try:
payload = json.loads(ready_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
return False
return (
isinstance(payload, dict)
and payload.get("manifest_revision") == MANIFEST_REVISION
)
def installed_devices(args):
devices = []
machine = platform.machine().lower()
if sys.platform == "darwin" and machine in {"arm64", "aarch64"}:
devices.append("mps")
elif args.torch_variant.startswith("cu"):
devices.append("cuda")
devices.append("cpu")
return devices
def verify_command(python):
return [python, "-c", CHECK_CODE]
def run_verify(
python,
status_path,
start=0.0,
end=1.0,
clock=time.monotonic,
stage=None,
stages=None,
stage_label=None,
):
"""Run the environment import check with heartbeat progress; return the device report."""
command = verify_command(python)
print("Running:", subprocess.list2cmdline([str(value) for value in command]))
process = subprocess.Popen(
[str(value) for value in command],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
)
output_queue = queue.Queue()
def read_output():
for raw_line in process.stdout:
output_queue.put(raw_line)
output_queue.put(None)
threading.Thread(target=read_output, daemon=True).start()
report = {}
started = clock()
last_heartbeat = 0.0
while True:
try:
raw_line = output_queue.get(timeout=0.2)
empty = False
except queue.Empty:
raw_line = None
empty = True
if raw_line is None and not empty:
break
if empty:
elapsed = clock() - started
if elapsed - last_heartbeat >= 5.0:
last_heartbeat = elapsed
creep = min(elapsed / 60.0, CREEP_CAP)
progress = start + creep * (end - start)
write_status(
status_path,
progress,
"Checking Environment",
stage=stage,
stages=stages,
stage_label=stage_label,
eta_seconds=max(60.0 - elapsed, 0.0),
)
continue
line = raw_line.rstrip()
print(line)
if line.startswith("SNAPFORM_DEVICES="):
try:
report = json.loads(line[len("SNAPFORM_DEVICES=") :])
except json.JSONDecodeError:
report = {}
return_code = process.wait()
if return_code != 0:
raise subprocess.CalledProcessError(return_code, command)
return report
def uv_download_target():
machine = platform.machine().lower()
if sys.platform == "win32":
if machine in {"arm64", "aarch64"}:
return "aarch64-pc-windows-msvc", "zip"
return "x86_64-pc-windows-msvc", "zip"
if sys.platform == "darwin":
arch = "aarch64" if machine in {"arm64", "aarch64"} else "x86_64"
return f"{arch}-apple-darwin", "tar.gz"
if sys.platform.startswith("linux"):
arch = "aarch64" if machine in {"arm64", "aarch64"} else "x86_64"
return f"{arch}-unknown-linux-gnu", "tar.gz"
raise RuntimeError(f"Unsupported platform for uv download: {sys.platform}")
def bundled_vendor_directory():
packaged = Path(__file__).with_name("vendor")
if packaged.is_dir():
return packaged
project_root = Path(__file__).resolve().parents[2]
if (project_root / "pyproject.toml").is_file():
return project_root / "vendor"
return packaged
def bundled_uv_archive():
target, archive = uv_download_target()
path = bundled_vendor_directory() / f"uv-{target}.{archive}"
return path if path.is_file() else None
def vendor_overlay_directory(filename):
names = {
"depth-anything-3.tar.gz": "depth-anything-3",
"moge.tar.gz": "moge",
"utils3d.tar.gz": "utils3d",
}
overlay_name = names.get(filename)
if overlay_name is None:
return None
project_root = Path(__file__).resolve().parents[2]
overlay = project_root / "patches" / overlay_name
return overlay if overlay.is_dir() else None
def copy_archive_with_overlay(source, target, overlay):
"""开发目录使用原始 vendor 时,应用与正式打包相同的源码覆盖。"""
with tarfile.open(source, "r:gz") as input_archive:
members = input_archive.getmembers()
top_level = members[0].name.split("/", 1)[0]
with tarfile.open(target, "w:gz") as output_archive:
for member in members:
if member.isdir() or member.islnk() or member.issym():
output_archive.addfile(member)
continue
relative = member.name[len(top_level) + 1 :]
replacement = overlay / relative
if replacement.is_file():
info = tarfile.TarInfo(member.name)
info.size = replacement.stat().st_size
info.mode = member.mode
with replacement.open("rb") as stream:
output_archive.addfile(info, stream)
continue
with input_archive.extractfile(member) as stream:
output_archive.addfile(member, stream)
def copy_vendor_inputs(bundled_dir, vendor_dir):
"""Copy bundled installation inputs into a temporary directory."""
for name in (
"depth-anything-3.tar.gz",
"ben2.tar.gz",
"utils3d.tar.gz",
"moge.tar.gz",
):
bundled = Path(bundled_dir) / name
if not bundled.is_file():
continue
target = Path(vendor_dir) / name
overlay = vendor_overlay_directory(name)
if overlay is None and target.is_file() and filecmp.cmp(
bundled,
target,
shallow=False,
):
continue
Path(vendor_dir).mkdir(parents=True, exist_ok=True)
if overlay is not None and tarfile.is_tarfile(bundled):
copy_archive_with_overlay(bundled, target, overlay)
else:
shutil.copy2(bundled, target)
def remove_legacy_vendor(storage_root):
"""Remove installation inputs persisted by older development builds."""
root = Path(storage_root)
shutil.rmtree(root / "vendor", ignore_errors=True)
for name in (
"depth-anything-3.tar.gz",
"ben2.tar.gz",
"utils3d.tar.gz",
"moge.tar.gz",
):
(root / name).unlink(missing_ok=True)
def archive_is_valid(path):
if zipfile.is_zipfile(path):
return True
try:
with tarfile.open(path) as archive:
archive.getmembers()
return True
except (OSError, tarfile.TarError):
return False
def extract_uv(archive_path, uv_path):
executable_name = "uv.exe" if sys.platform == "win32" else "uv"
with tempfile.TemporaryDirectory() as directory:
if zipfile.is_zipfile(archive_path):
with zipfile.ZipFile(archive_path) as archive:
archive.extractall(directory)
else:
with tarfile.open(archive_path) as archive:
archive.extractall(directory)
source = next(Path(directory).rglob(executable_name))
shutil.copy2(source, uv_path)
if sys.platform != "win32":
os.chmod(uv_path, 0o755)
def install_uv(
uv,
status_path,
stage=None,
stages=None,
stage_label=None,
start=0.0,
end=1.0,
):
"""Extract the bundled uv binary; fail clearly when it is missing."""
stage_fields = {
"stage": stage,
"stages": stages,
"stage_label": stage_label,
}
bundled_uv = bundled_uv_archive()
if bundled_uv is None:
raise RuntimeError(
"Bundled uv is missing; reinstall the extension matching your "
f"platform (searched {bundled_vendor_directory()})"
)
if not archive_is_valid(bundled_uv):
raise RuntimeError(f"Bundled uv archive is invalid: {bundled_uv}")
print(f"Extracting bundled uv from {bundled_uv}")
write_status(
status_path,
start,
"extracting bundled uv",
**stage_fields,
)
Path(uv).parent.mkdir(parents=True, exist_ok=True)
extract_uv(bundled_uv, uv)
def run_stage(
status_path,
stage,
stages,
label,
command,
environment,
kind,
start=0.0,
end=1.0,
):
if stage is None:
opening_message = f"Downloading {label}"
elif kind == "install":
opening_message = "Preparing packages"
else:
opening_message = ""
write_status(
status_path,
start,
opening_message,
stage=stage,
stages=stages,
stage_label=label,
eta_seconds=estimated_remaining(label, start),
)
installed = []
if kind == "install":
installed = run_install(
status_path,
stage,
stages,
label,
command,
environment,
start=start,
end=end,
)
elif kind == "download":
progress_command = [
*command,
"--progress-start",
str(start),
"--progress-end",
str(end),
]
if stage is not None:
progress_command.extend(
[
"--stage",
str(stage),
"--stages",
str(stages),
"--stage-label",
label,
]
)
run(progress_command, environment)
else:
run(command, environment)
write_status(
status_path,
end,
f"{label} ready" if stage is None else "",
stage=stage,
stages=stages,
stage_label=label,
)
return installed
def install(args):
if sys.platform == "darwin" and platform.machine().lower() not in {
"arm64",
"aarch64",
}:
raise RuntimeError(
"SnapForm requires Apple Silicon macOS; PyTorch stopped "
"shipping Intel Mac (x86_64) wheels after 2.2.2"
)
resolve_indexes(args)
venv_dir = Path(args.venv_dir)
manifest_path = Path(
getattr(args, "manifest", venv_dir.parent / "manifest.json")
)
venv_dir.parent.mkdir(parents=True, exist_ok=True)
if not env_installed(venv_dir, manifest_path):
shutil.rmtree(venv_dir, ignore_errors=True)
manifest_path.unlink(missing_ok=True)
uv = Path(args.uv)
if not uv.is_file():
resolved = shutil.which(args.uv)
uv = Path(resolved) if resolved else uv
uv_missing = not uv.is_file()
python = venv_python(venv_dir)
vendor_temp = tempfile.TemporaryDirectory(prefix="snapform_vendor_")
vendor_dir = Path(vendor_temp.name)
copy_vendor_inputs(
bundled_vendor_directory(),
vendor_dir,
)
stages = install_stages(args, venv_dir, python, uv, vendor_dir)
env_stages = [stage for stage in stages if stage[3] != "download"]
model_stages = [stage for stage in stages if stage[3] == "download"]
model_span = (
(MODELS_SLICE_END - MODELS_SLICE_START) / len(model_stages)
if model_stages
else 0.0
)
installed_packages = []
component_stages = env_stages[2:]
component_ranges = (
[XFORMERS_SLICE, AI_COMPONENTS_SLICE]
if len(component_stages) == 2
else [(XFORMERS_SLICE[0], AI_COMPONENTS_SLICE[1])]
)
for index, (label, command, environment, kind) in enumerate(env_stages):
if index == 0:
stage = 1
stage_start, stage_end = ENVIRONMENT_SLICE
elif index == 1:
stage = 2
stage_start, stage_end = PYTORCH_SLICE
else:
stage = 3
component_index = index - 2
stage_start, stage_end = component_ranges[component_index]
if uv_missing and index == 0:
midpoint = stage_start + (stage_end - stage_start) * 0.5
install_uv(
uv,
args.status,
stage=stage,
stages=RUNTIME_STAGE_COUNT,
stage_label=label,
start=stage_start,
end=midpoint,
)
stage_start = midpoint
stage_installed = run_stage(
args.status,
stage,
RUNTIME_STAGE_COUNT,
label,
command,
environment,
kind,
start=stage_start,
end=stage_end,
)
if isinstance(stage_installed, list):
installed_packages.extend(stage_installed)
vendor_temp.cleanup()
remove_legacy_vendor(venv_dir.parent)
devices = installed_devices(args)
venv_dir.mkdir(parents=True, exist_ok=True)
manifest_path.parent.mkdir(parents=True, exist_ok=True)
ready_path = manifest_path
ready_path.write_text(
json.dumps(
{
"manifest_revision": MANIFEST_REVISION,
"python": "3.12",
"torch": TORCH_VERSION,
"xformers": XFORMERS_VERSION,
"fastapi": FASTAPI_VERSION,
"uvicorn": UVICORN_VERSION,
"torch_variant": args.torch_variant,
"devices": devices,
"da3_commit": args.da3_commit,
"ben2_commit": args.ben2_commit,
"packages": sorted(set(installed_packages)),
},
indent=2,
),
encoding="utf-8",
)
model_offset = MODELS_SLICE_START
failed_models = []
for label, command, environment, kind in model_stages:
stage_start = model_offset
stage_end = stage_start + model_span
try:
run_stage(
args.status,
None,
None,
label,
command,
environment,
kind,
start=stage_start,
end=stage_end,
)
except Exception as error:
print(f"Model download failed: {label}: {error}")
failed_models.append(label)
write_status(
args.status,
stage_end,
f"Skipping {label}; continuing",
)
model_offset = stage_end
if failed_models:
labels = ", ".join(failed_models)
write_status(
args.status,
1.0,
f"Environment is installed; retry {labels} from Models",
)
return
write_status(
args.status,
1.0,
"Environment and recommended models ready",
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--uv", required=True)
parser.add_argument("--venv-dir", required=True)
parser.add_argument("--manifest", required=True)
parser.add_argument("--models-dir", required=True)
parser.add_argument("--torch-variant", choices=tuple(TORCH_INDEXES), required=True)
parser.add_argument("--da3-commit", required=True)
parser.add_argument("--ben2-commit", required=True)
parser.add_argument("--status", required=True)
parser.add_argument("--pypi-index", default=None)
return parser.parse_args()
def main():
configure_standard_streams()
args = parse_args()
try:
install(args)
except Exception as error:
write_status(
args.status,
1.0,
"Environment installation failed",
str(error),
)
raise
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,59 @@
from pathlib import Path
import bpy
from blendjob import JobRuntime
DEFAULT_STORAGE_ROOT = Path.home() / ".snapform"
def configured_storage_root():
addon = bpy.context.preferences.addons.get(__package__)
configured = getattr(getattr(addon, "preferences", None), "storage_root", "")
if configured:
return Path(bpy.path.abspath(configured))
return DEFAULT_STORAGE_ROOT
ENVIRONMENT = {
"python": "3.12",
"packages": [
"numpy==2.4.2",
"pillow==12.1.1",
"av==17.0.0",
"trimesh==4.11.2",
"huggingface-hub==1.4.1",
],
"platform_packages": {
"windows": ["onnxruntime-directml==1.24.4"],
"default": ["onnxruntime==1.24.4"],
},
}
def post_install(job_runtime):
errors = []
models = (
("MOGE2_VITS_NORMAL", "MoGe-2 ViT-S Normal"),
("BEN2_BASE", "BEN2 Base"),
)
for model, label in models:
try:
job_runtime.request("download-model", {"model": model})
except RuntimeError as error:
errors.append(f"{label}: {error}")
if errors:
raise RuntimeError("; ".join(errors))
runtime = JobRuntime(
"server:server",
entrypoint_root=Path(__file__).parent,
storage_root=configured_storage_root,
environment=ENVIRONMENT,
post_install=post_install,
namespace="snapform",
)
JobOperatorBase = runtime.JobOperatorBase

View File

@@ -0,0 +1,188 @@
from dataclasses import dataclass
@dataclass(frozen=True)
class ModelSpec:
key: str
label: str
directory_name: str
repository: str
license_name: str
description: str
supports_gs: bool
download_patterns: tuple[str, ...] = ()
ready_patterns: tuple[str, ...] = ("config.json", "*.safetensors")
enabled: bool = True
r2_directory: str = ""
r2_files: tuple[tuple[str, int, str], ...] = ()
MODELS = {
"SMALL": ModelSpec(
key="SMALL",
label="DA3 Small",
directory_name="DA3-SMALL",
repository="depth-anything/DA3-SMALL",
license_name="Apache 2.0",
description="Fastest any-view model using ONNX Runtime",
supports_gs=False,
download_patterns=("model.onnx",),
ready_patterns=("model.onnx",),
r2_directory="DA3-SMALL",
r2_files=(
("model.onnx", 105484029, "7a9d81475b3e2334ace0d6d2c2a816b3c93081437e1866cc434ce1789125f631"),
),
),
"BASE": ModelSpec(
key="BASE",
label="DA3 Base",
directory_name="DA3-BASE",
repository="depth-anything/DA3-BASE",
license_name="Apache 2.0",
description="Balanced any-view model using ONNX Runtime",
supports_gs=False,
download_patterns=("model.onnx",),
ready_patterns=("model.onnx",),
r2_directory="DA3-BASE",
r2_files=(
("model.onnx", 412856697, "31f38fc4990539d7902ba267924b9731db1acb00d840c248a60b1865810ff9e6"),
),
),
"GIANT_1_1": ModelSpec(
key="GIANT_1_1",
label="DA3 Giant 1.1",
directory_name="da3_giant_1_1",
repository="depth-anything/DA3-GIANT-1.1",
license_name="CC BY-NC 4.0",
description="Non-commercial Giant model with Gaussian head",
supports_gs=True,
enabled=False,
),
"NESTED_GIANT_LARGE_1_1": ModelSpec(
key="NESTED_GIANT_LARGE_1_1",
label="DA3 Nested Giant Large 1.1",
directory_name="da3_nested_giant_large_1_1",
repository="depth-anything/DA3NESTED-GIANT-LARGE-1.1",
license_name="CC BY-NC 4.0",
description="Non-commercial nested model with metric depth and Gaussian head",
supports_gs=True,
enabled=False,
),
}
BEN2_MODEL = ModelSpec(
key="BEN2_BASE",
label="BEN2 Base",
directory_name="BEN2-ONNX",
repository="onnx-community/BEN2-ONNX",
license_name="Apache 2.0",
description="Fast high-quality FP16 ONNX background removal model",
supports_gs=False,
download_patterns=("onnx/model_fp16.onnx",),
ready_patterns=("onnx/model_fp16.onnx",),
r2_directory="BEN2-ONNX",
r2_files=((
"onnx/model_fp16.onnx",
219121675,
"dfdc25f421f32a0d1268e0f2ff2153d340e8f1d52d3dd16f5dc33c1ce85cedf1",
),),
)
MOGE2_MODELS = {
"MOGE2_VITS_NORMAL": ModelSpec(
key="MOGE2_VITS_NORMAL",
label="MoGe-2 ViT-S Normal",
directory_name="moge-2-vits-normal-onnx",
repository="Ruicheng/moge-2-vits-normal-onnx",
license_name="MIT",
description="Fastest MoGe-2 model using ONNX Runtime",
supports_gs=False,
download_patterns=("model.onnx",),
ready_patterns=("model.onnx",),
r2_directory="moge-2-vits-normal-onnx",
r2_files=((
"model.onnx",
140852051,
"24eacb5dc7a2c54c7bc98f7de085ffbed79ad006ea5b664c2c2cdc02ff3a52f0",
),),
),
"MOGE2_VITB_NORMAL": ModelSpec(
key="MOGE2_VITB_NORMAL",
label="MoGe-2 ViT-B Normal",
directory_name="moge-2-vitb-normal-onnx",
repository="Ruicheng/moge-2-vitb-normal-onnx",
license_name="MIT",
description="Balanced MoGe-2 model using ONNX Runtime",
supports_gs=False,
download_patterns=("model.onnx",),
ready_patterns=("model.onnx",),
r2_directory="moge-2-vitb-normal-onnx",
r2_files=((
"model.onnx",
419411850,
"bbf14e07a30f11e69d36ab861590123f5598ababcbc8946a063eb4a966f35a21",
),),
),
"MOGE2_VITL_NORMAL": ModelSpec(
key="MOGE2_VITL_NORMAL",
label="MoGe-2 ViT-L Normal",
directory_name="moge-2-vitl-normal-onnx",
repository="Ruicheng/moge-2-vitl-normal-onnx",
license_name="MIT",
description="Highest-quality MoGe-2 model using ONNX Runtime",
supports_gs=False,
download_patterns=("model.onnx",),
ready_patterns=("model.onnx",),
r2_directory="moge-2-vitl-normal-onnx",
r2_files=((
"model.onnx",
1324265014,
"afbc4ccc3450298f3afb35b90f015f4c4f552dea21dc6470d5f7b78b77e2d751",
),),
),
}
MOGE2_MODEL = MOGE2_MODELS["MOGE2_VITS_NORMAL"]
DOWNLOADABLE_MODELS = {
**MODELS,
BEN2_MODEL.key: BEN2_MODEL,
**MOGE2_MODELS,
}
MODEL_NUMBERS = {
model_key: number for number, model_key in enumerate(MODELS)
}
def get_model(model_key):
try:
return MODELS[model_key]
except KeyError as error:
raise ValueError(f"Unknown DA3 model: {model_key}") from error
def get_downloadable_model(model_key):
try:
return DOWNLOADABLE_MODELS[model_key]
except KeyError as error:
raise ValueError(f"Unknown model: {model_key}") from error
def model_directory_name(model_key):
"""Return the model's folder name inside the shared models directory."""
return get_downloadable_model(model_key).directory_name
def model_record(model, ready=False):
"""Return public model metadata exposed by the Models Resource."""
return {
"key": model.key,
"label": model.label,
"description": model.description,
"license": model.license_name,
"supports_gs": model.supports_gs,
"enabled": model.enabled,
"family": "moge2" if model.key.startswith("MOGE2_") else (
"ben2" if model.key == BEN2_MODEL.key else "da3"
),
"ready": bool(ready),
}

View File

@@ -1,254 +0,0 @@
from dataclasses import dataclass
@dataclass(frozen=True)
class ModelSpec:
key: str
label: str
directory_name: str
repository: str
license_name: str
description: str
supports_gs: bool
download_patterns: tuple[str, ...] = ()
ready_patterns: tuple[str, ...] = ("config.json", "*.safetensors")
enabled: bool = True
r2_directory: str = ""
r2_files: tuple[tuple[str, int, str], ...] = ()
MODELS = {
"SMALL": ModelSpec(
key="SMALL",
label="DA3 Small",
directory_name="da3_small",
repository="depth-anything/DA3-SMALL",
license_name="Apache 2.0",
description="Fastest any-view model with Apache 2.0 weights",
supports_gs=False,
r2_directory="DA3Small",
r2_files=(
("config.json", 1202, "a486e29e82b7ab4a7d4cefc1ea4526cfe2ae438a572c8ca98917cfbcde7447d2"),
("model.safetensors", 137248940, "364492e38a3a06d221ac75da7f6621ada3f2361cd24fde11ba79091e9f40efcf"),
),
),
"BASE": ModelSpec(
key="BASE",
label="DA3 Base",
directory_name="da3_base",
repository="depth-anything/DA3-BASE",
license_name="Apache 2.0",
description="Balanced any-view model with Apache 2.0 weights",
supports_gs=False,
r2_directory="DA3Base",
r2_files=(
("config.json", 1205, "5e34115ebc17bd2d8d43033c5f72e9446ac8833fd61d3fa160b7e67e0bb5b7b5"),
("configuration.json", 74, "b684d553257b2ca379fd5dfdf96a6e4a633eec3b969c97119c9875465a7c07a8"),
("model.safetensors", 541518028, "e01067dc1659613083d9145a9a2547ccdbe6ccbbf83c4fe7b3e8a4e2bdae78b5"),
),
),
"LARGE_1_1": ModelSpec(
key="LARGE_1_1",
label="DA3 Large 1.1",
directory_name="da3_large_1_1",
repository="depth-anything/DA3-LARGE-1.1",
license_name="CC BY-NC 4.0",
description="Higher quality non-commercial model with larger VRAM use",
supports_gs=False,
enabled=False,
),
"GIANT_1_1": ModelSpec(
key="GIANT_1_1",
label="DA3 Giant 1.1",
directory_name="da3_giant_1_1",
repository="depth-anything/DA3-GIANT-1.1",
license_name="CC BY-NC 4.0",
description="Non-commercial Giant model with Gaussian head",
supports_gs=True,
enabled=False,
),
"NESTED_GIANT_LARGE_1_1": ModelSpec(
key="NESTED_GIANT_LARGE_1_1",
label="DA3 Nested Giant Large 1.1",
directory_name="da3_nested_giant_large_1_1",
repository="depth-anything/DA3NESTED-GIANT-LARGE-1.1",
license_name="CC BY-NC 4.0",
description="Non-commercial nested model with metric depth and Gaussian head",
supports_gs=True,
enabled=False,
),
}
BEN2_MODEL = ModelSpec(
key="BEN2_BASE",
label="BEN2 Base",
directory_name="ben2_base",
repository="PramaLLC/BEN2",
license_name="MIT",
description="Fast high-quality background removal model",
supports_gs=False,
download_patterns=("config.json", "model.safetensors"),
r2_directory="BEN2Base",
r2_files=(
("config.json", 124, "01ac876925577b52fa7c6af3f5b1d1420bc9b40016b6c25e2de89b38d8fff865"),
("model.safetensors", 380577976, "ea8b7907176a09667c86343dc7d00de6a6d871076cb90bb5f753618fd6fb3ebb"),
),
)
def moge2_model(key, label, directory_name, repository, description, r2_directory, size, checksum):
return ModelSpec(
key=key,
label=label,
directory_name=directory_name,
repository=repository,
license_name="MIT",
description=description,
supports_gs=False,
download_patterns=("model.pt",),
ready_patterns=("model.pt",),
r2_directory=r2_directory,
r2_files=(("model.pt", size, checksum),),
)
MOGE2_MODELS = {
"MOGE2_VITS_NORMAL": moge2_model(
"MOGE2_VITS_NORMAL", "MoGe-2 ViT-S Normal", "moge2_vits_normal",
"Ruicheng/moge-2-vits-normal", "Fastest MoGe-2 model", "MoGe2ViTSNormal",
140550416, "79a16621928c2bf0ed04659218c55c01075e950507f40bb3332fb4c873d3e1dc",
),
"MOGE2_VITB_NORMAL": moge2_model(
"MOGE2_VITB_NORMAL", "MoGe-2 ViT-B Normal", "moge2_vitb_normal",
"Ruicheng/moge-2-vitb-normal", "Balanced MoGe-2 model", "MoGe2ViTBNormal",
419110160, "16b8110e86d5dc5a849db120ca96ef3a223fd30b0c9146d1d81db504073da5f6",
),
"MOGE2_VITL_NORMAL": moge2_model(
"MOGE2_VITL_NORMAL", "MoGe-2 ViT-L Normal", "moge2_vitl_normal",
"Ruicheng/moge-2-vitl-normal", "Highest-quality MoGe-2 model", "MoGe2ViTLNormal",
1323815904, "280741fd09bc3f403ccff9967784c2a391b52d2c0742ae3efdb21d9f90cc1a01",
),
}
MOGE2_MODEL = MOGE2_MODELS["MOGE2_VITB_NORMAL"]
DOWNLOADABLE_MODELS = {
**MODELS,
BEN2_MODEL.key: BEN2_MODEL,
**MOGE2_MODELS,
}
MODEL_NUMBERS = {
model_key: number for number, model_key in enumerate(MODELS)
}
_ENUM_ITEMS_CACHE = {}
def enum_items(supports_gs=None, show_download_status=False):
model_ready = None
if show_download_status:
from .client import model_ready
models = [
model
for model in MODELS.values()
if model.enabled
and (supports_gs is None or model.supports_gs == supports_gs)
]
download_status = (
tuple(model_ready(model.key) for model in models)
if model_ready is not None
else ()
)
cache_key = (
supports_gs,
show_download_status,
download_status,
)
cached_items = _ENUM_ITEMS_CACHE.get(cache_key)
if cached_items is not None:
return cached_items
items = []
for index, model in enumerate(models):
label = model.label
if download_status and not download_status[index]:
label = f"{label} (Missing)"
items.append(
(
model.key,
label,
f"{model.description}; license: {model.license_name}",
MODEL_NUMBERS[model.key],
)
)
_ENUM_ITEMS_CACHE[cache_key] = items
return items
def model_enum_items(_owner, _context):
return enum_items(show_download_status=True)
def gs_model_enum_items(_owner, _context):
return enum_items(supports_gs=True, show_download_status=True)
def moge2_model_enum_items(_owner, _context):
from .client import model_ready
return [
(
model.key,
model.label if model_ready(model.key) else f"{model.label} (Missing)",
f"{model.description}; license: {model.license_name}",
index,
)
for index, model in enumerate(MOGE2_MODELS.values())
]
def depth_plane_model_enum_items(_owner, _context):
from .client import model_ready
models = [
*MOGE2_MODELS.values(),
*(model for model in MODELS.values() if model.enabled),
]
items = []
for index, model in enumerate(models):
label = model.label if model_ready(model.key) else f"{model.label} (Missing)"
items.append(
(
model.key,
label,
f"{model.description}; license: {model.license_name}",
index,
)
)
items.append(("ORIGINAL", "Original", "Use the original image as depth", len(items)))
return items
def selected_model_key(settings, export_format=None):
selected_format = export_format or settings.export_format
if selected_format == "gs_ply":
return settings.gs_model
return settings.model
def get_model(model_key):
try:
return MODELS[model_key]
except KeyError as error:
raise ValueError(f"Unknown DA3 model: {model_key}") from error
def get_downloadable_model(model_key):
try:
return DOWNLOADABLE_MODELS[model_key]
except KeyError as error:
raise ValueError(f"Unknown model: {model_key}") from error
def model_directory_name(model_key):
"""Return the model's folder name inside the shared models directory."""
return get_downloadable_model(model_key).directory_name

File diff suppressed because it is too large Load Diff

View File

@@ -1,14 +1,16 @@
import bpy
from .models import BEN2_MODEL, MODELS, MOGE2_MODELS, selected_model_key
from .operators import active_job
from . import client
from .client import model_ready, env_installed
from .job_runtime import runtime
from .properties import BEN2_MODEL_KEY, model_catalog, model_ready
class AiServicePanel(bpy.types.Panel):
bl_label = "AI Service"
bl_idname = "SNAPFORM_PT_service"
def selected_model_key(settings):
return settings.gs_model if settings.export_format == "gs_ply" else settings.model
class ServerPanel(bpy.types.Panel):
bl_label = "Job Server"
bl_idname = "SNAPFORM_PT_server"
bl_space_type = "VIEW_3D"
bl_region_type = "UI"
bl_category = "SnapForm"
@@ -16,23 +18,22 @@ class AiServicePanel(bpy.types.Panel):
def draw(self, _context):
layout = self.layout
running = active_job() is not None or client.service_busy()
if not env_installed():
running = runtime.server_busy()
if not runtime.environment_ready():
layout.label(
text="Environment is not installed",
icon="ERROR",
)
install_row = layout.row()
install_row.enabled = not running
operator = install_row.operator(
"snapform.run_job",
install_row.operator(
"snapform.install_environment",
text="Install Environment",
icon="IMPORT",
)
operator.job_type = "INSTALL_ENV"
return
status = client.service_status()
status = runtime.server_status()
state = status.get("state", "STOPPED")
icons = {
"READY": "RADIOBUT_ON",
@@ -46,7 +47,7 @@ class AiServicePanel(bpy.types.Panel):
if state in {"READY", "BUSY"}:
row.label(
text=(
f"Running (Clients: {status.get('clients', 0)})"
"Running"
if state == "READY"
else "Running (Busy)"
),
@@ -54,41 +55,40 @@ class AiServicePanel(bpy.types.Panel):
)
if state == "READY":
controls = row.row(align=True)
controls.enabled = status.get("clients", 0) <= 1
operator = controls.operator(
"snapform.control_service",
controls.operator(
"snapform.restart_server",
text="",
icon="FILE_REFRESH",
)
operator.action = "RESTART"
operator = controls.operator(
"snapform.control_service",
controls.operator(
"snapform.stop_server",
text="",
icon="PAUSE",
)
operator.action = "STOP"
elif state == "STOPPED":
operator = row.operator(
"snapform.control_service",
row.operator(
"snapform.start_server",
text="Start",
icon="PLAY",
)
operator.action = "START"
else:
row.label(
text=status.get("message", state.title()),
icon=icons.get(state, "QUESTION"),
)
if state == "ERROR":
operator = row.operator(
"snapform.control_service",
row.operator(
"snapform.start_server",
text="Start",
icon="PLAY",
)
operator.action = "START"
if state in {"READY", "BUSY"}:
cached = status.get("cached_models", {})
cached = (
status.get("resources", {})
.get("model_manager", {})
.get("loaded", {})
)
cached_names = [name for name in cached.values() if name]
cached_row = layout.row(align=True)
cached_row.label(
@@ -100,12 +100,11 @@ class AiServicePanel(bpy.types.Panel):
)
clear = cached_row.row(align=True)
clear.enabled = state == "READY"
operator = clear.operator(
"snapform.control_service",
clear.operator(
"snapform.clear_models",
text="",
icon="TRASH",
)
operator.action = "CLEAR"
class ModelsPanel(bpy.types.Panel):
@@ -119,34 +118,41 @@ class ModelsPanel(bpy.types.Panel):
@classmethod
def poll(cls, _context):
return env_installed()
return runtime.environment_ready()
def draw(self, _context):
layout = self.layout
running = active_job() is not None or client.service_busy()
running = runtime.server_busy()
for model in MODELS.values():
if model.enabled:
models = model_catalog()
for model in models:
if model.get("family") == "moge2":
self._draw_model(layout, model, running)
for model in MOGE2_MODELS.values():
self._draw_model(layout, model, running)
self._draw_model(layout, BEN2_MODEL, running)
for model in models:
if model.get("family") == "da3" and model.get("enabled"):
self._draw_model(layout, model, running)
layout.separator(type="LINE")
ben2 = next(
(model for model in models if model.get("key") == BEN2_MODEL_KEY),
None,
)
if ben2:
self._draw_model(layout, ben2, running)
def _draw_model(self, layout, model, running):
row = layout.row(align=True)
row.label(text=model.label)
if model_ready(model.key):
row.label(text=model["label"])
if model.get("ready"):
row.label(text="Ready", icon="CHECKMARK")
else:
download_row = row.row(align=True)
download_row.enabled = env_installed() and not running
download_row.enabled = runtime.environment_ready() and not running
operator = download_row.operator(
"snapform.run_job",
"snapform.download_model",
text="Download",
icon="IMPORT",
)
operator.job_type = "DOWNLOAD_MODEL"
operator.model_key = model.key
operator.model = model["key"]
class DebugPanel(bpy.types.Panel):
@@ -185,19 +191,18 @@ def draw_da3_input_settings(layout, settings, model_property):
layout.prop(settings, "video_frames")
layout.prop(settings, "process_res")
layout.prop(settings, "process_res_method")
layout.prop(settings, "use_ray_pose")
layout.prop(settings, "ref_view_strategy")
class Da3OutputDebugPanel(bpy.types.Panel):
bl_label = "DA3 Output Test"
bl_label = "DA3"
bl_idname = "SNAPFORM_PT_debug_da3_output"
bl_parent_id = DebugPanel.bl_idname
bl_space_type = "VIEW_3D"
bl_region_type = "UI"
bl_category = "SnapForm"
bl_options = {"DEFAULT_CLOSED"}
bl_order = 0
bl_order = 1
def draw(self, context):
layout = self.layout
@@ -227,50 +232,47 @@ class Da3OutputDebugPanel(bpy.types.Panel):
mesh_box.prop(settings, "mesh_max_edge_length")
export_row = layout.row()
export_row.enabled = (
active_job() is None
and not client.service_busy()
not runtime.server_busy()
and model_ready(selected_model_key(settings))
)
operator = export_row.operator(
"snapform.run_job",
export_row.operator(
"snapform.debug_da3",
text="Generate Output",
icon="EXPORT",
)
operator.job_type = "RECONSTRUCT"
class Moge2ApiDebugPanel(bpy.types.Panel):
bl_label = "MoGe-2 API Test"
bl_label = "MoGe-2"
bl_idname = "SNAPFORM_PT_debug_moge2_api"
bl_parent_id = DebugPanel.bl_idname
bl_space_type = "VIEW_3D"
bl_region_type = "UI"
bl_category = "SnapForm"
bl_options = {"DEFAULT_CLOSED"}
bl_order = 1
bl_order = 0
def draw(self, context):
layout = self.layout
settings = context.scene.snapform_settings
layout.prop(settings, "moge_export_format")
layout.prop(settings, "input_path")
layout.prop(settings, "moge_resolution_level")
layout.prop(settings, "moge_model")
layout.prop(settings, "moge_resolution_level")
run_row = layout.row()
run_row.enabled = (
active_job() is None
and not client.service_busy()
not runtime.server_busy()
and model_ready(settings.moge_model)
)
operator = run_row.operator(
"snapform.run_job",
run_row.operator(
"snapform.debug_moge2",
text="Generate MoGe-2 Output",
icon="EXPORT",
)
operator.job_type = "MOGE2_DEBUG"
class Ben2RemovalDebugPanel(bpy.types.Panel):
bl_label = "BEN2 Removal Test"
bl_label = "BEN2"
bl_idname = "SNAPFORM_PT_debug_ben2_removal"
bl_parent_id = DebugPanel.bl_idname
bl_space_type = "VIEW_3D"
@@ -286,13 +288,11 @@ class Ben2RemovalDebugPanel(bpy.types.Panel):
layout.prop(settings, "background_input_path")
background_row = layout.row()
background_row.enabled = (
active_job() is None
and not client.service_busy()
and model_ready(BEN2_MODEL.key)
not runtime.server_busy()
and model_ready(BEN2_MODEL_KEY)
)
operator = background_row.operator(
"snapform.run_job",
background_row.operator(
"snapform.debug_ben2",
text="Remove Background",
icon="IMAGE_ALPHA",
)
operator.job_type = "REMOVE_BACKGROUND"

View File

@@ -1,8 +1,7 @@
from pathlib import Path
import bpy
from .client import device_enum_items
from .job_runtime import DEFAULT_STORAGE_ROOT, runtime
from .properties import device_enum_items
class SnapFormPreferences(bpy.types.AddonPreferences):
@@ -15,7 +14,7 @@ class SnapFormPreferences(bpy.types.AddonPreferences):
"all Blender versions can use the same directory"
),
subtype="DIR_PATH",
default=str(Path.home() / ".snapform"),
default=str(DEFAULT_STORAGE_ROOT),
)
device: bpy.props.EnumProperty(
name="Device",
@@ -28,29 +27,24 @@ class SnapFormPreferences(bpy.types.AddonPreferences):
)
def draw(self, _context):
from .operators import active_job
from .client import venv_directory, env_installed
layout = self.layout
layout.prop(self, "storage_root")
layout.prop(self, "device")
layout.separator()
layout.label(text=f"Environment: {venv_directory()}")
if env_installed():
layout.label(text=f"Environment: {runtime.environment_directory()}")
if runtime.environment_ready():
layout.label(text="Environment is installed", icon="CHECKMARK")
else:
install_row = layout.row()
install_row.enabled = active_job() is None
operator = install_row.operator(
"snapform.run_job",
install_row.enabled = runtime.active_job is None
install_row.operator(
"snapform.install_environment",
text="Install Environment",
icon="IMPORT",
)
operator.job_type = "INSTALL_ENV"
layout.separator()
operator = layout.operator(
"snapform.control_service",
text="Open Service Log",
layout.operator(
"snapform.open_server_log",
text="Open Server Log",
icon="TEXT",
)
operator.action = "LOG"

View File

@@ -1,7 +1,150 @@
import platform
import sys
import bpy
from .models import gs_model_enum_items, model_enum_items, moge2_model_enum_items
from .client import device_enum_items
from .job_runtime import runtime
from . import model_catalog as shared_model_catalog
BEN2_MODEL_KEY = "BEN2_BASE"
DEFAULT_DA3_MODEL_KEY = "BASE"
DEFAULT_MOGE2_MODEL_KEY = "MOGE2_VITS_NORMAL"
REQUIRED_MODEL_KEYS = (DEFAULT_MOGE2_MODEL_KEY, BEN2_MODEL_KEY)
DEVICE_LABELS = {
"cuda": ("CUDA", "NVIDIA CUDA"),
"directml": ("DirectML", "Windows GPU (DirectML)"),
"mps": ("MPS", "Apple Metal (MPS)"),
"cpu": ("CPU", "CPU"),
}
def supported_devices():
devices = runtime.environment_manifest().get("devices")
if isinstance(devices, list) and devices:
return [str(device).lower() for device in devices]
if sys.platform == "win32":
return ["directml", "cpu"]
if sys.platform == "darwin" and platform.machine().lower() in {
"arm64",
"aarch64",
}:
return ["mps", "cpu"]
return ["cpu"]
def device_enum_items(_owner, _context):
return [
(name.upper(), *DEVICE_LABELS[name], index)
for index, name in enumerate(supported_devices())
]
def models_status(refresh=False):
status = runtime.server_status().get("resources", {}).get("model_manager", {})
return {
"catalog": tuple(
shared_model_catalog.model_record(
model,
_model_files_ready(model),
)
for model in shared_model_catalog.DOWNLOADABLE_MODELS.values()
),
"loaded": status.get("loaded", {}),
}
def _model_files_ready(model):
directory = runtime.storage_root() / "models" / model.directory_name
return all(
any(directory.glob(pattern))
for pattern in model.ready_patterns
)
def model_ready(model_key):
return any(
model.get("key") == model_key and model.get("ready")
for model in models_status().get("catalog", ())
)
def required_models_ready():
return all(model_ready(model_key) for model_key in REQUIRED_MODEL_KEYS)
def model_catalog():
return models_status().get("catalog", ())
def model_record(model_key):
return next(
(model for model in model_catalog() if model.get("key") == model_key),
None,
)
def _model_items(models):
return [
(
model["key"],
model["label"] if model.get("ready") else f"{model['label']} (Missing)",
f"{model['description']}; license: {model['license']}",
index,
)
for index, model in enumerate(models)
]
def model_enum_items(_owner, _context):
return _model_items(
model
for model in model_catalog()
if model.get("family") == "da3" and model.get("enabled")
)
def gs_model_enum_items(_owner, _context):
return _model_items(
model
for model in model_catalog()
if model.get("family") == "da3"
and model.get("enabled")
and model.get("supports_gs")
)
def moge2_model_enum_items(_owner, _context):
return _model_items(
model for model in model_catalog() if model.get("family") == "moge2"
)
def depth_plane_model_enum_items(_owner, _context):
models = [
model
for model in model_catalog()
if model.get("family") in {"moge2", "da3"} and model.get("enabled")
]
models.sort(key=lambda model: model.get("key") != DEFAULT_MOGE2_MODEL_KEY)
items = _model_items(models)
items.append(
("ORIGINAL", "Original", "Use the original image as depth", len(items))
)
return items
def surface_model_enum_items(_owner, _context):
models = [
model
for model in model_catalog()
if model.get("family") in {"moge2", "da3"}
and model.get("enabled")
]
models.sort(key=lambda model: model.get("key") != DEFAULT_MOGE2_MODEL_KEY)
return _model_items(models)
class SnapFormSettings(bpy.types.PropertyGroup):
@@ -76,7 +219,17 @@ class SnapFormSettings(bpy.types.PropertyGroup):
name="Model",
description="MoGe-2 model used for depth estimation",
items=moge2_model_enum_items,
default=1,
default=0,
)
moge_export_format: bpy.props.EnumProperty(
name="Output Format",
description="Additional MoGe-2 debug output generated with results.npz",
items=(
("maps", "Maps", "Export depth, normal, and mask previews"),
("glb", "Textured Surface (GLB)", "Export the official-style textured mesh"),
("ply", "Point Cloud (PLY)", "Export the official-style colored point cloud"),
),
default="maps",
)
conf_thresh_percentile: bpy.props.FloatProperty(
name="Confidence Threshold",
@@ -98,11 +251,6 @@ class SnapFormSettings(bpy.types.PropertyGroup):
description="Include DA3 camera wireframes in the GLB export",
default=True,
)
use_ray_pose: bpy.props.BoolProperty(
name="Use Ray Pose",
description="Use the slower and more accurate ray-based pose estimation",
default=False,
)
ref_view_strategy: bpy.props.EnumProperty(
name="Ref View Strategy",
description="DA3 reference view selection strategy",
@@ -175,16 +323,3 @@ class SnapFormSettings(bpy.types.PropertyGroup):
default=1,
min=1,
)
status: bpy.props.StringProperty(
name="Status",
default="Ready",
options={"SKIP_SAVE"},
)
progress: bpy.props.FloatProperty(
name="Progress",
default=0.0,
min=0.0,
max=1.0,
subtype="FACTOR",
options={"SKIP_SAVE"},
)

View File

@@ -0,0 +1,57 @@
"""SnapForm Job Server without Blender dependencies."""
from blendjob import JobServer
from .model_manager import ModelManager, ModelSessionCache
from .background import remove_background as _remove_background
from .da3 import generate_da3 as _generate_da3
from .media import extract_frames as _extract_frames
from .moge2 import generate_moge2 as _generate_moge2
server = JobServer("SnapForm Job Server")
@server.resource("model_manager")
def create_model_manager(job_server):
return ModelManager(job_server.storage_root / "models")
@server.job("download-model")
def download_model(context, parameters):
manager = context.resource("model_manager")
return manager.download(context, parameters["model"])
@server.job("download-required-models")
def download_required_models(context, parameters):
manager = context.resource("model_manager")
return manager.download_missing(context, parameters["models"])
@server.job("debug-ben2")
@server.job("remove-background")
def remove_background(context, parameters):
return _remove_background(context, parameters)
@server.job("extract-frames")
def extract_frames(context, parameters):
return _extract_frames(context, parameters)
@server.job("debug-moge2")
@server.job("generate-moge2-surface")
@server.job("generate-moge2-depth")
def generate_moge2(context, parameters):
return _generate_moge2(context, parameters)
@server.job("debug-da3")
@server.job("generate-da3-surface")
@server.job("generate-da3-depth")
def generate_da3(context, parameters):
return _generate_da3(context, parameters)
__all__ = ("ModelManager", "ModelSessionCache", "server")

View File

@@ -0,0 +1,170 @@
import shutil
import time
from pathlib import Path
from .media import (
IMAGE_SUFFIXES,
check_cancelled,
copy_frames,
prepare_background_input,
)
def infer_foreground(
input_path,
model_dir,
device,
model_cache=None,
cancel_check=None,
):
from PIL import Image
input_path = Path(input_path)
check_cancelled(cancel_check)
if input_path.suffix.lower() not in IMAGE_SUFFIXES:
raise RuntimeError("BEN2 requires a JPG, JPEG, PNG, or WEBP image")
from .onnx_ben2 import create_session, infer
owns_model = model_cache is None
requested_device = getattr(device, "type", device)
if owns_model:
load_started = time.perf_counter()
model = create_session(model_dir, requested_device)
load_ms = (time.perf_counter() - load_started) * 1000.0
else:
model, load_ms = model_cache.get_ben2(model_dir, device)
check_cancelled(cancel_check)
with Image.open(input_path) as opened_image:
image = opened_image.convert("RGB")
inference_started = time.perf_counter()
foreground = infer(model, image)
foreground = preserve_source_rgb(image, foreground)
inference_ms = (time.perf_counter() - inference_started) * 1000.0
check_cancelled(cancel_check)
return foreground, load_ms, inference_ms
def preserve_source_rgb(source, foreground):
"""Combine the untouched source RGB with BEN2's foreground alpha."""
source_rgba = source.convert("RGBA")
alpha = foreground.convert("RGBA").getchannel("A")
if alpha.size != source_rgba.size:
from PIL import Image
alpha = alpha.resize(source_rgba.size, Image.Resampling.LANCZOS)
source_rgba.putalpha(alpha)
return source_rgba
def infer_foreground_frames(
images,
model_dir,
device,
output_directory,
progress_callback,
start_progress,
progress_span,
model_cache=None,
cancel_check=None,
):
"""Run BEN2 on every frame and write foreground.0001.png etc."""
total_load_ms = 0.0
total_inference_ms = 0.0
for index, image_path in enumerate(images):
check_cancelled(cancel_check)
progress_callback(
start_progress + progress_span * index / len(images),
f"Removing background ({index + 1}/{len(images)})",
)
foreground, load_ms, inference_ms = infer_foreground(
image_path,
model_dir,
device,
model_cache=model_cache,
cancel_check=cancel_check,
)
output_path = output_directory / f"foreground.{index + 1:04d}.png"
foreground.convert("RGBA").save(
output_path,
format="PNG",
compress_level=1,
)
if not output_path.is_file():
raise RuntimeError(f"BEN2 did not write: {output_path}")
total_load_ms += load_ms
total_inference_ms += inference_ms
return total_load_ms, total_inference_ms
def remove_background(context, parameters):
"""Run BEN2 and write transparent foreground images into the Job directory."""
total_started = time.perf_counter()
context.progress(0.02, "Starting Environment")
context.check_cancelled()
output_directory = context.directory
output_directory.mkdir(parents=True, exist_ok=True)
device = parameters["device"]
model_manager = context.resource("model_manager")
model_directory = model_manager.directory(parameters["model"])
video_frames = parameters.get("video_frames", 16)
start_frame = parameters.get("start_frame", 1)
images, temporary_directory = prepare_background_input(
parameters["input"],
video_frames,
output_directory,
start_frame=start_frame,
cancel_check=context.check_cancelled,
)
try:
context.progress(0.12, f"Loading BEN2 for {device.upper()}")
if len(images) == 1:
context.progress(0.35, "Removing background")
foreground, load_ms, inference_ms = infer_foreground(
images[0],
model_directory,
device,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
context.check_cancelled()
context.progress(0.92, "Writing transparent PNG")
output_path = output_directory / "foreground.png"
foreground.convert("RGBA").save(
output_path,
format="PNG",
compress_level=1,
)
if not output_path.is_file():
raise RuntimeError("BEN2 did not write a foreground image")
else:
load_ms, inference_ms = infer_foreground_frames(
images,
model_directory,
device,
output_directory,
context.progress,
0.12,
0.7,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
copy_frames(images, output_directory)
finally:
if temporary_directory is not None:
shutil.rmtree(temporary_directory, ignore_errors=True)
context.check_cancelled()
total_ms = (time.perf_counter() - total_started) * 1000.0
frame_text = "" if len(images) == 1 else f" from {len(images)} frame(s)"
message = (
f"Background removed{frame_text} in {inference_ms:.0f} ms "
f"(model load {load_ms:.0f} ms, total {total_ms:.0f} ms)"
)
print(message)
context.progress(1.0, message)
if tuple(output_directory.glob("foreground.*.png")):
return {"foreground": "."}
return {"foreground": "foreground.png"}

232
src/snapform/server/da3.py Normal file
View File

@@ -0,0 +1,232 @@
import shutil
from .background import infer_foreground, infer_foreground_frames
from .geometry import (
export_da3_depth_preview,
export_da3_mini_npz,
export_da3_point_cloud,
export_image_mesh_data,
export_prediction_attributes,
export_textured_mesh_glb,
)
from .media import (
copy_frames,
flatten_exported_paths,
prepare_input,
wait_for_exported_paths,
)
def merge_da3_frame_predictions(predictions):
"""Combine independent single-frame DA3 results for existing exporters."""
import numpy as np
if not predictions:
raise ValueError("DA3 returned no frame predictions")
merged = predictions[0]
for name in (
"depth",
"conf",
"extrinsics",
"intrinsics",
"processed_images",
):
values = [getattr(prediction, name) for prediction in predictions]
setattr(merged, name, np.concatenate(values, axis=0))
return merged
def generate_da3(context, parameters):
"""Run DA3 inference and return the artifact used by the requesting Job."""
context.progress(0.02, "Starting Environment")
context.check_cancelled()
from .onnx_da3 import infer
from .onnx_runtime import provider_label
output_directory = context.directory
output_directory.mkdir(parents=True, exist_ok=True)
context.progress( 0.08, "Preparing input")
model_manager = context.resource("model_manager")
model_directory = model_manager.directory(parameters["model"])
device = parameters["device"]
export_format = parameters["export_format"]
images, temporary_directory = prepare_input(
parameters["input"],
parameters.get("video_frames", 16),
output_directory,
start_frame=parameters.get("start_frame", 1),
cancel_check=context.check_cancelled,
)
try:
context.check_cancelled()
remove_background_model = parameters.get("remove_background_model", "")
remove_background_model_dir = (
model_manager.directory(remove_background_model)
if remove_background_model
else ""
)
foreground_images = []
if remove_background_model_dir:
context.progress( 0.12, "Removing image background")
if len(images) == 1:
foreground, load_ms, inference_ms = infer_foreground(
images[0],
remove_background_model_dir,
device,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
context.check_cancelled()
foreground.convert("RGBA").save(
output_directory / "foreground.png",
format="PNG",
compress_level=1,
)
if not (output_directory / "foreground.png").is_file():
raise RuntimeError("BEN2 did not write a foreground image")
foreground_images = [output_directory / "foreground.png"]
else:
load_ms, inference_ms = infer_foreground_frames(
images,
remove_background_model_dir,
device,
output_directory,
context.progress,
0.12,
0.05,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
foreground_images = sorted(
output_directory.glob("foreground.*.png")
)
if remove_background_model_dir:
print(
f"BEN2 inference {inference_ms:.0f} ms; "
f"model load {load_ms:.0f} ms"
)
context.progress( 0.15, "Loading DA3 ONNX model")
model = model_manager.get_da3(model_directory, device)
context.progress(
0.20,
f"Loaded DA3 on {provider_label(model)}",
)
context.check_cancelled()
context.progress(
0.25,
f"Estimating depth for {len(images)} frame(s)",
)
inference_options = {
"process_res": parameters["process_res"],
"process_res_method": parameters["process_res_method"],
"ref_view_strategy": parameters["ref_view_strategy"],
}
try:
if parameters.get("inference_mode", "joint") == "per_frame":
predictions = []
for index, image in enumerate(images):
context.check_cancelled()
context.progress(
0.25 + 0.60 * index / len(images),
f"Estimating DA3 depth ({index + 1}/{len(images)})",
)
predictions.append(
infer(model, [str(image)], **inference_options)
)
prediction = merge_da3_frame_predictions(predictions)
else:
prediction = infer(
model,
[str(path) for path in images],
**inference_options,
)
except UnicodeDecodeError as error:
# DirectML reports failures with a locale-specific message that
# ONNX Runtime cannot always decode, masking the real error.
if provider_label(model) == "DirectML":
raise RuntimeError(
"DA3 inference failed on DirectML. Switch the device to "
"CPU in SnapForm Preferences and retry. "
f"Original error: {error}"
) from error
raise
context.check_cancelled()
context.progress( 0.88, f"Exporting {export_format}")
if export_format == "mesh":
export_textured_mesh_glb(
prediction,
images,
output_directory,
parameters.get("conf_thresh_percentile", 40.0),
bool(parameters.get("mesh_filter_edges", True)),
float(parameters.get("mesh_max_edge_length", 0.1)),
)
elif export_format == "mesh_data":
def mesh_progress(step, total, label):
context.progress(
0.88 + 0.09 * step / total,
f"Preparing surface data {step}/{total}: {label}",
)
export_image_mesh_data(
prediction,
images[0],
output_directory,
foreground_path=(
foreground_images[0] if foreground_images else None
),
progress_callback=mesh_progress,
)
elif export_format == "mini_npz":
export_da3_mini_npz(prediction, output_directory)
elif export_format == "depth_vis":
export_da3_depth_preview(prediction, output_directory)
elif export_format == "glb":
export_da3_point_cloud(
prediction,
output_directory,
parameters["conf_thresh_percentile"],
parameters["num_max_points"],
)
else:
raise ValueError(
f"DA3 ONNX does not support export format: {export_format}"
)
if export_format in {"glb", "gs_ply", "depth_vis", "mesh"}:
context.progress(
0.97,
"Writing structured debug attributes",
)
export_prediction_attributes(prediction, output_directory)
context.check_cancelled()
result_paths = wait_for_exported_paths(
output_directory,
export_format,
cancel_check=context.check_cancelled,
)
if not result_paths:
raise RuntimeError(
f"No {export_format} files were exported"
)
result_paths = flatten_exported_paths(
output_directory,
result_paths,
cancel_check=context.check_cancelled,
)
context.check_cancelled()
copy_frames(images, output_directory)
context.progress(
1.0,
f"Exported {len(result_paths)} {export_format} file(s)",
)
if context.job_type == "generate-da3-surface":
return {"surface": "mesh.npz"}
if context.job_type == "generate-da3-depth":
return {"depth": "results.npz"}
return {"directory": "."}
finally:
if temporary_directory is not None:
shutil.rmtree(temporary_directory, ignore_errors=True)

View File

@@ -0,0 +1,616 @@
from pathlib import Path
SCENE_FILENAME = "scene.glb"
def unproject_depth_frame(depth, intrinsics, extrinsics):
import numpy as np
height, width = depth.shape
x, y = np.meshgrid(
np.arange(width, dtype=np.float32),
np.arange(height, dtype=np.float32),
)
pixels = np.stack((x, y, np.ones_like(x)), axis=-1).reshape(-1, 3)
rays = (np.linalg.inv(intrinsics) @ pixels.T).T
camera_points = rays * depth.reshape(-1, 1)
camera_points = np.column_stack(
(camera_points, np.ones(camera_points.shape[0], dtype=np.float32))
)
world_to_camera = np.eye(4, dtype=np.float32)
world_to_camera[: extrinsics.shape[0], :4] = extrinsics
world_points = (np.linalg.inv(world_to_camera) @ camera_points.T).T
return world_points[:, :3].reshape(height, width, 3)
def unproject_camera_depth(depth, intrinsics):
import numpy as np
height, width = depth.shape
x, y = np.meshgrid(
np.arange(width, dtype=np.float32),
np.arange(height, dtype=np.float32),
)
pixels = np.stack((x, y, np.ones_like(x)), axis=-1).reshape(-1, 3)
rays = (np.linalg.inv(intrinsics) @ pixels.T).T
return (rays * depth.reshape(-1, 1)).reshape(height, width, 3)
def compact_mesh(vertices, faces):
import numpy as np
vertices = np.asarray(vertices, dtype=np.float32)
faces = np.asarray(faces, dtype=np.int32)
used_vertices = np.unique(faces)
index_map = np.full(len(vertices), -1, dtype=np.int32)
index_map[used_vertices] = np.arange(len(used_vertices), dtype=np.int32)
vertices = vertices[used_vertices]
faces = index_map[faces]
return vertices, faces
def project_camera_uv(vertices, intrinsics, image_size):
import numpy as np
width, height = image_size
z = vertices[:, 2]
if np.any(~np.isfinite(z)) or np.any(z <= 0.0):
raise ValueError("Simplified mesh contains invalid camera depth")
u = intrinsics[0, 0] * vertices[:, 0] / z + intrinsics[0, 2]
v = intrinsics[1, 1] * vertices[:, 1] / z + intrinsics[1, 2]
denominator_x = max(width - 1, 1)
denominator_y = max(height - 1, 1)
return np.column_stack(
(
np.clip(u / denominator_x, 0.0, 1.0),
np.clip(1.0 - v / denominator_y, 0.0, 1.0),
)
).astype(np.float32)
def project_camera_pixels(vertices, intrinsics):
import numpy as np
z = vertices[:, 2]
if np.any(~np.isfinite(z)) or np.any(z <= 0.0):
raise ValueError("Mesh contains invalid camera depth")
return np.column_stack(
(
intrinsics[0, 0] * vertices[:, 0] / z + intrinsics[0, 2],
intrinsics[1, 1] * vertices[:, 1] / z + intrinsics[1, 2],
)
)
def sample_image_nearest(values, pixels):
import numpy as np
height, width = values.shape
x = np.clip(np.rint(pixels[:, 0]).astype(np.int32), 0, width - 1)
y = np.clip(np.rint(pixels[:, 1]).astype(np.int32), 0, height - 1)
return np.asarray(values)[y, x]
def confidence_percentile_map(depth, confidence):
import numpy as np
valid = np.isfinite(depth) & (depth > 0.0) & np.isfinite(confidence)
result = np.zeros(depth.shape, dtype=np.float32)
if not valid.any():
return result
sorted_values = np.sort(np.asarray(confidence)[valid])
ranks = np.searchsorted(
sorted_values,
np.asarray(confidence)[valid],
side="right",
)
result[valid] = ranks.astype(np.float32) / len(sorted_values)
return result
def depth_discontinuity_ratio(depth):
import numpy as np
padded = np.pad(depth, 1, mode="edge")
windows = np.lib.stride_tricks.sliding_window_view(padded, (3, 3))
difference = np.nanmax(windows, axis=(-2, -1))
difference -= np.nanmin(windows, axis=(-2, -1))
return difference / np.maximum(depth, 1e-6)
def triangle_edge_stretch_values(points, triangles):
import numpy as np
flat_points = np.asarray(points, dtype=np.float32).reshape(-1, 3)
triangle_points = flat_points[triangles]
triangle_depth = triangle_points[..., 2]
safe_depth = np.where(
np.abs(triangle_depth) > 1e-8,
triangle_depth,
np.nan,
)
triangle_rays = triangle_points / safe_depth[..., None]
next_points = np.roll(triangle_points, -1, axis=1)
next_rays = np.roll(triangle_rays, -1, axis=1)
next_depth = np.roll(triangle_depth, -1, axis=1)
local_depth = (triangle_depth + next_depth) * 0.5
flat_edges = (
triangle_rays * local_depth[..., None]
- next_rays * local_depth[..., None]
)
flat_lengths = np.linalg.norm(flat_edges, axis=-1)
actual_lengths = np.linalg.norm(
triangle_points - next_points,
axis=-1,
)
stretch = actual_lengths / np.maximum(flat_lengths, 1e-8)
stretch[np.isclose(stretch, 1.0, rtol=1e-5, atol=1e-6)] = 1.0
return stretch
def image_mesh_attributes(
vertices,
faces,
intrinsics,
confidence_percentiles,
face_depth_discontinuity,
):
import numpy as np
point_percentile = sample_image_nearest(
confidence_percentiles,
project_camera_pixels(vertices, intrinsics),
).astype(np.float32)
face_confidence = point_percentile[faces].min(axis=1)
return {
"point_depth": vertices[:, 2].astype(np.float32),
"face_confidence_percentile": face_confidence.astype(np.float32),
"face_depth_discontinuity": np.asarray(
face_depth_discontinuity,
dtype=np.float32,
),
}
def export_prediction_attributes(prediction, output_directory):
import numpy as np
values = {}
for name in ("depth", "conf", "intrinsics", "extrinsics"):
value = getattr(prediction, name, None)
if value is not None:
values[name] = np.asarray(value)
if "depth" in values and "conf" in values:
values["confidence_percentile"] = np.stack(
[
confidence_percentile_map(depth, confidence)
for depth, confidence in zip(
values["depth"],
values["conf"],
)
]
)
output_path = Path(output_directory) / "prediction.attributes.npz"
np.savez(output_path, **values)
return output_path
def export_da3_mini_npz(prediction, output_directory):
import numpy as np
values = {"depth": np.round(prediction.depth, 8)}
for name in ("conf", "extrinsics", "intrinsics"):
value = getattr(prediction, name, None)
if value is not None:
values[name] = np.round(value, 2) if name == "conf" else value
output_path = Path(output_directory) / "results.npz"
np.savez_compressed(output_path, **values)
return output_path
def export_da3_depth_preview(prediction, output_directory):
import numpy as np
from PIL import Image
target = Path(output_directory) / "depth_vis"
target.mkdir(parents=True, exist_ok=True)
for index, depth in enumerate(prediction.depth):
valid = np.isfinite(depth) & (depth > 0.0)
normalized = np.zeros(depth.shape, dtype=np.float32)
if valid.any():
low, high = np.percentile(depth[valid], (2.0, 98.0))
normalized[valid] = np.clip(
(depth[valid] - low) / max(float(high - low), 1e-6),
0.0,
1.0,
)
color = np.stack(
(
normalized,
1.0 - np.abs(normalized * 2.0 - 1.0),
1.0 - normalized,
),
axis=-1,
)
color = np.asarray(color * 255.0, dtype=np.uint8)
source = np.asarray(prediction.processed_images[index], dtype=np.uint8)
combined = np.concatenate((source, color), axis=1)
Image.fromarray(combined, mode="RGB").save(
target / f"{index:04d}.jpg",
quality=95,
)
def export_da3_point_cloud(
prediction,
output_directory,
confidence_percentile=40.0,
num_max_points=1_000_000,
):
import numpy as np
import trimesh
depth = np.asarray(prediction.depth, dtype=np.float32)
confidence = np.asarray(prediction.conf, dtype=np.float32)
threshold = confidence_percentile_threshold(
depth,
confidence,
confidence_percentile,
)
point_sets = []
color_sets = []
for index in range(depth.shape[0]):
points = unproject_depth_frame(
depth[index],
prediction.intrinsics[index],
prediction.extrinsics[index],
)
valid = np.isfinite(points).all(axis=-1) & (depth[index] > 0.0)
if threshold is not None:
valid &= confidence[index] >= threshold
point_sets.append(points[valid])
color_sets.append(prediction.processed_images[index][valid])
points = np.concatenate(point_sets, axis=0)
colors = np.concatenate(color_sets, axis=0)
maximum = max(int(num_max_points), 1)
if len(points) > maximum:
indices = np.linspace(0, len(points) - 1, maximum, dtype=np.int64)
points = points[indices]
colors = colors[indices]
scene = trimesh.Scene(trimesh.points.PointCloud(points, colors=colors))
scene.export(Path(output_directory) / SCENE_FILENAME)
def remove_background_faces(faces, foreground_mask):
"""Return only triangles whose vertices all belong to the foreground."""
foreground_points = foreground_mask.reshape(-1)
return faces[foreground_points[faces].all(axis=1)]
def export_image_mesh_data(
prediction,
image_path,
output_directory,
foreground_path=None,
progress_callback=None,
):
import numpy as np
from PIL import Image
total_steps = 6
def progress(step, label):
if progress_callback is not None:
progress_callback(step, total_steps, label)
progress(1, "Validating DA3 data")
required = ("depth", "conf", "intrinsics")
missing = [
name
for name in required
if getattr(prediction, name, None) is None
]
if missing:
raise ValueError(f"DA3 mesh output is missing: {', '.join(missing)}")
depth_frames = np.asarray(prediction.depth, dtype=np.float32)
confidence_frames = np.asarray(prediction.conf, dtype=np.float32)
if depth_frames.shape[0] != 1:
raise ValueError("Convert to Surface currently supports one image")
raw_depth = depth_frames[0]
confidence = confidence_frames[0]
intrinsics = np.asarray(prediction.intrinsics[0], dtype=np.float32)
progress(2, "Unprojecting the depth grid")
points = unproject_camera_depth(raw_depth, intrinsics)
progress(3, "Preparing confidence and background data")
foreground_mask = np.ones(raw_depth.shape, dtype=bool)
if foreground_path is not None:
foreground_mask = foreground_mesh_mask(
foreground_path,
raw_depth.shape,
)
confidence_percentiles = confidence_percentile_map(
raw_depth,
confidence,
)
progress(4, "Building source triangles")
faces = mesh_triangle_faces(
points,
raw_depth,
confidence,
None,
False,
0.0,
)
if foreground_path is not None:
faces = remove_background_faces(faces, foreground_mask)
if not len(faces):
raise ValueError(
"DA3 output does not contain any valid surface faces"
)
progress(5, "Preparing source UV and attributes")
depth_discontinuity = depth_discontinuity_ratio(raw_depth)
face_depth_discontinuity = depth_discontinuity.reshape(-1)[faces].max(axis=1)
vertices, faces = compact_mesh(
points.reshape(-1, 3),
faces,
)
height, width = raw_depth.shape
uv = project_camera_uv(vertices, intrinsics, (width, height))
attributes = image_mesh_attributes(
vertices,
faces,
intrinsics,
confidence_percentiles,
face_depth_discontinuity,
)
reference_depth = float(np.median(vertices[:, 2]))
output_directory = Path(output_directory)
output_directory.mkdir(parents=True, exist_ok=True)
progress(6, "Writing source surface data")
result_path = output_directory / "mesh.npz"
np.savez_compressed(
result_path,
vertices=vertices,
faces=faces,
uv=uv,
intrinsics=intrinsics,
image_size=np.asarray((width, height), dtype=np.int32),
reference_depth=np.asarray(reference_depth, dtype=np.float32),
**attributes,
)
texture_path = output_directory / "texture.png"
texture_source = foreground_path or image_path
with Image.open(texture_source) as image:
image.convert("RGBA").save(texture_path, format="PNG", compress_level=1)
return result_path, texture_path
def depth_edge_mask(depth, threshold=12.0 / 255.0):
import numpy as np
values = np.asarray(depth, dtype=np.float32)
edges = np.zeros(values.shape, dtype=bool)
for axis in (0, 1):
first = np.take(values, range(values.shape[axis] - 1), axis=axis)
second = np.take(values, range(1, values.shape[axis]), axis=axis)
valid = (
np.isfinite(first)
& np.isfinite(second)
& (first > 0.0)
& (second > 0.0)
)
scale = np.maximum(np.minimum(np.abs(first), np.abs(second)), 1e-6)
discontinuity = valid & (np.abs(first - second) / scale > threshold)
lower = [slice(None), slice(None)]
upper = [slice(None), slice(None)]
lower[axis] = slice(0, -1)
upper[axis] = slice(1, None)
edges[tuple(lower)] |= discontinuity
edges[tuple(upper)] |= discontinuity
return edges
def edge_culling_stretch_limit(culling):
factor = min(max(float(culling) / 100.0, 0.0001), 1.0)
return 1.0 / factor
def foreground_mesh_mask(foreground_path, depth_shape):
import numpy as np
from PIL import Image, ImageFilter
height, width = depth_shape
with Image.open(foreground_path) as source:
alpha = source.convert("RGBA").getchannel("A")
alpha = alpha.resize(
(width, height),
resample=Image.Resampling.BILINEAR,
)
alpha = alpha.filter(ImageFilter.GaussianBlur(radius=0.75))
return np.asarray(alpha) >= 128
def confidence_percentile_threshold(depth, confidence, percentile):
import numpy as np
if float(percentile) <= 0.0:
return None
valid = np.isfinite(depth) & (depth > 0.0) & np.isfinite(confidence)
if not valid.any():
raise ValueError("DA3 returned no valid depth confidence values")
return float(np.percentile(confidence[valid], percentile))
def mesh_triangle_faces(
points,
depth,
confidence,
confidence_threshold,
filter_edges,
max_edge_length,
edge_threshold=12.0 / 255.0,
edge_stretch_limit=None,
vertex_mask=None,
):
import numpy as np
height, width = depth.shape
indices = np.arange(height * width, dtype=np.int32).reshape(height, width)
quads = np.stack(
(
indices[:-1, :-1],
indices[1:, :-1],
indices[1:, 1:],
indices[:-1, 1:],
),
axis=-1,
).reshape(-1, 4)
valid = (
np.isfinite(points).all(axis=-1)
& np.isfinite(depth)
& (depth > 0.0)
)
if confidence_threshold is not None:
valid &= np.isfinite(confidence) & (
confidence >= confidence_threshold
)
if filter_edges:
valid &= ~depth_edge_mask(depth, edge_threshold)
if vertex_mask is not None:
if np.shape(vertex_mask) != depth.shape:
raise ValueError("Foreground mask and depth dimensions do not match")
valid &= np.asarray(vertex_mask, dtype=bool)
triangles = np.concatenate(
(quads[:, (0, 1, 2)], quads[:, (0, 2, 3)]),
axis=0,
)
face_valid = valid.reshape(-1)[triangles].all(axis=1)
if edge_stretch_limit is not None:
face_valid &= triangle_edge_stretch_mask(
points,
triangles,
edge_stretch_limit,
)
if max_edge_length > 0.0:
flat_points = points.reshape(-1, 3)
triangle_points = flat_points[triangles]
next_points = np.roll(triangle_points, -1, axis=1)
edge_lengths = np.linalg.norm(
triangle_points - next_points,
axis=-1,
)
face_valid &= (edge_lengths <= max_edge_length).all(axis=1)
triangles = triangles[face_valid]
if not len(triangles):
return np.empty((0, 3), dtype=np.int32)
return triangles
def triangle_edge_stretch_mask(points, triangles, stretch_limit):
stretch = triangle_edge_stretch_values(points, triangles)
tolerance = float(stretch_limit) * (1.0 + 1e-5)
return (stretch <= tolerance).all(axis=1)
def export_textured_mesh_glb(
prediction,
image_paths,
output_directory,
confidence_percentile,
filter_edges,
max_edge_length,
):
import numpy as np
import trimesh
from PIL import Image
required = ("depth", "conf", "intrinsics", "extrinsics")
missing = [name for name in required if getattr(prediction, name) is None]
if missing:
raise ValueError(f"DA3 mesh output is missing: {', '.join(missing)}")
depth = np.asarray(prediction.depth, dtype=np.float32)
confidence = np.asarray(prediction.conf, dtype=np.float32)
threshold = confidence_percentile_threshold(
depth,
confidence,
confidence_percentile,
)
meshes = []
referenced_points = []
for index in range(depth.shape[0]):
points = unproject_depth_frame(
depth[index],
prediction.intrinsics[index],
prediction.extrinsics[index],
)
faces = mesh_triangle_faces(
points,
depth[index],
confidence[index],
threshold,
filter_edges,
max_edge_length,
)
if not len(faces):
continue
height, width = depth[index].shape
u = np.linspace(0.0, 1.0, width, dtype=np.float32)
v = np.linspace(1.0, 0.0, height, dtype=np.float32)
uu, vv = np.meshgrid(u, v)
uv = np.stack((uu, vv), axis=-1).reshape(-1, 2)
with Image.open(image_paths[index]) as source_image:
texture_image = source_image.convert("RGB").copy()
visual = trimesh.visual.texture.TextureVisuals(
uv=uv,
image=texture_image,
)
mesh = trimesh.Trimesh(
vertices=points.reshape(-1, 3),
faces=faces,
visual=visual,
process=False,
)
mesh.vertex_attributes["o_depth"] = depth[index].reshape(-1)
mesh.vertex_attributes["o_confidence"] = confidence[
index
].reshape(-1)
mesh.vertex_attributes["o_confidence_percentile"] = (
confidence_percentile_map(
depth[index],
confidence[index],
).reshape(-1)
)
mesh.remove_unreferenced_vertices()
meshes.append(mesh)
referenced_points.append(mesh.vertices)
if not meshes:
raise ValueError(
"Surface filters removed every face; lower Confidence Threshold or disable filters"
)
world_to_camera = np.asarray(prediction.extrinsics[0], dtype=np.float64)
if world_to_camera.shape == (3, 4):
homogeneous = np.eye(4, dtype=np.float64)
homogeneous[:3, :4] = world_to_camera
world_to_camera = homogeneous
cv_to_gltf = np.eye(4, dtype=np.float64)
cv_to_gltf[1, 1] = -1.0
cv_to_gltf[2, 2] = -1.0
alignment = cv_to_gltf @ world_to_camera
aligned_points = trimesh.transform_points(
np.concatenate(referenced_points, axis=0),
alignment,
)
center = np.median(aligned_points, axis=0)
alignment[:3, 3] -= center
scene = trimesh.Scene()
for index, mesh in enumerate(meshes):
mesh.apply_transform(alignment)
scene.add_geometry(mesh, node_name=f"DepthMesh.{index + 1:04d}")
output_directory = Path(output_directory)
output_directory.mkdir(parents=True, exist_ok=True)
output_path = output_directory / "mesh.glb"
scene.export(output_path)
return output_path

View File

@@ -0,0 +1,306 @@
import shutil
import tempfile
import time
from pathlib import Path
IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png", ".webp"}
VIDEO_SUFFIXES = {
".mp4",
".mov",
".avi",
".mkv",
".webm",
".m4v",
".mts",
".mpg",
".mpeg",
}
SCENE_FILENAME = "scene.glb"
class TaskCancelled(RuntimeError):
pass
def check_cancelled(cancel_check):
if cancel_check is not None and cancel_check():
raise TaskCancelled("Task cancelled")
def collect_images(directory):
directory = Path(directory)
return sorted(
path
for path in directory.iterdir()
if path.is_file() and path.suffix.lower() in IMAGE_SUFFIXES
)
def first_frames(images, maximum_frames, start_frame=1):
"""Return up to maximum_frames image paths in order from start_frame."""
if maximum_frames <= 0:
return images
start_index = max(start_frame - 1, 0)
return images[start_index:start_index + maximum_frames]
def extract_video_frames(
video_path,
output_directory,
maximum_frames,
av,
start_frame=1,
cancel_check=None,
):
output_directory = Path(output_directory)
output_directory.mkdir(parents=True, exist_ok=True)
try:
container = av.open(str(video_path))
except Exception as error:
raise RuntimeError(f"Unable to open video: {video_path}") from error
image_paths = []
skipped = 0
try:
stream = container.streams.video[0]
for frame in container.decode(stream):
check_cancelled(cancel_check)
if skipped < start_frame - 1:
skipped += 1
continue
image_path = output_directory / f"frame.{len(image_paths) + 1:04d}.jpg"
frame.to_image().save(image_path, format="JPEG")
image_paths.append(image_path)
if len(image_paths) >= maximum_frames:
break
finally:
container.close()
if not image_paths:
raise RuntimeError(f"Unable to read any video frame from: {video_path}")
return image_paths
def prepare_input(
input_path,
video_frames,
temporary_root,
start_frame=1,
cancel_check=None,
):
check_cancelled(cancel_check)
input_path = Path(input_path)
if input_path.is_dir():
images = collect_images(input_path)
if not images:
raise RuntimeError(f"No image frames found in: {input_path}")
return first_frames(images, video_frames, start_frame), None
suffix = input_path.suffix.lower()
if suffix in IMAGE_SUFFIXES:
return [input_path], None
if suffix not in VIDEO_SUFFIXES:
supported = ", ".join(sorted(IMAGE_SUFFIXES | VIDEO_SUFFIXES))
raise RuntimeError(f"Unsupported input type; expected one of: {supported}")
import av
temporary_directory = Path(
tempfile.mkdtemp(prefix="snapform-video-", dir=temporary_root)
)
try:
images = extract_video_frames(
input_path,
temporary_directory,
video_frames,
av,
start_frame=start_frame,
cancel_check=cancel_check,
)
except Exception:
shutil.rmtree(temporary_directory, ignore_errors=True)
raise
return images, temporary_directory
def prepare_background_input(
input_path,
video_frames,
temporary_root,
start_frame=1,
cancel_check=None,
):
"""Resolve a BEN2 input to frame images; returns (images, temporary_directory)."""
input_path = Path(input_path)
if input_path.is_dir():
images = collect_images(input_path)
if not images:
raise RuntimeError(f"No image frames found in: {input_path}")
return first_frames(images, video_frames, start_frame), None
if input_path.suffix.lower() in IMAGE_SUFFIXES:
return [input_path], None
return prepare_input(
input_path,
video_frames,
temporary_root,
start_frame=start_frame,
cancel_check=cancel_check,
)
def copy_frames(images, output_directory):
"""Copy a multi-frame input as numbered frame files for the Blender sequence."""
if len(images) <= 1:
return
for index, image_path in enumerate(images):
suffix = Path(image_path).suffix.lower() or ".jpg"
destination = (
Path(output_directory) / f"frame.{index + 1:04d}{suffix}"
)
if Path(image_path).resolve() == destination.resolve():
continue
shutil.copy2(image_path, destination)
def extract_frames(context, parameters):
"""Copy selected source frames into the current job directory."""
context.directory.mkdir(parents=True, exist_ok=True)
images, temporary_directory = prepare_input(
parameters["input"],
parameters.get("video_frames", 16),
context.directory,
start_frame=parameters.get("start_frame", 1),
cancel_check=context.check_cancelled,
)
try:
copy_frames(images, context.directory)
finally:
if temporary_directory is not None:
shutil.rmtree(temporary_directory, ignore_errors=True)
message = f"Extracted {len(images)} frame(s)"
context.progress(1.0, message)
return {"frames": len(images)}
def exported_paths(output_directory, export_format):
output_directory = Path(output_directory)
if export_format == "glb":
scene_path = output_directory / SCENE_FILENAME
return [scene_path] if scene_path.is_file() else []
if export_format == "gs_ply":
return sorted((output_directory / "gs_ply").glob("*.ply"))
if export_format == "depth_vis":
return sorted((output_directory / "depth_vis").glob("*.jpg"))
if export_format == "mesh":
result_path = output_directory / "mesh.glb"
return [result_path] if result_path.is_file() else []
if export_format == "mesh_data":
result_path = output_directory / "mesh.npz"
return [result_path] if result_path.is_file() else []
if export_format == "mini_npz":
result_path = output_directory / "results.npz"
if result_path.is_file():
return [result_path]
result_path = (
output_directory / "exports" / "mini_npz" / "results.npz"
)
return [result_path] if result_path.is_file() else []
return []
def npz_is_complete(path):
"""Return True only when the npz archive can be fully read."""
try:
import numpy as np
with np.load(path, allow_pickle=False) as archive:
return "depth" in archive.files
except Exception:
return False
def wait_for_exported_paths(
output_directory,
export_format,
timeout=30.0,
cancel_check=None,
):
# DA3 的 NPZ exporter 在后台线程写入,文件出现时可能尚未写完整,
# 直接返回会让下游读到半截 zip 而报 EOFError,因此需等文件可完整读取
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
check_cancelled(cancel_check)
result_paths = exported_paths(output_directory, export_format)
if result_paths:
if (
export_format in ("mini_npz", "npz")
and not npz_is_complete(result_paths[0])
):
time.sleep(0.1)
continue
return result_paths
time.sleep(0.05)
return []
def move_exported_file(
source,
destination,
timeout=10.0,
cancel_check=None,
):
"""Move an async export after Windows releases its file handle."""
source = Path(source)
destination = Path(destination)
deadline = time.monotonic() + timeout
while True:
check_cancelled(cancel_check)
try:
source.replace(destination)
return destination
except OSError as error:
is_file_lock = isinstance(error, PermissionError) or getattr(
error,
"winerror",
None,
) in {32, 33}
if not is_file_lock or time.monotonic() >= deadline:
raise
time.sleep(0.1)
def flatten_exported_paths(
output_directory,
result_paths,
cancel_check=None,
):
"""Move files produced in exporter subdirectories to the job root."""
output_directory = Path(output_directory)
flattened_paths = []
for source in result_paths:
source = Path(source)
check_cancelled(cancel_check)
if source.parent == output_directory:
flattened_paths.append(source)
continue
destination = output_directory / source.name
if destination.exists():
raise RuntimeError(f"Export filename already exists: {destination.name}")
flattened_paths.append(
move_exported_file(
source,
destination,
cancel_check=cancel_check,
)
)
for directory in sorted(
{path.parent for path in map(Path, result_paths)},
key=lambda path: len(path.parts),
reverse=True,
):
while directory != output_directory:
try:
directory.rmdir()
except OSError:
break
directory = directory.parent
return flattened_paths

View File

@@ -0,0 +1,445 @@
import hashlib
import http.client
import os
import threading
import time
from pathlib import Path
from urllib.parse import quote
from urllib.request import (
HTTPSHandler,
ProxyHandler,
Request,
build_opener,
urlopen,
)
HF_ENDPOINT = "https://huggingface.co"
R2_ENDPOINT = "https://models.omoolab.xyz"
CONNECT_TIMEOUT = 10.0
READ_TIMEOUT = 60.0
DOWNLOAD_CHUNK_SIZE = 256 * 1024
class _ShortConnectHTTPSConnection(http.client.HTTPSConnection):
"""Bound connection and idle reads without limiting total download time."""
def connect(self):
try:
self.timeout = CONNECT_TIMEOUT
super().connect()
finally:
self.timeout = None
self.sock.settimeout(READ_TIMEOUT)
class _ShortConnectHTTPSHandler(HTTPSHandler):
def https_open(self, request):
return self.do_open(_ShortConnectHTTPSConnection, request)
_download_opener = build_opener(ProxyHandler(), _ShortConnectHTTPSHandler())
class DownloadProgressReporter:
def __init__(
self,
progress_callback,
repository,
clock=time.monotonic,
start=0.05,
span=0.90,
label=None,
):
self.progress_callback = progress_callback
self.repository = repository
self.clock = clock
self.lock = threading.Lock()
self.start = start
self.span = span
self.label = label
self.progress = start
self.last_write_time = 0.0
self.started_at = self.clock()
def report(self, current, total, rate=None):
if not total or current <= 0:
return
factor = min(max(current / total, 0.0), 1.0)
progress = self.start + factor * self.span
now = self.clock()
with self.lock:
if progress <= self.progress:
return
if now - self.last_write_time < 0.20 and factor < 1.0:
return
self.progress = progress
self.last_write_time = now
effective_rate = rate
elapsed = now - self.started_at
if not effective_rate and elapsed > 0:
effective_rate = current / elapsed
speed = format_rate(effective_rate) if effective_rate else ""
prefix = self.label or self.repository
message = f"Downloading {prefix}"
if speed:
message = f"{message} {speed}"
self.progress_callback(
progress,
message,
)
def create_download_progress_class(
progress_callback,
repository,
tqdm_type=None,
clock=time.monotonic,
start=0.05,
span=0.90,
label=None,
):
if tqdm_type is None:
from tqdm.auto import tqdm as tqdm_type
reporter = DownloadProgressReporter(
progress_callback,
repository,
clock,
start,
span,
label,
)
class DownloadProgress(tqdm_type):
def __init__(self, *args, **kwargs):
description = str(kwargs.get("desc", ""))
# huggingface_hub 1.x 的字节进度条描述是 "Downloading (incomplete total...)",
# 用前缀匹配而非精确匹配,否则真实下载进度会被忽略。
self.snapform_reports_progress = (
description.startswith("Downloading")
or description.startswith("Fetching ")
)
super().__init__(*args, **kwargs)
def update(self, amount=1):
result = super().update(amount)
if self.snapform_reports_progress:
reporter.report(self.n, self.total, self._download_rate())
return result
def refresh(self, *args, **kwargs):
result = super().refresh(*args, **kwargs)
if self.snapform_reports_progress:
reporter.report(self.n, self.total, self._download_rate())
return result
def _download_rate(self):
try:
return getattr(self, "format_dict", {}).get("rate")
except Exception:
return None
return DownloadProgress
def progress_range(args):
start = getattr(args, "progress_start", None)
end = getattr(args, "progress_end", None)
if start is not None and end is not None:
return float(start), float(end) - float(start)
return 0.05, 0.90
def download_model(args):
label = getattr(args, "label", None) or args.repository
model_dir = Path(args.model_dir)
model_dir.mkdir(parents=True, exist_ok=True)
start, span = progress_range(args)
r2_directory = getattr(args, "r2_directory", "")
r2_files = getattr(args, "r2_file", [])
if r2_directory and r2_files:
_download_declared_or_fail(
args,
model_dir,
label,
start,
span,
)
return
_download_snapshot_or_fail(args, model_dir, label, start, span)
def _download_declared_or_fail(args, model_dir, label, start, span):
"""Download declared files from Hugging Face, then the R2 mirror."""
sources = (
(HF_ENDPOINT, download_from_hf, f"Downloading {label}"),
(
R2_ENDPOINT,
download_from_r2,
f"Downloading {label} (mirror)",
),
)
failures = []
for source, downloader, message in sources:
args.progress(start, message)
try:
downloader(
args,
model_dir,
start=start,
span=span,
label=label,
)
_verify_model_complete(
model_dir,
getattr(args, "required_pattern", None),
)
except Exception as error:
cancel_check = getattr(args, "cancel_check", None)
if cancel_check is not None:
cancel_check()
failures.append((source, error))
continue
args.progress(start + span, f"{label} ready")
return
detail = "; ".join(
f"{source}: {error}" for source, error in failures
)
raise RuntimeError(f"All download sources failed: {detail}")
def _download_snapshot_or_fail(args, model_dir, label, start, span):
"""Fallback download for repositories without declared files, via huggingface_hub."""
failures = []
try:
from huggingface_hub import snapshot_download
except Exception as error:
snapshot_download = None
failures.append((HF_ENDPOINT, error))
progress_class = None
if snapshot_download:
progress_class = create_download_progress_class(
args.progress,
args.repository,
start=start,
span=span,
label=label,
)
for endpoint in download_endpoints() if snapshot_download else ():
os.environ["HF_ENDPOINT"] = endpoint
# snapshot_download 先拉取文件清单并校验 ETag,还没有字节下载,
# 用 Checking 提示当前阶段,避免进度条看起来卡住。
message = f"Checking {label} files"
if endpoint != HF_ENDPOINT:
message = f"{message} (mirror)"
args.progress(start, message)
try:
snapshot_download(
repo_id=args.repository,
local_dir=model_dir,
tqdm_class=progress_class,
allow_patterns=args.allow_pattern or None,
endpoint=endpoint,
)
_verify_model_complete(
model_dir,
getattr(args, "required_pattern", None),
)
except Exception as error:
cancel_check = getattr(args, "cancel_check", None)
if cancel_check is not None:
cancel_check()
failures.append((endpoint, error))
continue
args.progress(start + span, f"{label} ready")
return
detail = "; ".join(
f"{source}: {error}" for source, error in failures
)
raise RuntimeError(f"All download sources failed: {detail}")
def download_endpoints():
"""Return the configured Hugging Face endpoint or official source."""
endpoint = os.environ.get("HF_ENDPOINT") or HF_ENDPOINT
return [endpoint.rstrip("/")]
def _verify_model_complete(model_dir, required_patterns=None):
"""Refuse to report success when the required model files are missing."""
model_dir = Path(model_dir)
required_patterns = required_patterns or (
"config.json",
"*.safetensors",
)
missing = [
pattern
for pattern in required_patterns
if not any(model_dir.glob(pattern))
]
if missing:
raise RuntimeError(
"Model download reported success but files are missing: "
+ ", ".join(missing)
)
def _download_file_from_url(
url,
destination,
expected_size,
expected_checksum,
progress=None,
retries=1,
cancel_check=None,
):
"""Download and verify one model file atomically from a direct URL."""
request = Request(url, headers={"User-Agent": "SnapForm"})
opener = _download_opener.open if request.type == "https" else urlopen
last_error = None
for _attempt in range(retries):
temporary = destination.with_name(f"{destination.name}.part")
try:
checksum = hashlib.sha256()
downloaded = 0
with opener(request) as response:
with open(temporary, "wb") as output:
while True:
if cancel_check is not None:
cancel_check()
chunk = response.read(DOWNLOAD_CHUNK_SIZE)
if not chunk:
break
output.write(chunk)
checksum.update(chunk)
downloaded += len(chunk)
if progress is not None:
progress(len(chunk))
if downloaded != expected_size:
raise RuntimeError(
f"Size mismatch for {destination.name}: "
f"{downloaded} != {expected_size}"
)
if checksum.hexdigest().lower() != expected_checksum.lower():
raise RuntimeError(f"SHA-256 mismatch for {destination.name}")
temporary.replace(destination)
return
except Exception as error:
if cancel_check is not None:
cancel_check()
last_error = error
try:
temporary.unlink()
except OSError:
pass
raise last_error
def r2_file_url(directory, filename):
return f"{R2_ENDPOINT}/{quote(directory)}/{quote(filename)}"
def hf_file_url(repository, filename):
"""Hugging Face resolve URL for a direct, metadata-free download."""
endpoint = (os.environ.get("HF_ENDPOINT") or HF_ENDPOINT).rstrip("/")
return f"{endpoint}/{repository}/resolve/main/{quote(filename)}"
def _parse_declared_files(args):
selected = []
for value in args.r2_file:
filename, size, checksum = value.split("|", 2)
selected.append((filename, int(size), checksum))
return selected
def _download_declared_files(
args,
model_dir,
url_builder,
start,
span,
label,
retries=1,
):
"""Download each declared file through url_builder with aggregated progress."""
repository = args.repository
selected = _parse_declared_files(args)
reporter = DownloadProgressReporter(
args.progress,
repository,
start=start,
span=span,
label=label or repository,
)
total_size = sum(size for _path, size, _checksum in selected)
downloaded = 0
def on_chunk(count):
nonlocal downloaded
downloaded += count
reporter.report(downloaded, total_size)
for path, size, checksum in selected:
destination = Path(model_dir) / path
destination.parent.mkdir(parents=True, exist_ok=True)
_download_file_from_url(
url_builder(path),
destination,
size,
checksum,
progress=on_chunk,
retries=retries,
cancel_check=getattr(args, "cancel_check", None),
)
def download_from_r2(
args,
model_dir,
start=0.05,
span=0.90,
label=None,
):
"""Download the declared model files from OmooLab R2."""
_download_declared_files(
args,
model_dir,
lambda filename: r2_file_url(args.r2_directory, filename),
start,
span,
f"{label} (mirror)" if label else "mirror",
)
def download_from_hf(
args,
model_dir,
start=0.05,
span=0.90,
label=None,
):
"""Download the declared model files directly from Hugging Face resolve URLs."""
_download_declared_files(
args,
model_dir,
lambda filename: hf_file_url(args.repository, filename),
start,
span,
label,
)
def format_rate(rate):
size = float(rate)
if size <= 0:
return ""
units = ("B/s", "KB/s", "MB/s", "GB/s")
unit_index = 0
while size >= 1024 and unit_index < len(units) - 1:
size /= 1024
unit_index += 1
if unit_index == 0:
return f"{size:.0f} {units[unit_index]}"
return f"{size:.1f} {units[unit_index]}"

View File

@@ -0,0 +1,194 @@
import gc
import time
from pathlib import Path
from types import SimpleNamespace
import model_catalog
from .model_download import download_model
class ModelSessionCache:
"""Cache ONNX sessions shared by Jobs in one Server process."""
def __init__(self):
self.ben2_model = None
self.ben2_key = None
self.da3_model = None
self.da3_key = None
self.moge_model = None
self.moge_key = None
def get_ben2(self, model_dir, device):
requested_device = getattr(device, "type", device)
key = (str(Path(model_dir).resolve()), requested_device, "onnx")
if self.ben2_model is not None and self.ben2_key == key:
return self.ben2_model, 0.0
self.release_ben2()
return self._load_ben2(model_dir, requested_device, key)
def _load_ben2(self, model_dir, requested_device, key):
from .onnx_ben2 import create_session
started = time.perf_counter()
model = create_session(model_dir, requested_device)
self.ben2_model = model
self.ben2_key = key
elapsed_ms = (time.perf_counter() - started) * 1000.0
return model, elapsed_ms
def get_da3(self, model_dir, device):
requested_device = getattr(device, "type", device)
key = (str(Path(model_dir).resolve()), requested_device, "onnx")
if self.moge_model is not None:
self.release_moge()
if self.da3_model is not None and self.da3_key == key:
return self.da3_model
self.release_da3()
return self._load_da3(model_dir, requested_device, key)
def _load_da3(self, model_dir, requested_device, key):
from .onnx_da3 import create_session
model = create_session(model_dir, requested_device)
self.da3_model = model
self.da3_key = key
return model
def get_moge2(self, model_dir, requested_device):
from .onnx_moge2 import create_session
key = (str(Path(model_dir).resolve()), requested_device, "onnx")
if self.da3_model is not None:
self.release_da3()
if self.moge_model is not None and self.moge_key == key:
return self.moge_model
self.release_moge()
model = create_session(model_dir, requested_device)
self.moge_model = model
self.moge_key = key
return model
def release_ben2(self):
self.ben2_model = None
self.ben2_key = None
self._release_runtime_cache()
def release_da3(self):
self.da3_model = None
self.da3_key = None
self._release_runtime_cache()
def release_moge(self):
self.moge_model = None
self.moge_key = None
self._release_runtime_cache()
def close(self):
self.ben2_model = None
self.da3_model = None
self.moge_model = None
self.ben2_key = None
self.da3_key = None
self.moge_key = None
self._release_runtime_cache()
def snapshot(self):
return {
"ben2": Path(self.ben2_key[0]).name if self.ben2_key else "",
"da3": Path(self.da3_key[0]).name if self.da3_key else "",
"moge2": Path(self.moge_key[0]).name if self.moge_key else "",
}
def _release_runtime_cache(self):
gc.collect()
class ModelManager:
"""Own downloaded model files and loaded inference sessions."""
def __init__(self, models_directory):
self.models_directory = Path(models_directory)
self.sessions = ModelSessionCache()
def directory(self, model_key):
model = model_catalog.get_downloadable_model(model_key)
directory = self.models_directory / model.directory_name
old_moge_directory = self.models_directory / "moge2_base_normal"
if (
model_key == "MOGE2_VITB_NORMAL"
and not directory.is_dir()
and old_moge_directory.is_dir()
):
old_moge_directory.rename(directory)
legacy_directory = self.models_directory / model_key.lower()
if (
not directory.is_dir()
and legacy_directory.is_dir()
and legacy_directory != directory
):
legacy_directory.rename(directory)
return directory
def ready(self, model_key):
model = model_catalog.get_downloadable_model(model_key)
directory = self.directory(model_key)
return all(any(directory.glob(pattern)) for pattern in model.ready_patterns)
def get_ben2(self, model_dir, device):
return self.sessions.get_ben2(model_dir, device)
def get_da3(self, model_dir, device):
return self.sessions.get_da3(model_dir, device)
def get_moge2(self, model_dir, device):
return self.sessions.get_moge2(model_dir, device)
def snapshot(self):
catalog = [
model_catalog.model_record(model, self.ready(model.key))
for model in model_catalog.DOWNLOADABLE_MODELS.values()
]
return {"loaded": self.sessions.snapshot(), "catalog": catalog}
def clear(self):
self.sessions.close()
def close(self):
self.sessions.close()
def download(self, context, model_key):
return self._download(context, model_key, context.progress)
def download_missing(self, context, model_keys):
missing = [key for key in model_keys if not self.ready(key)]
if not missing:
context.progress(1.0, "Required models are ready")
return {"downloaded": []}
total = len(missing)
for index, model_key in enumerate(missing):
def report(value, message, *, offset=index):
context.progress((offset + value) / total, message)
self._download(context, model_key, report)
return {"downloaded": missing}
def _download(self, context, model_key, progress):
model = model_catalog.get_downloadable_model(model_key)
args = SimpleNamespace(
repository=model.repository,
model_dir=str(self.directory(model_key)),
progress=progress,
cancel_check=context.check_cancelled,
label=model.label,
allow_pattern=list(model.download_patterns),
required_pattern=list(model.ready_patterns),
r2_directory=model.r2_directory,
r2_file=[
f"{filename}|{size}|{checksum}"
for filename, size, checksum in model.r2_files
],
)
context.check_cancelled()
download_model(args)
context.check_cancelled()

View File

@@ -0,0 +1,304 @@
import shutil
from pathlib import Path
from .background import infer_foreground, infer_foreground_frames
from .geometry import (
depth_discontinuity_ratio,
depth_edge_mask,
foreground_mesh_mask,
mesh_triangle_faces,
)
from .media import copy_frames, prepare_input
def _save_moge_preview(output_directory, index, depth, normal, mask):
import numpy as np
from PIL import Image
suffix = "" if index is None else f".{index + 1:04d}"
valid = mask & np.isfinite(depth) & (depth > 0)
preview = np.zeros(depth.shape, dtype=np.uint8)
if np.any(valid):
low, high = np.percentile(depth[valid], (2.0, 98.0))
span = max(float(high - low), 1e-6)
preview[valid] = np.clip(
(depth[valid] - low) / span * 255.0,
0.0,
255.0,
).astype(np.uint8)
Image.fromarray(preview, mode="L").save(
output_directory / f"depth{suffix}.png"
)
normal_rgb = np.clip(
(normal * np.array([1.0, -1.0, -1.0], dtype=np.float32) + 1.0)
* 127.5,
0.0,
255.0,
).astype(np.uint8)
Image.fromarray(normal_rgb, mode="RGB").save(
output_directory / f"normal{suffix}.png"
)
Image.fromarray(mask.astype(np.uint8) * 255, mode="L").save(
output_directory / f"mask{suffix}.png"
)
def export_moge_debug_mesh(
output_directory,
points,
depth,
mask,
normal,
image,
export_format,
):
import numpy as np
import trimesh
clean_mask = mask & ~depth_edge_mask(depth, threshold=0.04)
faces = mesh_triangle_faces(
points,
depth,
np.ones(depth.shape, dtype=np.float32),
None,
False,
0.0,
vertex_mask=clean_mask,
)
vertices = points.reshape(-1, 3)
colors = np.asarray(image.reshape(-1, 3) * 255.0, dtype=np.uint8)
vertex_normals = normal.reshape(-1, 3)
vertices = vertices * np.array([1.0, -1.0, -1.0], dtype=np.float32)
vertex_normals = vertex_normals * np.array([1.0, -1.0, -1.0], dtype=np.float32)
mesh = trimesh.Trimesh(
vertices=vertices,
faces=faces,
vertex_colors=colors,
vertex_normals=vertex_normals,
process=False,
)
mesh.remove_unreferenced_vertices()
filename = "mesh.glb" if export_format == "glb" else "pointcloud.ply"
mesh.export(Path(output_directory) / filename)
def export_moge_surface_archive(
output_path,
points,
depth,
mask,
normal,
intrinsics,
foreground_mask=None,
):
import numpy as np
height, width = depth.shape
finite_mask = np.isfinite(depth) & (depth > 0.0)
finite_mask &= np.isfinite(points).all(axis=-1)
if foreground_mask is not None:
finite_mask &= foreground_mask
depth_discontinuity = depth_discontinuity_ratio(depth)
faces = mesh_triangle_faces(
points,
depth,
np.ones(depth.shape, dtype=np.float32),
None,
False,
0.0,
vertex_mask=finite_mask,
)
vertices = points.reshape(-1, 3)
u = np.linspace(0.0, 1.0, width, dtype=np.float32)
v = np.linspace(1.0, 0.0, height, dtype=np.float32)
uu, vv = np.meshgrid(u, v)
vertex_uvs = np.stack((uu, vv), axis=-1).reshape(-1, 2)
pixel_intrinsics = intrinsics.copy()
pixel_intrinsics[0] *= width
pixel_intrinsics[1] *= height
pixel_intrinsics[2] = (0.0, 0.0, 1.0)
point_depth = depth.reshape(-1).astype(np.float32)
face_mask = mask.reshape(-1)[faces].all(axis=1)
face_depth_discontinuity = depth_discontinuity.reshape(-1)[faces].max(axis=1)
masked_depth = point_depth[mask.reshape(-1).astype(bool)]
reference_depth = np.median(masked_depth if len(masked_depth) else point_depth)
np.savez(
output_path,
vertices=vertices.astype(np.float32),
faces=faces.astype(np.int32),
uv=vertex_uvs.astype(np.float32),
intrinsics=pixel_intrinsics.astype(np.float32),
image_size=np.array((width, height), dtype=np.int32),
reference_depth=np.float32(reference_depth),
point_depth=point_depth,
normal_map=normal.astype(np.float32),
face_mask=face_mask,
face_depth_discontinuity=face_depth_discontinuity,
)
def generate_moge2(context, parameters):
"""Run MoGe-2 inference and return the artifact used by the requesting Job."""
import numpy as np
from PIL import Image
from .onnx_moge2 import (
infer as infer_moge2,
provider_label,
)
context.progress(0.02, "Starting MoGe-2")
output_directory = context.directory
output_directory.mkdir(parents=True, exist_ok=True)
model_manager = context.resource("model_manager")
model_directory = model_manager.directory(parameters["model"])
device = parameters["device"]
images, temporary_directory = prepare_input(
parameters["input"],
parameters.get("video_frames", 16),
output_directory,
start_frame=parameters.get("start_frame", 1),
cancel_check=context.check_cancelled,
)
try:
model = model_manager.get_moge2(model_directory, device)
context.progress(
0.08,
f"Loading MoGe-2 on {provider_label(model)}",
)
depths = []
normals = []
masks = []
points = []
intrinsics = []
source_images = []
for index, image_path in enumerate(images):
context.check_cancelled()
context.progress(
0.15 + 0.68 * index / len(images),
f"Estimating MoGe-2 geometry ({index + 1}/{len(images)})",
)
with Image.open(image_path) as opened:
image = np.asarray(opened.convert("RGB"), dtype=np.float32) / 255.0
source_images.append(image)
prediction = infer_moge2(
model,
image,
int(parameters["resolution_level"]),
)
depth = prediction["depth"]
normal = prediction["normal"]
mask = prediction["mask"].astype(bool)
point = prediction["points"]
intrinsic = prediction["intrinsics"]
depths.append(depth)
normals.append(normal)
masks.append(mask)
intrinsics.append(intrinsic)
if bool(
parameters.get("include_points", False)
or parameters.get("surface_mesh", False)
):
points.append(point)
debug_format = parameters.get("export_format", "")
if not debug_format or debug_format == "maps":
_save_moge_preview(
output_directory,
None if len(images) == 1 else index,
depth,
normal,
mask,
)
shapes = {array.shape for array in depths}
if len(shapes) != 1:
raise RuntimeError("MoGe-2 frame sizes must match")
result = {
"depth": np.stack(depths).astype(np.float32),
"normal": np.stack(normals).astype(np.float32),
"mask": np.stack(masks),
"intrinsics": np.stack(intrinsics).astype(np.float32),
}
if points:
result["points"] = np.stack(points).astype(np.float32)
debug_format = parameters.get("export_format", "")
if debug_format in {"glb", "ply"} and len(images) == 1:
context.progress(
0.84,
f"Exporting MoGe-2 {debug_format.upper()}",
)
export_moge_debug_mesh(
output_directory,
points[0],
depths[0],
masks[0],
normals[0],
source_images[0],
debug_format,
)
foreground_mask = None
remove_background_model = parameters.get("remove_background_model", "")
if remove_background_model:
remove_background_model_dir = model_manager.directory(
remove_background_model
)
context.progress( 0.87, "Removing image background")
if len(images) == 1:
foreground, _, _ = infer_foreground(
images[0],
remove_background_model_dir,
device,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
foreground_path = output_directory / "foreground.png"
foreground.save(foreground_path)
if points:
foreground_mask = foreground_mesh_mask(
foreground_path,
result["depth"][0].shape,
)
else:
infer_foreground_frames(
images,
remove_background_model_dir,
device,
output_directory,
context.progress,
0.87,
0.08,
model_cache=model_manager,
cancel_check=context.check_cancelled,
)
if bool(parameters.get("surface_mesh", False)):
if len(images) != 1 or not points:
raise RuntimeError("MoGe-2 Surface requires one point map")
context.progress( 0.94, "Building official MoGe-2 surface")
export_moge_surface_archive(
output_directory / "mesh.npz",
points[0],
result["depth"][0],
result["mask"][0],
result["normal"][0],
result["intrinsics"][0],
foreground_mask=(
foreground_mask
if remove_background_model
else None
),
)
np.savez(output_directory / "results.npz", **result)
copy_frames(images, output_directory)
context.progress(
1.0,
f"Exported MoGe-2 data for {len(images)} frame(s)",
)
if context.job_type == "generate-moge2-surface":
return {"surface": "mesh.npz"}
if context.job_type == "generate-moge2-depth":
return {"depth": "results.npz"}
return {"directory": "."}
finally:
if temporary_directory is not None:
shutil.rmtree(temporary_directory, ignore_errors=True)

View File

@@ -0,0 +1,51 @@
"""BEN2 ONNX inference with Pillow and NumPy preprocessing."""
from pathlib import Path
try:
from .onnx_runtime import create_session as create_runtime_session
except ImportError:
from onnx_runtime import create_session as create_runtime_session
ONNX_PATH = Path("onnx/model_fp16.onnx")
INPUT_SIZE = (1024, 1024)
def model_path(model_directory):
return Path(model_directory) / ONNX_PATH
def create_session(model_directory, requested_device="auto"):
return create_runtime_session(model_path(model_directory), requested_device)
def preprocess(image):
import numpy as np
from PIL import Image
resized = image.convert("RGB").resize(INPUT_SIZE, Image.Resampling.LANCZOS)
array = np.asarray(resized, dtype=np.float32) / 255.0
mean = np.array((0.485, 0.456, 0.406), dtype=np.float32)
std = np.array((0.229, 0.224, 0.225), dtype=np.float32)
array = (array - mean) / std
return np.ascontiguousarray(array.transpose(2, 0, 1)[None])
def infer(session, image):
import numpy as np
from PIL import Image
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
alpha = session.run([output_name], {input_name: preprocess(image)})[0]
alpha = np.asarray(alpha, dtype=np.float32).squeeze()
minimum = float(alpha.min())
span = max(float(alpha.max()) - minimum, np.finfo(np.float32).eps)
alpha = np.clip((alpha - minimum) / span * 255.0, 0.0, 255.0)
mask = Image.fromarray(alpha.astype(np.uint8), mode="L")
mask = mask.resize(image.size, Image.Resampling.LANCZOS)
foreground = image.convert("RGBA")
foreground.putalpha(mask)
return foreground

View File

@@ -0,0 +1,240 @@
"""Depth Anything 3 inference with ONNX Runtime and NumPy."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
import numpy as np
try:
from .onnx_runtime import create_session as create_onnx_session
except ImportError:
from onnx_runtime import create_session as create_onnx_session
ONNX_PATH = "model.onnx"
PATCH_SIZE = 14
# DA3 cross-view attention is quadratic in (views x patches); the peak
# attention tensor alone can exhaust memory long before ONNX Runtime reports
# a generic allocation failure.
ATTENTION_BUDGET_BYTES = 4 * 1024 * 1024 * 1024
STRATEGY_CODES = {
"first": 0,
"middle": 1,
"saddle_balanced": 2,
"saddle_sim_range": 3,
}
IMAGE_MEAN = np.asarray((0.485, 0.456, 0.406), dtype=np.float32)
IMAGE_STD = np.asarray((0.229, 0.224, 0.225), dtype=np.float32)
@dataclass
class Prediction:
depth: np.ndarray
conf: np.ndarray
extrinsics: np.ndarray
intrinsics: np.ndarray
processed_images: np.ndarray
is_metric: int = 0
sky: np.ndarray | None = None
gaussians: object | None = None
aux: dict | None = None
scale_factor: float | None = None
def create_session(model_dir, requested_device="auto"):
path = Path(model_dir) / ONNX_PATH
return create_onnx_session(path, requested_device)
def nearest_multiple(value, multiple=PATCH_SIZE):
lower = value // multiple * multiple
upper = lower + multiple
return upper if abs(upper - value) <= abs(value - lower) else lower
def target_size(size, process_res, method):
width, height = size
boundary = max(width, height)
if method.startswith("lower_bound"):
boundary = min(width, height)
scale = float(process_res) / boundary
width = max(1, int(round(width * scale)))
height = max(1, int(round(height * scale)))
if method.endswith("resize"):
return (
max(PATCH_SIZE, nearest_multiple(width)),
max(PATCH_SIZE, nearest_multiple(height)),
)
if method.endswith("crop"):
return (
max(PATCH_SIZE, width // PATCH_SIZE * PATCH_SIZE),
max(PATCH_SIZE, height // PATCH_SIZE * PATCH_SIZE),
)
raise ValueError(f"Unsupported process_res_method: {method}")
def resize_and_crop(image, process_res, method):
from PIL import Image
width, height = image.size
boundary = max(width, height)
if method.startswith("lower_bound"):
boundary = min(width, height)
scale = float(process_res) / boundary
resized_size = (
max(1, int(round(width * scale))),
max(1, int(round(height * scale))),
)
resampling = (
Image.Resampling.BICUBIC
if scale > 1.0
else Image.Resampling.BOX
)
image = image.resize(resized_size, resample=resampling)
final_size = target_size((width, height), process_res, method)
if method.endswith("resize"):
if image.size != final_size:
upscale = final_size[0] > image.width or final_size[1] > image.height
resampling = (
Image.Resampling.BICUBIC
if upscale
else Image.Resampling.BOX
)
image = image.resize(final_size, resample=resampling)
return image
left = (image.width - final_size[0]) // 2
top = (image.height - final_size[1]) // 2
return image.crop((left, top, left + final_size[0], top + final_size[1]))
def preprocess(image_paths, process_res=504, process_res_method="upper_bound_resize"):
from PIL import Image
processed = []
for path in image_paths:
with Image.open(path) as source:
image = resize_and_crop(
source.convert("RGB"),
process_res,
process_res_method,
)
processed.append(np.asarray(image, dtype=np.uint8))
min_height = min(image.shape[0] for image in processed)
min_width = min(image.shape[1] for image in processed)
cropped = []
for image in processed:
top = (image.shape[0] - min_height) // 2
left = (image.shape[1] - min_width) // 2
cropped.append(
image[top : top + min_height, left : left + min_width]
)
images = np.stack(cropped)
normalized = images.astype(np.float32) / 255.0
normalized = (normalized - IMAGE_MEAN) / IMAGE_STD
model_input = normalized.transpose(0, 3, 1, 2)[None]
return np.ascontiguousarray(model_input), images
def quaternion_to_matrix(quaternion):
i, j, k, real = np.moveaxis(quaternion, -1, 0)
two_scale = 2.0 / np.sum(quaternion * quaternion, axis=-1)
values = (
1 - two_scale * (j * j + k * k),
two_scale * (i * j - k * real),
two_scale * (i * k + j * real),
two_scale * (i * j + k * real),
1 - two_scale * (i * i + k * k),
two_scale * (j * k - i * real),
two_scale * (i * k - j * real),
two_scale * (j * k + i * real),
1 - two_scale * (i * i + j * j),
)
return np.stack(values, axis=-1).reshape(quaternion.shape[:-1] + (3, 3))
def camera_from_pose(pose_encoding, height, width):
translation = pose_encoding[..., :3]
rotation = quaternion_to_matrix(pose_encoding[..., 3:7])
rotation_inverse = np.swapaxes(rotation, -1, -2)
translation_inverse = -rotation_inverse @ translation[..., None]
extrinsics = np.concatenate(
(rotation_inverse, translation_inverse),
axis=-1,
).astype(np.float32)
field_of_view_height = pose_encoding[..., 7]
field_of_view_width = pose_encoding[..., 8]
focal_y = (height / 2.0) / np.maximum(
np.tan(field_of_view_height / 2.0),
1e-6,
)
focal_x = (width / 2.0) / np.maximum(
np.tan(field_of_view_width / 2.0),
1e-6,
)
intrinsics = np.zeros(pose_encoding.shape[:-1] + (3, 3), dtype=np.float32)
intrinsics[..., 0, 0] = focal_x
intrinsics[..., 1, 1] = focal_y
intrinsics[..., 0, 2] = width / 2.0
intrinsics[..., 1, 2] = height / 2.0
intrinsics[..., 2, 2] = 1.0
return extrinsics, intrinsics
def estimate_attention_bytes(views, height, width):
"""Estimate the peak cross-view attention tensor size in bytes."""
patches = (height // PATCH_SIZE) * (width // PATCH_SIZE)
tokens = views * patches
# 12 attention heads in DA3 Base; float32 activations.
return tokens * tokens * 12 * 4
def infer(
session,
image_paths,
process_res=504,
process_res_method="upper_bound_resize",
ref_view_strategy="saddle_balanced",
):
inputs = session.get_inputs()
model_input, processed_images = preprocess(
image_paths,
process_res,
process_res_method,
)
_, views, _, height, width = model_input.shape
attention_bytes = estimate_attention_bytes(views, height, width)
if attention_bytes > ATTENTION_BUDGET_BYTES:
raise RuntimeError(
f"DA3 cannot process {views} frame(s) at {width}x{height}: "
"the cross-view attention would need about "
f"{attention_bytes // 2**30} GB. Reduce the frame count or "
"the Resolution Scale."
)
strategy_code = np.asarray(
[STRATEGY_CODES.get(ref_view_strategy, 2)],
dtype=np.int64,
)
input_names = [value.name for value in inputs]
outputs = session.run(
None,
{
input_names[0]: model_input,
input_names[1]: strategy_code,
},
)
depth, confidence, pose_encoding = outputs[:3]
height, width = depth.shape[-2:]
extrinsics, intrinsics = camera_from_pose(
pose_encoding[0],
height,
width,
)
return Prediction(
depth=np.asarray(depth[0], dtype=np.float32),
conf=np.asarray(confidence[0], dtype=np.float32),
extrinsics=extrinsics,
intrinsics=intrinsics,
processed_images=processed_images,
aux={},
)

View File

@@ -0,0 +1,155 @@
"""MoGe-2 ONNX inference and NumPy post-processing."""
from pathlib import Path
from .onnx_runtime import create_session as create_runtime_session
from .onnx_runtime import provider_label, select_providers
ONNX_FILENAME = "model.onnx"
MIN_TOKENS = 1200
MAX_TOKENS = 3600
def is_onnx_model(model_directory):
return (Path(model_directory) / ONNX_FILENAME).is_file()
def create_session(model_directory, requested_device="auto"):
path = Path(model_directory) / ONNX_FILENAME
return create_runtime_session(path, requested_device)
def normalized_view_plane_uv(width, height, dtype):
import numpy as np
aspect_ratio = width / height
span_x = aspect_ratio / (1.0 + aspect_ratio**2) ** 0.5
span_y = 1.0 / (1.0 + aspect_ratio**2) ** 0.5
u = np.linspace(
-span_x * (width - 1) / width,
span_x * (width - 1) / width,
width,
dtype=dtype,
)
v = np.linspace(
-span_y * (height - 1) / height,
span_y * (height - 1) / height,
height,
dtype=dtype,
)
u, v = np.meshgrid(u, v, indexing="xy")
return np.stack((u, v), axis=-1)
def solve_focal_shift(uv, points):
import numpy as np
uv = uv.reshape(-1, 2)
xy = points[..., :2].reshape(-1, 2)
z = points[..., 2].reshape(-1)
matrix = np.stack((xy, -uv), axis=-1).reshape(-1, 2)
target = (uv * z[:, None]).reshape(-1)
focal, shift = np.linalg.lstsq(matrix, target, rcond=None)[0]
epsilon = np.finfo(np.float32).eps
for _iteration in range(20):
denominator = z + shift
denominator = np.where(
np.abs(denominator) < epsilon,
np.copysign(epsilon, denominator),
denominator,
)
projected = xy / denominator[:, None]
residual = (focal * projected - uv).reshape(-1)
jacobian = np.stack(
(
projected,
-focal * xy / np.square(denominator)[:, None],
),
axis=-1,
).reshape(-1, 2)
delta = np.linalg.lstsq(jacobian, -residual, rcond=None)[0]
focal += delta[0]
shift += delta[1]
if np.max(np.abs(delta)) < 1e-6:
break
return np.float32(focal), shift
def recover_focal_shift(points, mask, downsample_size=(64, 64)):
import numpy as np
height, width = points.shape[:2]
uv = normalized_view_plane_uv(width, height, points.dtype)
target_height, target_width = downsample_size
y = np.floor(np.arange(target_height) * height / target_height).astype(int)
x = np.floor(np.arange(target_width) * width / target_width).astype(int)
points_low = points[y[:, None], x[None, :]]
uv_low = uv[y[:, None], x[None, :]]
mask_low = mask[y[:, None], x[None, :]]
points_low = points_low[mask_low]
uv_low = uv_low[mask_low]
if len(points_low) < 2:
return np.float32(1.0), np.float32(0.0)
return solve_focal_shift(uv_low, points_low)
def depth_to_points(depth, intrinsics):
import numpy as np
height, width = depth.shape
u = np.linspace(0.5 / width, 1.0 - 0.5 / width, width, dtype=depth.dtype)
v = np.linspace(0.5 / height, 1.0 - 0.5 / height, height, dtype=depth.dtype)
u, v = np.meshgrid(u, v, indexing="xy")
x = (u - intrinsics[0, 2]) / intrinsics[0, 0] * depth
y = (v - intrinsics[1, 2]) / intrinsics[1, 1] * depth
return np.stack((x, y, depth), axis=-1)
def postprocess(raw_output):
import numpy as np
points = np.asarray(raw_output["points"][0], dtype=np.float32)
normal = np.asarray(raw_output["normal"][0], dtype=np.float32)
mask = np.asarray(raw_output["mask"][0] > 0.5)
metric_scale = np.float32(np.asarray(raw_output["metric_scale"]).reshape(-1)[0])
height, width = mask.shape
aspect_ratio = width / height
focal, shift = recover_focal_shift(points, mask)
diagonal_factor = (1.0 + aspect_ratio**2) ** 0.5
fx = focal / 2.0 * diagonal_factor / aspect_ratio
fy = focal / 2.0 * diagonal_factor
intrinsics = np.array(
((fx, 0.0, 0.5), (0.0, fy, 0.5), (0.0, 0.0, 1.0)),
dtype=np.float32,
)
depth = points[..., 2] + shift
mask &= depth > 0.0
depth = depth * metric_scale
points = depth_to_points(depth, intrinsics)
return {
"points": points.astype(np.float32),
"depth": depth.astype(np.float32),
"normal": normal,
"mask": mask,
"intrinsics": intrinsics,
}
def infer(session, image, resolution_level):
import numpy as np
level = min(max(int(resolution_level), 0), 9)
num_tokens = int(MIN_TOKENS + level / 9.0 * (MAX_TOKENS - MIN_TOKENS))
image_input = np.ascontiguousarray(image.transpose(2, 0, 1)[None], dtype=np.float32)
input_feed = {
"image": image_input,
"num_tokens": np.asarray(num_tokens, dtype=np.int64),
}
output_names = {output.name for output in session.get_outputs()}
scale_name = "metric_scale" if "metric_scale" in output_names else "scale"
names = ("points", "normal", "mask", scale_name)
values = session.run(list(names), input_feed)
raw_output = dict(zip(names, values))
raw_output["metric_scale"] = raw_output.pop(scale_name)
return postprocess(raw_output)

View File

@@ -0,0 +1,55 @@
"""Shared ONNX Runtime session helpers."""
def select_providers(requested, available):
available = set(available)
providers_for_device = {
"cuda": ("CUDAExecutionProvider", "DmlExecutionProvider"),
"directml": ("DmlExecutionProvider",),
"mps": ("CoreMLExecutionProvider",),
"cpu": ("CPUExecutionProvider",),
}
if requested != "auto":
candidates = providers_for_device[requested]
for provider in candidates:
if provider in available:
return [provider]
if "CPUExecutionProvider" in available:
return ["CPUExecutionProvider"]
raise RuntimeError(
f"{requested.upper()} was requested but the ONNX Runtime "
f"providers {', '.join(candidates)} are not available"
)
priorities = (
"CUDAExecutionProvider",
"CoreMLExecutionProvider",
"DmlExecutionProvider",
"OpenVINOExecutionProvider",
"CPUExecutionProvider",
)
providers = [provider for provider in priorities if provider in available]
if not providers:
raise RuntimeError("ONNX Runtime has no usable execution provider")
return providers
def create_session(path, requested_device="auto"):
import onnxruntime
providers = select_providers(
requested_device,
onnxruntime.get_available_providers(),
)
return onnxruntime.InferenceSession(str(path), providers=providers)
def provider_label(session):
provider = session.get_providers()[0]
labels = {
"CUDAExecutionProvider": "CUDA",
"CoreMLExecutionProvider": "CoreML",
"DmlExecutionProvider": "DirectML",
"OpenVINOExecutionProvider": "OpenVINO",
"CPUExecutionProvider": "CPU",
}
return labels.get(provider, provider)

View File

@@ -1,623 +0,0 @@
import argparse
import gc
import hmac
import json
import os
import sys
import threading
import time
import traceback
from contextlib import asynccontextmanager
from pathlib import Path
from types import SimpleNamespace
SCRIPT_DIRECTORY = Path(__file__).resolve().parent
if str(SCRIPT_DIRECTORY) not in sys.path:
sys.path.insert(0, str(SCRIPT_DIRECTORY))
import worker
CLIENT_TIMEOUT_SECONDS = 10.0
JOB_COMMANDS = frozenset(
{"remove-background", "reconstruct", "extract-frames", "moge"}
)
IDLE_SHUTDOWN_SECONDS = 20.0
class ServiceLock:
def __init__(self, path):
self.path = Path(path)
self.file = None
def acquire(self):
self.path.parent.mkdir(parents=True, exist_ok=True)
lock_file = self.path.open("a+b")
lock_file.seek(0, os.SEEK_END)
if lock_file.tell() == 0:
lock_file.write(b"0")
lock_file.flush()
lock_file.seek(0)
try:
if sys.platform == "win32":
import msvcrt
msvcrt.locking(lock_file.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self.file = lock_file
return True
def release(self):
if self.file is None:
return
try:
self.file.seek(0)
if sys.platform == "win32":
import msvcrt
msvcrt.locking(self.file.fileno(), msvcrt.LK_UNLCK, 1)
else:
import fcntl
fcntl.flock(self.file.fileno(), fcntl.LOCK_UN)
finally:
self.file.close()
self.file = None
class ModelCache:
def __init__(self):
import torch
self.torch = torch
self.ben2_model = None
self.ben2_key = None
self.da3_model = None
self.da3_key = None
self.moge_model = None
self.moge_key = None
def get_ben2(self, model_dir, device):
key = (str(Path(model_dir).resolve()), device.type)
if self.ben2_model is not None and self.ben2_key == key:
return self.ben2_model, 0.0
self.release_ben2()
try:
return self._load_ben2(model_dir, device, key)
except self.torch.cuda.OutOfMemoryError:
self.release_da3()
self.release_moge()
return self._load_ben2(model_dir, device, key)
def _load_ben2(self, model_dir, device, key):
from ben2 import BEN_Base
started = time.perf_counter()
model = BEN_Base.from_pretrained(model_dir)
model = model.to(device=device)
model.eval()
self.ben2_model = model
self.ben2_key = key
elapsed_ms = (time.perf_counter() - started) * 1000.0
return model, elapsed_ms
def get_da3(self, model_dir, device):
key = (str(Path(model_dir).resolve()), device.type)
if self.da3_model is not None and self.da3_key == key:
return self.da3_model
self.release_da3()
try:
return self._load_da3(model_dir, device, key)
except self.torch.cuda.OutOfMemoryError:
self.release_ben2()
self.release_moge()
return self._load_da3(model_dir, device, key)
def _load_da3(self, model_dir, device, key):
from depth_anything_3.api import DepthAnything3
model = DepthAnything3.from_pretrained(model_dir)
model = model.to(device=device)
model.eval()
self.da3_model = model
self.da3_key = key
return model
def get_moge(self, model_dir, device):
key = (str(Path(model_dir).resolve()), device.type)
if self.moge_model is not None and self.moge_key == key:
return self.moge_model
self.release_moge()
try:
return self._load_moge(model_dir, device, key)
except self.torch.cuda.OutOfMemoryError:
self.release_da3()
self.release_ben2()
return self._load_moge(model_dir, device, key)
def _load_moge(self, model_dir, device, key):
worker.prepare_moge_runtime()
from moge.model.v2 import MoGeModel
checkpoint = Path(model_dir) / "model.pt"
model = MoGeModel.from_pretrained(checkpoint)
model = model.to(device=device)
model.eval()
self.moge_model = model
self.moge_key = key
return model
def release_ben2(self):
self.ben2_model = None
self.ben2_key = None
self._release_cuda_cache()
def release_da3(self):
self.da3_model = None
self.da3_key = None
self._release_cuda_cache()
def release_moge(self):
self.moge_model = None
self.moge_key = None
self._release_cuda_cache()
def close(self):
self.ben2_model = None
self.da3_model = None
self.moge_model = None
self.ben2_key = None
self.da3_key = None
self.moge_key = None
self._release_cuda_cache()
def status(self):
return {
"ben2": (
Path(self.ben2_key[0]).name if self.ben2_key is not None else ""
),
"da3": (
Path(self.da3_key[0]).name
if self.da3_key is not None
else ""
),
"moge2": (
Path(getattr(self, "moge_key", ("",))[0]).name
if getattr(self, "moge_key", None) is not None
else ""
),
}
def _release_cuda_cache(self):
gc.collect()
if self.torch.cuda.is_available():
self.torch.cuda.empty_cache()
class ServiceState:
def __init__(self, token, instance_id):
self.token = token
self.instance_id = instance_id
self.models = ModelCache()
self.lock = threading.Lock()
self.clients = {}
self.active_job = None
self.started_at = time.time()
self.last_without_clients = time.monotonic()
self.shutdown_event = threading.Event()
self.monitor_stop = threading.Event()
self.monitor_thread = None
def register_client(self, client_id, label):
with self.lock:
self.clients[client_id] = {
"label": label,
"last_seen": time.monotonic(),
}
self.last_without_clients = None
def heartbeat(self, client_id):
with self.lock:
client = self.clients.get(client_id)
if client is None:
return False
client["last_seen"] = time.monotonic()
return True
def unregister_client(self, client_id):
with self.lock:
self.clients.pop(client_id, None)
self._mark_empty_clients()
def reserve(self, job_id, client_id, command, status_path):
with self.lock:
if client_id not in self.clients or self.active_job is not None:
return False
self.active_job = {
"job_id": job_id,
"client_id": client_id,
"command": command,
"status": status_path,
"cancel_requested": False,
}
return True
def snapshot(self):
self.expire_clients()
with self.lock:
return {
"ready": True,
"instance_id": self.instance_id,
"busy": self.active_job is not None,
"active_job": dict(self.active_job) if self.active_job else None,
"clients": len(self.clients),
"cached_models": self.models.status(),
"started_at": self.started_at,
}
def clear_models(self):
with self.lock:
if self.active_job is not None:
return False
self.models.close()
return True
def request_shutdown(self, client_id):
with self.lock:
if client_id not in self.clients:
return "missing"
if len(self.clients) > 1:
return "shared"
if self.active_job is not None:
return "busy"
self.shutdown_event.set()
return "accepted"
def cancel_job(self, job_id, client_id):
with self.lock:
active = self.active_job
if active is None or active["job_id"] != job_id:
return None
if active["client_id"] != client_id:
return None
active["cancel_requested"] = True
return dict(active)
def is_job_cancelled(self, job_id):
with self.lock:
active = self.active_job
return bool(
active is not None
and active["job_id"] == job_id
and active["cancel_requested"]
)
def run_job(self, payload):
job_id = payload["job_id"]
parameters = payload["parameters"]
status_path = parameters["status"]
worker.write_status(
status_path,
0.0,
"Service accepted the task",
state="running",
)
try:
cancel_check = lambda: self.is_job_cancelled(job_id)
worker.check_cancelled(cancel_check)
args = SimpleNamespace(**parameters)
if payload["command"] == "remove-background":
worker.remove_background(
args,
model_cache=self.models,
cancel_check=cancel_check,
)
elif payload["command"] == "extract-frames":
worker.extract_frames(
args,
cancel_check=cancel_check,
)
elif payload["command"] == "moge":
worker.run_moge(
args,
model_cache=self.models,
cancel_check=cancel_check,
)
else:
worker.reconstruct(
args,
model_cache=self.models,
cancel_check=cancel_check,
)
worker.check_cancelled(cancel_check)
except worker.TaskCancelled:
worker.write_status(
status_path,
1.0,
"Task cancelled",
state="cancelled",
)
except Exception as error:
traceback.print_exc()
worker.write_status(
status_path,
1.0,
"Task failed",
str(error),
state="failed",
)
finally:
with self.lock:
if (
self.active_job is not None
and self.active_job["job_id"] == job_id
):
self.active_job = None
self._mark_empty_clients()
def start_monitor(self):
self.monitor_thread = threading.Thread(
target=self._monitor,
name="SnapFormServiceMonitor",
daemon=True,
)
self.monitor_thread.start()
def close(self):
self.monitor_stop.set()
if self.monitor_thread is not None:
self.monitor_thread.join(timeout=1.0)
self.models.close()
def expire_clients(self):
deadline = time.monotonic() - CLIENT_TIMEOUT_SECONDS
with self.lock:
expired = [
client_id
for client_id, client in self.clients.items()
if client["last_seen"] < deadline
]
for client_id in expired:
self.clients.pop(client_id, None)
self._mark_empty_clients()
def _mark_empty_clients(self):
if self.clients:
self.last_without_clients = None
elif self.last_without_clients is None:
self.last_without_clients = time.monotonic()
def _monitor(self):
while not self.monitor_stop.wait(1.0):
self.expire_clients()
with self.lock:
idle = self.active_job is None
empty_since = self.last_without_clients
if not idle or empty_since is None:
continue
if time.monotonic() - empty_since >= IDLE_SHUTDOWN_SECONDS:
self.shutdown_event.set()
return
def create_app(token, instance_id="test"):
from fastapi import BackgroundTasks, FastAPI, Header, HTTPException
state = ServiceState(token, instance_id)
@asynccontextmanager
async def lifespan(_app):
state.start_monitor()
yield
state.close()
app = FastAPI(
title="SnapForm Service",
docs_url=None,
redoc_url=None,
openapi_url=None,
lifespan=lifespan,
)
app.state.snapform_service_state = state
def authorize(x_snapform_token: str = Header(default="")):
if not hmac.compare_digest(x_snapform_token, state.token):
raise HTTPException(status_code=401, detail="Invalid service token")
@app.get("/health")
def health(x_snapform_token: str = Header(default="")):
authorize(x_snapform_token)
return state.snapshot()
@app.post("/clients", status_code=201)
def register_client(
payload: dict,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
client_id = str(payload.get("client_id", ""))
if not client_id:
raise HTTPException(status_code=422, detail="Missing client id")
state.register_client(client_id, str(payload.get("label", "Blender")))
return {"client_id": client_id, "lease_seconds": CLIENT_TIMEOUT_SECONDS}
@app.post("/clients/{client_id}/heartbeat")
def heartbeat(
client_id: str,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
if not state.heartbeat(client_id):
raise HTTPException(status_code=404, detail="Client is not registered")
return {"client_id": client_id, "state": "active"}
@app.delete("/clients/{client_id}")
def unregister_client(
client_id: str,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
state.unregister_client(client_id)
return {"client_id": client_id, "state": "removed"}
@app.post("/jobs", status_code=202)
def submit_job(
payload: dict,
background_tasks: BackgroundTasks,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
job_id = str(payload.get("job_id", ""))
client_id = str(payload.get("client_id", ""))
command = payload.get("command")
parameters = payload.get("parameters")
if not job_id or command not in JOB_COMMANDS:
raise HTTPException(status_code=422, detail="Invalid job request")
if not isinstance(parameters, dict) or not parameters.get("status"):
raise HTTPException(status_code=422, detail="Missing job parameters")
if not state.reserve(
job_id,
client_id,
command,
parameters["status"],
):
raise HTTPException(status_code=409, detail="Service is busy")
worker.write_status(
parameters["status"],
0.0,
"Task queued",
state="queued",
)
background_tasks.add_task(state.run_job, payload)
return {"job_id": job_id, "state": "queued"}
@app.delete("/jobs/{job_id}", status_code=202)
def cancel_job(
job_id: str,
payload: dict,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
client_id = str(payload.get("client_id", ""))
active = state.cancel_job(job_id, client_id)
if active is None:
raise HTTPException(status_code=404, detail="Active job was not found")
try:
current = json.loads(
Path(active["status"]).read_text(encoding="utf-8")
)
progress = float(current.get("progress", 0.0))
except (OSError, ValueError, TypeError, json.JSONDecodeError):
progress = 0.0
worker.write_status(
active["status"],
progress,
"Cancelling task",
state="cancelling",
)
return {"job_id": job_id, "state": "cancelling"}
@app.post("/models/clear")
def clear_models(x_snapform_token: str = Header(default="")):
authorize(x_snapform_token)
if not state.clear_models():
raise HTTPException(status_code=409, detail="Service is busy")
return {"state": "cleared"}
@app.post("/shutdown", status_code=202)
def shutdown_service(
payload: dict,
x_snapform_token: str = Header(default=""),
):
authorize(x_snapform_token)
result = state.request_shutdown(str(payload.get("client_id", "")))
if result == "shared":
raise HTTPException(status_code=409, detail="Other Blender clients are connected")
if result == "busy":
raise HTTPException(status_code=409, detail="Service is busy")
if result != "accepted":
raise HTTPException(status_code=404, detail="Client is not registered")
return {"state": "stopping"}
return app
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--host", default="127.0.0.1")
parser.add_argument("--port", type=int, required=True)
parser.add_argument("--token", required=True)
parser.add_argument("--instance-id", required=True)
parser.add_argument("--discovery", required=True)
parser.add_argument("--lock", required=True)
return parser.parse_args()
def write_discovery(args):
path = Path(args.discovery)
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_name(f"{path.name}.tmp")
temporary.write_text(
json.dumps(
{
"port": args.port,
"token": args.token,
"pid": os.getpid(),
"instance_id": args.instance_id,
}
),
encoding="utf-8",
)
temporary.replace(path)
def remove_discovery(path, instance_id):
path = Path(path)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (FileNotFoundError, json.JSONDecodeError, OSError):
return
if payload.get("instance_id") != instance_id:
return
path.unlink(missing_ok=True)
def main():
import uvicorn
args = parse_args()
service_lock = ServiceLock(args.lock)
if not service_lock.acquire():
return
try:
app = create_app(args.token, args.instance_id)
config = uvicorn.Config(
app,
host=args.host,
port=args.port,
access_log=False,
log_level="warning",
)
server = uvicorn.Server(config)
def watch_shutdown():
app.state.snapform_service_state.shutdown_event.wait()
server.should_exit = True
threading.Thread(target=watch_shutdown, daemon=True).start()
write_discovery(args)
server.run()
finally:
remove_discovery(args.discovery, args.instance_id)
service_lock.release()
if __name__ == "__main__":
main()

View File

@@ -1,55 +0,0 @@
from .operators import active_job
def draw_status_bar(self, context):
scene = getattr(context, "scene", None)
job = active_job()
if job is None or scene is None:
return
settings = getattr(scene, "snapform_settings", None)
if settings is None:
return
progress = min(max(settings.progress, 0.0), 1.0)
row = self.layout.row(align=True)
progress_row = row.row(align=True)
progress_row.ui_units_x = 18.0
progress_row.progress(
factor=progress,
type="BAR",
text=settings.status,
)
row.operator(
"snapform.cancel_job",
text="",
icon="X",
)
def tag_ui_regions(context):
window_manager = getattr(context, "window_manager", None)
if window_manager is None:
return
for window in window_manager.windows:
screen = getattr(window, "screen", None)
if screen is None:
continue
for area in screen.areas:
if area.type in {"STATUSBAR", "VIEW_3D"}:
area.tag_redraw()
for region in getattr(area, "regions", ()):
region.tag_redraw()
def redraw_ui(context, force=False):
tag_ui_regions(context)
if not force:
return
workspace = getattr(context, "workspace", None)
if workspace is not None:
try:
workspace.status_text_set_internal(None)
except (AttributeError, RuntimeError, TypeError):
pass

File diff suppressed because it is too large Load Diff

194
sync_models.py Normal file
View File

@@ -0,0 +1,194 @@
"""Prepare SnapForm ONNX models and sync them to OmooLab R2."""
import argparse
import hashlib
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from urllib.request import Request, urlopen
PROJECT_ROOT = Path(__file__).resolve().parent
MODELS_DIR = PROJECT_ROOT / "models"
R2_ENDPOINT = "https://models.omoolab.xyz"
CHUNK_SIZE = 1024 * 1024
@dataclass(frozen=True)
class ModelFile:
folder: str
size: int
sha256: str
sources: tuple[str, ...]
@property
def destination(self):
filename = "model_fp16.onnx" if self.folder.endswith("/onnx") else "model.onnx"
return MODELS_DIR / self.folder / filename
def hf_url(repository, filename):
return f"https://huggingface.co/{repository}/resolve/main/{filename}"
def r2_url(folder, filename="model.onnx"):
return f"{R2_ENDPOINT}/{folder}/{filename}"
MODEL_FILES = (
ModelFile(
"DA3-SMALL",
105484029,
"7a9d81475b3e2334ace0d6d2c2a816b3c93081437e1866cc434ce1789125f631",
(r2_url("DA3-SMALL"),),
),
ModelFile(
"DA3-BASE",
412856697,
"31f38fc4990539d7902ba267924b9731db1acb00d840c248a60b1865810ff9e6",
(r2_url("DA3-BASE"),),
),
ModelFile(
"BEN2-ONNX/onnx",
219121675,
"dfdc25f421f32a0d1268e0f2ff2153d340e8f1d52d3dd16f5dc33c1ce85cedf1",
(
hf_url("onnx-community/BEN2-ONNX", "onnx/model_fp16.onnx"),
r2_url("BEN2-ONNX", "onnx/model_fp16.onnx"),
),
),
ModelFile(
"moge-2-vits-normal-onnx",
140852051,
"24eacb5dc7a2c54c7bc98f7de085ffbed79ad006ea5b664c2c2cdc02ff3a52f0",
(
hf_url("Ruicheng/moge-2-vits-normal-onnx", "model.onnx"),
r2_url("moge-2-vits-normal-onnx"),
),
),
ModelFile(
"moge-2-vitb-normal-onnx",
419411850,
"bbf14e07a30f11e69d36ab861590123f5598ababcbc8946a063eb4a966f35a21",
(
hf_url("Ruicheng/moge-2-vitb-normal-onnx", "model.onnx"),
r2_url("moge-2-vitb-normal-onnx"),
),
),
ModelFile(
"moge-2-vitl-normal-onnx",
1324265014,
"afbc4ccc3450298f3afb35b90f015f4c4f552dea21dc6470d5f7b78b77e2d751",
(
hf_url("Ruicheng/moge-2-vitl-normal-onnx", "model.onnx"),
r2_url("moge-2-vitl-normal-onnx"),
),
),
)
def file_sha256(path):
digest = hashlib.sha256()
with Path(path).open("rb") as stream:
while chunk := stream.read(CHUNK_SIZE):
digest.update(chunk)
return digest.hexdigest()
def verify_existing(model):
path = model.destination
if not path.is_file():
return False
if path.stat().st_size != model.size:
raise RuntimeError(f"Existing model has an unexpected size: {path}")
if file_sha256(path) != model.sha256:
raise RuntimeError(f"Existing model has an unexpected SHA-256: {path}")
print(f"Skip existing: {path.relative_to(PROJECT_ROOT)}")
return True
def download_once(model, url):
destination = model.destination
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_suffix(".onnx.part")
temporary.unlink(missing_ok=True)
request = Request(url, headers={"User-Agent": "SnapForm model sync"})
print(f"Download: {url}")
downloaded = 0
digest = hashlib.sha256()
try:
with urlopen(request, timeout=60) as response, temporary.open("wb") as output:
while chunk := response.read(CHUNK_SIZE):
output.write(chunk)
digest.update(chunk)
downloaded += len(chunk)
print(
f"\r {downloaded / 1024**2:.1f} / "
f"{model.size / 1024**2:.1f} MiB",
end="",
flush=True,
)
print()
if downloaded != model.size:
raise RuntimeError(f"size mismatch: {downloaded} != {model.size}")
if digest.hexdigest() != model.sha256:
raise RuntimeError("SHA-256 mismatch")
temporary.replace(destination)
except Exception:
temporary.unlink(missing_ok=True)
raise
def prepare_model(model):
if verify_existing(model):
return
failures = []
for url in model.sources:
try:
download_once(model, url)
return
except Exception as error:
failures.append(f"{url}: {error}")
print(f"Source failed: {error}", file=sys.stderr)
raise RuntimeError("All sources failed:\n" + "\n".join(failures))
def sync_models():
executable = shutil.which("rclone")
if executable is None:
raise RuntimeError("rclone is not available on PATH")
subprocess.run(
[
executable,
"sync",
"models",
"omoolab-r2:ai-models",
"--progress",
],
cwd=PROJECT_ROOT,
check=True,
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"--download-only",
action="store_true",
help="Prepare and verify models without running rclone",
)
return parser.parse_args()
def main():
args = parse_args()
for model in MODEL_FILES:
prepare_model(model)
if not args.download_only:
sync_models()
if __name__ == "__main__":
main()

File diff suppressed because it is too large Load Diff

58
tests/test_ben2_onnx.py Normal file
View File

@@ -0,0 +1,58 @@
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
import numpy as np
from PIL import Image
SOURCE_DIR = (
Path(__file__).resolve().parents[1]
/ "src"
/ "snapform"
/ "server"
)
sys.path.insert(0, str(SOURCE_DIR))
import onnx_ben2
class FakeSession:
def get_inputs(self):
return [SimpleNamespace(name="input")]
def get_outputs(self):
return [SimpleNamespace(name="alpha")]
def run(self, output_names, feeds):
self.output_names = output_names
self.feeds = feeds
gradient = np.linspace(0.0, 1.0, 1024, dtype=np.float32)
return [np.tile(gradient, (1024, 1))[None, None]]
class Ben2OnnxTest(unittest.TestCase):
def test_preprocess_matches_model_layout(self):
image = Image.new("RGB", (40, 20), (64, 128, 192))
result = onnx_ben2.preprocess(image)
self.assertEqual(result.shape, (1, 3, 1024, 1024))
self.assertEqual(result.dtype, np.float32)
self.assertTrue(result.flags.c_contiguous)
def test_infer_restores_original_size_and_alpha(self):
session = FakeSession()
image = Image.new("RGB", (40, 20), (10, 20, 30))
result = onnx_ben2.infer(session, image)
self.assertEqual(result.mode, "RGBA")
self.assertEqual(result.size, image.size)
self.assertEqual(session.feeds["input"].shape, (1, 3, 1024, 1024))
alpha = np.asarray(result.getchannel("A"))
self.assertLess(int(alpha[:, 0].max()), int(alpha[:, -1].min()))
if __name__ == "__main__":
unittest.main()

66
tests/test_da3_onnx.py Normal file
View File

@@ -0,0 +1,66 @@
import sys
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
import numpy as np
from PIL import Image
SOURCE_DIR = (
Path(__file__).resolve().parents[1]
/ "src"
/ "snapform"
/ "server"
)
sys.path.insert(0, str(SOURCE_DIR))
import onnx_da3
class FakeSession:
def get_inputs(self):
return [SimpleNamespace(name="images"), SimpleNamespace(name="strategy")]
def run(self, output_names, feeds):
self.feeds = feeds
_, views, _, height, width = feeds["images"].shape
depth = np.ones((1, views, height, width), dtype=np.float32)
confidence = np.full_like(depth, 0.75)
pose = np.zeros((1, views, 9), dtype=np.float32)
pose[..., 6] = 1.0
pose[..., 7:] = np.pi / 2.0
return depth, confidence, pose
class Da3OnnxTest(unittest.TestCase):
def test_target_size_uses_patch_multiple(self):
self.assertEqual(
onnx_da3.target_size((800, 600), 504, "upper_bound_resize"),
(504, 378),
)
def test_infer_returns_camera_and_image_arrays(self):
with tempfile.TemporaryDirectory() as directory:
image_path = Path(directory) / "input.png"
Image.new("RGB", (80, 60), (10, 20, 30)).save(image_path)
session = FakeSession()
result = onnx_da3.infer(
session,
[image_path],
process_res=56,
ref_view_strategy="middle",
)
self.assertEqual(result.depth.shape, (1, 42, 56))
self.assertEqual(result.conf.shape, result.depth.shape)
self.assertEqual(result.processed_images.shape, (1, 42, 56, 3))
self.assertEqual(result.extrinsics.shape, (1, 3, 4))
self.assertEqual(result.intrinsics.shape, (1, 3, 3))
self.assertEqual(int(session.feeds["strategy"][0]), 1)
np.testing.assert_allclose(result.extrinsics[0, :3, :3], np.eye(3))
if __name__ == "__main__":
unittest.main()

View File

@@ -1,23 +1,24 @@
import importlib.util
import json
import os
import tempfile
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
PROJECT_ROOT = Path(__file__).resolve().parents[1]
def load_worker():
specification = importlib.util.spec_from_file_location(
"snapform_download_worker",
PROJECT_ROOT / "src" / "snapform" / "worker.py",
)
module = importlib.util.module_from_spec(specification)
specification.loader.exec_module(module)
return module
def load_download_module():
addon_directory = PROJECT_ROOT / "src" / "snapform"
sys.path.insert(0, str(addon_directory))
try:
return __import__(
"server.model_download",
fromlist=["model_download"],
)
finally:
sys.path.remove(str(addon_directory))
class FakeTqdm:
@@ -36,29 +37,63 @@ class FakeTqdm:
class DownloadProgressTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.worker = load_worker()
cls.download = load_download_module()
def create_progress(self, description="Downloading bytes", **options):
reports = []
progress_class = self.download.create_download_progress_class(
lambda value, message: reports.append((value, message)),
"owner/model",
tqdm_type=options.pop("tqdm_type", FakeTqdm),
clock=options.pop("clock", lambda: 1.0),
**options,
)
progress = progress_class(
desc=description,
total=100,
initial=0,
unit="B",
)
return progress, reports
def test_https_connection_bounds_connect_and_idle_read_time(self):
connect_timeouts = []
read_timeouts = []
def connect(connection):
connect_timeouts.append(connection.timeout)
connection.sock = SimpleNamespace(
settimeout=read_timeouts.append,
)
connection = self.download._ShortConnectHTTPSConnection("example.com")
with patch.object(
self.download.http.client.HTTPSConnection,
"connect",
new=connect,
):
connection.connect()
self.assertEqual(connect_timeouts, [10.0])
self.assertEqual(read_timeouts, [60.0])
self.assertIsNone(connection.timeout)
def test_reports_aggregated_download_bytes(self):
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
progress_class = self.worker.create_download_progress_class(
status_path,
"owner/model",
tqdm_type=FakeTqdm,
clock=lambda: 1.0,
)
progress = progress_class(
desc="Downloading bytes",
total=100,
initial=0,
unit="B",
)
progress, reports = self.create_progress()
progress.update(25)
progress.update(25)
status = json.loads(status_path.read_text(encoding="utf-8"))
self.assertAlmostEqual(status["progress"], 0.275)
self.assertEqual(status["message"], "Downloading owner/model")
self.assertAlmostEqual(reports[-1][0], 0.275)
self.assertEqual(reports[-1][1], "Downloading owner/model")
def test_reports_hugging_face_aggregated_bar(self):
progress, reports = self.create_progress(
"Downloading (incomplete total...)"
)
progress.update(25)
self.assertAlmostEqual(reports[-1][0], 0.275)
def test_reports_download_speed(self):
class FakeTqdmWithRate(FakeTqdm):
@@ -66,129 +101,48 @@ class DownloadProgressTest(unittest.TestCase):
def format_dict(self):
return {"rate": 14_000_000}
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
progress_class = self.worker.create_download_progress_class(
status_path,
"owner/model",
tqdm_type=FakeTqdmWithRate,
clock=lambda: 1.0,
label="da3 small",
)
progress = progress_class(
desc="Downloading bytes",
total=100,
initial=0,
unit="B",
)
progress, reports = self.create_progress(
tqdm_type=FakeTqdmWithRate,
label="da3 small",
)
progress.update(50)
progress.update(50)
status = json.loads(status_path.read_text(encoding="utf-8"))
self.assertEqual(
status["message"],
"Downloading da3 small | 13.4 MB/s",
)
self.assertEqual(
reports[-1][1],
"Downloading da3 small 13.4 MB/s",
)
def test_reports_estimated_time_remaining(self):
def test_progress_callback_has_no_eta_or_state_protocol(self):
times = iter((0.0, 10.0))
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
progress_class = self.worker.create_download_progress_class(
status_path,
"owner/model",
tqdm_type=FakeTqdm,
clock=lambda: next(times),
)
progress = progress_class(
desc="Downloading bytes",
total=100,
initial=0,
unit="B",
)
progress, reports = self.create_progress(clock=lambda: next(times))
progress.update(50)
progress.update(50)
status = json.loads(status_path.read_text(encoding="utf-8"))
self.assertEqual(status["eta_seconds"], 10)
def test_staged_download_prefixes_speed_with_downloading(self):
class FakeTqdmWithRate(FakeTqdm):
@property
def format_dict(self):
return {"rate": 14_000_000}
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
progress_class = self.worker.create_download_progress_class(
status_path,
"owner/model",
tqdm_type=FakeTqdmWithRate,
clock=lambda: 1.0,
label="da3 small",
stage=6,
stages=8,
stage_label="da3 small",
)
progress = progress_class(
desc="Downloading bytes",
total=100,
initial=0,
unit="B",
)
progress.update(50)
status = json.loads(status_path.read_text(encoding="utf-8"))
self.assertEqual(
status["message"],
"Downloading packages | 13.4 MB/s",
)
self.assertEqual(status["stage"], 6)
self.assertEqual(status["stages"], 8)
self.assertEqual(status["stage_label"], "da3 small")
self.assertEqual(len(reports[-1]), 2)
def test_ignores_unrelated_progress_bars(self):
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
progress_class = self.worker.create_download_progress_class(
status_path,
"owner/model",
tqdm_type=FakeTqdm,
clock=lambda: 1.0,
)
progress = progress_class(
desc="Reconstructing",
total=100,
initial=0,
unit="B",
)
progress, reports = self.create_progress("Reconstructing")
progress.update(25)
progress.update(25)
self.assertFalse(status_path.exists())
self.assertEqual(reports, [])
def test_download_endpoints_uses_direct_endpoint(self):
with patch.dict(os.environ, {"HF_ENDPOINT": ""}):
endpoints = self.worker.download_endpoints()
endpoints = self.download.download_endpoints()
self.assertEqual(
endpoints,
["https://huggingface.co"],
)
self.assertEqual(endpoints, ["https://huggingface.co"])
def test_download_endpoints_respects_environment_override(self):
with patch.dict(os.environ, {"HF_ENDPOINT": "https://hf.example"}):
endpoints = self.worker.download_endpoints()
endpoints = self.download.download_endpoints()
self.assertEqual(
endpoints,
["https://hf.example"],
)
self.assertEqual(endpoints, ["https://hf.example"])
def test_download_endpoints_strips_trailing_slash(self):
with patch.dict(os.environ, {"HF_ENDPOINT": "https://hf.example/"}):
endpoints = self.worker.download_endpoints()
endpoints = self.download.download_endpoints()
self.assertEqual(endpoints, ["https://hf.example"])

View File

@@ -26,7 +26,11 @@ def load_image_depth():
with patch.dict(sys.modules, {"bpy": fake_bpy}):
specification = importlib.util.spec_from_file_location(
"snapform_image_depth_test",
PROJECT_ROOT / "src" / "snapform" / "image_depth.py",
PROJECT_ROOT
/ "src"
/ "snapform"
/ "blender"
/ "depth_plane.py",
)
module = importlib.util.module_from_spec(specification)
specification.loader.exec_module(module)
@@ -601,6 +605,21 @@ class ImageDepthTest(unittest.TestCase):
"img.001.depth",
)
def test_moge_map_names_use_source_image_name(self):
source_object = SimpleNamespace(
name="obj_img",
data=SimpleNamespace(name="img.png"),
)
self.assertEqual(
self.image_depth.normal_data_name(source_object),
"img.normal",
)
self.assertEqual(
self.image_depth.mask_data_name(source_object),
"img.mask",
)
def test_copy_depth_image_renames_still_source(self):
class FakeImage:
def __init__(self, name):

File diff suppressed because it is too large Load Diff

View File

@@ -21,7 +21,7 @@ class ModelTest(unittest.TestCase):
def setUpClass(cls):
cls.models = load_module(
"snapform_models",
"src/snapform/models.py",
"src/snapform/model_catalog.py",
)
def test_base_model_is_default_friendly_apache_model(self):
@@ -30,12 +30,6 @@ class ModelTest(unittest.TestCase):
self.assertEqual(model.repository, "depth-anything/DA3-BASE")
self.assertEqual(model.license_name, "Apache 2.0")
def test_large_model_exposes_noncommercial_license(self):
model = self.models.get_model("LARGE_1_1")
self.assertEqual(model.license_name, "CC BY-NC 4.0")
self.assertFalse(model.enabled)
def test_giant_models_expose_gaussian_head(self):
for model_key in ("GIANT_1_1", "NESTED_GIANT_LARGE_1_1"):
with self.subTest(model_key=model_key):
@@ -54,19 +48,20 @@ class ModelTest(unittest.TestCase):
def test_ben2_download_only_includes_required_hugging_face_files(self):
model = self.models.get_downloadable_model("BEN2_BASE")
self.assertEqual(model.repository, "PramaLLC/BEN2")
self.assertEqual(model.license_name, "MIT")
self.assertEqual(
model.download_patterns,
("config.json", "model.safetensors"),
)
self.assertEqual(model.repository, "onnx-community/BEN2-ONNX")
self.assertEqual(model.license_name, "Apache 2.0")
self.assertEqual(model.download_patterns, ("onnx/model_fp16.onnx",))
self.assertEqual(model.ready_patterns, ("onnx/model_fp16.onnx",))
self.assertEqual(model.r2_files[0][0], "onnx/model_fp16.onnx")
self.assertNotIn("BEN2_BASE", self.models.MODELS)
def test_da3_model_directories_share_the_da3_prefix(self):
def test_da3_model_directories_use_repository_names(self):
for model_key, model in self.models.MODELS.items():
if not model.enabled:
continue
with self.subTest(model_key=model_key):
self.assertTrue(
model.directory_name.startswith("da3_"),
model.directory_name.startswith("DA3-"),
model.directory_name,
)
self.assertEqual(
@@ -74,20 +69,37 @@ class ModelTest(unittest.TestCase):
model.directory_name,
)
def test_ben2_model_directory_uses_ben2_prefix(self):
def test_ben2_model_directory_uses_repository_name(self):
self.assertEqual(
self.models.model_directory_name("BEN2_BASE"),
"ben2_base",
"BEN2-ONNX",
)
def test_moge2_uses_single_checkpoint_and_mit_license(self):
model = self.models.get_downloadable_model("MOGE2_VITB_NORMAL")
def test_enabled_da3_models_use_onnx_and_r2_checksums(self):
for model in self.models.MODELS.values():
if not model.enabled:
continue
with self.subTest(model=model.key):
self.assertTrue(model.download_patterns[0].endswith(".onnx"))
self.assertEqual(model.ready_patterns, ("model.onnx",))
self.assertEqual(model.r2_files[0][0], "model.onnx")
self.assertEqual(model.repository, "Ruicheng/moge-2-vitb-normal")
self.assertEqual(model.license_name, "MIT")
self.assertEqual(model.download_patterns, ("model.pt",))
self.assertEqual(model.ready_patterns, ("model.pt",))
self.assertFalse(model.supports_gs)
def test_moge2_models_use_official_onnx_models(self):
for key, variant in (("MOGE2_VITS_NORMAL", "vits"),
("MOGE2_VITB_NORMAL", "vitb"),
("MOGE2_VITL_NORMAL", "vitl")):
with self.subTest(key=key):
model = self.models.get_downloadable_model(key)
self.assertEqual(
model.repository,
f"Ruicheng/moge-2-{variant}-normal-onnx",
)
self.assertEqual(model.license_name, "MIT")
self.assertEqual(model.download_patterns, ("model.onnx",))
self.assertEqual(model.ready_patterns, ("model.onnx",))
self.assertEqual(model.r2_directory, model.repository.split("/")[-1])
self.assertEqual(model.r2_files[0][0], "model.onnx")
self.assertFalse(model.supports_gs)
def test_moge2_models_match_uploaded_r2_variants(self):
self.assertEqual(
@@ -96,5 +108,9 @@ class ModelTest(unittest.TestCase):
)
self.assertEqual(
[model.r2_directory for model in self.models.MOGE2_MODELS.values()],
["MoGe2ViTSNormal", "MoGe2ViTBNormal", "MoGe2ViTLNormal"],
[
"moge-2-vits-normal-onnx",
"moge-2-vitb-normal-onnx",
"moge-2-vitl-normal-onnx",
],
)

136
tests/test_moge_onnx.py Normal file
View File

@@ -0,0 +1,136 @@
import importlib.util
import sys
import unittest
from pathlib import Path
from types import ModuleType
from unittest.mock import patch
import numpy as np
PROJECT_ROOT = Path(__file__).resolve().parents[1]
def load_module():
snapform = ModuleType("snapform")
snapform.__path__ = [str(PROJECT_ROOT / "src" / "snapform")]
server = ModuleType("snapform.server")
server.__path__ = [str(PROJECT_ROOT / "src" / "snapform" / "server")]
specification = importlib.util.spec_from_file_location(
"snapform.server.onnx_moge2",
PROJECT_ROOT / "src" / "snapform" / "server" / "onnx_moge2.py",
)
module = importlib.util.module_from_spec(specification)
with patch.dict(sys.modules, {"snapform": snapform, "snapform.server": server}):
specification.loader.exec_module(module)
return module
class FakeInput:
def __init__(self, name):
self.name = name
class FakeSession:
def __init__(self, outputs):
self.outputs = outputs
self.calls = []
def run(self, names, inputs):
self.calls.append((names, inputs))
return [self.outputs[name] for name in names]
def get_outputs(self):
return [FakeInput(name) for name in self.outputs]
class MogeOnnxTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.moge = load_module()
def test_auto_provider_prefers_accelerated_runtime(self):
providers = self.moge.select_providers(
"auto",
["CPUExecutionProvider", "DmlExecutionProvider"],
)
self.assertEqual(
providers,
["DmlExecutionProvider", "CPUExecutionProvider"],
)
def test_explicit_unavailable_provider_falls_back_to_cpu(self):
providers = self.moge.select_providers(
"cuda",
["CPUExecutionProvider"],
)
self.assertEqual(providers, ["CPUExecutionProvider"])
def test_infer_uses_dynamic_token_input_and_numpy_postprocess(self):
height, width = 8, 12
raw = {
"points": np.zeros((1, height, width, 3), dtype=np.float32),
"normal": np.zeros((1, height, width, 3), dtype=np.float32),
"mask": np.ones((1, height, width), dtype=np.float32),
"scale": np.ones((1,), dtype=np.float32),
}
raw["points"][..., 2] = 1.0
raw["normal"][..., 2] = 1.0
session = FakeSession(raw)
expected = {
"points": np.zeros((height, width, 3), dtype=np.float32),
"depth": np.ones((height, width), dtype=np.float32),
"normal": raw["normal"][0],
"mask": np.ones((height, width), dtype=bool),
"intrinsics": np.eye(3, dtype=np.float32),
}
with patch.object(self.moge, "postprocess", return_value=expected):
result = self.moge.infer(
session,
np.zeros((height, width, 3), dtype=np.float32),
resolution_level=9,
)
self.assertIs(result, expected)
names, inputs = session.calls[0]
self.assertEqual(names, ["points", "normal", "mask", "scale"])
self.assertEqual(inputs["image"].shape, (1, 3, height, width))
self.assertEqual(inputs["num_tokens"].dtype, np.int64)
self.assertEqual(inputs["num_tokens"], 3600)
def test_postprocess_recovers_camera_projection_and_metric_scale(self):
height, width = 64, 80
focal = np.float32(0.9)
shift = np.float32(0.3)
metric_scale = np.float32(2.0)
uv = self.moge.normalized_view_plane_uv(width, height, np.float32)
depth = np.linspace(1.0, 2.4, height * width, dtype=np.float32).reshape(
height,
width,
)
raw_points = np.concatenate(
(uv * depth[..., None] / focal, (depth - shift)[..., None]),
axis=-1,
)
raw = {
"points": raw_points[None],
"normal": np.broadcast_to(
np.array((0.0, 0.0, 1.0), dtype=np.float32),
(1, height, width, 3),
),
"mask": np.ones((1, height, width), dtype=np.float32),
"metric_scale": np.array((metric_scale,), dtype=np.float32),
}
result = self.moge.postprocess(raw)
np.testing.assert_allclose(result["depth"], depth * metric_scale, rtol=1e-3)
np.testing.assert_allclose(result["points"][..., 2], result["depth"], rtol=1e-5)
self.assertTrue(result["mask"].all())
if __name__ == "__main__":
unittest.main()

View File

@@ -1,7 +1,4 @@
import ast
import importlib.util
import io
import tarfile
import tempfile
import tomllib
import unittest
@@ -55,32 +52,41 @@ class PackTest(unittest.TestCase):
self.assertIn("__init__.py", names)
self.assertIn("blender_manifest.toml", names)
self.assertIn("image_depth.py", names)
self.assertIn("blender/depth_plane.py", names)
self.assertIn("blender/surface.py", names)
self.assertIn(
"assets/O_SnapForm.blend",
names,
)
self.assertIn("worker.py", names)
self.assertIn("service.py", names)
self.assertNotIn("snapform/__init__.py", names)
self.assertNotIn("importer.py", names)
self.assertIn("vendor/depth-anything-3.tar.gz", names)
self.assertIn("vendor/ben2.tar.gz", names)
self.assertIn("server/media.py", names)
self.assertIn("server/geometry.py", names)
self.assertIn("server/background.py", names)
self.assertIn("server/moge2.py", names)
self.assertIn("server/da3.py", names)
self.assertNotIn("server/processing.py", names)
self.assertNotIn("server/progress.py", names)
self.assertIn("model_catalog.py", names)
self.assertIn("server/model_download.py", names)
self.assertIn("server/onnx_ben2.py", names)
self.assertIn("server/onnx_da3.py", names)
self.assertIn("server/onnx_moge2.py", names)
self.assertNotIn("server/jobs.py", names)
self.assertIn("server/__init__.py", names)
self.assertIn(
f"vendor/{pack.PLATFORMS[platform]}",
"wheels/blendjob-0.1.6-py3-none-any.whl",
names,
)
self.assertFalse(
any(name.startswith("blendjob/") for name in names)
)
self.assertNotIn("install_env.py", names)
self.assertNotIn("snapform/__init__.py", names)
self.assertNotIn("importer.py", names)
self.assertFalse(
any(name.startswith("sources/") for name in names)
)
bundled_uv = {
name
for name in names
if name.startswith("vendor/uv-")
}
self.assertEqual(
bundled_uv,
{f"vendor/{pack.PLATFORMS[platform]}"},
self.assertFalse(
any(name.startswith("vendor/") for name in names)
)
self.assertIn(f'platforms = ["{platform}"]', manifest)
self.assertNotIn(
@@ -108,95 +114,5 @@ class PackTest(unittest.TestCase):
{"windows-x64", "macos-arm64", "linux-x64"},
)
def test_bundled_depth_anything_3_has_no_xformers_dependency(self):
pack = load_pack()
with tarfile.open(
fileobj=io.BytesIO(
pack.bundled_artifact_bytes("depth-anything-3.tar.gz")
),
mode="r:gz",
) as archive:
raw_pyproject = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"pyproject.toml"
).read()
pyproject = raw_pyproject.decode("utf-8")
self.assertNotIn("xformers", pyproject)
self.assertFalse(raw_pyproject.startswith(b"\xef\xbb\xbf"))
def test_bundled_depth_anything_3_exports_are_lazy(self):
pack = load_pack()
with tarfile.open(
fileobj=io.BytesIO(
pack.bundled_artifact_bytes("depth-anything-3.tar.gz")
),
mode="r:gz",
) as archive:
source = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"src/depth_anything_3/utils/export/__init__.py"
).read().decode("utf-8")
tree = ast.parse(source)
module_imports = [
ast.get_source_segment(source, node)
for node in tree.body
if isinstance(node, (ast.Import, ast.ImportFrom))
]
self.assertEqual(
module_imports,
["from depth_anything_3.specs import Prediction"],
)
def test_official_depth_anything_3_baseline_is_pristine(self):
pack = load_pack()
archive_path = pack.VENDOR_DIR / "depth-anything-3.tar.gz"
with tarfile.open(archive_path, "r:gz") as archive:
raw_pyproject = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"pyproject.toml"
).read()
export_source = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"src/depth_anything_3/utils/export/__init__.py"
).read().decode("utf-8")
pyproject = raw_pyproject.decode("utf-8")
self.assertIn("xformers", pyproject)
self.assertFalse(raw_pyproject.startswith(b"\xef\xbb\xbf"))
self.assertIn(
"from depth_anything_3.utils.export.gs import",
export_source,
)
def test_bundled_depth_anything_3_qr_falls_back_on_mps(self):
pack = load_pack()
with tarfile.open(
fileobj=io.BytesIO(
pack.bundled_artifact_bytes("depth-anything-3.tar.gz")
),
mode="r:gz",
) as archive:
source = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"src/depth_anything_3/utils/ray_utils.py"
).read().decode("utf-8")
self.assertIn('use_cpu = device.type == "mps"', source)
self.assertIn("A_tilde = A_tilde.cpu()", source)
self.assertIn("Q_tilde = Q_tilde.to(device=device)", source)
def test_official_depth_anything_3_ray_utils_is_pristine(self):
pack = load_pack()
archive_path = pack.VENDOR_DIR / "depth-anything-3.tar.gz"
with tarfile.open(archive_path, "r:gz") as archive:
source = archive.extractfile(
"Depth-Anything-3-3d835ec1a5802d64a8b8b15f817a1ab54809bfe4/"
"src/depth_anything_3/utils/ray_utils.py"
).read().decode("utf-8")
self.assertIn(
"Q_tilde, R_tilde = torch.linalg.qr(A_tilde)",
source,
)
self.assertNotIn("use_cpu", source)
if __name__ == "__main__":
unittest.main()

View File

@@ -1,67 +0,0 @@
import hashlib
import importlib.util
import io
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
PROJECT_ROOT = Path(__file__).resolve().parents[1]
def load_prepare_vendor():
specification = importlib.util.spec_from_file_location(
"snapform_prepare_vendor",
PROJECT_ROOT / "prepare_vendor.py",
)
module = importlib.util.module_from_spec(specification)
specification.loader.exec_module(module)
return module
class PrepareVendorTest(unittest.TestCase):
def test_downloads_and_verifies_artifact(self):
prepare = load_prepare_vendor()
content = b"locked artifact"
artifact = {
"filename": "artifact.tar.gz",
"url": "https://example.invalid/artifact.tar.gz",
"sha256": hashlib.sha256(content).hexdigest(),
}
with tempfile.TemporaryDirectory() as directory, patch.object(
prepare.urllib.request,
"urlopen",
return_value=io.BytesIO(content),
):
path = prepare.download_artifact(artifact, Path(directory))
self.assertEqual(path.read_bytes(), content)
def test_rejects_hash_mismatch_without_leaving_partial_file(self):
prepare = load_prepare_vendor()
artifact = {
"filename": "artifact.tar.gz",
"url": "https://example.invalid/artifact.tar.gz",
"sha256": hashlib.sha256(b"expected").hexdigest(),
}
with tempfile.TemporaryDirectory() as directory, patch.object(
prepare.urllib.request,
"urlopen",
return_value=io.BytesIO(b"unexpected"),
):
destination = Path(directory)
with self.assertRaisesRegex(RuntimeError, "SHA-256 mismatch"):
prepare.download_artifact(artifact, destination)
self.assertFalse((destination / artifact["filename"]).exists())
self.assertFalse(
(destination / f"{artifact['filename']}.part").exists()
)
def test_rejects_unknown_platform(self):
prepare = load_prepare_vendor()
with self.assertRaisesRegex(ValueError, "Unsupported platform"):
prepare.prepare(["windows-arm64"])
if __name__ == "__main__":
unittest.main()

371
tests/test_server.py Normal file
View File

@@ -0,0 +1,371 @@
import importlib
import sys
import tempfile
import threading
import time
import unittest
from pathlib import Path
from types import ModuleType, SimpleNamespace
from unittest.mock import patch
PROJECT_ROOT = Path(__file__).resolve().parents[1]
def load_server_module():
addon_directory = PROJECT_ROOT / "src" / "snapform"
sys.path.insert(0, str(addon_directory))
try:
return importlib.import_module("server")
finally:
sys.path.remove(str(addon_directory))
class FakeCuda:
class OutOfMemoryError(RuntimeError):
pass
@staticmethod
def is_available():
return False
@staticmethod
def empty_cache():
return None
class ServerTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.server_module = load_server_module()
def setUp(self):
self.jobs_temp = tempfile.TemporaryDirectory()
self.addCleanup(self.jobs_temp.cleanup)
self.storage_root = Path(self.jobs_temp.name)
def test_required_model_download_skips_ready_model(self):
manager = self.server_module.ModelManager(Path("."))
downloads = []
context = SimpleNamespace(progress=lambda *_args: None)
with patch.object(
manager,
"ready",
side_effect=lambda key: key == "BEN2_BASE",
), patch.object(
manager,
"_download",
side_effect=lambda _context, key, _progress: downloads.append(key),
):
result = manager.download_missing(
context,
("MOGE2_VITS_NORMAL", "BEN2_BASE"),
)
self.assertEqual(downloads, ["MOGE2_VITS_NORMAL"])
self.assertEqual(result, {"downloaded": ["MOGE2_VITS_NORMAL"]})
def test_models_are_owned_by_the_server_resource(self):
self.assertFalse((PROJECT_ROOT / "src" / "snapform" / "models.py").exists())
with tempfile.TemporaryDirectory() as directory:
manager = self.server_module.ModelManager(Path(directory))
snapshot = manager.snapshot()
self.assertIn("catalog", snapshot)
self.assertTrue(snapshot["catalog"])
def create_cache(self):
cache = self.server_module.ModelSessionCache.__new__(
self.server_module.ModelSessionCache
)
cache.ben2_model = None
cache.ben2_key = None
cache.da3_model = None
cache.da3_key = None
cache.moge_model = None
cache.moge_key = None
return cache
def create_server(self):
source = self.server_module.server
server = self.server_module.JobServer(
"SnapForm Job Server", storage_root=self.storage_root
)
server.handlers.update(source.handlers)
server.resource_factories.update(source.resource_factories)
server.bind(self.storage_root)
return server
def wait_for_job(self, context, timeout=2.0):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if context.snapshot()["state"] in {"succeeded", "failed", "cancelled"}:
return context.snapshot()
time.sleep(0.01)
self.fail(f"Job did not finish: {context.snapshot()}")
def test_ben2_model_is_reused_for_matching_runtime_key(self):
loads = []
ben2 = ModuleType("server.onnx_ben2")
ben2.create_session = lambda model_dir, device: (
loads.append((model_dir, device)) or object()
)
cache = self.create_cache()
device = SimpleNamespace(type="cuda")
with patch.dict(sys.modules, {"server.onnx_ben2": ben2}):
first, _load_ms = cache.get_ben2("model", device)
second, cached_load_ms = cache.get_ben2("model", device)
self.assertIs(first, second)
self.assertEqual(loads, [("model", "cuda")])
self.assertEqual(cache.snapshot()["ben2"], "model")
self.assertEqual(cached_load_ms, 0.0)
def test_da3_model_is_reused_for_matching_runtime_key(self):
loads = []
da3 = ModuleType("server.onnx_da3")
da3.create_session = lambda model_dir, device: (
loads.append((model_dir, device)) or object()
)
cache = self.create_cache()
device = SimpleNamespace(type="cuda")
with patch.dict(
sys.modules,
{
"server.onnx_da3": da3,
},
):
first = cache.get_da3("model", device)
second = cache.get_da3("model", device)
self.assertIs(first, second)
self.assertEqual(loads, [("model", "cuda")])
self.assertEqual(cache.snapshot()["da3"], "model")
def test_depth_model_cache_keeps_only_da3_or_moge(self):
cache = self.create_cache()
cache.ben2_model = object()
cache.ben2_key = ("ben2", "cuda")
cache.moge_model = object()
cache.moge_key = ("moge", "cuda")
device = SimpleNamespace(type="cuda")
da3 = object()
def load_da3(_model_dir, _device, key):
cache.da3_model = da3
cache.da3_key = key
return da3
with patch.object(cache, "_load_da3", side_effect=load_da3):
self.assertIs(cache.get_da3("da3", device), da3)
self.assertIsNone(cache.moge_model)
self.assertIsNotNone(cache.ben2_model)
moge = object()
fake_moge = ModuleType("server.onnx_moge2")
fake_moge.create_session = lambda _model_dir, _device: moge
with patch.dict(sys.modules, {"server.onnx_moge2": fake_moge}):
self.assertIs(cache.get_moge2("moge", "cuda"), moge)
self.assertIsNone(cache.da3_model)
self.assertIsNotNone(cache.ben2_model)
def test_dedicated_server_runs_only_one_job_at_a_time(self):
server = self.server_module.JobServer("Test", storage_root=self.storage_root)
first_started = threading.Event()
release_first = threading.Event()
order = []
@server.job("example")
def example(_context, parameters):
order.append(parameters["name"])
if parameters["name"] == "first":
first_started.set()
release_first.wait(1.0)
first = server.submit("example", {"name": "first"}, job_id="job")
self.assertTrue(first_started.wait(1.0))
second = server.submit("example", {"name": "second"}, job_id="other")
self.assertIsNotNone(first)
self.assertIsNotNone(second)
self.assertTrue(server.snapshot("instance")["busy"])
self.assertEqual(server.snapshot("instance")["queued_jobs"], 1)
self.assertNotIn("clients", server.snapshot("instance"))
release_first.set()
self.wait_for_job(first)
self.wait_for_job(second)
self.assertEqual(order, ["first", "second"])
server.close()
def test_server_registers_tasks_instead_of_model_names(self):
server = self.create_server()
self.assertEqual(
set(server.handlers),
{
"download-model",
"download-required-models",
"remove-background",
"extract-frames",
"generate-da3-depth",
"generate-moge2-depth",
"generate-da3-surface",
"generate-moge2-surface",
"debug-da3",
"debug-moge2",
"debug-ben2",
},
)
def test_queued_job_can_be_cancelled_before_it_runs(self):
server = self.server_module.JobServer("Test", storage_root=self.storage_root)
first_started = threading.Event()
release_first = threading.Event()
executed = []
@server.job("example")
def example(_context, parameters):
executed.append(parameters["name"])
if parameters["name"] == "first":
first_started.set()
release_first.wait(1.0)
first = server.submit("example", {"name": "first"})
self.assertTrue(first_started.wait(1.0))
second = server.submit("example", {"name": "second"})
self.assertIs(server.cancel(second.job_id), second)
release_first.set()
self.wait_for_job(first)
second_status = self.wait_for_job(second)
self.assertEqual(second_status["state"], "cancelled")
self.assertEqual(executed, ["first"])
server.close()
def test_cancelling_job_keeps_server_and_model_cache_alive(self):
fake_models = SimpleNamespace(
snapshot=lambda: {
"loaded": {"ben2": "BEN2_BASE", "da3": ""}
},
clear=lambda: None,
close=lambda: None,
)
with patch.object(
self.server_module, "ModelManager", return_value=fake_models
):
server = self.create_server()
context = server.submit("remove-background", {}, job_id="job")
self.assertIsNotNone(server.cancel("job"))
self.assertTrue(context.is_cancelled())
status = self.wait_for_job(context)
self.assertEqual(status["state"], "cancelled")
self.assertFalse(server.shutdown_event.is_set())
self.assertEqual(
server.snapshot("instance")["resources"]["model_manager"]["loaded"]["ben2"],
"BEN2_BASE",
)
def test_job_progress_is_kept_in_server_memory(self):
server = self.server_module.JobServer(
"Test", storage_root=self.storage_root
)
@server.job("example")
def example(context, _parameters):
context.progress(0.5, "Half way")
context = server.submit("example", {})
self.wait_for_job(context)
self.assertRegex(context.job_id, r"^[0-9a-f]{32}$")
self.assertEqual(context.directory.name, context.job_id)
self.assertEqual(context.snapshot()["state"], "succeeded")
self.assertEqual(context.snapshot()["progress"], 1.0)
def test_runtime_storage_root_owns_the_jobs_directory(self):
storage_root = self.storage_root
server = self.server_module.JobServer("Test")
server.bind(storage_root)
@server.job("example")
def example(_context, _parameters):
return {"value": 42}
context = server.submit("example", {})
self.wait_for_job(context)
self.assertEqual(server.jobs_directory, storage_root / "jobs")
self.assertEqual(context.storage_root, storage_root.resolve())
self.assertEqual(context.snapshot()["result"], {"value": 42})
def test_job_server_owns_and_clears_shared_resources(self):
events = []
resource = SimpleNamespace(
snapshot=lambda: {"loaded": "model"},
clear=lambda: events.append("clear"),
close=lambda: events.append("close"),
)
server = self.server_module.JobServer(
"Test", storage_root=self.storage_root
)
server.add_resource("model_manager", resource)
@server.job("example")
def example(context, _parameters):
self.assertIs(context.resource("model_manager"), resource)
release.wait(1.0)
release = threading.Event()
context = server.submit("example", {}, job_id="job")
self.assertFalse(server.clear_resource("model_manager"))
release.set()
self.wait_for_job(context)
self.assertTrue(server.clear_resource("model_manager"))
server.close()
self.assertEqual(
server.snapshot("instance")["resources"]["model_manager"]["loaded"],
"model",
)
self.assertEqual(events, ["clear", "close"])
def test_server_dispatches_frame_extraction_without_models(self):
fake_models = SimpleNamespace(
snapshot=lambda: {"loaded": {"ben2": "", "da3": ""}},
clear=lambda: None,
close=lambda: None,
)
with patch.object(
self.server_module, "ModelManager", return_value=fake_models
):
server = self.create_server()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
input_directory = root / "frames"
input_directory.mkdir()
(input_directory / "frame-0001.jpg").touch()
(input_directory / "frame-0002.jpg").touch()
context = server.submit(
"extract-frames",
{
"input": str(input_directory),
"video_frames": 16,
"start_frame": 1,
},
job_id="job",
)
status = self.wait_for_job(context)
self.assertTrue(
(context.directory / "frame.0001.jpg").is_file()
)
self.assertEqual(status["state"], "succeeded")
if __name__ == "__main__":
unittest.main()

View File

@@ -1,255 +0,0 @@
import importlib.util
import sys
import tempfile
import unittest
from pathlib import Path
from types import ModuleType, SimpleNamespace
from unittest.mock import patch
PROJECT_ROOT = Path(__file__).resolve().parents[1]
def load_service():
specification = importlib.util.spec_from_file_location(
"snapform_service",
PROJECT_ROOT / "src" / "snapform" / "service.py",
)
module = importlib.util.module_from_spec(specification)
specification.loader.exec_module(module)
return module
class FakeCuda:
class OutOfMemoryError(RuntimeError):
pass
@staticmethod
def is_available():
return False
@staticmethod
def empty_cache():
return None
class ServiceTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.service = load_service()
def create_cache(self):
cache = self.service.ModelCache.__new__(self.service.ModelCache)
cache.torch = SimpleNamespace(cuda=FakeCuda())
cache.ben2_model = None
cache.ben2_key = None
cache.da3_model = None
cache.da3_key = None
return cache
def test_ben2_model_is_reused_for_matching_runtime_key(self):
loads = []
class FakeModel:
@classmethod
def from_pretrained(cls, model_dir):
loads.append(model_dir)
return cls()
def to(self, device):
self.device = device
return self
def eval(self):
return None
ben2 = ModuleType("ben2")
ben2.BEN_Base = FakeModel
cache = self.create_cache()
device = SimpleNamespace(type="cuda")
with patch.dict(sys.modules, {"ben2": ben2}):
first, _load_ms = cache.get_ben2("model", device)
second, cached_load_ms = cache.get_ben2("model", device)
self.assertIs(first, second)
self.assertEqual(loads, ["model"])
self.assertEqual(cache.status()["ben2"], "model")
self.assertEqual(cached_load_ms, 0.0)
def test_da3_model_is_reused_for_matching_runtime_key(self):
loads = []
class FakeModel:
@classmethod
def from_pretrained(cls, model_dir):
loads.append(model_dir)
return cls()
def to(self, device):
self.device = device
return self
def eval(self):
return None
package = ModuleType("depth_anything_3")
api = ModuleType("depth_anything_3.api")
api.DepthAnything3 = FakeModel
cache = self.create_cache()
device = SimpleNamespace(type="cuda")
with patch.dict(
sys.modules,
{
"depth_anything_3": package,
"depth_anything_3.api": api,
},
):
first = cache.get_da3("model", device)
second = cache.get_da3("model", device)
self.assertIs(first, second)
self.assertEqual(loads, ["model"])
self.assertEqual(cache.status()["da3"], "model")
def test_service_tracks_clients_and_rejects_shared_shutdown(self):
fake_models = SimpleNamespace(
status=lambda: {"ben2": "", "da3": ""},
close=lambda: None,
)
with patch.object(self.service, "ModelCache", return_value=fake_models):
state = self.service.ServiceState("token", "instance")
state.register_client("first", "Blender A")
state.register_client("second", "Blender B")
self.assertEqual(state.snapshot()["clients"], 2)
self.assertEqual(state.request_shutdown("first"), "shared")
self.assertTrue(
state.reserve("job", "first", "reconstruct", "status.json")
)
self.assertFalse(
state.reserve("other", "second", "reconstruct", "other.json")
)
def test_service_accepts_moge_command(self):
self.assertIn("moge", self.service.JOB_COMMANDS)
def test_cancelling_job_keeps_service_and_model_cache_alive(self):
fake_models = SimpleNamespace(
status=lambda: {"ben2": "BEN2_BASE", "da3": ""},
close=lambda: None,
)
with patch.object(self.service, "ModelCache", return_value=fake_models):
state = self.service.ServiceState("token", "instance")
state.register_client("client", "Blender")
with tempfile.TemporaryDirectory() as directory:
status_path = Path(directory) / "status.json"
state.reserve(
"job",
"client",
"remove-background",
str(status_path),
)
active = state.cancel_job("job", "client")
self.assertTrue(active["cancel_requested"])
self.assertTrue(state.is_job_cancelled("job"))
state.run_job(
{
"job_id": "job",
"command": "remove-background",
"parameters": {"status": str(status_path)},
}
)
status = self.service.json.loads(
status_path.read_text(encoding="utf-8")
)
self.assertEqual(status["state"], "cancelled")
self.assertFalse(state.shutdown_event.is_set())
self.assertEqual(state.snapshot()["cached_models"]["ben2"], "BEN2_BASE")
def test_expired_clients_are_removed(self):
fake_models = SimpleNamespace(
status=lambda: {"ben2": "", "da3": ""},
close=lambda: None,
)
with patch.object(self.service, "ModelCache", return_value=fake_models):
state = self.service.ServiceState("token", "instance")
state.register_client("client", "Blender")
state.clients["client"]["last_seen"] -= (
self.service.CLIENT_TIMEOUT_SECONDS + 1.0
)
state.expire_clients()
self.assertEqual(state.snapshot()["clients"], 0)
def test_discovery_is_only_removed_by_matching_service(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "service.json"
args = SimpleNamespace(
discovery=str(path),
port=51234,
token="token",
instance_id="owner",
)
with patch.object(self.service.os, "getpid", return_value=42):
self.service.write_discovery(args)
payload = self.service.json.loads(path.read_text(encoding="utf-8"))
self.assertEqual(payload["port"], 51234)
self.assertEqual(payload["pid"], 42)
self.service.remove_discovery(path, "other")
self.assertTrue(path.is_file())
self.service.remove_discovery(path, "owner")
self.assertFalse(path.exists())
def test_service_dispatches_frame_extraction_without_models(self):
fake_models = SimpleNamespace(
status=lambda: {"ben2": "", "da3": ""},
close=lambda: None,
)
with patch.object(self.service, "ModelCache", return_value=fake_models):
state = self.service.ServiceState("token", "instance")
state.register_client("client", "Blender")
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
input_directory = root / "frames"
input_directory.mkdir()
(input_directory / "frame-0001.jpg").touch()
(input_directory / "frame-0002.jpg").touch()
result_directory = root / "result"
status_path = root / "status.json"
state.reserve(
"job",
"client",
"extract-frames",
str(status_path),
)
state.run_job(
{
"job_id": "job",
"command": "extract-frames",
"parameters": {
"input": str(input_directory),
"output_dir": str(result_directory),
"status": str(status_path),
"video_frames": 16,
"start_frame": 1,
},
}
)
status = self.service.json.loads(
status_path.read_text(encoding="utf-8")
)
self.assertTrue(
(result_directory / "frame.0001.jpg").is_file()
)
self.assertEqual(status["state"], "succeeded")
if __name__ == "__main__":
unittest.main()

File diff suppressed because it is too large Load Diff

View File

@@ -4,6 +4,7 @@ import bpy
GROUP_NAME = "O DA3 Surface"
MOGE_SURFACE_GROUP_NAME = "O MoGe2 Surface"
DEPTH_PLANE_GROUP_NAME = "O DA3 Depth Plane"
MOGE_DEPTH_PLANE_GROUP_NAME = "O MoGe2 Depth Plane"
MATERIAL_DISPLACEMENT_ATTRIBUTE = "o_material_displacement"
@@ -24,12 +25,15 @@ def input_socket(node, name, fallback):
def new_socket(interface, name, in_out, socket_type, parent=None):
return interface.new_socket(
socket = interface.new_socket(
name=name,
in_out=in_out,
socket_type=socket_type,
parent=parent,
)
if in_out == "INPUT" and hasattr(socket, "structure_type"):
socket.structure_type = "SINGLE"
return socket
def new_compare(nodes, data_type, operation, location, value):
@@ -357,8 +361,16 @@ def add_uv_reprojection(group, source, geometry, location):
return store_uv.outputs["Geometry"]
def build_group():
group = bpy.data.node_groups.new(GROUP_NAME, "GeometryNodeTree")
def build_group(
group_name=GROUP_NAME,
include_confidence=True,
include_discontinuity=True,
include_mask=False,
discontinuity_name="Depth Continuity",
discontinuity_attribute="o_depth_discontinuity",
discontinuity_default=50.0,
):
group = bpy.data.node_groups.new(group_name, "GeometryNodeTree")
group.is_modifier = True
interface = group.interface
geometry_input_socket = new_socket(
@@ -375,7 +387,7 @@ def build_group():
"INPUT",
"NodeSocketFloat",
)
resolution.default_value = 0.01
resolution.default_value = 0.1
resolution.min_value = 0.000001
resolution.subtype = "DISTANCE"
remesh_iterations = new_socket(
@@ -393,37 +405,49 @@ def build_group():
"INPUT",
"NodeSocketFloat",
)
smooth_amount.default_value = 1.0
smooth_amount.default_value = 0.5
smooth_amount.min_value = 0.0
smooth_amount.max_value = 1.0
smooth_amount.subtype = "FACTOR"
culling_panel = interface.new_panel(
name="Culling",
default_closed=False,
)
confidence = new_socket(
interface,
"Confidence",
"INPUT",
"NodeSocketFloat",
parent=culling_panel,
)
confidence.default_value = 0.0
confidence.min_value = 0.0
confidence.max_value = 100.0
confidence.subtype = "PERCENTAGE"
edge = new_socket(
interface,
"Stretch",
"INPUT",
"NodeSocketFloat",
parent=culling_panel,
)
edge.default_value = 10.0
edge.min_value = 0.0
edge.max_value = 100.0
edge.subtype = "PERCENTAGE"
culling_panel = None
if include_confidence or include_discontinuity or include_mask:
culling_panel = interface.new_panel(
name="Culling",
default_closed=False,
)
if include_discontinuity:
discontinuity = new_socket(
interface,
discontinuity_name,
"INPUT",
"NodeSocketFloat",
parent=culling_panel,
)
discontinuity.default_value = discontinuity_default
discontinuity.min_value = 0.0
discontinuity.max_value = 100.0
discontinuity.subtype = "PERCENTAGE"
if include_confidence:
confidence = new_socket(
interface,
"Confidence",
"INPUT",
"NodeSocketFloat",
parent=culling_panel,
)
confidence.default_value = 0.0
confidence.min_value = 0.0
confidence.max_value = 100.0
confidence.subtype = "PERCENTAGE"
if include_mask:
mask_socket = new_socket(
interface,
"Mask",
"INPUT",
"NodeSocketBool",
parent=culling_panel,
)
mask_socket.default_value = False
options_panel = interface.new_panel(
name="Options",
@@ -444,8 +468,20 @@ def build_group():
nodes = group.nodes
links = group.links
geometry_input = new_group_input(nodes, (-650.0, 50.0), {"Geometry"})
confidence_input = new_group_input(nodes, (-1800.0, 800.0), {"Confidence"})
stretch_input = new_group_input(nodes, (-1800.0, 50.0), {"Stretch"})
confidence_input = None
if include_confidence:
confidence_input = new_group_input(
nodes,
(-1800.0, 800.0),
{"Confidence"},
)
discontinuity_input = None
if include_discontinuity:
discontinuity_input = new_group_input(
nodes,
(-1800.0, 50.0),
{discontinuity_name},
)
remesh_input = new_group_input(
nodes,
(-100.0, -500.0),
@@ -457,112 +493,107 @@ def build_group():
group_output = nodes.new("NodeGroupOutput")
group_output.location = (5650.0, 100.0)
confidence_attribute = nodes.new("GeometryNodeInputNamedAttribute")
confidence_attribute.data_type = "FLOAT"
confidence_attribute.inputs["Name"].default_value = "o_face_confidence"
confidence_attribute.location = (-1800.0, 600.0)
confidence_scale = nodes.new("ShaderNodeMath")
confidence_scale.operation = "MULTIPLY"
confidence_scale.inputs[1].default_value = 0.01
confidence_scale.location = (-1600.0, 800.0)
confidence_compare = new_compare(
nodes,
"FLOAT",
"LESS_THAN",
(-1550.0, 600.0),
0.0,
)
links.new(
confidence_attribute.outputs["Attribute"],
input_socket(confidence_compare, "A", 0),
)
links.new(
confidence_input.outputs["Confidence"],
confidence_scale.inputs[0],
)
links.new(
confidence_scale.outputs[0],
input_socket(confidence_compare, "B", 1),
)
geometry = geometry_input.outputs["Geometry"]
if include_confidence:
confidence_attribute = nodes.new("GeometryNodeInputNamedAttribute")
confidence_attribute.data_type = "FLOAT"
confidence_attribute.inputs["Name"].default_value = "o_face_confidence"
confidence_attribute.location = (-1800.0, 600.0)
confidence_scale = nodes.new("ShaderNodeMath")
confidence_scale.operation = "MULTIPLY"
confidence_scale.inputs[1].default_value = 0.01
confidence_scale.location = (-1600.0, 800.0)
confidence_compare = new_compare(
nodes,
"FLOAT",
"LESS_THAN",
(-1550.0, 600.0),
0.0,
)
links.new(
confidence_attribute.outputs["Attribute"],
input_socket(confidence_compare, "A", 0),
)
links.new(
confidence_input.outputs["Confidence"],
confidence_scale.inputs[0],
)
links.new(
confidence_scale.outputs[0],
input_socket(confidence_compare, "B", 1),
)
delete_confidence = nodes.new("GeometryNodeDeleteGeometry")
delete_confidence.domain = "FACE"
delete_confidence.location = (-1050.0, 600.0)
links.new(geometry, delete_confidence.inputs["Geometry"])
links.new(
confidence_compare.outputs["Result"],
delete_confidence.inputs["Selection"],
)
geometry = delete_confidence.outputs["Geometry"]
edge_attribute = nodes.new("GeometryNodeInputNamedAttribute")
edge_attribute.data_type = "FLOAT"
edge_attribute.inputs["Name"].default_value = "o_edge_stretch"
edge_attribute.location = (-1800.0, 350.0)
edge_scale = nodes.new("ShaderNodeMath")
edge_scale.operation = "MULTIPLY"
edge_scale.inputs[1].default_value = 0.01
edge_scale.location = (-1600.0, -50.0)
edge_maximum = nodes.new("ShaderNodeMath")
edge_maximum.operation = "MAXIMUM"
edge_maximum.inputs[1].default_value = 0.0001
edge_maximum.location = (-1600.0, 100.0)
edge_reciprocal = nodes.new("ShaderNodeMath")
edge_reciprocal.operation = "DIVIDE"
edge_reciprocal.inputs[0].default_value = 1.0
edge_reciprocal.location = (-1400.0, 100.0)
edge_compare = new_compare(
nodes,
"FLOAT",
"GREATER_EQUAL",
(-1200.0, 350.0),
1.0,
)
links.new(stretch_input.outputs["Stretch"], edge_scale.inputs[0])
links.new(edge_scale.outputs[0], edge_maximum.inputs[0])
links.new(edge_maximum.outputs[0], edge_reciprocal.inputs[1])
links.new(edge_attribute.outputs["Attribute"], input_socket(edge_compare, "A", 0))
links.new(edge_reciprocal.outputs[0], input_socket(edge_compare, "B", 1))
first_or = nodes.new("FunctionNodeBooleanMath")
first_or.operation = "OR"
first_or.location = (-950.0, 450.0)
links.new(confidence_compare.outputs["Result"], first_or.inputs[0])
links.new(edge_compare.outputs["Result"], first_or.inputs[1])
full_stretch = new_compare(
nodes,
"FLOAT",
"GREATER_EQUAL",
(-950.0, -20.0),
0.999999,
)
links.new(
edge_scale.outputs[0],
input_socket(full_stretch, "A", 0),
)
culling_selection = nodes.new("FunctionNodeBooleanMath")
culling_selection.operation = "OR"
culling_selection.location = (-550.0, 300.0)
links.new(first_or.outputs[0], culling_selection.inputs[0])
links.new(full_stretch.outputs["Result"], culling_selection.inputs[1])
delete_geometry = nodes.new("GeometryNodeDeleteGeometry")
delete_geometry.domain = "FACE"
delete_geometry.location = (-350.0, 100.0)
links.new(geometry_input.outputs["Geometry"], delete_geometry.inputs["Geometry"])
links.new(culling_selection.outputs[0], delete_geometry.inputs["Selection"])
vertex_neighbors = nodes.new("GeometryNodeInputMeshVertexNeighbors")
vertex_neighbors.location = (-550.0, -320.0)
orphan_vertex = new_compare(
nodes,
"INT",
"EQUAL",
(-350.0, -320.0),
0,
)
links.new(
vertex_neighbors.outputs["Face Count"],
input_socket(orphan_vertex, "A", 0),
)
delete_orphans = nodes.new("GeometryNodeDeleteGeometry")
delete_orphans.domain = "POINT"
delete_orphans.location = (-300.0, 100.0)
links.new(delete_geometry.outputs["Geometry"], delete_orphans.inputs["Geometry"])
links.new(orphan_vertex.outputs["Result"], delete_orphans.inputs["Selection"])
culled_geometry = geometry
if include_discontinuity:
discontinuity_data = nodes.new("GeometryNodeInputNamedAttribute")
discontinuity_data.data_type = "FLOAT"
discontinuity_data.inputs["Name"].default_value = discontinuity_attribute
discontinuity_data.location = (-1800.0, 350.0)
culling_scale = nodes.new("ShaderNodeMath")
culling_scale.operation = "MULTIPLY"
culling_scale.inputs[1].default_value = 0.01
culling_scale.location = (-1650.0, -50.0)
inverse_culling = nodes.new("ShaderNodeMath")
inverse_culling.operation = "SUBTRACT"
inverse_culling.inputs[0].default_value = 0.5
inverse_culling.location = (-1450.0, -50.0)
exponent_scale = nodes.new("ShaderNodeMath")
exponent_scale.operation = "MULTIPLY"
exponent_scale.inputs[1].default_value = 10.0
exponent_scale.location = (-1250.0, -50.0)
threshold_power = nodes.new("ShaderNodeMath")
threshold_power.operation = "POWER"
threshold_power.inputs[0].default_value = 2.0
threshold_power.location = (-1050.0, -50.0)
threshold_scale = nodes.new("ShaderNodeMath")
threshold_scale.operation = "MULTIPLY"
threshold_scale.inputs[1].default_value = 0.04
threshold_scale.location = (-850.0, -50.0)
discontinuity_compare = new_compare(
nodes,
"FLOAT",
"GREATER_THAN",
(-650.0, 350.0),
0.0,
)
links.new(
discontinuity_input.outputs[discontinuity_name],
culling_scale.inputs[0],
)
links.new(culling_scale.outputs[0], inverse_culling.inputs[1])
links.new(inverse_culling.outputs[0], exponent_scale.inputs[0])
links.new(exponent_scale.outputs[0], threshold_power.inputs[1])
links.new(threshold_power.outputs[0], threshold_scale.inputs[0])
links.new(
discontinuity_data.outputs["Attribute"],
input_socket(discontinuity_compare, "A", 0),
)
links.new(
threshold_scale.outputs[0],
input_socket(discontinuity_compare, "B", 1),
)
delete_geometry = nodes.new("GeometryNodeDeleteGeometry")
delete_geometry.domain = "FACE"
delete_geometry.location = (-350.0, 100.0)
links.new(geometry, delete_geometry.inputs["Geometry"])
links.new(
discontinuity_compare.outputs["Result"],
delete_geometry.inputs["Selection"],
)
culled_geometry = delete_geometry.outputs["Geometry"]
remeshed = add_remesh_zone(
group,
delete_orphans.outputs["Geometry"],
culled_geometry,
remesh_input.outputs["Resolution"],
remesh_input.outputs["Relax Iterations"],
topology_input.outputs["Fix Topology"],
@@ -622,6 +653,53 @@ def build_group():
return group
def configure_moge_surface_group(group):
nodes = group.nodes
links = group.links
controls = new_group_input(
nodes,
(-1800.0, 1050.0),
{"Mask"},
)
mask_attribute = nodes.new("GeometryNodeInputNamedAttribute")
mask_attribute.data_type = "BOOLEAN"
mask_attribute.inputs["Name"].default_value = "o_mask"
mask_attribute.location = (-1550.0, 1050.0)
invert_mask = nodes.new("FunctionNodeBooleanMath")
invert_mask.operation = "NOT"
invert_mask.location = (-1350.0, 1050.0)
use_mask = nodes.new("FunctionNodeBooleanMath")
use_mask.operation = "AND"
use_mask.location = (-1150.0, 1050.0)
links.new(mask_attribute.outputs["Attribute"], invert_mask.inputs[0])
links.new(controls.outputs["Mask"], use_mask.inputs[0])
links.new(invert_mask.outputs[0], use_mask.inputs[1])
first_processing = min(
(
node
for node in nodes
if node.bl_idname == "GeometryNodeDeleteGeometry"
),
key=lambda node: node.location.x,
)
geometry_input = first_processing.inputs["Geometry"]
previous_link = geometry_input.links[0]
previous_geometry = previous_link.from_socket
links.remove(previous_link)
delete_mask = nodes.new("GeometryNodeDeleteGeometry")
delete_mask.domain = "FACE"
delete_mask.location = (-750.0, 650.0)
links.new(previous_geometry, delete_mask.inputs["Geometry"])
links.new(use_mask.outputs[0], delete_mask.inputs["Selection"])
links.new(delete_mask.outputs["Geometry"], geometry_input)
group.description = (
"Cull MoGe-2 mask and depth-edge faces, repair topology, preserve boundaries, "
"and reproject UVs."
)
group.use_fake_user = True
return group
def build_sequence_frame(nodes, links, scene_frame, frames_input, y):
relative_frame = nodes.new("ShaderNodeMath")
relative_frame.operation = "SUBTRACT"
@@ -1246,6 +1324,16 @@ def main():
bpy.ops.wm.read_factory_settings(use_empty=True)
bpy.context.preferences.filepaths.save_version = 0
build_group()
moge_surface_group = build_group(
MOGE_SURFACE_GROUP_NAME,
include_confidence=False,
include_discontinuity=True,
include_mask=True,
discontinuity_name="Depth Continuity",
discontinuity_attribute="o_depth_discontinuity",
discontinuity_default=50.0,
)
configure_moge_surface_group(moge_surface_group)
build_depth_plane_group()
build_moge_depth_plane_group()
OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)

View File

@@ -6,20 +6,30 @@ import bpy
sys.path.insert(0, str(Path(__file__).parents[1] / "src"))
from snapform.image_depth import (
from snapform.blender.depth_plane import (
DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME,
SURFACE_MATERIAL_NODE_GROUP_NAME,
add_depth_plane_processing,
create_material,
get_modifier_input,
load_depth_plane_node_group,
modifier_input_driver_path,
modifier_input_identifier,
set_modifier_input,
)
from snapform.image_mesh import load_geometry_node_group
from snapform.blender.surface import load_geometry_node_group
def main():
bpy.ops.wm.read_factory_settings(use_empty=True)
group = load_depth_plane_node_group()
geometry_textures = [
node
for node in group.nodes
if node.bl_idname == "GeometryNodeImageTexture"
]
assert geometry_textures
assert all(node.extension == "EXTEND" for node in geometry_textures)
background_depth = next(
item
for item in group.interface.items_tree
@@ -212,6 +222,11 @@ def main():
node.bl_idname in forbidden_outer_nodes
for node in material.node_tree.nodes
)
assert all(
node.extension == "EXTEND"
for node in material.node_tree.nodes
if node.bl_idname == "ShaderNodeTexImage"
)
surface_material = create_material("Surface", foreground, 1.0)
surface_group = bpy.data.node_groups[SURFACE_MATERIAL_NODE_GROUP_NAME]
surface_group_nodes = [
@@ -251,6 +266,11 @@ def main():
node.bl_idname in forbidden_outer_nodes
for node in surface_material.node_tree.nodes
)
assert all(
node.extension == "EXTEND"
for node in surface_material.node_tree.nodes
if node.bl_idname == "ShaderNodeTexImage"
)
modifier = add_depth_plane_processing(
obj,
depth,
@@ -258,24 +278,28 @@ def main():
material,
animate_frame=True,
)
modifier[modifier_input_identifier(group, "Depth Scale")] = -2.0
set_modifier_input(
modifier,
modifier_input_identifier(group, "Depth Scale"),
-2.0,
)
for name, expected in (
("Frame Start", 1),
("Frames", 1),
("Cyclic", False),
):
assert modifier[modifier_input_identifier(group, name)] == expected
identifier = modifier_input_identifier(group, name)
assert get_modifier_input(modifier, identifier) == expected
assert not any(
node.bl_idname == "GeometryNodeInputSceneTime"
for node in group.nodes
)
frame_identifier = modifier_input_identifier(group, "Frame")
frame_path = modifier_input_driver_path(modifier, frame_identifier)
frame_driver = next(
fcurve
for fcurve in obj.animation_data.drivers
if fcurve.data_path == (
f'modifiers["{modifier.name}"]["{frame_identifier}"]'
)
if fcurve.data_path == frame_path
)
assert frame_driver.driver.type == "SCRIPTED"
assert frame_driver.driver.expression == "frame"

View File

@@ -7,7 +7,11 @@ import numpy as np
sys.path.insert(0, str(Path(__file__).parents[1] / "src"))
from snapform.image_mesh import create_matching_camera, load_geometry_node_group
from snapform.blender.depth_plane import (
get_modifier_input,
set_modifier_input,
)
from snapform.blender.surface import create_matching_camera, load_geometry_node_group
def set_input(modifier, group, name, value):
@@ -18,18 +22,12 @@ def set_input(modifier, group, name, value):
and item.in_out == "INPUT"
and item.name == name
)
modifier[item.identifier] = value
set_modifier_input(modifier, item.identifier, value)
def add_face_attributes(mesh, count, background_values=None):
for name in ("o_face_confidence", "o_edge_stretch"):
attribute = mesh.attributes.new(name, "FLOAT", "FACE")
attribute.data.foreach_set("value", [1.0] * count)
background = mesh.attributes.new("o_background_face", "BOOLEAN", "FACE")
background.data.foreach_set(
"value",
background_values or [False] * count,
)
def add_face_attributes(mesh, count):
confidence = mesh.attributes.new("o_face_confidence", "FLOAT", "FACE")
confidence.data.foreach_set("value", [1.0] * count)
def main():
@@ -45,6 +43,13 @@ def main():
and item.name == "Geometry"
)
assert geometry_input.structure_type == "SINGLE"
input_sockets = [
item for item in group.interface.items_tree
if item.item_type == "SOCKET" and item.in_out == "INPUT"
]
assert all(item.structure_type == "SINGLE" for item in input_sockets)
resolution = next(item for item in input_sockets if item.name == "Resolution")
assert abs(resolution.default_value - 0.1) < 1e-6
assert group.asset_data is None
assert not any(node.bl_idname == "GeometryNodeGroup" for node in group.nodes)
assert not any(node.bl_idname == "GeometryNodeTriangulate" for node in group.nodes)
@@ -56,7 +61,7 @@ def main():
visible_outputs = [
output for output in node.outputs if output.name and not output.hide
]
assert 1 <= len(visible_outputs) <= 3
assert 1 <= len(visible_outputs) <= 4
uv_stores = [
node
for node in group.nodes
@@ -73,13 +78,24 @@ def main():
assert not panels["Culling"].default_closed
assert panels["Options"].default_closed
assert [item.name for item in panels["Culling"].interface_items] == [
"Depth Continuity",
"Confidence",
"Stretch",
]
assert [item.name for item in panels["Options"].interface_items] == [
"Fix Topology",
]
for name, default_value in (("Confidence", 0.0), ("Stretch", 10.0)):
smooth = next(
item
for item in group.interface.items_tree
if item.item_type == "SOCKET"
and item.in_out == "INPUT"
and item.name == "Smooth"
)
assert smooth.default_value == 0.5
for name, default_value in (
("Confidence", 0.0),
("Depth Continuity", 50.0),
):
item = next(
item
for item in panels["Culling"].interface_items
@@ -98,9 +114,21 @@ def main():
[],
[(0, 2, 1), (1, 2, 3)],
)
discontinuity = mesh.attributes.new(
"o_depth_discontinuity",
"FLOAT",
"FACE",
)
discontinuity.data.foreach_set("value", [0.03] * len(mesh.polygons))
uv = mesh.uv_layers.new(name="UVMap")
uv.data.foreach_set("uv", [0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1])
add_face_attributes(mesh, 2, [False, True])
add_face_attributes(mesh, 2)
assert not {
"o_background",
"o_background_face",
"o_confidence",
"o_confidence_percentile",
}.intersection(mesh.attributes.keys())
obj = bpy.data.objects.new("Validation Object", mesh)
bpy.context.scene.collection.objects.link(obj)
@@ -121,13 +149,13 @@ def main():
finally:
evaluated.to_mesh_clear()
set_input(modifier, group, "Stretch", 100.0)
stretch_item = next(
set_input(modifier, group, "Depth Continuity", 100.0)
discontinuity_item = next(
item
for item in group.interface.items_tree
if item.item_type == "SOCKET" and item.name == "Stretch"
if item.item_type == "SOCKET" and item.name == "Depth Continuity"
)
assert modifier[stretch_item.identifier] == 100.0
assert get_modifier_input(modifier, discontinuity_item.identifier) == 100.0
obj.update_tag(refresh={"DATA"})
bpy.context.view_layer.update()
culled = obj.evaluated_get(bpy.context.evaluated_depsgraph_get())

View File

@@ -9,7 +9,7 @@ import numpy as np
PROJECT_ROOT = Path(__file__).parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "src"))
from snapform.image_depth import (
from snapform.blender.depth_plane import (
MOGE_DEPTH_PLANE_NODE_GROUP_NAME,
MOGE_MATERIAL_NODE_GROUP_NAME,
build_moge2_depth_plane,
@@ -72,6 +72,22 @@ def main():
material_group = bpy.data.node_groups[MOGE_MATERIAL_NODE_GROUP_NAME]
assert not any(node.bl_idname == "ShaderNodeNormalMap" for node in material_group.nodes)
assert any(node.bl_idname == "ShaderNodeBump" for node in material_group.nodes)
material = plane.data.materials[0]
image_textures = [
node
for node in material.node_tree.nodes
if node.bl_idname == "ShaderNodeTexImage"
]
assert image_textures
assert all(node.extension == "EXTEND" for node in image_textures)
assert any(node.image.name == "MoGe Test.depth" for node in image_textures)
geometry_textures = [
node
for node in geometry_group.nodes
if node.bl_idname == "GeometryNodeImageTexture"
]
assert geometry_textures
assert all(node.extension == "EXTEND" for node in geometry_textures)
print("MoGe-2 depth plane validation passed")

View File

@@ -0,0 +1,226 @@
import sys
import tempfile
from pathlib import Path
import bpy
import numpy as np
PROJECT_ROOT = Path(__file__).parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "src"))
from snapform.blender.depth_plane import MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME
from snapform.blender.surface import (
MOGE_NODE_GROUP_NAME,
build_image_mesh,
prepare_moge_surface_archive,
)
def main():
bpy.ops.wm.read_factory_settings(use_empty=True)
arguments = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else []
if arguments:
validate_real_result(Path(arguments[0]), Path(arguments[1]))
return
height, width = 48, 64
u, v = np.meshgrid(
np.linspace(-0.5, 0.5, width, dtype=np.float32),
np.linspace(-0.375, 0.375, height, dtype=np.float32),
)
depth = np.full((1, height, width), 2.0, dtype=np.float32)
points = np.stack((u * 2.0, v * 2.0, depth[0]), axis=-1)[None]
mask = np.ones((1, height, width), dtype=bool)
mask[:, :2] = False
intrinsics = np.array(
[[[1.0, 0.0, 0.5], [0.0, 1.0, 0.5], [0.0, 0.0, 1.0]]],
dtype=np.float32,
)
image = bpy.data.images.new("MoGe Surface Test", width=width, height=height)
source = bpy.data.objects.new("MoGe Surface Test", None)
source.empty_display_type = "IMAGE"
source.data = image
bpy.context.collection.objects.link(source)
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary)
results_path = directory / "results.npz"
mesh_path = directory / "mesh.npz"
np.savez(
results_path,
depth=depth,
mask=mask,
points=points,
intrinsics=intrinsics,
normal=np.broadcast_to(
np.array([0.0, 0.0, -1.0], dtype=np.float32),
(1, height, width, 3),
),
)
prepare_moge_surface_archive(results_path, mesh_path)
surface = build_image_mesh(
bpy.context,
source,
mesh_path,
False,
True,
False,
True,
)
assert len(surface.data.vertices) > 0
assert surface.modifiers[0].node_group.name == MOGE_NODE_GROUP_NAME
inputs = {
item.name
for item in surface.modifiers[0].node_group.interface.items_tree
if getattr(item, "in_out", None) == "INPUT"
}
assert "Stretch" not in inputs
assert "Depth Continuity" in inputs
assert "Mask" in inputs
assert "Confidence" not in inputs
assert "o_mask" in surface.data.attributes
assert surface.data.attributes["o_mask"].domain == "FACE"
assert not all(item.value for item in surface.data.attributes["o_mask"].data)
attribute_names = set(surface.data.attributes.keys())
assert attribute_names >= {"o_depth", "o_mask"}
assert "o_depth_discontinuity" in attribute_names
assert surface.data.attributes["o_depth_discontinuity"].data_type == "FLOAT"
assert "o_edge_stretch" not in attribute_names
assert not {
"o_confidence",
"o_confidence_percentile",
"o_face_confidence",
"o_background",
"o_background_face",
}.intersection(attribute_names)
mask_socket = next(
item
for item in surface.modifiers[0].node_group.interface.items_tree
if getattr(item, "in_out", None) == "INPUT" and item.name == "Mask"
)
assert mask_socket.default_value is False
assert mask_socket.structure_type == "SINGLE"
assert mask_socket.parent.name == "Culling"
depth_edge_socket = next(
item
for item in surface.modifiers[0].node_group.interface.items_tree
if getattr(item, "in_out", None) == "INPUT"
and item.name == "Depth Continuity"
)
assert depth_edge_socket.default_value == 50.0
assert depth_edge_socket.subtype == "PERCENTAGE"
assert depth_edge_socket.structure_type == "SINGLE"
culling_panel = next(
item for item in surface.modifiers[0].node_group.interface.items_tree
if item.item_type == "PANEL" and item.name == "Culling"
)
assert [item.name for item in culling_panel.interface_items] == [
"Depth Continuity",
"Mask",
]
input_sockets = [
item for item in surface.modifiers[0].node_group.interface.items_tree
if item.item_type == "SOCKET" and item.in_out == "INPUT"
]
assert all(item.structure_type == "SINGLE" for item in input_sockets)
resolution_socket = next(
item for item in input_sockets if item.name == "Resolution"
)
assert abs(resolution_socket.default_value - 0.1) < 1e-6
smooth_socket = next(
item
for item in surface.modifiers[0].node_group.interface.items_tree
if getattr(item, "in_out", None) == "INPUT" and item.name == "Smooth"
)
assert smooth_socket.default_value == 0.5
assert not any(
node.bl_idname == "GeometryNodeInputNamedAttribute"
and node.inputs["Name"].default_value == "o_face_confidence"
for node in surface.modifiers[0].node_group.nodes
)
assert MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME in bpy.data.node_groups
material_group = bpy.data.node_groups[MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME]
assert any(node.bl_idname == "ShaderNodeBump" for node in material_group.nodes)
normal_map = next(
node for node in material_group.nodes
if node.bl_idname == "ShaderNodeNormalMap"
)
assert normal_map.space == "OBJECT"
normal_strength = next(
item for item in material_group.interface.items_tree
if getattr(item, "in_out", None) == "INPUT"
and item.name == "Normal Strength"
)
assert normal_strength.default_value == 1.0
assert normal_strength.min_value == 0.0
assert normal_map.inputs["Strength"].is_linked
assert not surface.data.has_custom_normals
normal_texture = next(
node for node in surface.data.materials[0].node_tree.nodes
if node.bl_idname == "ShaderNodeTexImage"
and node.image.colorspace_settings.name == "Non-Color"
)
assert normal_texture.image.name == "MoGe Surface Test.normal"
assert any(child.type == "CAMERA" for child in surface.children)
print("MoGe-2 surface validation passed")
def validate_real_result(results_path, image_path):
image = bpy.data.images.load(str(image_path), check_existing=False)
source = bpy.data.objects.new("MoGe Real Surface", None)
source.empty_display_type = "IMAGE"
source.data = image
bpy.context.collection.objects.link(source)
mesh_path = results_path.with_name("mesh.npz")
if not mesh_path.is_file():
prepare_moge_surface_archive(results_path, mesh_path)
surface = build_image_mesh(
bpy.context,
source,
mesh_path,
False,
True,
False,
True,
)
assert len(surface.data.vertices) > 100
assert len(surface.data.polygons) > 100
assert surface.modifiers[0].node_group.name == MOGE_NODE_GROUP_NAME
assert MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME in bpy.data.node_groups
assert not surface.data.has_custom_normals
material_group = bpy.data.node_groups[MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME]
assert any(node.bl_idname == "ShaderNodeBump" for node in material_group.nodes)
normal_map = next(
node for node in material_group.nodes
if node.bl_idname == "ShaderNodeNormalMap"
)
assert normal_map.space == "OBJECT"
assert normal_map.inputs["Strength"].is_linked
attribute_names = set(surface.data.attributes.keys())
assert {"o_depth", "o_mask"}.issubset(attribute_names)
assert "o_depth_discontinuity" in attribute_names
assert "o_edge_stretch" not in attribute_names
assert not {
"o_confidence",
"o_confidence_percentile",
"o_face_confidence",
"o_background",
"o_background_face",
}.intersection(attribute_names)
camera = next(child for child in surface.children if child.type == "CAMERA")
dimensions = surface.dimensions
assert max(dimensions) < 10.0
assert min(dimensions.x, dimensions.y) > 0.1
assert 1.0 <= camera.data.lens <= 500.0
assert abs(camera.data.shift_x) <= 1.0
assert abs(camera.data.shift_y) <= 1.0
print(
f"Real MoGe-2 surface validation passed: "
f"{len(surface.data.vertices)} vertices, {len(surface.data.polygons)} faces, "
f"dimensions={tuple(round(value, 4) for value in dimensions)}, "
f"lens={camera.data.lens:.3f}, shift=({camera.data.shift_x:.4f}, "
f"{camera.data.shift_y:.4f})"
)
if __name__ == "__main__":
main()

108
uv.lock generated
View File

@@ -6,6 +6,21 @@ resolution-markers = [
"python_full_version < '3.12'",
]
[[package]]
name = "blendjob"
version = "0.1.6"
source = { path = "src/snapform/wheels/blendjob-0.1.6-py3-none-any.whl" }
wheels = [
{ filename = "blendjob-0.1.6-py3-none-any.whl", hash = "sha256:ed7c0ee4a0c952c25a7b6606bcd860b5c782ecb04ae6f9e7b1c475a03ab9c6b1" },
]
[package.metadata]
requires-dist = [
{ name = "fastapi", marker = "extra == 'server'", specifier = "==0.139.2" },
{ name = "uvicorn", marker = "extra == 'server'", specifier = "==0.51.0" },
]
provides-extras = ["server"]
[[package]]
name = "colorama"
version = "0.4.6"
@@ -169,6 +184,91 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" },
]
[[package]]
name = "pillow"
version = "12.3.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1c/3d/bb7fca845737cf9d7dbde16ed1843984665ff2e0a518f5db43e77ec540b9/pillow-12.3.0.tar.gz", hash = "sha256:3b8182a766685eaa002637e28b4ec8d6b18819a0c71f579bf0dbaa5830297cce", size = 47025035, upload-time = "2026-07-01T11:56:38.965Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/fb/c8/0a78b0e02d7ac54bc03e5321c9220da52f0c2ea83b21f7c40e7f3169c502/pillow-12.3.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:00808c5e14ef63ac5161091d242999076604ff74b883423a11e5d7bbb38bf756", size = 5392415, upload-time = "2026-07-01T11:53:47.162Z" },
{ url = "https://files.pythonhosted.org/packages/b2/5b/a02d30018abd97ced9f5a6c63d28597694a00d066516b9c1c6de45859fc9/pillow-12.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:37d6d0a00072fd2948eb22bce7e1475f34569d90c87c59f7a2ec59541b77f7a6", size = 4785266, upload-time = "2026-07-01T11:53:49.079Z" },
{ url = "https://files.pythonhosted.org/packages/c8/98/766667a4be768150a202836acd9fad19c06824ca86c4286d3cf6b274964e/pillow-12.3.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bcb46e2f9feff8d06323983bd83ed00c201fdcab3d74973e7072a889b3979fcd", size = 6263814, upload-time = "2026-07-01T11:53:51.32Z" },
{ url = "https://files.pythonhosted.org/packages/3b/2d/ede717bc1144f63886c21fd349bb95860b0d1a21149ff16f2bb362b612b6/pillow-12.3.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23d27a3e0307ec2244cc51e7287b919aa68d097504ebe19df4e76a98a3eea5bd", size = 6934408, upload-time = "2026-07-01T11:53:53.487Z" },
{ url = "https://files.pythonhosted.org/packages/a3/48/9c58b685e69d49c31af6c8eb9012055fab7e665785165c84796e2c73ce72/pillow-12.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:4f883547d4b7f0495ebe7056b0cc2aea76094e7a4abc8e933540f3271df27d9c", size = 6337160, upload-time = "2026-07-01T11:53:55.457Z" },
{ url = "https://files.pythonhosted.org/packages/ff/fa/dc2a5c0ba6df93f67c31d34b808b7ce440b40cdbf96f0b81cde1d1e6fa93/pillow-12.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:236ff70b9312fb68943c703aa842ca6a758abfa45ac187a5e7c1452e96ef72b5", size = 7045172, upload-time = "2026-07-01T11:53:57.736Z" },
{ url = "https://files.pythonhosted.org/packages/86/a5/444817a4d4c4c2417df00513086ca196f388d8f9ef40c2e4ccd1ad1af54b/pillow-12.3.0-cp311-cp311-win32.whl", hash = "sha256:10e41f0fbf1eec8cfd234b8fe17a4caac7c9d0db4c204d3c173a8f9f6ef3232b", size = 6472232, upload-time = "2026-07-01T11:53:59.767Z" },
{ url = "https://files.pythonhosted.org/packages/63/c6/4bad1b18d132a50b27e1365e1ab163616f7a5bb56d330f66f9d1d9d4f9d4/pillow-12.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:8e95e1385e4998ae9694eeaa4730ba5457ff61185b3a55e2e7bea0880aef452a", size = 7233653, upload-time = "2026-07-01T11:54:02.066Z" },
{ url = "https://files.pythonhosted.org/packages/fd/16/00f91ab7760dc842f5aad55217e80fc4a7067a0604535249bc8a2d6d9870/pillow-12.3.0-cp311-cp311-win_arm64.whl", hash = "sha256:ebaea975e03d3141d9d3a507df75c9b3ec90fa9d2ffd07567b3a978d9d790b26", size = 2568195, upload-time = "2026-07-01T11:54:04.622Z" },
{ url = "https://files.pythonhosted.org/packages/37/bf/fb3ebff8ddcb76aac5a01389251bbbb9519922a9b520d8247c1ca864a25d/pillow-12.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ba09209fbe443b4acccebe845d8a138b89a8f4fbaeedd44953490b5315d5e965", size = 5345969, upload-time = "2026-07-01T11:54:06.397Z" },
{ url = "https://files.pythonhosted.org/packages/d8/66/9a386a92561f402389a4fc70c18838bf6d35eb5eb5c6850b4b2dc64f5048/pillow-12.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ffd0c5368496f41b0944be820fcb7a838aa6e623d250b01acf2643939c3f99d7", size = 4780323, upload-time = "2026-07-01T11:54:09.351Z" },
{ url = "https://files.pythonhosted.org/packages/25/27/ac8f99618ffd3dde21db0f4d4b1d2ab00c0880595bfd17df103f7f39fd0c/pillow-12.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9c7f76c0673154f044e9d78c8655fb4213f6ca31a836df48b40fe5d187717b9", size = 6266838, upload-time = "2026-07-01T11:54:11.71Z" },
{ url = "https://files.pythonhosted.org/packages/84/21/a35af28dcc61f37ed850a2d64c65c701321dfbf25085e469d5559360cbbf/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:78cb2c6865a35ab8ff8b75fd122f6033b92a62c82801110e48ddd6c936a45d91", size = 6940830, upload-time = "2026-07-01T11:54:13.732Z" },
{ url = "https://files.pythonhosted.org/packages/eb/51/8b08617af3ad95e33ce6d7dd2c99ed6c8298f7fb131636303956be022e25/pillow-12.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e491916b378fba47242221bb9ead245211b70d504f495d105d17b14a24b4907c", size = 6344383, upload-time = "2026-07-01T11:54:15.756Z" },
{ url = "https://files.pythonhosted.org/packages/1d/72/cf78ac9780bb93c28328f408973845a309d4d145041665f734572ced1b52/pillow-12.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0dd2064cbc55aaec028ef5fbb60fa47bb6c3e7918e07ff17935284b227a9d2df", size = 7052934, upload-time = "2026-07-01T11:54:17.721Z" },
{ url = "https://files.pythonhosted.org/packages/20/20/25e0f4dc178a6bc0696793720055519a0de89e7661dae886992decbd2f81/pillow-12.3.0-cp312-cp312-win32.whl", hash = "sha256:dbce0b29841537a2fa4a214c2bbf14de3587c9680caa9b4e217568472490b28f", size = 6472684, upload-time = "2026-07-01T11:54:19.839Z" },
{ url = "https://files.pythonhosted.org/packages/45/89/da2f7971a317f83d807fdd4065c0af40208e59e692cc43d315a71a0e96d1/pillow-12.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a2b55dd6b2a4c4b7d87ffa56bdb33fdc5fdb9a462173861a7bc097f17d91cb09", size = 7227137, upload-time = "2026-07-01T11:54:22.025Z" },
{ url = "https://files.pythonhosted.org/packages/de/47/4845a0a6c0dbf1db8456bd9fc791f13c5ced7ced20606d08a0aacfd25b49/pillow-12.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:331b624368d4f1d069149002f25f44bc61c8919ce8ddb3c45bdad8f6e2d89510", size = 2568267, upload-time = "2026-07-01T11:54:24.051Z" },
{ url = "https://files.pythonhosted.org/packages/9d/ac/31fb64e1e7efb5a4b50cd3d92049ba89ac6e4d8d3bb6a74e15048ca3353e/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:21900ce7ba264168cd50defae43cd75d25c833ad4ad6e73ffc5596d12e25ac89", size = 4161684, upload-time = "2026-07-01T11:54:25.934Z" },
{ url = "https://files.pythonhosted.org/packages/87/b4/9805e23d2b4d77842b468513841fda254ee42f0289d25088340e4ff46e2d/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:4e8c2a84d977f50b9daed6eeaf3baef67d00d5d74d932288f02cb94518ee3ace", size = 4255487, upload-time = "2026-07-01T11:54:27.935Z" },
{ url = "https://files.pythonhosted.org/packages/df/39/ecf519435a200c693fe053a6ee4d835b41cf963a4dfc2551c4e637cb2a71/pillow-12.3.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:ae26d61dfa7a47befdc7572b521024e8745f3d809bd95ca9505a7bba9ef849ec", size = 3696433, upload-time = "2026-07-01T11:54:29.813Z" },
{ url = "https://files.pythonhosted.org/packages/42/92/2fc3ffad878ae8dd5469ec1bc8eb83b71f48e13efdf68f02709003982a32/pillow-12.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7a743ff716f746fc19a9557f60dab1600d4613255f8a7aeb3cdde4db7eb15a66", size = 5345889, upload-time = "2026-07-01T11:54:31.97Z" },
{ url = "https://files.pythonhosted.org/packages/10/76/8803c13605b763d33d156c4678fc77f8443389c0c51c8aef707bb02015f4/pillow-12.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d69141514cc30b774ceea5e3ed3a6635c8d8a96edf664689b890f4089111fb35", size = 4780109, upload-time = "2026-07-01T11:54:34.026Z" },
{ url = "https://files.pythonhosted.org/packages/1f/01/e18aff37cb0b4aac47ac90f016d347a49aca667ef97f190b06ac2aabc928/pillow-12.3.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f7401aebd7f581d7f83a439d87d474999317ee099218e5ad25d125290990ba65", size = 6263736, upload-time = "2026-07-01T11:54:36.131Z" },
{ url = "https://files.pythonhosted.org/packages/f7/62/de5bdd77d935331f4f802edc11e4d82950f642caad6cb2f949837b8560e2/pillow-12.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3", size = 6937129, upload-time = "2026-07-01T11:54:38.216Z" },
{ url = "https://files.pythonhosted.org/packages/70/4d/105627a13300c5e0df1d174230b32fd1273062c96f7745fd552b945d1e1d/pillow-12.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:571b9fcb07b97ef3a492028fb3d2dc0993ca23a06138b0315286566d29ef718a", size = 6339562, upload-time = "2026-07-01T11:54:40.354Z" },
{ url = "https://files.pythonhosted.org/packages/6b/1d/f13de01a553988ab895ba1c722e06cf3144d4f57656fd5b81b6d881f1179/pillow-12.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:756c768d0c9c2955feb7a56c37ea24aea2e369f8d36a88da270b6a9f19e62b5e", size = 7049439, upload-time = "2026-07-01T11:54:42.489Z" },
{ url = "https://files.pythonhosted.org/packages/c9/f9/066794cca041b969964f779ee5fa66a9498bbf34248ac39c5d7954e4198f/pillow-12.3.0-cp313-cp313-win32.whl", hash = "sha256:a876864214e136f0eb367788dbd7df045f4806801518e2cfe9e13229cfe06d8f", size = 6473287, upload-time = "2026-07-01T11:54:44.9Z" },
{ url = "https://files.pythonhosted.org/packages/a6/9b/7a58e61d62be561da3a356fe2384d4059a6345fc130e23ef1c36a5b81d24/pillow-12.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:1cca606cd25738df4ed873d5ad46bbdb3d83b5cbca291f6b4ff13a4df6b0bbe8", size = 7239691, upload-time = "2026-07-01T11:54:47.141Z" },
{ url = "https://files.pythonhosted.org/packages/aa/b0/c4ed4f0ef8f8fa5ee8351537db6650bb8189f7e118842978dd6589065692/pillow-12.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:b629de27fda84b42cde7edef0d85f13b958b47f6e9bbcbba9b673c562a89bd8b", size = 2568185, upload-time = "2026-07-01T11:54:49.137Z" },
{ url = "https://files.pythonhosted.org/packages/dc/01/001f65b68192f0228cc1dbbc8d2530ab5d58b61037ba0587f946fea607cd/pillow-12.3.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:9cf95fe4d0f84c82d282745d9bb08ad9f926efa00be4697e767b814ce40d4330", size = 4161736, upload-time = "2026-07-01T11:54:51.156Z" },
{ url = "https://files.pythonhosted.org/packages/1a/d2/0219746d0fd16fc8a84498e79452375be3797d3ce4044596ce565164b84f/pillow-12.3.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:8728f216dcdb6e6d555cf971cb34076139ad74b31fc2c14da4fafc741c5f6217", size = 4255435, upload-time = "2026-07-01T11:54:53.414Z" },
{ url = "https://files.pythonhosted.org/packages/c8/02/8d0bc62ef0302318c46ff2a512822d2610e81c7aa46c9b3abe6cbaca5ad0/pillow-12.3.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:a45650e8ce7fafffd731db8550230db6b0d306d181a90b67d3e6bca2f1990930", size = 3696262, upload-time = "2026-07-01T11:54:55.739Z" },
{ url = "https://files.pythonhosted.org/packages/85/e2/73c77d218410b14f5f2d565e8a998d5317b7b9c75368d29985139f7a46f0/pillow-12.3.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ba54cfebe86920a559a7c4d6b9050791c20513650a1952ebe3368c7dc70306f8", size = 5350344, upload-time = "2026-07-01T11:54:57.657Z" },
{ url = "https://files.pythonhosted.org/packages/c7/da/32c752228ae345f489e3a42499d817b6c3996da7e8a3bc7a04fc806b243b/pillow-12.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e158cb00350dc278f3b91551101aa7d12415a66ebf2c91d8d5ac14e56ddd3ad0", size = 4780131, upload-time = "2026-07-01T11:54:59.713Z" },
{ url = "https://files.pythonhosted.org/packages/b1/9d/8b2c807dbef61a5197c047afe99823787eb66f63daf9fb2432f91d6f0462/pillow-12.3.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9aeb04d6aef139de265b29683e119b638208f88cf73cdd1658aa07221165321", size = 6263757, upload-time = "2026-07-01T11:55:01.778Z" },
{ url = "https://files.pythonhosted.org/packages/5c/44/c85361f65dbe00eea8576ee467c768d25129989efb76e94f205e9ca9bb46/pillow-12.3.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:251bf95b67017e27b13d82f5b326234ca62d70f9cf4c2b9032de2358a3b12c7b", size = 6936962, upload-time = "2026-07-01T11:55:03.93Z" },
{ url = "https://files.pythonhosted.org/packages/18/7e/e483414b35800b86b6f08dbbc7803fb5cd52c4d6f897f47d53ea2c7e6f65/pillow-12.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fe3cca2e4e8a592be0f269a1ca4835c25199d9f3ce815c8491048f785b0a0198", size = 6339171, upload-time = "2026-07-01T11:55:05.989Z" },
{ url = "https://files.pythonhosted.org/packages/f0/f4/68c491844841ede6bed70189546b3ee9731cf9f2cbad396faff5e1ccba45/pillow-12.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:23aceaa007d6172b02c277f0cd359c79492bbb14f7072b4ede9fbcaf20648130", size = 7048116, upload-time = "2026-07-01T11:55:08.131Z" },
{ url = "https://files.pythonhosted.org/packages/a3/34/77f3f793fed8efc7d243f21b33c5a3f0d1c97ee70346d3db855587e155ff/pillow-12.3.0-cp314-cp314-win32.whl", hash = "sha256:af8d94b0db561cf68b88a267c5c44b49e134f525d0dc2cb7ed413a66bc23559a", size = 6467209, upload-time = "2026-07-01T11:55:10.408Z" },
{ url = "https://files.pythonhosted.org/packages/f1/e0/492879f69d94f91f60fc8cd05ba03650e9520afebb2fb7aa12777d7c7f38/pillow-12.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:fdafc9cce40277e0f7a0feabce0ee50dd2fa1800f3b38015e51296b5e814048d", size = 7237707, upload-time = "2026-07-01T11:55:12.745Z" },
{ url = "https://files.pythonhosted.org/packages/c9/ac/6b11f2875f1c2ac040d84e1bbf9cf22a88038f901ca1037898b280b38365/pillow-12.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:e91206ee562682b51b98ef4b26a6ef48fd84e15fd4c4bc5ec768eb641d206838", size = 2565995, upload-time = "2026-07-01T11:55:14.736Z" },
{ url = "https://files.pythonhosted.org/packages/52/69/c2208e56af9bfc1913afb24020297a691eb1d4ef688474c8a04913f65e04/pillow-12.3.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:164b31cd1a0490ab6efae01aa5df49da7061be0af1b30e035b6e9a1bfe34ee6e", size = 5352503, upload-time = "2026-07-01T11:55:17.076Z" },
{ url = "https://files.pythonhosted.org/packages/07/70/e5686d753e898a45d778ff1718dba8516ead6ab6b95d85fc8c4b70650cf2/pillow-12.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5afb51d599ea772b8365ae807ae557f18bccfe46ab261fd1c2a9ed700fc6eb17", size = 4782956, upload-time = "2026-07-01T11:55:19.448Z" },
{ url = "https://files.pythonhosted.org/packages/d5/37/25c6692f06927ee973ff18c8d9ee98ad0b4d84ee67a09610c2dd1447958e/pillow-12.3.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3edce1d53195db527e0191f84b71d02022de0540bf43a16ed734ed7537b07385", size = 6322855, upload-time = "2026-07-01T11:55:21.613Z" },
{ url = "https://files.pythonhosted.org/packages/cc/91/420637fcb8f1bc11029e403b4538e6694744428d8246118e45719f944556/pillow-12.3.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bf16ba1b4d0b6b7c8e534936632270cf70eb00dbe09005bc345b2677b726855c", size = 6989642, upload-time = "2026-07-01T11:55:24.006Z" },
{ url = "https://files.pythonhosted.org/packages/10/08/b94d7811281ccf0d143a1cf768d1c49e1e54af63e7b708ab2ee3eb87face/pillow-12.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:24870b09b224f7ae3c39ed07d10e819d06f8720bc551847b1d623832b5b0e28d", size = 6391281, upload-time = "2026-07-01T11:55:26.252Z" },
{ url = "https://files.pythonhosted.org/packages/d2/87/24233f785f55474dc02ce3e739c5528a77e3a862e9333d1dd7a25cc31f70/pillow-12.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:30f2aa603c41533cc25c05acd0da21636e84a315768feb631c937177db558931", size = 7096716, upload-time = "2026-07-01T11:55:28.318Z" },
{ url = "https://files.pythonhosted.org/packages/23/26/fcb2f6e37175b04f53570b59937867e2b80ee1685e744023153028fc14f9/pillow-12.3.0-cp314-cp314t-win32.whl", hash = "sha256:4b0a7fe987b14c31ebda6083f74f22b561fd3739bc0ac51e019622e3d72668c7", size = 6474125, upload-time = "2026-07-01T11:55:30.956Z" },
{ url = "https://files.pythonhosted.org/packages/90/de/3634abee5f1c9e13c56787b7d5517b0ba8d6de51700b95578cf338349c9f/pillow-12.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:962864dc93511324d51ddbb5b9f8731bf71675b93ca612a07441896f4688fb8c", size = 7242939, upload-time = "2026-07-01T11:55:34.044Z" },
{ url = "https://files.pythonhosted.org/packages/ce/2a/fd13f8eb24de5714a6eb444a3d67e2842c6c576e159a43793adf23051351/pillow-12.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0740a512dc522224c77d9aa5a8d70d8b7d73fb91f2c21125d8d025d3b8990e45", size = 2567506, upload-time = "2026-07-01T11:55:35.988Z" },
{ url = "https://files.pythonhosted.org/packages/5d/dc/8fdce34ec725a33c81c6ba122b904d6b9024e50ea9ac7bede62fab54506c/pillow-12.3.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:0feb2e9d6ad6c9e3c06effe9d00f3f1e618a6643273576b016f591e9315a7139", size = 4162063, upload-time = "2026-07-01T11:55:37.941Z" },
{ url = "https://files.pythonhosted.org/packages/76/66/2044b9a63d3b84ff048228dfcb7cd9bf0df983e8470971bf7d4c57b693de/pillow-12.3.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:9e881fca225083806662a5c43d627d215f258ff43c890f831966c7d7ba9c7402", size = 4255549, upload-time = "2026-07-01T11:55:40.022Z" },
{ url = "https://files.pythonhosted.org/packages/52/7e/1f67e6f4ece6b582ee4b539decbcc9f848dc245a93ed8cd7338bafef72f1/pillow-12.3.0-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:4998562bf62a445225f22e07c896bb04b35b1b1f2eb6d760584c9c51d7a5f78c", size = 3696331, upload-time = "2026-07-01T11:55:41.98Z" },
{ url = "https://files.pythonhosted.org/packages/12/40/d306fc2c8e4d45d7f175c77edca7063be7b86fe7fe6e68f4353bf71d808c/pillow-12.3.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:dc624f6bc473dacdf7ef7eb8678d0d08edf15cd94fad6ae5c7d6cc67a4e4902f", size = 5350370, upload-time = "2026-07-01T11:55:44.028Z" },
{ url = "https://files.pythonhosted.org/packages/dd/44/668fb1437e8ce420f62d6106eb66e44a5971602a4d794615bdf79315d82d/pillow-12.3.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:71d6097b330eea8fd15097780c8e89cb1a8ce7838669f48c5bacd6f663dd4701", size = 4780147, upload-time = "2026-07-01T11:55:46.073Z" },
{ url = "https://files.pythonhosted.org/packages/0c/08/93fa2e70e30a2d81547e481b6ee2bb9522117221fb1e0ce4b5df70967677/pillow-12.3.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:28ce87c5ab450a9dd970b52e5aca5fe63ed432d18a2eaddd1979a00a1ba24ace", size = 6273659, upload-time = "2026-07-01T11:55:48.264Z" },
{ url = "https://files.pythonhosted.org/packages/f8/6d/043e96ff814fc31a33077e4cba86082167db520c93632afdf2042febbb0c/pillow-12.3.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6b02afb9b97f65fbca5f31db6a2a3ba21aa93030225f150fa3f249717e938fb4", size = 6947439, upload-time = "2026-07-01T11:55:50.503Z" },
{ url = "https://files.pythonhosted.org/packages/af/92/ba71d2ee2ac0edf3fa33bd9d5ee9ee080da70b1766f3ca3934f9938ddac9/pillow-12.3.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:1182d52bc2d5e5d7d0949503aa7e36d12f42205dc287e4883f407b1988820d39", size = 6353577, upload-time = "2026-07-01T11:55:52.697Z" },
{ url = "https://files.pythonhosted.org/packages/0f/ce/e63064e2122923ff687c8ad792d0d736a7b3920a56a46982e81a7fdd25d6/pillow-12.3.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e795b7eb908249c4e43c7c99fac7c2c75dab0c43566e37db472a355f63693d71", size = 7060394, upload-time = "2026-07-01T11:55:55.149Z" },
{ url = "https://files.pythonhosted.org/packages/54/76/a09cc3ccc8d773a7283d34c38bec1708f9e3cc932093cbc4c5e71ac4060b/pillow-12.3.0-cp315-cp315-win32.whl", hash = "sha256:57b3d78c95ba9059768b10e28b813002261d3f3dfc55cc48b0c988f625175827", size = 6467375, upload-time = "2026-07-01T11:55:57.769Z" },
{ url = "https://files.pythonhosted.org/packages/3e/03/1846c49ba3b1d5550392a4bbd06d6fb4578e1cd91a803198b5c90f5f7d53/pillow-12.3.0-cp315-cp315-win_amd64.whl", hash = "sha256:fa4ecea169a355be7a3ade2c783e2ed12f0e40d2c5621cda8b3297faf7fbb9f5", size = 7237048, upload-time = "2026-07-01T11:55:59.975Z" },
{ url = "https://files.pythonhosted.org/packages/fb/bb/89f35dcc79610423f9f195504d7def7f0d1416a711541b42867e25fe3412/pillow-12.3.0-cp315-cp315-win_arm64.whl", hash = "sha256:877c3f311ff35410f690861c4409e7ccbf0cd2f878e50628a28e5a0bb689e658", size = 2566006, upload-time = "2026-07-01T11:56:02.143Z" },
{ url = "https://files.pythonhosted.org/packages/30/88/707027ba09942dfa2c28759b5c222d769290a41c6d20ea60ec250801941f/pillow-12.3.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:e9871b1ffbfa9656b60aeee92ed5136a5742696006fa322b29ea3d8da0ecc9cf", size = 5352509, upload-time = "2026-07-01T11:56:04.2Z" },
{ url = "https://files.pythonhosted.org/packages/b0/6d/00352fa25332c2569cd387851f568cc5a4b75a9adbfb37ac4fbce4c02eec/pillow-12.3.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:53aa02d20d10c3d814d536aa4e5ac9b84ca0ff5a88377963b085ad6822f93e64", size = 4783167, upload-time = "2026-07-01T11:56:06.631Z" },
{ url = "https://files.pythonhosted.org/packages/13/4f/9e049dfa21af7c22427275720e2490267ba8138120add5c4c574deb69782/pillow-12.3.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:446c34dcc4324b084a53b705127dc15717b22c5e140ae0a3c38349d4efec071e", size = 6329237, upload-time = "2026-07-01T11:56:08.868Z" },
{ url = "https://files.pythonhosted.org/packages/36/16/cf6eeaae8d0fce8dd390a33437cf68c5d5bd73834a2bc6e2f14efda0ab45/pillow-12.3.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cf1845d02ad822a369a49f2bb9345b1614744267682e7a03527dc3bf6eea1777", size = 6997047, upload-time = "2026-07-01T11:56:11.379Z" },
{ url = "https://files.pythonhosted.org/packages/1e/69/dbf769bdd55f48bf5733cac28edc6364ffaa072ec9ba336266e4fe66be55/pillow-12.3.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:186941b6aef820ad110fb01fb06eb925374dc3a21b17e37ec9a53b250c6fe2d1", size = 6400440, upload-time = "2026-07-01T11:56:13.908Z" },
{ url = "https://files.pythonhosted.org/packages/a0/e1/ffc9cfc2eea0d178da8018e18e959301ad9d6bc9f3edb7181e748a474b97/pillow-12.3.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:f13c32a3abd6079a66d9526e18dad9b6d280384d49d7c54040cd57b6424041d9", size = 7105895, upload-time = "2026-07-01T11:56:16.575Z" },
{ url = "https://files.pythonhosted.org/packages/18/f0/a5595c1e8c3ae44b9828cb2f0fa8155e5095ef04d6327b8f61cf44a3df85/pillow-12.3.0-cp315-cp315t-win32.whl", hash = "sha256:1657923d2d45afb66526e5b933e5b3052e6bdea196c90d3abb2424e18c77dae8", size = 6474384, upload-time = "2026-07-01T11:56:18.855Z" },
{ url = "https://files.pythonhosted.org/packages/e4/04/62bcd9f844984c5938d3b05264a61d797a29d3e0812341a8204af70bbdee/pillow-12.3.0-cp315-cp315t-win_amd64.whl", hash = "sha256:8cd2f7bdda092d99c9fc2fb7391354f306d01443d22785d0cbfafa2e2c8bb418", size = 7243537, upload-time = "2026-07-01T11:56:21.214Z" },
{ url = "https://files.pythonhosted.org/packages/3d/68/1f3066acedf37673694a7141381d8f811ae97f30d34413d236abe7d489f1/pillow-12.3.0-cp315-cp315t-win_arm64.whl", hash = "sha256:06ff022112bc9cbf83b60f8e028d94ad87b60621706487e65f673de61610ab59", size = 2567491, upload-time = "2026-07-01T11:56:23.506Z" },
{ url = "https://files.pythonhosted.org/packages/75/18/2e8b40223153ccbc60df07f9e8928dc0c76202aa4e55ae9f53962b6510d6/pillow-12.3.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:b3c777e849237620b022f7f297dd67705f9f5cf1685f09f02e46f93e92725468", size = 5302510, upload-time = "2026-07-01T11:56:25.736Z" },
{ url = "https://files.pythonhosted.org/packages/46/3e/51fabf59d5ab801ceab709453d3ab6b180083496579549de4c45ced6528a/pillow-12.3.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:b343699e8308bdc51978310e1c959c584e7869cc8c40780058c87da7781a1e94", size = 4736058, upload-time = "2026-07-01T11:56:28.041Z" },
{ url = "https://files.pythonhosted.org/packages/bf/20/22fe9384b7949e25fb1293bcfc84fb82590ff4ea6b37c95b24d26d793d86/pillow-12.3.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbd139c8447d25dd750ab79ee274cc5e1fe80fc56340ab10b18a195e1b6eca3e", size = 5237776, upload-time = "2026-07-01T11:56:30.263Z" },
{ url = "https://files.pythonhosted.org/packages/08/14/f6ba68107680ffa74b39985f3f30884e41318fbc4250caa423c79b4788bb/pillow-12.3.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e7e480451b9fa137494bccd3a7d69adbe8ac65a87d97be61e11f1b1050a5bac3", size = 5860358, upload-time = "2026-07-01T11:56:32.68Z" },
{ url = "https://files.pythonhosted.org/packages/36/54/0169bc772ec491108b62f644f8ecf1fe5d8ae5ebafde2ee2142210166903/pillow-12.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:04f01d28a6aaff387bf842a13be313df23ba0597a44f1a976c9feb3c6ff4711a", size = 7231786, upload-time = "2026-07-01T11:56:35.046Z" },
]
[[package]]
name = "pluggy"
version = "1.6.0"
@@ -205,20 +305,26 @@ wheels = [
[[package]]
name = "snapform"
version = "0.10.11"
version = "0.12.3"
source = { virtual = "." }
dependencies = [
{ name = "blendjob" },
]
[package.dev-dependencies]
dev = [
{ name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" },
{ name = "numpy", version = "2.5.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" },
{ name = "pillow" },
{ name = "pytest" },
]
[package.metadata]
requires-dist = [{ name = "blendjob", path = "src/snapform/wheels/blendjob-0.1.6-py3-none-any.whl" }]
[package.metadata.requires-dev]
dev = [
{ name = "numpy" },
{ name = "pillow" },
{ name = "pytest" },
]

View File

@@ -1,46 +0,0 @@
schema_version = 1
[artifacts.depth-anything-3]
revision = "3d835ec1a5802d64a8b8b15f817a1ab54809bfe4"
filename = "depth-anything-3.tar.gz"
url = "https://codeload.github.com/ByteDance-Seed/Depth-Anything-3/tar.gz/3d835ec1a5802d64a8b8b15f817a1ab54809bfe4"
sha256 = "98aa2dd53ab44b96cef5190ae4841c6ae51797d099b784e08e12a1a55bd3a69b"
patches = "depth-anything-3"
[artifacts.ben2]
revision = "2c99a5da477b5523585bfa5c893888a6e818a8f6"
filename = "ben2.tar.gz"
url = "https://codeload.github.com/PramaLLC/BEN2/tar.gz/2c99a5da477b5523585bfa5c893888a6e818a8f6"
sha256 = "01fd26716f3ff3a3f18ad2336c768af5b759edfc7c38bb3bf733b7c86eee6282"
[artifacts.moge]
revision = "925b8ed835a7a9cdb7578ba15c658a0afc969030"
filename = "moge.tar.gz"
url = "https://codeload.github.com/microsoft/MoGe/tar.gz/925b8ed835a7a9cdb7578ba15c658a0afc969030"
sha256 = "c6d90b2564d643f0f3ae65b5ffe163abaea3b55d194c6e2ba9976d67f552bd26"
patches = "moge"
[artifacts.utils3d]
revision = "3fab839f0be9931dac7c8488eb0e1600c236e183"
filename = "utils3d.tar.gz"
url = "https://codeload.github.com/EasternJournalist/utils3d/tar.gz/3fab839f0be9931dac7c8488eb0e1600c236e183"
sha256 = "f1f167b1f68809bea391a66dd0d2ad0e14f7382cbddf087b655181c31a285abb"
patches = "utils3d"
[uv]
version = "0.11.3"
[platforms.windows-x64]
filename = "uv-x86_64-pc-windows-msvc.zip"
url = "https://github.com/astral-sh/uv/releases/download/0.11.3/uv-x86_64-pc-windows-msvc.zip"
sha256 = "ae681c0aaec7cc96af184648cb88d73f8393ed60fa5880abdd6bdb910f9b227c"
[platforms.macos-arm64]
filename = "uv-aarch64-apple-darwin.tar.gz"
url = "https://github.com/astral-sh/uv/releases/download/0.11.3/uv-aarch64-apple-darwin.tar.gz"
sha256 = "2bc3d0c7bf2bd08325b1e170abac6f7e5b3346e1d4eab3370d17cefec934996f"
[platforms.linux-x64]
filename = "uv-x86_64-unknown-linux-gnu.tar.gz"
url = "https://github.com/astral-sh/uv/releases/download/0.11.3/uv-x86_64-unknown-linux-gnu.tar.gz"
sha256 = "c0f3236f146e55472663cfbcc9be3042a9f1092275bbe3fe2a56a6cbfd3da5ce"