Refactor image mesh processing and background removal functionality

- Updated NODE_GROUP_NAME from "O Refine DA3 Surface" to "O DA3 Surface".
- Introduced `remove_surface_background` function to filter out background faces and compact geometry.
- Enhanced `build_image_mesh` to conditionally remove background faces based on a new property.
- Added `snapform_remove_background` and `snapform_keep_original` properties to operators for better control over background handling.
- Updated UI to reflect new properties and improved resolution scaling logic.
- Refactored tests to cover new background removal functionality and ensure proper geometry handling.
- Adjusted geometry nodes and validation scripts to remove references to background handling.
This commit is contained in:
2026-08-12 16:15:14 +08:00
parent 7cd5681a8b
commit c3a151ad2c
13 changed files with 477 additions and 271 deletions

View File

@@ -31,8 +31,10 @@ DA3、BEN2 源码包和 uv 已内置在扩展里,安装时直接从本地解
2. 右键打开 `SnapForm` 子菜单,选择 `Convert to Depth Plane`
3. 在对话框中选择:
- `Depth from`:`Original` 直接用原图,`AI Generated` 让 AI 估算深度
- `Background Depth Distance` 为 `0` 时不剔除背景;大于 `0` 时把背景设为固定距离,AI 与 Original 均支持
- 其余选项保持默认即可
- `Resolution Scale` 使用 `1–100%`,按原图长边计算 Process Res;结果会限制在 `252–1008` 并自动对齐到 14 的倍数
- `Remove Background` 默认关闭;开启后在转换期间使用 BEN2 删除背景
- `Unshaded` 保留为直接显示贴图颜色的选项
- `Keep Original` 默认关闭,位于分割线下方;开启后保留源 Image Empty
4. 点击确认,稍等片刻就得到带位移的立体面片
### 去除背景
@@ -44,14 +46,17 @@ DA3、BEN2 源码包和 uv 已内置在扩展里,安装时直接从本地解
选中单张图片的 Image Empty,右键打开 `SnapForm` 子菜单并选择 `Convert to Surface`。SnapForm 会按 DA3
相机内参与深度逐像素反投影,并把生成网格匹配到原 Empty 的可见尺寸、Offset 与 Transform。DA3
confidence、边拉伸与 BEN2 背景结果会保存为 Surface attributes,再由一个可复用的 Geometry Nodes
修改器完成剔面、快速重拓扑、Relax 与 UV 重投。匹配相机默认不创建;开启后会作为 Surface 的子级。
confidence 与边拉伸结果会保存为 Surface attributes,再由一个可复用的 Geometry Nodes
修改器完成剔面、快速重拓扑、Relax 与 UV 重投。开启 Remove Background 时,背景三角面会在转换期间
直接删除,不再交给 Geometry Nodes。匹配相机默认不创建;开启后会作为 Surface 的子级。
当前此功能仅支持单张图片。
转换对话框只保留模型设置、`Create Matching Camera`、`Unshaded` 与 `Delete Original`。剔除和重拓扑参数直接在
转换对话框提供模型与 `Resolution Scale`、`Create Matching Camera`、`Remove Background`、`Unshaded`,
以及分割线下方的 `Keep Original`。
界面会显示计算所得的实际 Process Res。剔除和重拓扑参数直接在
Geometry Nodes 修改器中调整;`Unshaded` 可使用与 Depth Plane 相同的 Emission 材质。普通材质的
Bump Scale 会在材质组内乘以 `0.001`,并按透视表面的深度空间缩放。Surface attributes 统一使用 `o_` 前缀;Host 准备过程显示
`Post-processing n/6`,可看到当前阶段与总步骤。
`Preparing surface data n/6`,可看到当前阶段与总步骤。
### 视频与序列帧

View File

@@ -39,7 +39,7 @@
- 视频生成的序列帧以外部文件引用保留在 job 目录(Blender 无法整体 Pack 序列帧);
Geometry Nodes 与材质 Image Texture 均可按场景帧播放序列帧
- 使用浮点深度贴图、`O DA3 Depth Plane` Geometry Nodes 和原图 Bump 构建立体面片
- 转换对话框选择推理模式和处理分辨率;细分、背景与置换方式在 Geometry Nodes 修改器中调整
- 转换对话框选择推理模式、处理分辨率与背景移除;细分和置换方式在 Geometry Nodes 修改器中调整
- Debug 的 DA3 输出或 BEN2 背景剔除完成后只打开对应 job 文件夹,不修改当前场景
- 完整保留每次任务生成的所有官方文件
- 安装和模型下载使用后台子进程;推理提交给常驻 AI Host,状态栏实时显示阶段、百分比进度条与取消按钮
@@ -103,18 +103,19 @@ Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因
1. 在 3D View 选中 `Object Type = Empty`、`Display As = Image` 的对象。
2. 右键打开 `SnapForm` 子菜单并选择 `Convert to Depth Plane`,通过 `Depth from` 选择深度来源:
- `AI Generated`:显示 Model、Use Ray Pose 和 Resolution Scale。实际 Process Res 为图片长边像素数
乘以 Resolution Scale;范围为 `0.1–1.0`,默认 `1.0`。计算后的 Process Res 会直接显示在
`Resolution Scale (… px)` 标签中。
- `AI Generated`:显示 Model、Use Ray Pose 和 Resolution Scale。Process Res 先以原图长边乘以
Resolution Scale,再钳制到 `252–1008`,最后对齐到最接近的 `14` 倍数。Resolution Scale 以百分比
显示,范围为 `1–100%`,默认 `100%`;界面只在标签中实时显示最终 Process Res。
转换导出 `mini_npz`,使用前需要在
`Host > Models` 下载所选模型,并固定使用 `process_res_method="upper_bound_resize"`;设备和参考视图策略
继续使用场景设置。Model 下拉菜单只列出已经下载的 DA3 模型;若没有可用模型,对话框会引导到
`Host > Models` 下载。
- `Original`:直接把原图作为深度贴图,不显示估算参数。单张图片运行 BEN2 取得前景 Alpha;视频或
序列帧同时抽取原始帧与前景序列,把原始帧作为 `.depth` 序列。
- `Original`:直接把原图作为深度贴图,不显示估算参数。仅在开启 `Remove Background` 时运行 BEN2
取得前景 Alpha;视频或序列帧按需同时抽取原始帧与前景序列,把原始帧作为 `.depth` 序列。
`Original` 是默认深度来源。选择 `AI Generated` 时,其 Model、Use Ray Pose 与 Resolution Scale
参数会集中显示在独立边框中。两种模式都有默认折叠的 Options,内部只保留 `Unshaded`;
`Delete Original` 保持在折叠项外。转换固定需要 `BEN2 Base`,只把 `.foreground` 的 Alpha 作为独立
参数会集中显示在独立边框中。`Remove Background` 与 `Unshaded` 直接显示,不再使用 Options 折叠;
`Keep Original` 默认关闭,并在分割线下方保持为最后一项。仅当开启 `Remove Background` 时才需要
`BEN2 Base`,并把 `.foreground` 的 Alpha 作为独立
Mask,不在 Host 中改写深度值。保存 `.foreground` 时保留 BEN2 Alpha,但 RGB 使用未经修改的原始输入;
因此材质直接用 `.foreground` 同时提供颜色和 Mask,不再额外加载原图 Image Texture。
Image Empty 引用视频或序列帧时,AI Generated 会按 Empty 的 Image Frames 数值逐帧估计
@@ -123,40 +124,39 @@ Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因
一致;Original 模式的视频或序列帧使用 `frame.0001.jpg` 等原图序列作为颜色与深度,
`foreground.0001.png` 只提供前景 Alpha。
AI Generated 的深度仍以 EXR 序列帧(`depth.0001.exr` 起)保存在 job 目录根下。
`Unshaded` 默认关闭,并默认删除原对象。
`Unshaded`、`Remove Background` 与 `Keep Original` 均默认关闭,因此默认删除原对象。
3. AI Generated 完成后会自动创建面片。面片几何先由 `mini_npz` 的相机内参与深度确定,再通过对象
Transform 匹配原 Image Empty 的可见尺寸、位置、旋转和 Offset。Original 模式直接使用源 Empty
的可见尺寸、Offset 与 Transform。AI Generated 会把用于匹配画面尺寸的缩放预乘到 `.depth` 数值,
XY 缩放烘焙到 Mesh,Z 不参与平面匹配;最终对象 Scale 为 `(1, 1, 1)`,视觉深度与旧逻辑一致。
开启 `Delete Original` 时,成功生成后
删除源 Empty。生成面片 Object 使用源 Empty 的原名;关闭 `Delete Original` 时不处理
关闭 `Keep Original` 时,成功生成后删除源 Empty。生成面片 Object 使用源 Empty 的原名;开启
`Keep Original` 时不处理
同名冲突,由 Blender 自动为生成面片追加编号。
4. Blender 只创建保留图像纵横比的最低密度基础网格,再把 `O DA3 Depth Plane` 作为 Geometry Nodes
修改器添加到对象。节点组与 `O Refine DA3 Surface` 一同保存在 `O_SnapForm.blend`,关闭
修改器添加到对象。节点组与 `O DA3 Surface` 一同保存在 `O_SnapForm.blend`,关闭
`show_group_selector`,不为每个对象复制节点组。修改器公开以下参数:
- `Subdivision`:默认 `5`。节点读取基础平面的 Bounding Box,短边分段数为 `2^Subdivision`,
- `Displacement`:位于 Geometry 之后的第一项,菜单包含 `Geometry`(默认)与 `Material`。`Geometry`
在节点组内沿法线修改网格,`Material` 保持网格平面并启用材质置换;
- `Subdivision`:位于第二项,默认 `5`。节点读取基础平面的 Bounding Box,短边分段数为 `2^Subdivision`,
长边按宽高比计算并四舍五入;例如 16:9 图片在等级 `5` 时生成 `57×32` 个面;
- `Remove Background`:默认关闭的 Panel Toggle。关闭时折叠隐藏 `Background Depth`,开启后才显示;
它不删除几何,而是通过点域的
`o_remove_background` Attribute 控制材质 Alpha 是否乘以前景 Mask;
- `Background Depth`:默认 `0`,允许正负值。只有开启 `Remove Background` 时,前景 Mask 外的
结构或材质置换才使用该值;关闭时所有像素继续使用原始深度;
- `Displacement`:菜单包含 `Geometry`(默认)与 `Material`。`Geometry` 在节点组内沿法线修改网格,
`Material` 保持网格平面并启用材质置换。
- `Background Depth`:默认 `0`,允许正负值,Structure Type 为 `Single`,直接显示且不再放在 Panel 中。前景 Mask 外的结构或
材质置换使用该值;是否存在背景只取决于转换所得色彩贴图的 `Alpha > 0.5` 二值结果;
- `Depth Scale`:默认 `1`,允许正负值,Structure Type 为 `Single`,同时控制 Geometry 与 Material
两条置换路径。
Grid 重建后通过隐藏的 Material 输入重新赋予原图材质,并对所有面执行 Shade Smooth。
材质使用显式指向 `UVMap` 的 UV Map 节点连接颜色、前景 Mask 与深度 Image Texture,不依赖活动 UV。
开启背景移除时,前景深度先与 `Background Depth` 取 Maximum,再按前景 Alpha 与背景深度混合,
前景深度先与 `Background Depth` 取 Maximum,再按色彩贴图 Alpha 与背景深度混合,
深度分支在混合前使用 `Alpha > 0.5` 转为二值 Mask,避免半透明前景被背景深度拉扯;
材质表面 Alpha 仍保留柔和边缘。材质置换使用相同的二值和钳制顺序。
材质表面 Alpha 同样使用二值 Mask。材质置换使用相同的二值和钳制顺序。
节点组还会把 `o_material_displacement` 与 `o_background_depth` 写入点域,材质通过
`Geometry` 类型的 Attribute 节点读取。点域会为顶点保存重复值,但能在应用修改器后继续保留参数;
相比自适应网格本身,其占用很小。Depth Image 与 Foreground Image 是修改器隐藏数据输入,
由导入流程自动设置。
视频与序列帧还会为 Depth 与 Foreground 共同写入隐藏的 `Frame Start`、`Frames` 与 `Cyclic`。
导入内容已经是截取后的帧,因此 Offset 在节点组内恒定为 `0`,不再作为输入。节点组使用 Scene Time
计算从 `1` 开始的 Image Texture Frame:场景帧与 Start 都为 `1` 时读取图片第 `1` 帧;非循环输入
视频与序列帧还会为 Depth 与 Foreground 共同写入隐藏的 `Frame`、`Frame Start`、`Frames` 与 `Cyclic`。
节点组内部不再使用 Scene Time。只有原 Image Empty 是视频或序列时,导入流程才直接给修改器的 `Frame`
socket 添加 Scripted Expression Driver `frame`;静态图保持默认值 `1`,不创建 Driver。导入内容已经
是截取后的帧,因此 Offset 在节点组内恒定为 `0`。
节点组根据 `Frame` 计算从 `1` 开始的 Image Texture Frame:场景帧与 Start 都为 `1` 时读取图片第 `1` 帧;非循环输入
钳制到有效范围;循环输入使用 Floored Modulo,让 `Frame Start` 前后的负、正相对帧都映射到
`1…Frames`,因此开始帧之前也会循环播放。Geometry 置换会像材质 Image Texture 一样随场景帧刷新。
@@ -165,13 +165,17 @@ Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因
1. 在 3D View 选中仅包含一帧的 Image Empty,右键打开 `SnapForm` 子菜单并选择
`Convert to Surface`。该入口独立于
`Convert to Depth Plane`,当前不处理视频或序列帧。
2. Host 请求 DA3 的 `depth`、`conf` 与 `intrinsics`,同时固定运行 BEN2 取得前景 alpha。Depth 不做
smoothing,也不在 Host 中剔面或减面;Host 只反投影完整有效拓扑,并计算 confidence percentile、
camera-space edge stretch 与 background 标记。
3. Blender 把勾选 `Usage > Modifier` 的 `O Refine DA3 Surface` 节点组直接作为 Geometry Nodes 修改器
对话框中的 Resolution Scale 与 Depth Plane 使用相同设计:原图长边乘以 `1–100%` 的比例后,
将实际 Process Res 钳制在 `252–1008` 并按 `14` 对齐。
2. Host 请求 DA3 的 `depth`、`conf` 与 `intrinsics`。仅在开启 `Remove Background` 时运行 BEN2 取得
前景 Alpha,并在输出前删除包含背景顶点的三角面;关闭时不加载 BEN2。Depth 不做 smoothing,Host
反投影深度网格,建立源三角面,计算 confidence percentile 与 camera-space edge stretch,并写入
基于原图像素坐标的源 `UVMap`。这些步骤只准备修改器的输入网格,不执行 Relax、拓扑修复、平滑、
小岛清理或最终 UV 重投影。运行时显示的六个 `Preparing surface data` 步骤均指这段 Host 端源数据准备。
3. Blender 把勾选 `Usage > Modifier` 的 `O DA3 Surface` 节点组直接作为 Geometry Nodes 修改器
添加到 Surface,不另设修改器名称。节点组内没有子节点组,依次完成:
- 修改器顶层依次显示 `Resolution`、`Relax Iterations`、`Smooth`;默认展开的 `Culling` Panel
包含 `Confidence`、`Stretch`、`Background`。两个百分比输入都使用 `0–100` 的节点值,并在组内
只包含 `Confidence` 与 `Stretch`。两个百分比输入都使用 `0–100` 的节点值,并在组内
转换为 `0–1` 阈值;`Stretch = 100%` 明确选择全部面,剔面后同时删除
Face Count 为零的孤立顶点;
- 每轮 Remesh Iteration 依次执行 Connected Merge、循环拓扑修复与一次 Smooth,使新拓扑在进入下一轮
@@ -188,8 +192,8 @@ Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因
修改器关闭 `show_group_selector`。`Fix Topology` 位于默认折叠的 `Options` Panel。
节点图按 Culling、Remesh、Topology Fix、Island Cleanup 与 UV 分区摆放;各区就近使用独立的
Group Input,并隐藏没有连线的接口,避免跨越整个节点图的输入连线。
5. Host 准备过程固定显示 6 个编号阶段:验证 DA3 数据、反投影、准备 DA3/BEN2 attributes、构建原始
拓扑、投影 UV/attributes、写出网格。状态栏显示 `Post-processing n/6: ...`。
5. Host 准备过程固定显示 6 个编号阶段:验证 DA3 数据、反投影、准备 DA3 与可选 BEN2 数据、构建原始
源三角面、源 UV/attributes、写出网格。状态栏显示 `Preparing surface data n/6: ...`。
6. Host 在 job 根目录写出 `mesh.npz` 与 `texture.png`。NPZ 保存 camera-space `vertices`、
`faces`、`uv`、`intrinsics`、`image_size` 与有效深度中位数 `reference_depth`。Blender 读取这些
数据后直接创建 Surface 与原图材质,不需要 Open3D 或 OpenCV 进入 Blender Python 环境。
@@ -200,14 +204,26 @@ Cyclic 沿用原 Empty 的设置。序列帧无法被 Blender 整体 Pack,因
边界重合,主点同时匹配 Empty Offset。网格对象直接沿用源 Empty 的 `matrix_world`,因此位置、旋转、
缩放及父级关系产生的世界变换保持一致。更近的顶点沿 Empty 局部 `+Z` 方向凸出,三角面正面朝向
观察相机,启用 Backface Culling 时可从正面看到。
8. 转换对话框不再提供后处理参数,只保留模型设置、`Create Matching Camera`、`Unshaded` 与
`Delete Original`。`Unshaded` 使用与 Depth Plane 相同的 Emission/Transparent 材质路径;普通材质的
8. 转换对话框不再提供后处理参数,只保留模型与 Resolution Scale 设置、`Create Matching Camera`、
`Remove Background`、`Unshaded`,以及分割线下方的 `Keep Original`。背景移除默认关闭;开启后 Host
使用 BEN2,并在写出 Surface 前直接删除背景三角面,不再由 Geometry Nodes 的 Background 开关处理。
`Unshaded` 使用与 Depth Plane 相同的 Emission/Transparent 材质路径;普通材质的
Bump Scale 按 `abs(object.scale.z) × camera scale_z` 设置,并在材质组内乘以固定基准 `0.001`,
与透视表面实际深度尺度一致;它允许负数以反转凹凸方向。
相机默认关闭;开启后会在 Empty 局部 `+Z` 侧创建一台朝向局部 `-Z` 的透视相机,并成为生成 Surface
的子级,使二者移动时保持投影关系。开启 `Delete Original` 时,网格成功创建后删除源 Empty。
的子级,使二者移动时保持投影关系。关闭 `Keep Original` 时,网格成功创建后删除源 Empty。
表面、材质与相机会归入同一个 `Convert to Surface` Undo 步骤。
Process Res 的设计依据 DA3 官方实现:API 与 CLI 默认使用 `process_res=504` 和
`process_res_method="upper_bound_resize"`;输入预处理的 `PATCH_SIZE=14`,会把宽高分别调整到最接近的
`14` 倍数。DA3 论文以 `504×504` 为 base resolution,公开的训练尺寸长边为 `504`、`672`、`756` 或
`896`。官方 CLI 虽给出 `1024` 示例,但没有声明更高分辨率会提升质量,也没有规定 `1000` 为硬上限。
因此转换入口使用相对原图的 Resolution Scale,同时通过上下限避免 Process Res 过低或过高,并把训练范围
之外的值明确视为实验设置。参考
[DA3 API](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)、
[输入预处理实现](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/src/depth_anything_3/utils/io/input_processor.py)
与[技术报告](https://depth-anything-3.github.io/assets/da3_tech_report_2025.pdf)。
Debug 的 Textured Surface GLB 会把 depth、confidence 与 confidence percentile 写入自定义顶点
attributes。其他 DA3 Debug 三维输出会同时写出未压缩的 `prediction.attributes.npz`,保存完整
depth、confidence、intrinsics、extrinsics 与 confidence percentile。当前 trimesh 的 glTF 点图元
@@ -245,24 +261,27 @@ Height = sqrt(Area / Aspect)
材质始终导入深度 Image Texture,并连接到 Displacement 节点和 Material Output;材质 Surface
Displacement 设置为 `Displacement Only`,AI Generated 与 Original 都使用 `Midlevel = 0`、`Scale = 1`。
`o_material_displacement` 为 `False` 时,材质把 Height 乘以零,置换只在几何节点中生效;为 `True` 时,
几何节点关闭结构置换,材质 Height 恢复。开启 `Remove Background` 时,前景 Alpha 同时用于选择
原始深度或有符号的 `Background Depth`,使两种置换路径保持一致;关闭时背景深度不参与计算。
几何节点关闭结构置换,材质 Height 恢复。色彩贴图的 Alpha 二值结果同时用于选择原始深度或有符号的
`Background Depth`,使两种置换路径保持一致;未移除背景的普通不透明图片因此全部视为前景。
视频与序列帧的颜色、深度与前景 Image Texture
通过 image_user 的 Frames / Offset / Start 把第 i 帧映射到场景 `Start+i` 帧,并启用 Auto Refresh。
颜色贴图到 Principled BSDF 之间使用独立的 Shader Node Group。Surface 使用 `O DA3 Surface`;
Depth Plane 使用显示名为 `O DA3 Depth Plane` 的材质组,并把背景剔除、背景深度与材质置换 Attribute
读取全部收进组内。两者都依次接收 `Color`、`Alpha`、Float 类型的 `Depth`、`Bump Scale`,输出 `Color`、
`Roughness`、`Alpha`、`Normal`、`Displacement`。Surface 把原图 Color 同时送入 `Color` 与 `Depth`,
颜色贴图到 Principled BSDF 之间使用独立的 Shader Node Group。Surface 使用 `O DA3 Surface Layer`;
Depth Plane 使用 `O DA3 Depth Plane Layer` 材质组,并把背景剔除、背景深度与材质置换 Attribute
读取全部收进组内。两者都依次接收 `Color`、`Alpha`、Float 类型的 `Depth`、`Bump Scale`。
`O DA3 Surface Layer` 只输出 `Color`、`Roughness`、`Alpha` 与 `Normal`,不创建 Displacement 节点或输出;
`O DA3 Depth Plane Layer` 另外输出 `Displacement`。Surface 把原图 Color 同时送入 `Color` 与 `Depth`,
保持图像 Bump 效果。外层材质只保留贴图、材质组、Principled BSDF 与 Material Output,方便继续调整。
Depth Plane 的 Bump Height 直接使用材质组的 `Color` 输入,只受独立的 `Bump Scale` 影响,不使用深度贴图,
也不经过 `Displacement` 模式开关、`Depth Scale` 或 `Background Depth`;材质 Displacement 仍使用这些参数处理后的深度。
无论选择哪种置换方式,原始彩色贴图都会连接
Principled BSDF 的 `Base Color`,并通过
`Bump.Height → Bump.Normal → Principled.Normal` 提供表面细节。Bump 的基础 Distance 为
`0.001`,会乘以面片 Transform 的统一缩放倍率。原图 Color 还会经过 `Float Curve` 连接到
Principled BSDF 的 `Roughness`,曲线两端输出分别为 `0.5` 和 `0.1`;材质 IOR 保持
Principled BSDF 默认值 `1.5`。
开启 `Unshaded` 后,原图 Color 连接 Emission,原图 Alpha 作为 Mix Shader 系数,在
开启 `Unshaded` 后,色彩贴图 Color 连接 Emission,二值 Alpha 作为 Mix Shader 系数,在
Transparent BSDF 与 Emission 之间混合后连接 Material Output;不创建 Principled BSDF、Bump 或
Float Curve。`Remove Background` 开启时,原图 Alpha 会再乘以前景 Mask Alpha;关闭时仍完整显示原图。
Float Curve。开启 `Remove Background` 时色彩贴图使用 BEN2 输出;关闭时使用原图。
这里的“原始距离”是 DA3 模型输出的尺度。普通 Small、Base、Large 和 Giant 模型的深度不应
被解释为现实世界的米;需要现实尺度时应选择带 metric depth 的

Binary file not shown.

View File

@@ -15,12 +15,14 @@ ORIGINAL_DISPLACEMENT_MIDLEVEL = 0.0
BUMP_BASE_DISTANCE = 0.001
DEPTH_PLANE_NODE_GROUP_NAME = "O DA3 Depth Plane"
NODE_ASSET_NAME = "O_SnapForm.blend"
REMOVE_BACKGROUND_ATTRIBUTE = "o_remove_background"
MATERIAL_DISPLACEMENT_ATTRIBUTE = "o_material_displacement"
BACKGROUND_DEPTH_ATTRIBUTE = "o_background_depth"
DEPTH_SCALE_ATTRIBUTE = "o_depth_scale"
SURFACE_MATERIAL_NODE_GROUP_NAME = "O DA3 Surface"
DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME = "O DA3 Depth Plane Material"
SURFACE_MATERIAL_NODE_GROUP_NAME = "O DA3 Surface Layer"
DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME = "O DA3 Depth Plane Layer"
DA3_PATCH_SIZE = 14
DA3_MIN_PROCESS_RES = 252
DA3_MAX_PROCESS_RES = 1008
IMAGE_NAME_SUFFIXES = (
".jpeg",
".jpg",
@@ -180,13 +182,45 @@ def removed_background_data_name(source_object):
return f"{image_base_name(source_object.data)}.foreground"
def scaled_process_resolution(image, scale):
def scaled_process_resolution(image, scale_percent):
"""Return the scaled long edge clamped and aligned for DA3 inference."""
width, height = image.size
if width <= 0 or height <= 0:
raise ValueError("The source image has no readable dimensions")
if scale < 0.1 or scale > 1.0:
raise ValueError("Resolution Scale must be between 0.1 and 1.0")
return int(max(width, height) * scale + 0.5)
if scale_percent < 1.0 or scale_percent > 100.0:
raise ValueError("Resolution Scale must be between 1% and 100%")
process_res = max(width, height) * scale_percent / 100.0
process_res = max(DA3_MIN_PROCESS_RES, process_res)
process_res = min(DA3_MAX_PROCESS_RES, process_res)
return int(process_res / DA3_PATCH_SIZE + 0.5) * DA3_PATCH_SIZE
def process_resolution_scale_property():
return bpy.props.FloatProperty(
name="Resolution Scale",
description=(
"Scale the source image long edge for DA3 processing; the result "
"is limited to 252-1008 px and aligned to a multiple of 14"
),
default=100.0,
min=1.0,
max=100.0,
step=100,
precision=0,
subtype="PERCENTAGE",
)
def draw_process_resolution(layout, owner, image):
process_res = scaled_process_resolution(
image,
owner.snapform_process_res_factor,
)
layout.prop(
owner,
"snapform_process_res_factor",
text=f"Resolution Scale ({process_res} px)",
)
def save_packed_image(image, scene):
@@ -880,13 +914,15 @@ def material_node_group(depth_plane=False):
socket.default_value = 1.0
socket.min_value = -10000.0
socket.max_value = 10000.0
for name, socket_type in (
output_sockets = [
("Color", "NodeSocketColor"),
("Roughness", "NodeSocketFloat"),
("Alpha", "NodeSocketFloat"),
("Normal", "NodeSocketVector"),
("Displacement", "NodeSocketVector"),
):
]
if depth_plane:
output_sockets.append(("Displacement", "NodeSocketVector"))
for name, socket_type in output_sockets:
interface.new_socket(
name=name,
in_out="OUTPUT",
@@ -908,23 +944,20 @@ def material_node_group(depth_plane=False):
roughness_curve.location = (-250.0, 350.0)
bump = nodes.new("ShaderNodeBump")
bump.location = (-250.0, 0.0)
displacement = nodes.new("ShaderNodeDisplacement")
displacement.inputs["Midlevel"].default_value = 0.0
displacement.inputs["Scale"].default_value = 1.0
displacement.location = (-250.0, -300.0)
displacement = None
processed_alpha = group_input.outputs["Alpha"]
processed_depth = group_input.outputs["Depth"]
bump_depth = group_input.outputs["Color"]
if depth_plane:
displacement = nodes.new("ShaderNodeDisplacement")
displacement.inputs["Midlevel"].default_value = 0.0
displacement.inputs["Scale"].default_value = 1.0
group_input.location = (-1200.0, 100.0)
group_output.location = (1050.0, 100.0)
roughness_curve.location = (-300.0, 450.0)
bump.location = (750.0, 50.0)
displacement.location = (750.0, -300.0)
remove_attribute = nodes.new("ShaderNodeAttribute")
remove_attribute.attribute_type = "GEOMETRY"
remove_attribute.attribute_name = REMOVE_BACKGROUND_ATTRIBUTE
remove_attribute.location = (-950.0, 0.0)
background_attribute = nodes.new("ShaderNodeAttribute")
background_attribute.attribute_type = "GEOMETRY"
background_attribute.attribute_name = BACKGROUND_DEPTH_ATTRIBUTE
@@ -937,10 +970,6 @@ def material_node_group(depth_plane=False):
strength_attribute.attribute_type = "GEOMETRY"
strength_attribute.attribute_name = DEPTH_SCALE_ATTRIBUTE
strength_attribute.location = (200.0, -650.0)
alpha_mix = nodes.new("ShaderNodeMix")
alpha_mix.data_type = "FLOAT"
alpha_mix.inputs[2].default_value = 1.0
alpha_mix.location = (-650.0, 50.0)
foreground_mask = nodes.new("ShaderNodeMath")
foreground_mask.operation = "GREATER_THAN"
foreground_mask.inputs[1].default_value = 0.5
@@ -951,31 +980,23 @@ def material_node_group(depth_plane=False):
masked_depth = nodes.new("ShaderNodeMix")
masked_depth.data_type = "FLOAT"
masked_depth.location = (-350.0, -350.0)
background_mix = nodes.new("ShaderNodeMix")
background_mix.data_type = "FLOAT"
background_mix.location = (-50.0, -350.0)
enabled_depth = nodes.new("ShaderNodeMath")
enabled_depth.operation = "MULTIPLY"
enabled_depth.location = (250.0, -350.0)
scaled_depth = nodes.new("ShaderNodeMath")
scaled_depth.operation = "MULTIPLY"
scaled_depth.location = (450.0, -350.0)
links.new(remove_attribute.outputs["Fac"], alpha_mix.inputs[0])
links.new(group_input.outputs["Alpha"], alpha_mix.inputs[3])
links.new(group_input.outputs["Alpha"], foreground_mask.inputs[0])
links.new(group_input.outputs["Depth"], clamp_foreground.inputs[0])
links.new(background_attribute.outputs["Fac"], clamp_foreground.inputs[1])
links.new(foreground_mask.outputs[0], masked_depth.inputs[0])
links.new(background_attribute.outputs["Fac"], masked_depth.inputs[2])
links.new(clamp_foreground.outputs[0], masked_depth.inputs[3])
links.new(remove_attribute.outputs["Fac"], background_mix.inputs[0])
links.new(group_input.outputs["Depth"], background_mix.inputs[2])
links.new(masked_depth.outputs["Result"], background_mix.inputs[3])
links.new(background_mix.outputs["Result"], enabled_depth.inputs[0])
links.new(masked_depth.outputs["Result"], enabled_depth.inputs[0])
links.new(material_attribute.outputs["Fac"], enabled_depth.inputs[1])
links.new(enabled_depth.outputs[0], scaled_depth.inputs[0])
links.new(strength_attribute.outputs["Fac"], scaled_depth.inputs[1])
processed_alpha = alpha_mix.outputs["Result"]
processed_alpha = foreground_mask.outputs[0]
processed_depth = scaled_depth.outputs[0]
links.new(group_input.outputs["Color"], group_output.inputs["Color"])
@@ -985,7 +1006,7 @@ def material_node_group(depth_plane=False):
group_output.inputs["Roughness"],
)
links.new(processed_alpha, group_output.inputs["Alpha"])
links.new(processed_depth, bump.inputs["Height"])
links.new(bump_depth, bump.inputs["Height"])
bump_distance = nodes.new("ShaderNodeMath")
bump_distance.operation = "MULTIPLY"
bump_distance.inputs[1].default_value = BUMP_BASE_DISTANCE
@@ -993,6 +1014,7 @@ def material_node_group(depth_plane=False):
links.new(group_input.outputs["Bump Scale"], bump_distance.inputs[0])
links.new(bump_distance.outputs[0], bump.inputs["Distance"])
links.new(bump.outputs["Normal"], group_output.inputs["Normal"])
if displacement is not None:
links.new(processed_depth, displacement.inputs["Height"])
links.new(
displacement.outputs["Displacement"],
@@ -1081,9 +1103,7 @@ def create_material(
material_inputs = nodes.new("ShaderNodeGroup")
is_depth_plane = displacement_image is not None
material_inputs.node_tree = material_node_group(is_depth_plane)
material_inputs.name = (
"O DA3 Depth Plane" if is_depth_plane else "O DA3 Surface"
)
material_inputs.name = material_inputs.node_tree.name
material_inputs.label = material_inputs.name
material_inputs.location = (-300.0, 350.0)
material_inputs.inputs["Bump Scale"].default_value = depth_scale
@@ -1243,21 +1263,32 @@ def add_depth_plane_processing(
frame_start=1,
frame_duration=1,
use_cyclic=False,
animate_frame=False,
):
node_group = load_depth_plane_node_group()
modifier = plane_object.modifiers.new(name=node_group.name, type="NODES")
modifier = plane_object.modifiers.new(
name=DEPTH_PLANE_NODE_GROUP_NAME,
type="NODES",
)
modifier.node_group = node_group
modifier.show_group_selector = False
for name, value in (
("Depth Image", depth_image),
("Foreground Image", foreground_image),
("Material", material),
("Frame", 1),
("Frame Start", frame_start),
("Frames", max(int(frame_duration or 1), 1)),
("Cyclic", use_cyclic),
):
identifier = modifier_input_identifier(node_group, name)
modifier[identifier] = value
if animate_frame:
frame_identifier = modifier_input_identifier(node_group, "Frame")
driver = modifier.driver_add(f'["{frame_identifier}"]')
driver.driver.type = "SCRIPTED"
driver.driver.expression = "frame"
plane_object.update_tag(refresh={"DATA"})
return modifier
@@ -1353,6 +1384,7 @@ def build_image_depth_plane(
frame_start=source_frame_start,
frame_duration=frame_count,
use_cyclic=source_use_cyclic,
animate_frame=is_animated_image(source_object.data),
)
plane_object["snapform_frame_count"] = frame_count
return finish_depth_plane(
@@ -1456,6 +1488,7 @@ def build_depth_plane(
frame_start=source_frame_start,
frame_duration=frame_count,
use_cyclic=source_use_cyclic,
animate_frame=is_animated_image(source_object.data),
)
plane_object["snapform_reference_depth"] = reference_depth
plane_object["snapform_camera_plane_width"] = bounds[1] - bounds[0]
@@ -1744,6 +1777,11 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
description="Display the source color through an Emission shader",
default=False,
)
snapform_remove_background: bpy.props.BoolProperty(
name="Remove Background",
description="Remove the background during conversion with BEN2",
default=False,
)
snapform_model: bpy.props.EnumProperty(
name="Model",
description="Model used to estimate image depth",
@@ -1754,33 +1792,24 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
description="Use ray-based camera pose estimation",
default=True,
)
snapform_process_res_factor: bpy.props.FloatProperty(
name="Resolution Scale",
description="Scale applied to the source image long-edge resolution",
default=1.0,
min=0.1,
max=1.0,
step=10,
precision=2,
)
snapform_delete_original: bpy.props.BoolProperty(
name="Delete Original",
description="Delete the source Image Empty after conversion",
default=True,
snapform_process_res_factor: process_resolution_scale_property()
snapform_keep_original: bpy.props.BoolProperty(
name="Keep Original",
description="Keep the source Image Empty after conversion",
default=False,
)
@classmethod
def poll(cls, context):
from .operators import active_job
from .runtime import host_busy, model_ready
from .models import BEN2_MODEL, MODELS
from .models import MODELS
return (
is_image_empty(context.object)
and active_job() is None
and not host_busy()
and any(model_ready(model_key) for model_key in MODELS)
and model_ready(BEN2_MODEL.key)
)
def invoke(self, context, _event):
@@ -1810,28 +1839,17 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
else:
estimate_box.prop(self, "snapform_model")
estimate_box.prop(self, "snapform_use_ray_pose")
process_res = scaled_process_resolution(
context.object.data,
self.snapform_process_res_factor,
)
estimate_box.prop(
self,
"snapform_process_res_factor",
text=f"Resolution Scale ({process_res} px)",
)
settings_header, settings_body = layout.panel(
"snapform_depth_plane_options",
default_closed=True,
)
settings_header.label(text="Options")
if settings_body is not None:
settings_body.prop(self, "snapform_unshaded")
layout.prop(self, "snapform_delete_original")
draw_process_resolution(estimate_box, self, context.object.data)
layout.prop(self, "snapform_remove_background")
layout.prop(self, "snapform_unshaded")
layout.separator(type="LINE")
layout.prop(self, "snapform_keep_original")
def execute(self, context):
from .operators import (
IMAGE_DEPTH_JOB,
ORIGINAL_BACKGROUND_JOB,
ORIGINAL_FRAMES_JOB,
)
source_object = context.object
@@ -1849,10 +1867,14 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
result = bpy.ops.snapform.run_job(
job_type=ORIGINAL_BACKGROUND_JOB,
job_type=(
ORIGINAL_BACKGROUND_JOB
if self.snapform_remove_background
else ORIGINAL_FRAMES_JOB
),
snapform_input_path=str(source_path),
snapform_source_object_name=source_object.name,
snapform_delete_source=self.snapform_delete_original,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_unshaded=self.snapform_unshaded,
)
@@ -1888,10 +1910,10 @@ class ConvertImageToDepthPlane(bpy.types.Operator):
snapform_model_key=self.snapform_model,
snapform_use_ray_pose=self.snapform_use_ray_pose,
snapform_process_res=process_res,
snapform_delete_source=self.snapform_delete_original,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_unshaded=self.snapform_unshaded,
snapform_remove_background=True,
snapform_remove_background=self.snapform_remove_background,
)
if "CANCELLED" in result:
if delete_input:

View File

@@ -4,10 +4,12 @@ import bpy
from .image_depth import (
create_material,
draw_process_resolution,
image_depth_model_items,
is_image_empty,
plane_bounds,
prepare_image_input,
process_resolution_scale_property,
remove_temporary_input,
scaled_process_resolution,
video_total_frames,
@@ -33,7 +35,7 @@ FACE_ATTRIBUTES = {
"o_edge_stretch": ("FLOAT", "face_edge_stretch"),
"o_background_face": ("BOOLEAN", "face_background"),
}
NODE_GROUP_NAME = "O Refine DA3 Surface"
NODE_GROUP_NAME = "O DA3 Surface"
NODE_ASSET_NAME = "O_SnapForm.blend"
@@ -202,6 +204,30 @@ def assign_snapform_attributes(mesh, attributes):
)
def remove_surface_background(vertices, faces, uv, attributes):
"""Remove background faces and compact matching point/face data."""
import numpy as np
background_faces = np.asarray(attributes["face_background"], dtype=bool)
kept_faces = faces[~background_faces]
if not len(kept_faces):
raise ValueError("Background removal deleted every surface face")
used_vertices = np.unique(kept_faces)
index_map = np.full(len(vertices), -1, dtype=np.int32)
index_map[used_vertices] = np.arange(len(used_vertices), dtype=np.int32)
filtered_attributes = {}
for _name, (_data_type, key) in POINT_ATTRIBUTES.items():
filtered_attributes[key] = attributes[key][used_vertices]
for _name, (_data_type, key) in FACE_ATTRIBUTES.items():
filtered_attributes[key] = attributes[key][~background_faces]
return (
vertices[used_vertices],
index_map[kept_faces],
uv[used_vertices],
filtered_attributes,
)
def node_asset_path():
return Path(__file__).with_name("assets") / NODE_ASSET_NAME
@@ -333,6 +359,7 @@ def build_image_mesh(
delete_source,
create_camera,
unshaded,
remove_background=False,
):
(
camera_vertices,
@@ -343,6 +370,13 @@ def build_image_mesh(
reference_depth,
attributes,
) = load_image_mesh_data(npz_path)
if remove_background:
camera_vertices, faces, uv, attributes = remove_surface_background(
camera_vertices,
faces,
uv,
attributes,
)
vertices, camera_parameters = empty_anchored_vertices(
source_object,
camera_vertices,
@@ -415,6 +449,10 @@ class BuildImageMesh(bpy.types.Operator):
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
snapform_remove_background: bpy.props.BoolProperty(
default=False,
options={"HIDDEN", "SKIP_SAVE"},
)
def execute(self, context):
source_object = bpy.data.objects.get(self.snapform_source_object_name)
@@ -429,6 +467,7 @@ class BuildImageMesh(bpy.types.Operator):
self.snapform_delete_source,
self.snapform_create_camera,
self.snapform_unshaded,
self.snapform_remove_background,
)
except (OSError, RuntimeError, ValueError) as error:
self.report({"ERROR"}, str(error))
@@ -452,15 +491,7 @@ class ConvertImageToMesh(bpy.types.Operator):
description="Use ray-based camera pose estimation",
default=True,
)
snapform_process_res_factor: bpy.props.FloatProperty(
name="Resolution Scale",
description="Scale applied to the source image long-edge resolution",
default=1.0,
min=0.1,
max=1.0,
step=10,
precision=2,
)
snapform_process_res_factor: process_resolution_scale_property()
snapform_create_camera: bpy.props.BoolProperty(
name="Create Matching Camera",
description="Create a perspective camera matching the source projection",
@@ -471,15 +502,20 @@ class ConvertImageToMesh(bpy.types.Operator):
description="Display the source color through an Emission shader",
default=False,
)
snapform_delete_original: bpy.props.BoolProperty(
name="Delete Original",
description="Delete the source Image Empty after conversion",
default=True,
snapform_remove_background: bpy.props.BoolProperty(
name="Remove Background",
description="Remove the background during conversion with BEN2",
default=False,
)
snapform_keep_original: bpy.props.BoolProperty(
name="Keep Original",
description="Keep the source Image Empty after conversion",
default=False,
)
@classmethod
def poll(cls, context):
from .models import BEN2_MODEL, MODELS
from .models import MODELS
from .operators import active_job
from .runtime import host_busy, model_ready
@@ -489,7 +525,6 @@ class ConvertImageToMesh(bpy.types.Operator):
and active_job() is None
and not host_busy()
and any(model_ready(model_key) for model_key in MODELS)
and model_ready(BEN2_MODEL.key)
)
def invoke(self, context, _event):
@@ -511,18 +546,12 @@ class ConvertImageToMesh(bpy.types.Operator):
layout = self.layout
layout.prop(self, "snapform_model")
layout.prop(self, "snapform_use_ray_pose")
process_res = scaled_process_resolution(
context.object.data,
self.snapform_process_res_factor,
)
layout.prop(
self,
"snapform_process_res_factor",
text=f"Resolution Scale ({process_res} px)",
)
draw_process_resolution(layout, self, context.object.data)
layout.prop(self, "snapform_create_camera")
layout.prop(self, "snapform_remove_background")
layout.prop(self, "snapform_unshaded")
layout.prop(self, "snapform_delete_original")
layout.separator(type="LINE")
layout.prop(self, "snapform_keep_original")
def execute(self, context):
from .operators import IMAGE_MESH_JOB
@@ -550,10 +579,11 @@ class ConvertImageToMesh(bpy.types.Operator):
snapform_model_key=self.snapform_model,
snapform_use_ray_pose=self.snapform_use_ray_pose,
snapform_process_res=process_res,
snapform_delete_source=self.snapform_delete_original,
snapform_delete_source=not self.snapform_keep_original,
snapform_delete_input=delete_input,
snapform_mesh_create_camera=self.snapform_create_camera,
snapform_unshaded=self.snapform_unshaded,
snapform_remove_background=self.snapform_remove_background,
)
if "CANCELLED" in result:
if delete_input:

View File

@@ -328,8 +328,6 @@ class RunJob(bpy.types.Operator):
remove_background = bool(
getattr(self, "snapform_remove_background", False)
)
if self.job_type == IMAGE_MESH_JOB:
remove_background = True
if self.job_type == "INSTALL_RUNTIME":
self._require_online_access()
return runtime.install_runtime_command(paths["status"]), paths
@@ -723,6 +721,7 @@ class RunJob(bpy.types.Operator):
snapform_delete_source=job.delete_source,
snapform_create_camera=job.mesh_create_camera,
snapform_unshaded=job.unshaded,
snapform_remove_background=job.remove_background,
)
if "FINISHED" not in result:
raise RuntimeError("Unable to build the perspective surface")

View File

@@ -473,6 +473,12 @@ def export_prediction_attributes(prediction, output_directory):
return output_path
def remove_background_faces(faces, foreground_mask):
"""Return only triangles whose vertices all belong to the foreground."""
foreground_points = foreground_mask.reshape(-1)
return faces[foreground_points[faces].all(axis=1)]
def export_image_mesh_data(
prediction,
image_path,
@@ -505,9 +511,9 @@ def export_image_mesh_data(
raw_depth = depth_frames[0]
confidence = confidence_frames[0]
intrinsics = np.asarray(prediction.intrinsics[0], dtype=np.float32)
progress(2, "Unprojecting camera geometry")
progress(2, "Unprojecting the depth grid")
points = unproject_camera_depth(raw_depth, intrinsics)
progress(3, "Preparing DA3 and BEN2 attributes")
progress(3, "Preparing confidence and background data")
foreground_mask = np.ones(raw_depth.shape, dtype=bool)
if foreground_path is not None:
foreground_mask = foreground_mesh_mask(
@@ -518,7 +524,7 @@ def export_image_mesh_data(
raw_depth,
confidence,
)
progress(4, "Building editable source topology")
progress(4, "Building source triangles")
faces = mesh_triangle_faces(
points,
raw_depth,
@@ -527,11 +533,13 @@ def export_image_mesh_data(
False,
0.0,
)
if foreground_path is not None:
faces = remove_background_faces(faces, foreground_mask)
if not len(faces):
raise ValueError(
"DA3 output does not contain any valid surface faces"
)
progress(5, "Projecting UV and structured attributes")
progress(5, "Preparing source UV and attributes")
vertices, faces = compact_mesh(
points.reshape(-1, 3),
faces,
@@ -551,7 +559,7 @@ def export_image_mesh_data(
reference_depth = float(np.median(valid_depth))
output_directory = Path(output_directory)
output_directory.mkdir(parents=True, exist_ok=True)
progress(6, "Writing surface data")
progress(6, "Writing source surface data")
result_path = output_directory / "mesh.npz"
np.savez_compressed(
result_path,
@@ -564,7 +572,8 @@ def export_image_mesh_data(
**attributes,
)
texture_path = output_directory / "texture.png"
with Image.open(image_path) as image:
texture_source = foreground_path or image_path
with Image.open(texture_source) as image:
image.convert("RGBA").save(texture_path, format="PNG", compress_level=1)
return result_path, texture_path
@@ -1650,7 +1659,7 @@ def reconstruct(args, model_cache=None, cancel_check=None):
write_status(
args.status,
0.88 + 0.09 * step / total,
f"Post-processing {step}/{total}: {label}",
f"Preparing surface data {step}/{total}: {label}",
)
export_image_mesh_data(

View File

@@ -214,7 +214,6 @@ class AddonTest(unittest.TestCase):
"snapform_edge_culling",
"snapform_face_reduction",
"snapform_depth_smoothing",
"snapform_remove_background",
"snapform_postprocess_method",
):
with self.subTest(property_name=property_name):
@@ -222,6 +221,18 @@ class AddonTest(unittest.TestCase):
self.assertFalse(annotations["snapform_create_camera"]["default"])
self.assertFalse(annotations["snapform_unshaded"]["default"])
self.assertEqual(annotations["snapform_unshaded"]["name"], "Unshaded")
self.assertFalse(annotations["snapform_remove_background"]["default"])
self.assertFalse(annotations["snapform_keep_original"]["default"])
self.assertEqual(
annotations["snapform_process_res_factor"]["default"],
100.0,
)
self.assertEqual(annotations["snapform_process_res_factor"]["min"], 1.0)
self.assertEqual(annotations["snapform_process_res_factor"]["max"], 100.0)
self.assertEqual(
annotations["snapform_process_res_factor"]["subtype"],
"PERCENTAGE",
)
def test_image_mesh_uses_o_prefixed_attributes_and_bundled_nodes(self):
image_mesh = self.addon.image_mesh
@@ -231,7 +242,7 @@ class AddonTest(unittest.TestCase):
self.assertTrue(all(name.startswith("o_") for name in names))
self.assertTrue(image_mesh.node_asset_path().is_file())
self.assertEqual(image_mesh.NODE_GROUP_NAME, "O Refine DA3 Surface")
self.assertEqual(image_mesh.NODE_GROUP_NAME, "O DA3 Surface")
self.assertEqual(image_mesh.NODE_ASSET_NAME, "O_SnapForm.blend")
def test_depth_plane_uses_bundled_nodes_and_material_attributes(self):
@@ -242,10 +253,6 @@ class AddonTest(unittest.TestCase):
image_depth.DEPTH_PLANE_NODE_GROUP_NAME,
"O DA3 Depth Plane",
)
self.assertEqual(
image_depth.REMOVE_BACKGROUND_ATTRIBUTE,
"o_remove_background",
)
self.assertEqual(
image_depth.MATERIAL_DISPLACEMENT_ATTRIBUTE,
"o_material_displacement",
@@ -279,6 +286,33 @@ class AddonTest(unittest.TestCase):
float(np.median(edge_lengths) * np.sqrt(10.0) * 2.0),
)
def test_surface_background_filter_compacts_geometry_and_attributes(self):
vertices = np.arange(18, dtype=np.float32).reshape(6, 3)
faces = np.asarray(((0, 1, 2), (2, 3, 4)), dtype=np.int32)
uv = np.arange(12, dtype=np.float32).reshape(6, 2)
attributes = {
key: np.arange(6 if name in self.addon.image_mesh.POINT_ATTRIBUTES else 2)
for name, (_data_type, key) in {
**self.addon.image_mesh.POINT_ATTRIBUTES,
**self.addon.image_mesh.FACE_ATTRIBUTES,
}.items()
}
attributes["face_background"] = np.asarray((True, False))
result = self.addon.image_mesh.remove_surface_background(
vertices,
faces,
uv,
attributes,
)
filtered_vertices, filtered_faces, filtered_uv, filtered_attributes = result
np.testing.assert_array_equal(filtered_vertices, vertices[[2, 3, 4]])
np.testing.assert_array_equal(filtered_faces, ((0, 1, 2),))
np.testing.assert_array_equal(filtered_uv, uv[[2, 3, 4]])
self.assertEqual(len(filtered_attributes["point_depth"]), 3)
self.assertEqual(len(filtered_attributes["face_background"]), 1)
def test_registers_image_background_operators(self):
self.assertIn(self.addon.RemoveImageBackground, self.addon.CLASSES)
self.assertIn(self.addon.ApplyRemovedBackground, self.addon.CLASSES)
@@ -481,7 +515,8 @@ class AddonTest(unittest.TestCase):
operator = self.addon.ConvertImageToDepthPlane()
operator.snapform_depth_source = "ORIGINAL"
operator.snapform_unshaded = False
operator.snapform_delete_original = False
operator.snapform_remove_background = True
operator.snapform_keep_original = True
source_object = SimpleNamespace(
name="Reference",
data=SimpleNamespace(name="img.png"),
@@ -513,7 +548,8 @@ class AddonTest(unittest.TestCase):
operator = self.addon.ConvertImageToDepthPlane()
operator.snapform_depth_source = "ORIGINAL"
operator.snapform_unshaded = False
operator.snapform_delete_original = False
operator.snapform_remove_background = True
operator.snapform_keep_original = True
source_object = SimpleNamespace(
name="Reference",
data=SimpleNamespace(
@@ -552,6 +588,39 @@ class AddonTest(unittest.TestCase):
self.assertFalse(calls[0]["snapform_delete_input"])
self.assertFalse(calls[0]["snapform_delete_source"])
def test_original_depth_without_background_uses_frame_extraction(self):
operator = self.addon.ConvertImageToDepthPlane()
operator.snapform_depth_source = "ORIGINAL"
operator.snapform_unshaded = False
operator.snapform_remove_background = False
operator.snapform_keep_original = False
source_object = SimpleNamespace(
name="Reference",
data=SimpleNamespace(name="img.png"),
)
context = SimpleNamespace(object=source_object, scene=object())
calls = []
self.fake_bpy.ops = SimpleNamespace(
snapform=SimpleNamespace(
run_job=lambda **options: calls.append(options)
or {"RUNNING_MODAL"},
),
)
with TemporaryDirectory() as directory:
input_path = Path(directory) / "input.png"
input_path.touch()
with patch.object(
self.addon.image_depth,
"prepare_image_input",
return_value=(input_path, False),
):
result = operator.execute(context)
self.assertEqual(result, {"FINISHED"})
self.assertEqual(calls[0]["job_type"], "ORIGINAL_FRAMES")
self.assertTrue(calls[0]["snapform_delete_source"])
def test_conversion_dialog_exposes_requested_settings(self):
operator = self.addon.ConvertImageToDepthPlane
@@ -562,10 +631,21 @@ class AddonTest(unittest.TestCase):
self.assertTrue(operator.__annotations__["snapform_use_ray_pose"]["default"])
self.assertEqual(
operator.__annotations__["snapform_process_res_factor"]["default"],
100.0,
)
self.assertEqual(
operator.__annotations__["snapform_process_res_factor"]["min"],
1.0,
)
self.assertTrue(
operator.__annotations__["snapform_delete_original"]["default"]
self.assertEqual(
operator.__annotations__["snapform_process_res_factor"]["max"],
100.0,
)
self.assertFalse(
operator.__annotations__["snapform_remove_background"]["default"]
)
self.assertFalse(
operator.__annotations__["snapform_keep_original"]["default"]
)
self.assertFalse(
operator.__annotations__["snapform_unshaded"]["default"]
@@ -578,7 +658,8 @@ class AddonTest(unittest.TestCase):
"snapform_model",
"snapform_use_ray_pose",
"snapform_process_res_factor",
"snapform_delete_original",
"snapform_remove_background",
"snapform_keep_original",
},
)
depth_sources = operator.__annotations__["snapform_depth_source"]
@@ -690,9 +771,10 @@ class AddonTest(unittest.TestCase):
operator.snapform_depth_source = "ESTIMATE"
operator.snapform_model = "BASE"
operator.snapform_use_ray_pose = True
operator.snapform_process_res_factor = 1.0
operator.snapform_process_res_factor = 50.0
operator.snapform_unshaded = False
operator.snapform_delete_original = False
operator.snapform_remove_background = True
operator.snapform_keep_original = True
calls = []
self.fake_bpy.ops = SimpleNamespace(
snapform=SimpleNamespace(
@@ -817,9 +899,10 @@ class AddonTest(unittest.TestCase):
operator.snapform_depth_source = "ESTIMATE"
operator.snapform_model = "GIANT_1_1"
operator.snapform_use_ray_pose = True
operator.snapform_process_res_factor = 0.5
operator.snapform_process_res_factor = 50.0
operator.snapform_unshaded = True
operator.snapform_delete_original = True
operator.snapform_remove_background = True
operator.snapform_keep_original = False
image = SimpleNamespace(
name="img.png",
filepath="",
@@ -855,7 +938,7 @@ class AddonTest(unittest.TestCase):
self.assertEqual(result, {"FINISHED"})
save_packed_image.assert_called_once_with(image, context.scene)
self.assertEqual(calls[0]["snapform_input_path"], str(packed_path))
self.assertEqual(calls[0]["snapform_process_res"], 960)
self.assertEqual(calls[0]["snapform_process_res"], 966)
self.assertEqual(calls[0]["snapform_model_key"], "GIANT_1_1")
self.assertTrue(calls[0]["snapform_remove_background"])
self.assertTrue(calls[0]["snapform_unshaded"])

View File

@@ -558,12 +558,29 @@ class ImageDepthTest(unittest.TestCase):
):
self.assertAlmostEqual(value, expected)
def test_process_resolution_scales_the_image_long_edge(self):
def test_process_resolution_scales_long_edge_and_aligns_to_patch_size(self):
image = SimpleNamespace(size=(1080, 1920))
self.assertEqual(
self.image_depth.scaled_process_resolution(image, 0.5),
960,
self.image_depth.scaled_process_resolution(image, 25.0),
476,
)
self.assertEqual(
self.image_depth.scaled_process_resolution(image, 50.0),
966,
)
def test_process_resolution_clamps_to_supported_range(self):
large_image = SimpleNamespace(size=(4000, 3000))
small_image = SimpleNamespace(size=(200, 100))
self.assertEqual(
self.image_depth.scaled_process_resolution(large_image, 100.0),
1008,
)
self.assertEqual(
self.image_depth.scaled_process_resolution(small_image, 1.0),
252,
)
def test_depth_data_uses_source_image_name(self):

View File

@@ -1340,6 +1340,20 @@ class WorkerTest(unittest.TestCase):
)
np.testing.assert_array_equal(compact_faces, [[0, 1, 2]])
def test_surface_background_faces_are_removed_before_export(self):
faces = np.asarray(
((0, 1, 4), (0, 4, 3), (1, 2, 5), (1, 5, 4)),
dtype=np.int32,
)
foreground = np.asarray(
((False, True, True), (False, True, True)),
dtype=bool,
)
filtered = self.worker.remove_background_faces(faces, foreground)
np.testing.assert_array_equal(filtered, faces[2:])
def test_image_mesh_data_keeps_camera_geometry_and_uv(self):
class FakeImage:
def __enter__(self):

View File

@@ -3,9 +3,8 @@ from pathlib import Path
import bpy
GROUP_NAME = "O Refine DA3 Surface"
GROUP_NAME = "O DA3 Surface"
DEPTH_PLANE_GROUP_NAME = "O DA3 Depth Plane"
REMOVE_BACKGROUND_ATTRIBUTE = "o_remove_background"
MATERIAL_DISPLACEMENT_ATTRIBUTE = "o_material_displacement"
BACKGROUND_DEPTH_ATTRIBUTE = "o_background_depth"
DEPTH_SCALE_ATTRIBUTE = "o_depth_scale"
@@ -424,14 +423,6 @@ def build_group():
edge.min_value = 0.0
edge.max_value = 100.0
edge.subtype = "PERCENTAGE"
background = new_socket(
interface,
"Background",
"INPUT",
"NodeSocketBool",
parent=culling_panel,
)
background.default_value = True
options_panel = interface.new_panel(
name="Options",
@@ -454,7 +445,6 @@ def build_group():
geometry_input = new_group_input(nodes, (-650.0, 50.0), {"Geometry"})
confidence_input = new_group_input(nodes, (-1800.0, 800.0), {"Confidence"})
stretch_input = new_group_input(nodes, (-1800.0, 50.0), {"Stretch"})
background_input = new_group_input(nodes, (-1800.0, -350.0), {"Background"})
remesh_input = new_group_input(
nodes,
(-100.0, -500.0),
@@ -523,25 +513,11 @@ def build_group():
links.new(edge_attribute.outputs["Attribute"], input_socket(edge_compare, "A", 0))
links.new(edge_reciprocal.outputs[0], input_socket(edge_compare, "B", 1))
background_attribute = nodes.new("GeometryNodeInputNamedAttribute")
background_attribute.data_type = "BOOLEAN"
background_attribute.inputs["Name"].default_value = "o_background_face"
background_attribute.location = (-1800.0, -200.0)
background_and = nodes.new("FunctionNodeBooleanMath")
background_and.operation = "AND"
background_and.location = (-1450.0, -200.0)
links.new(background_attribute.outputs["Attribute"], background_and.inputs[0])
links.new(background_input.outputs["Background"], background_and.inputs[1])
first_or = nodes.new("FunctionNodeBooleanMath")
first_or.operation = "OR"
first_or.location = (-950.0, 450.0)
links.new(confidence_compare.outputs["Result"], first_or.inputs[0])
links.new(edge_compare.outputs["Result"], first_or.inputs[1])
final_or = nodes.new("FunctionNodeBooleanMath")
final_or.operation = "OR"
final_or.location = (-750.0, 300.0)
links.new(first_or.outputs[0], final_or.inputs[0])
links.new(background_and.outputs[0], final_or.inputs[1])
full_stretch = new_compare(
nodes,
"FLOAT",
@@ -556,7 +532,7 @@ def build_group():
culling_selection = nodes.new("FunctionNodeBooleanMath")
culling_selection.operation = "OR"
culling_selection.location = (-550.0, 300.0)
links.new(final_or.outputs[0], culling_selection.inputs[0])
links.new(first_or.outputs[0], culling_selection.inputs[0])
links.new(full_stretch.outputs["Result"], culling_selection.inputs[1])
delete_geometry = nodes.new("GeometryNodeDeleteGeometry")
delete_geometry.domain = "FACE"
@@ -638,7 +614,7 @@ def build_group():
group.color_tag = "GEOMETRY"
group.description = (
"Cull DA3/BEN2 faces, repair merged topology, preserve boundaries, "
"Cull DA3 faces, repair merged topology, preserve boundaries, "
"and reproject UVs."
)
group.use_fake_user = True
@@ -707,6 +683,15 @@ def build_depth_plane_group():
"INPUT",
"NodeSocketMenu",
)
subdivision = new_socket(
interface,
"Subdivision",
"INPUT",
"NodeSocketInt",
)
subdivision.default_value = 5
subdivision.min_value = 0
subdivision.max_value = 10
displacement_strength = new_socket(
interface,
"Depth Scale",
@@ -717,39 +702,17 @@ def build_depth_plane_group():
displacement_strength.min_value = -10000.0
displacement_strength.max_value = 10000.0
displacement_strength.structure_type = "SINGLE"
subdivision = new_socket(
interface,
"Subdivision",
"INPUT",
"NodeSocketInt",
)
subdivision.default_value = 5
subdivision.min_value = 0
subdivision.max_value = 10
background_panel = interface.new_panel(
name="Remove Background",
default_closed=True,
)
remove_background = new_socket(
interface,
"Remove Background",
"INPUT",
"NodeSocketBool",
parent=background_panel,
)
remove_background.default_value = False
remove_background.is_panel_toggle = True
background_depth = new_socket(
interface,
"Background Depth",
"INPUT",
"NodeSocketFloat",
parent=background_panel,
)
background_depth.default_value = 0.0
background_depth.min_value = -10000.0
background_depth.max_value = 10000.0
background_depth.subtype = "DISTANCE"
background_depth.structure_type = "SINGLE"
data_panel = interface.new_panel(name="Data", default_closed=True)
depth_image = new_socket(
interface,
@@ -774,6 +737,7 @@ def build_depth_plane_group():
)
frame_data = []
for name, socket_type, default_value in (
("Frame", "NodeSocketInt", 1),
("Frame Start", "NodeSocketInt", 1),
("Frames", "NodeSocketInt", 1),
("Cyclic", "NodeSocketBool", False),
@@ -786,6 +750,8 @@ def build_depth_plane_group():
parent=data_panel,
)
socket.default_value = default_value
if name == "Frame":
socket.structure_type = "SINGLE"
if name == "Frames":
socket.min_value = 1
frame_data.append(socket)
@@ -809,6 +775,7 @@ def build_depth_plane_group():
nodes,
(50.0, 1750.0),
{
"Frame",
"Frame Start",
"Frames",
"Cyclic",
@@ -817,7 +784,7 @@ def build_depth_plane_group():
background_input = new_group_input(
nodes,
(350.0, 250.0),
{"Remove Background", "Background Depth"},
{"Background Depth"},
)
displacement_input = new_group_input(
nodes,
@@ -924,12 +891,10 @@ def build_depth_plane_group():
)
links.new(grid.outputs["UV Map"], depth_texture.inputs["Vector"])
links.new(grid.outputs["UV Map"], foreground_texture.inputs["Vector"])
scene_time = nodes.new("GeometryNodeInputSceneTime")
scene_time.location = (50.0, 2200.0)
sequence_frame = build_sequence_frame(
nodes,
links,
scene_time.outputs["Frame"],
frames_input.outputs["Frame"],
frames_input,
2200.0,
)
@@ -964,14 +929,6 @@ def build_depth_plane_group():
masked_depth.inputs[2],
)
links.new(clamp_foreground.outputs[0], masked_depth.inputs[3])
selected_depth = nodes.new("ShaderNodeMix")
selected_depth.data_type = "FLOAT"
selected_depth.location = (1000.0, 650.0)
remove_background_value = background_input.outputs["Remove Background"]
links.new(remove_background_value, selected_depth.inputs[0])
links.new(depth_texture.outputs["Color"], selected_depth.inputs[2])
links.new(masked_depth.outputs["Result"], selected_depth.inputs[3])
displacement_menu = nodes.new("GeometryNodeMenuSwitch")
displacement_menu.data_type = "FLOAT"
displacement_menu.enum_items.clear()
@@ -995,7 +952,7 @@ def build_depth_plane_group():
displacement_scale = nodes.new("ShaderNodeMath")
displacement_scale.operation = "MULTIPLY"
displacement_scale.location = (1200.0, 650.0)
links.new(selected_depth.outputs["Result"], displacement_scale.inputs[0])
links.new(masked_depth.outputs["Result"], displacement_scale.inputs[0])
geometry_strength = nodes.new("ShaderNodeMath")
geometry_strength.operation = "MULTIPLY"
geometry_strength.location = (1150.0, 450.0)
@@ -1024,11 +981,6 @@ def build_depth_plane_group():
shade_smooth.location = (1850.0, 650.0)
links.new(set_material.outputs["Geometry"], shade_smooth.inputs["Geometry"])
attribute_values = (
(
REMOVE_BACKGROUND_ATTRIBUTE,
"BOOLEAN",
remove_background_value,
),
(
MATERIAL_DISPLACEMENT_ATTRIBUTE,
"BOOLEAN",

View File

@@ -14,6 +14,7 @@ from snapform.image_depth import (
load_depth_plane_node_group,
modifier_input_identifier,
)
from snapform.image_mesh import load_geometry_node_group
def main():
@@ -27,6 +28,15 @@ def main():
and item.name == "Background Depth"
)
assert background_depth.subtype == "DISTANCE"
assert background_depth.structure_type == "SINGLE"
frame_input = next(
item
for item in group.interface.items_tree
if item.item_type == "SOCKET"
and item.in_out == "INPUT"
and item.name == "Frame"
)
assert frame_input.structure_type == "SINGLE"
geometry_input = next(
item
for item in group.interface.items_tree
@@ -45,22 +55,15 @@ def main():
assert visible_inputs[:6] == [
"Geometry",
"Displacement",
"Depth Scale",
"Subdivision",
"Remove Background",
"Depth Scale",
"Background Depth",
]
remove_background = next(
item
assert not getattr(background_depth.parent, "name", "")
assert not any(
item.name == "Remove Background"
for item in group.interface.items_tree
if item.item_type == "SOCKET"
and item.in_out == "INPUT"
and item.name == "Remove Background"
)
assert remove_background.socket_type == "NodeSocketBool"
assert remove_background.is_panel_toggle
assert remove_background.parent.name == "Remove Background"
assert background_depth.parent == remove_background.parent
depth_scale = next(
item
for item in group.interface.items_tree
@@ -77,7 +80,6 @@ def main():
if node.bl_idname == "GeometryNodeStoreNamedAttribute"
}
attribute_names = (
"o_remove_background",
"o_material_displacement",
"o_background_depth",
"o_depth_scale",
@@ -160,14 +162,31 @@ def main():
and node.node_tree == material_group
]
assert len(material_group_nodes) == 1
assert material_group_nodes[0].name == "O DA3 Depth Plane"
assert material_group_nodes[0].name == "O DA3 Depth Plane Layer"
bump = next(
node
for node in material_group.nodes
if node.bl_idname == "ShaderNodeBump"
)
bump_height_link = next(
link
for link in material_group.links
if link.to_node == bump and link.to_socket == bump.inputs["Height"]
)
assert bump_height_link.from_node.bl_idname == "NodeGroupInput"
assert bump_height_link.from_socket.name == "Color"
attributes = [
node
for node in material_group.nodes
if node.bl_idname == "ShaderNodeAttribute"
]
assert len(attributes) == 4
assert len(attributes) == 3
assert all(node.attribute_type == "GEOMETRY" for node in attributes)
assert {node.attribute_name for node in attributes} == {
"o_material_displacement",
"o_background_depth",
"o_depth_scale",
}
assert not any(
node.bl_idname == "ShaderNodeAttribute"
for node in material.node_tree.nodes
@@ -193,7 +212,25 @@ def main():
and node.node_tree == surface_group
]
assert len(surface_group_nodes) == 1
assert surface_group_nodes[0].name == "O DA3 Surface"
assert surface_group_nodes[0].name == "O DA3 Surface Layer"
surface_outputs = [
item.name
for item in surface_group.interface.items_tree
if item.item_type == "SOCKET"
and item.in_out == "OUTPUT"
]
assert surface_outputs == ["Color", "Roughness", "Alpha", "Normal"]
assert not any(
node.bl_idname == "ShaderNodeDisplacement"
for node in surface_group.nodes
)
refine_group = load_geometry_node_group()
assert not any(
item.item_type == "SOCKET"
and item.in_out == "INPUT"
and item.name == "Background"
for item in refine_group.interface.items_tree
)
surface_depth_input = next(
item
for item in surface_group.interface.items_tree
@@ -206,7 +243,13 @@ def main():
node.bl_idname in forbidden_outer_nodes
for node in surface_material.node_tree.nodes
)
modifier = add_depth_plane_processing(obj, depth, foreground, material)
modifier = add_depth_plane_processing(
obj,
depth,
foreground,
material,
animate_frame=True,
)
modifier[modifier_input_identifier(group, "Depth Scale")] = -2.0
for name, expected in (
("Frame Start", 1),
@@ -214,6 +257,20 @@ def main():
("Cyclic", False),
):
assert modifier[modifier_input_identifier(group, name)] == expected
assert not any(
node.bl_idname == "GeometryNodeInputSceneTime"
for node in group.nodes
)
frame_identifier = modifier_input_identifier(group, "Frame")
frame_driver = next(
fcurve
for fcurve in obj.animation_data.drivers
if fcurve.data_path == (
f'modifiers["{modifier.name}"]["{frame_identifier}"]'
)
)
assert frame_driver.driver.type == "SCRIPTED"
assert frame_driver.driver.expression == "frame"
image_textures = [
node
for node in group.nodes

View File

@@ -51,7 +51,7 @@ def main():
group_inputs = [
node for node in group.nodes if node.bl_idname == "NodeGroupInput"
]
assert len(group_inputs) >= 8
assert len(group_inputs) >= 7
for node in group_inputs:
visible_outputs = [
output for output in node.outputs if output.name and not output.hide
@@ -75,7 +75,6 @@ def main():
assert [item.name for item in panels["Culling"].interface_items] == [
"Confidence",
"Stretch",
"Background",
]
assert [item.name for item in panels["Options"].interface_items] == [
"Fix Topology",
@@ -116,7 +115,7 @@ def main():
evaluated = obj.evaluated_get(bpy.context.evaluated_depsgraph_get())
result = evaluated.to_mesh()
try:
assert len(result.polygons) == 1
assert len(result.polygons) == 2
assert "UVMap" in result.attributes
assert result.attributes["UVMap"].data_type == "FLOAT2"
finally: