icrdr/AnyImage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor and enhance DA3 and BEN2 debug panels, introduce mesh export functionality

Browse Source 
- Renamed DA3 and BEN2 debug panels for clarity and consistency.
- Added a new function to draw DA3 input settings to reduce code duplication.
- Implemented mesh export functionality, including confidence thresholding and edge filtering.
- Updated export format options to include new mesh type and adjusted related properties.
- Enhanced job handling to move generated inputs into job root and manage temporary directories.
- Improved error handling and file management during export processes.
- Added unit tests for new mesh functionality and ensured existing tests cover changes.
This commit is contained in:
马南
2026-08-09 21:31:08 +08:00
parent 8aac4822c6
commit 6d4d1c4ec2
10 changed files with 778 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@@ -45,8 +45,10 @@ SnapForm 支持视频（mp4、mkv、mov 等）和序列帧，会逐帧处理并
## 任务结果保存在哪里
任务结果保存在 `~/.snapform/jobs/` 下，每个任务一个文件夹。视频或序列帧生成的面片会引用这些文件夹
里的序列帧，请保留对应文件夹，不要随意删除。
任务结果保存在 `~/.snapform/jobs/` 下，每个任务一个文件夹，名称统一为
`动作-对象_日期-时间-微秒`。Packed Image 和序列帧的输入副本会直接归档到对应任务根目录，
不再使用公共 `inputs/` 文件夹；导出产物也尽量平铺。视频或序列帧生成的面片会引用任务文件夹里的
序列帧，请保留对应文件夹，不要随意删除。
## 常见问题

88
docs/technical.md
View File

@@ -23,13 +23,15 @@
- 视频按顺序截取前若干帧，可设置参与重建的最大帧数
- `Host > Models` 默认折叠，展开后可逐项查看和下载所有可用模型
- Runtime 已就绪时不额外占用面板空间；模型列表每个模型只占一行
- `Debug` 默认折叠，内部的 DA3 与 BEN2 测试项也分别默认折叠
- 使用一个导出类型下拉菜单和一个 `Export` 按钮
- `Debug` 默认折叠，内部的 `DA3 Output Test` 与 `BEN2 Removal Test` 也分别默认折叠
- 使用一个输出类型下拉菜单和一个 `Generate Output` 按钮
- 只显示当前格式的专属参数；GLB 与 PLY 设置不会同时出现
- Debug 的模型选项按格式过滤；PLY 只提供两个带 Gaussian head 的 Giant 模型
- Debug 与 Convert to Depth Plane 的模型选项直接显示下载状态；未下载模型带
`(Missing)`，已下载模型保持原名
- 分别调用 DA3 的 `glb`、`gs_ply`、`depth_vis` 和 `mini_npz` exporter
- `Textured Mesh (GLB)` 直接使用 DA3 Prediction 中的深度、置信度和相机参数，把相邻像素
组成三角面并写入 `mesh.glb`；每个输入帧生成一张独立纹理网格，不做多视图融合
- Image Empty 右键菜单可直接运行 DA3，根据 `mini_npz` 深度生成面片
- Image Empty 右键菜单可直接运行 BEN2，保留原图并换用新建的透明背景图片
- Image Empty 支持视频与序列帧输入：`Convert to Depth Plane` 逐帧估计深度，
@@ -39,7 +41,7 @@
- 使用浮点深度贴图、Simple Subdivision、Displace Modifier 和原图 Bump
构建立体面片
- 转换对话框可选择推理模式、处理分辨率和面片网格细分
- Debug 的 DA3 导出或 BEN2 背景剔除完成后只打开对应 job 文件夹，不向当前 Blender 文件添加任何数据
- Debug 的 DA3 输出或 BEN2 背景剔除完成后只打开对应 job 文件夹，不修改当前场景
- 完整保留每次任务生成的所有官方文件
- 安装和模型下载使用后台子进程；推理提交给常驻 AI Host，状态栏实时显示阶段、百分比进度条与取消按钮
- 后台任务运行时，Install、Download 和 Export 按钮保持原位置并变灰，任务完成后恢复
@@ -69,10 +71,10 @@
自行探测回退到 HF 镜像；需要手动指定镜像时可通过 `PIP_INDEX_URL` 环境变量覆盖。
5. 在 3D View 按 `N`，打开 `SnapForm` 页签。
6. 如需其他模型，展开 `Host > Models`，找到模型并点击该行的 `Download`。
7. 展开默认收起的 `Debug`，选择 `Export Format` 和本次执行使用的模型。
8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数，然后点击 `Export`。
7. 展开默认收起的 `Debug`，选择 `Output Type` 和本次执行使用的模型。
8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数，然后点击 `Generate Output`。
9. 测试背景剔除时，确认 `Host > Models` 中的 `BEN2 Base` 已就绪，然后在 `Debug` 的
`BEN2 Background Removal` 区域选择单张图片并点击 `Remove Background`。
`BEN2 Removal Test` 区域选择单张图片并点击 `Remove Background`。
完成后会打开 job 文件夹，其中的 `foreground.png` 带透明 Alpha；状态栏显示
BEN2 推理、模型加载与任务总耗时。当前测试固定关闭较慢的 foreground refinement。
@@ -218,33 +220,52 @@ Image Empty 使用 Packed Image 时，SnapForm 会通过 Blender 写出临时 PN
`DA3 Nested Giant Large 1.1`。Small、Base 和 Large 没有 Gaussian head，无法由
DA3 生成 `gs_ply`。
每个任务在 `~/.snapform/jobs/` 下用独立文件夹包裹（命名格式为 `任务类型_时间号`，例如
`depth-plane_20260807-153000`），日志、状态与输出产物都放在该文件夹内：
每个任务在 `~/.snapform/jobs/` 下用独立文件夹包裹。文件夹统一采用
`动作-对象_日期-时间-微秒`，每种任务都有唯一名称，例如
`generate-depth-plane_20260807-153000-123456`。微秒用于避免同一秒启动的任务共用目录。
Blender 为 Packed Image 或序列帧生成的输入副本也归档到对应 job 根目录：单图命名为
`input.png`，序列帧命名为 `frame.0001.*`。它们以前暂存在 `jobs/inputs/`，现在不再创建这个
公共中转文件夹。普通磁盘文件和视频仍直接读取原文件，不额外复制。
日志、运行中的状态文件、归档输入与输出产物均直接放在 job 根目录。各类任务名称为：
- `install-runtime`
- `download-model`
- `export-reconstruction`
- `generate-depth-plane`
- `remove-background`
- `remove-image-background`
- `generate-original-depth-plane`
- `extract-original-frames`
DA3 官方 exporter 产生的 `gs_ply/`、`depth_vis/`、`exports/mini_npz/` 会在完成后自动平铺，
不会保留只含一层产物的子目录：
```text
reconstruct_20260807-153000/
├── reconstruct_20260807-153000.log
├── reconstruct_20260807-153000.json
export-reconstruction_20260807-153000-123456/
├── export-reconstruction_20260807-153000-123456.log
├── export-reconstruction_20260807-153000-123456.json
└── scene.glb
reconstruct_20260807-153000/
├── reconstruct_20260807-153000.log
├── reconstruct_20260807-153000.json
└── gs_ply/*.ply
export-reconstruction_20260807-153000-123456/
├── export-reconstruction_20260807-153000-123456.log
├── export-reconstruction_20260807-153000-123456.json
└── *.ply
reconstruct_20260807-153000/
├── reconstruct_20260807-153000.log
├── reconstruct_20260807-153000.json
└── depth_vis/*.jpg
export-reconstruction_20260807-153000-123456/
├── export-reconstruction_20260807-153000-123456.log
├── export-reconstruction_20260807-153000-123456.json
└── *.jpg
reconstruct_20260807-153000/
├── reconstruct_20260807-153000.log
├── reconstruct_20260807-153000.json
export-reconstruction_20260807-153000-123456/
├── export-reconstruction_20260807-153000-123456.log
├── export-reconstruction_20260807-153000-123456.json
└── results.npz
remove-background_20260807-153000/
├── remove-background_20260807-153000.log
├── remove-background_20260807-153000.json
remove-background_20260807-153000-123456/
├── remove-background_20260807-153000-123456.log
├── remove-background_20260807-153000-123456.json
├── foreground.png 单图结果
├── foreground.0001.png… 视频/序列帧结果
├── frame.0001.jpg… 视频/序列帧的源帧副本
@@ -256,15 +277,16 @@ CC BY-NC 4.0，仅限非商业用途。
## 输出格式说明
`Export Format` 下拉菜单分别把以下原始 `export_format` 传给 DA3：
`Output Type` 提供以下结果：
- GLB：`glb`
- PLY：`gs_ply`，同时传入 `infer_gs=True`
- Depth Images：`depth_vis`
- Mini NPZ：`mini_npz`
- Point Cloud (GLB)：DA3 `glb`
- Gaussian Splats (PLY)：DA3 `gs_ply`，同时传入 `infer_gs=True`
- Depth Preview (Images)：DA3 `depth_vis`
- Depth Data (NPZ)：DA3 `mini_npz`
- Textured Mesh (GLB)：SnapForm 根据 DA3 Prediction 构建并输出 `mesh.glb`
界面使用 DA3 API 的参数名、选项和默认值。GLB 专属设置仅在选择 `GLB` 时显示，
`gs_views_interval` 仅在选择 `PLY (3DGS)` 时显示：
Point Cloud 专属设置仅在对应类型下显示，`gs_views_interval` 仅用于 Gaussian Splats。
Textured Mesh 使用 0–100% 的 Confidence Threshold 删除置信度最低的一定比例：
- `process_res=504`
- `process_res_method="upper_bound_resize"`
@@ -277,7 +299,7 @@ CC BY-NC 4.0，仅限非商业用途。
DA3 官方 `glb` 是彩色点云与可选相机线框；`gs_ply` 是带 Gaussian 属性的 PLY；
`depth_vis` 是原图与着色深度图的并排预览；`mini_npz` 保存深度、置信度和可用的相机参数。
GLB 任务会关闭其默认附带的深度预览，使四种 job 产物保持分离。
Point Cloud GLB 任务会关闭其默认附带的深度预览，使五种输出产物保持分离。
官方格式与限制可参阅
[DA3 Python API 文档](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)。

8
src/snapform/__init__.py
View File

@@ -17,8 +17,8 @@ from .operators import (
close_active_job,
)
from .panel import (
Ben2DebugPanel,
Da3DebugPanel,
Ben2RemovalDebugPanel,
Da3OutputDebugPanel,
DebugPanel,
HostPanel,
ModelsPanel,
@@ -42,8 +42,8 @@ CLASSES = (
HostPanel,
ModelsPanel,
DebugPanel,
Da3DebugPanel,
Ben2DebugPanel,
Da3OutputDebugPanel,
Ben2RemovalDebugPanel,
)

46
src/snapform/operators.py
View File

@@ -122,6 +122,7 @@ def close_active_job():
shutil.rmtree(temporary_path, ignore_errors=True)
else:
temporary_path.unlink(missing_ok=True)
temporary_path.parent.rmdir()
except OSError:
pass
_active_job = None
@@ -221,6 +222,12 @@ class RunJob(bpy.types.Operator):
self.report({"ERROR"}, str(error))
return {"CANCELLED"}
export_format = ""
if self.job_type == RECONSTRUCTION_JOB:
export_format = settings.export_format
elif self.job_type == IMAGE_DEPTH_JOB:
export_format = "mini_npz"
job = ActiveJob(
kind=self.job_type,
process=None,
@@ -228,11 +235,7 @@ class RunJob(bpy.types.Operator):
log_path=log_path,
status_path=paths["status"],
result_directory=paths["result"],
export_format=(
settings.export_format if self.job_type == RECONSTRUCTION_JOB
else "mini_npz" if self.job_type == IMAGE_DEPTH_JOB
else ""
),
export_format=export_format,
source_object_name=self.snapform_source_object_name,
depth_data_name=self.snapform_depth_data_name,
process_res=self.snapform_process_res,
@@ -310,6 +313,21 @@ class RunJob(bpy.types.Operator):
def _prepare_command(self, _context, settings):
paths = runtime.new_job_paths(self.job_type)
if (
getattr(self, "snapform_delete_input", False)
and self.job_type in {
IMAGE_DEPTH_JOB,
IMAGE_BACKGROUND_JOB,
ORIGINAL_BACKGROUND_JOB,
ORIGINAL_FRAMES_JOB,
}
):
archived_input = runtime.move_input_into_job(
self.snapform_input_path,
paths["result"],
)
self.snapform_input_path = str(archived_input)
self.snapform_delete_input = False
background_depth_distance = float(
getattr(self, "snapform_background_depth_distance", 0.0)
)
@@ -389,13 +407,14 @@ class RunJob(bpy.types.Operator):
),
paths,
)
if self.job_type not in {RECONSTRUCTION_JOB, IMAGE_DEPTH_JOB}:
if self.job_type not in {
RECONSTRUCTION_JOB,
IMAGE_DEPTH_JOB,
}:
raise RuntimeError(f"Unsupported task type: {self.job_type}")
export_format = (
"mini_npz"
if self.job_type == IMAGE_DEPTH_JOB
else settings.export_format
)
export_format = settings.export_format
if self.job_type == IMAGE_DEPTH_JOB:
export_format = "mini_npz"
use_ray_pose = None
process_res = None
model_key = selected_model_key(settings, export_format)
@@ -614,7 +633,10 @@ class RunJob(bpy.types.Operator):
self._redraw_ui(context, force=True)
return {"CANCELLED"}
if job.kind in {RECONSTRUCTION_JOB, BACKGROUND_REMOVAL_JOB}:
if job.kind in {
RECONSTRUCTION_JOB,
BACKGROUND_REMOVAL_JOB,
}:
try:
open_job_directory(job.result_directory)
except Exception as open_error:

68
src/snapform/panel.py
View File

@@ -160,9 +160,33 @@ class DebugPanel(bpy.types.Panel):
layout.prop(settings, "device")
class Da3DebugPanel(bpy.types.Panel):
bl_label = "DA3 Reconstruction"
bl_idname = "SNAPFORM_PT_debug_da3"
def draw_da3_input_settings(layout, settings, model_property):
layout.prop(settings, model_property)
layout.prop(settings, "input_path")
input_path = bpy.path.abspath(settings.input_path)
if input_path.lower().endswith(
(
".mp4",
".mov",
".avi",
".mkv",
".webm",
".m4v",
".mts",
".mpg",
".mpeg",
)
):
layout.prop(settings, "video_frames")
layout.prop(settings, "process_res")
layout.prop(settings, "process_res_method")
layout.prop(settings, "use_ray_pose")
layout.prop(settings, "ref_view_strategy")
class Da3OutputDebugPanel(bpy.types.Panel):
bl_label = "DA3 Output Test"
bl_idname = "SNAPFORM_PT_debug_da3_output"
bl_parent_id = DebugPanel.bl_idname
bl_space_type = "VIEW_3D"
bl_region_type = "UI"
@@ -178,28 +202,7 @@ class Da3DebugPanel(bpy.types.Panel):
model_property = (
"gs_model" if settings.export_format == "gs_ply" else "model"
)
layout.prop(settings, model_property)
layout.prop(settings, "input_path")
input_path = bpy.path.abspath(settings.input_path)
if input_path.lower().endswith(
(
".mp4",
".mov",
".avi",
".mkv",
".webm",
".m4v",
".mts",
".mpg",
".mpeg",
)
):
layout.prop(settings, "video_frames")
layout.prop(settings, "process_res")
layout.prop(settings, "process_res_method")
layout.prop(settings, "use_ray_pose")
layout.prop(settings, "ref_view_strategy")
draw_da3_input_settings(layout, settings, model_property)
if settings.export_format == "glb":
glb_box = layout.box()
@@ -211,7 +214,12 @@ class Da3DebugPanel(bpy.types.Panel):
ply_box = layout.box()
ply_box.label(text="PLY (3DGS) Settings")
ply_box.prop(settings, "gs_views_interval")
elif settings.export_format == "mesh":
mesh_box = layout.box()
mesh_box.label(text="Mesh Settings")
mesh_box.prop(settings, "conf_thresh_percentile")
mesh_box.prop(settings, "mesh_filter_edges")
mesh_box.prop(settings, "mesh_max_edge_length")
export_row = layout.row()
export_row.enabled = (
active_job() is None
@@ -220,15 +228,15 @@ class Da3DebugPanel(bpy.types.Panel):
)
operator = export_row.operator(
"snapform.run_job",
text="Export",
text="Generate Output",
icon="EXPORT",
)
operator.job_type = "RECONSTRUCT"
class Ben2DebugPanel(bpy.types.Panel):
bl_label = "BEN2 Background Removal"
bl_idname = "SNAPFORM_PT_debug_ben2"
class Ben2RemovalDebugPanel(bpy.types.Panel):
bl_label = "BEN2 Removal Test"
bl_idname = "SNAPFORM_PT_debug_ben2_removal"
bl_parent_id = DebugPanel.bl_idname
bl_space_type = "VIEW_3D"
bl_region_type = "UI"

48
src/snapform/properties.py
View File

@@ -66,8 +66,8 @@ class SnapFormSettings(bpy.types.PropertyGroup):
default="upper_bound_resize",
)
conf_thresh_percentile: bpy.props.FloatProperty(
name="Conf Thresh Percentile",
description="DA3 GLB confidence threshold percentile",
name="Confidence Threshold",
description="Remove the lowest-confidence percentile of points or mesh faces",
default=40.0,
min=0.0,
max=100.0,
@@ -110,24 +110,52 @@ class SnapFormSettings(bpy.types.PropertyGroup):
default="saddle_balanced",
)
export_format: bpy.props.EnumProperty(
name="Export Format",
description="Official DA3 export format",
name="Output Type",
description="Result generated by the DA3 debug task",
items=(
("glb", "GLB", "Export a colored point cloud as GLB"),
("gs_ply", "PLY (3DGS)", "Export Gaussian splats as PLY"),
(
"glb",
"Point Cloud (GLB)",
"Generate a colored point cloud and optional cameras",
),
(
"gs_ply",
"Gaussian Splats (PLY)",
"Generate a 3D Gaussian Splatting PLY",
),
(
"depth_vis",
"Depth Images",
"Export image and depth visualization pairs",
"Depth Preview (Images)",
"Generate image and colored-depth preview pairs",
),
(
"mini_npz",
"Mini NPZ",
"Export compact depth and camera data",
"Depth Data (NPZ)",
"Generate compact depth, confidence, and camera data",
),
(
"mesh",
"Textured Mesh (GLB)",
"Generate one filtered textured depth mesh per input image",
),
),
default="glb",
)
mesh_filter_edges: bpy.props.BoolProperty(
name="Filter Depth Edges",
description="Remove faces around sharp depth transitions",
default=True,
)
mesh_max_edge_length: bpy.props.FloatProperty(
name="Max Edge Length",
description=(
"Remove faces with an edge longer than this scene distance; "
"zero disables the test"
),
default=0.1,
min=0.0,
soft_max=2.0,
)
gs_views_interval: bpy.props.IntProperty(
name="GS Views Interval",
description="Export 3D Gaussians every N input views",

52
src/snapform/runtime.py
View File

@@ -6,6 +6,7 @@ import shutil
import socket
import subprocess
import sys
import tempfile
import threading
import time
import uuid
@@ -34,9 +35,12 @@ DEFAULT_STORAGE_ROOT = Path.home() / ".snapform"
JOB_FOLDER_NAMES = {
"INSTALL_RUNTIME": "install-runtime",
"DOWNLOAD_MODEL": "download-model",
"IMAGE_DEPTH": "depth-plane",
"IMAGE_BACKGROUND": "remove-background",
"ORIGINAL_FRAMES": "depth-plane",
"RECONSTRUCT": "export-reconstruction",
"IMAGE_DEPTH": "generate-depth-plane",
"REMOVE_BACKGROUND": "remove-background",
"IMAGE_BACKGROUND": "remove-image-background",
"ORIGINAL_BACKGROUND": "generate-original-depth-plane",
"ORIGINAL_FRAMES": "extract-original-frames",
}
@@ -196,15 +200,37 @@ def jobs_directory():
def temporary_input_path():
path = jobs_directory() / "inputs"
path.mkdir(parents=True, exist_ok=True)
return path / f"{uuid.uuid4().hex}.png"
directory = Path(tempfile.mkdtemp(prefix="snapform-input-"))
return directory / "input.png"
def temporary_input_directory():
path = jobs_directory() / "inputs" / f"{uuid.uuid4().hex}"
path.mkdir(parents=True, exist_ok=True)
return path
return Path(tempfile.mkdtemp(prefix="snapform-input-"))
def move_input_into_job(input_path, job_directory):
"""Move a Blender-generated input copy into the flat job directory."""
input_path = Path(input_path)
job_directory = Path(job_directory)
if input_path.is_file():
destination = job_directory / "input.png"
input_path.replace(destination)
try:
input_path.parent.rmdir()
except OSError:
pass
return destination
if not input_path.is_dir():
raise ValueError(f"Temporary input does not exist: {input_path}")
for source in input_path.iterdir():
if not source.is_file():
raise ValueError(f"Temporary input contains a directory: {source}")
destination = job_directory / source.name
if destination.exists():
raise ValueError(f"Job input filename already exists: {source.name}")
source.replace(destination)
input_path.rmdir()
return job_directory
def job_folder_name(job_type):
@@ -213,7 +239,7 @@ def job_folder_name(job_type):
def new_job_paths(job_type):
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S-%f")
stem = f"{job_folder_name(job_type)}_{timestamp}"
directory = jobs_directory() / stem
directory.mkdir(parents=True, exist_ok=True)
@@ -824,6 +850,12 @@ def reconstruct_request(
"ref_view_strategy": settings.ref_view_strategy,
"remove_background_model_dir": "",
"background_depth_distance": float(background_depth_distance),
"mesh_filter_edges": bool(
getattr(settings, "mesh_filter_edges", True)
),
"mesh_max_edge_length": float(
getattr(settings, "mesh_max_edge_length", 0.1)
),
}
if remove_background or background_depth_distance > 0.0:
parameters["remove_background_model_dir"] = str(

320
src/snapform/worker.py
View File

@@ -24,7 +24,7 @@ VIDEO_SUFFIXES = {
".mpeg",
}
SCENE_FILENAME = "scene.glb"
EXPORT_FORMATS = ("glb", "gs_ply", "depth_vis", "mini_npz")
EXPORT_FORMATS = ("glb", "gs_ply", "depth_vis", "mini_npz", "mesh")
HF_ENDPOINT = "https://huggingface.co"
MODELSCOPE_ENDPOINT = "https://modelscope.cn"
MODELSCOPE_REVISION = "master"
@@ -340,10 +340,211 @@ def copy_frames(images, output_directory):
return
for index, image_path in enumerate(images):
suffix = Path(image_path).suffix.lower() or ".jpg"
shutil.copy2(
image_path,
Path(output_directory) / f"frame.{index + 1:04d}{suffix}",
destination = (
Path(output_directory) / f"frame.{index + 1:04d}{suffix}"
)
if Path(image_path).resolve() == destination.resolve():
continue
shutil.copy2(image_path, destination)
def unproject_depth_frame(depth, intrinsics, extrinsics):
import numpy as np
height, width = depth.shape
x, y = np.meshgrid(
np.arange(width, dtype=np.float32),
np.arange(height, dtype=np.float32),
)
pixels = np.stack((x, y, np.ones_like(x)), axis=-1).reshape(-1, 3)
rays = (np.linalg.inv(intrinsics) @ pixels.T).T
camera_points = rays * depth.reshape(-1, 1)
camera_points = np.column_stack(
(camera_points, np.ones(camera_points.shape[0], dtype=np.float32))
)
world_to_camera = np.eye(4, dtype=np.float32)
world_to_camera[: extrinsics.shape[0], :4] = extrinsics
world_points = (np.linalg.inv(world_to_camera) @ camera_points.T).T
return world_points[:, :3].reshape(height, width, 3)
def depth_edge_mask(depth, threshold=12.0 / 255.0):
import numpy as np
padded = np.pad(depth.astype(np.float32), 1, mode="edge")
grad_x = (
-padded[:-2, :-2]
+ padded[:-2, 2:]
- 2.0 * padded[1:-1, :-2]
+ 2.0 * padded[1:-1, 2:]
- padded[2:, :-2]
+ padded[2:, 2:]
)
grad_y = (
-padded[:-2, :-2]
- 2.0 * padded[:-2, 1:-1]
- padded[:-2, 2:]
+ padded[2:, :-2]
+ 2.0 * padded[2:, 1:-1]
+ padded[2:, 2:]
)
magnitude = np.hypot(grad_x, grad_y)
finite = np.isfinite(magnitude)
if not finite.any():
return np.ones(depth.shape, dtype=bool)
minimum = float(magnitude[finite].min())
maximum = float(magnitude[finite].max())
if maximum <= minimum:
return np.zeros(depth.shape, dtype=bool)
normalized = (magnitude - minimum) / (maximum - minimum)
return normalized >= threshold
def confidence_percentile_threshold(depth, confidence, percentile):
import numpy as np
valid = np.isfinite(depth) & (depth > 0.0) & np.isfinite(confidence)
if not valid.any():
raise ValueError("DA3 returned no valid depth confidence values")
return float(np.percentile(confidence[valid], percentile))
def mesh_triangle_faces(
points,
depth,
confidence,
confidence_threshold,
filter_edges,
max_edge_length,
):
import numpy as np
height, width = depth.shape
indices = np.arange(height * width, dtype=np.int32).reshape(height, width)
quads = np.stack(
(
indices[:-1, :-1],
indices[1:, :-1],
indices[1:, 1:],
indices[:-1, 1:],
),
axis=-1,
).reshape(-1, 4)
valid = (
np.isfinite(points).all(axis=-1)
& np.isfinite(depth)
& (depth > 0.0)
& (confidence >= confidence_threshold)
)
if filter_edges:
valid &= ~depth_edge_mask(depth)
face_valid = valid.reshape(-1)[quads].all(axis=1)
if max_edge_length > 0.0:
flat_points = points.reshape(-1, 3)
quad_points = flat_points[quads]
next_points = np.roll(quad_points, -1, axis=1)
edge_lengths = np.linalg.norm(quad_points - next_points, axis=-1)
face_valid &= (edge_lengths <= max_edge_length).all(axis=1)
quads = quads[face_valid]
if not len(quads):
return np.empty((0, 3), dtype=np.int32)
return np.concatenate(
(quads[:, (0, 1, 2)], quads[:, (0, 2, 3)]),
axis=0,
)
def export_textured_mesh_glb(
prediction,
image_paths,
output_directory,
confidence_percentile,
filter_edges,
max_edge_length,
):
import numpy as np
import trimesh
from PIL import Image
required = ("depth", "conf", "intrinsics", "extrinsics")
missing = [name for name in required if getattr(prediction, name) is None]
if missing:
raise ValueError(f"DA3 mesh output is missing: {', '.join(missing)}")
depth = np.asarray(prediction.depth, dtype=np.float32)
confidence = np.asarray(prediction.conf, dtype=np.float32)
threshold = confidence_percentile_threshold(
depth,
confidence,
confidence_percentile,
)
meshes = []
referenced_points = []
for index in range(depth.shape[0]):
points = unproject_depth_frame(
depth[index],
prediction.intrinsics[index],
prediction.extrinsics[index],
)
faces = mesh_triangle_faces(
points,
depth[index],
confidence[index],
threshold,
filter_edges,
max_edge_length,
)
if not len(faces):
continue
height, width = depth[index].shape
u = np.linspace(0.0, 1.0, width, dtype=np.float32)
v = np.linspace(1.0, 0.0, height, dtype=np.float32)
uu, vv = np.meshgrid(u, v)
uv = np.stack((uu, vv), axis=-1).reshape(-1, 2)
with Image.open(image_paths[index]) as source_image:
texture_image = source_image.convert("RGB").copy()
visual = trimesh.visual.texture.TextureVisuals(
uv=uv,
image=texture_image,
)
mesh = trimesh.Trimesh(
vertices=points.reshape(-1, 3),
faces=faces,
visual=visual,
process=False,
)
mesh.remove_unreferenced_vertices()
meshes.append(mesh)
referenced_points.append(mesh.vertices)
if not meshes:
raise ValueError(
"Mesh filters removed every face; lower Confidence Threshold or disable filters"
)
world_to_camera = np.asarray(prediction.extrinsics[0], dtype=np.float64)
if world_to_camera.shape == (3, 4):
homogeneous = np.eye(4, dtype=np.float64)
homogeneous[:3, :4] = world_to_camera
world_to_camera = homogeneous
cv_to_gltf = np.eye(4, dtype=np.float64)
cv_to_gltf[1, 1] = -1.0
cv_to_gltf[2, 2] = -1.0
alignment = cv_to_gltf @ world_to_camera
aligned_points = trimesh.transform_points(
np.concatenate(referenced_points, axis=0),
alignment,
)
center = np.median(aligned_points, axis=0)
alignment[:3, 3] -= center
scene = trimesh.Scene()
for index, mesh in enumerate(meshes):
mesh.apply_transform(alignment)
scene.add_geometry(mesh, node_name=f"DepthMesh.{index + 1:04d}")
output_directory = Path(output_directory)
output_directory.mkdir(parents=True, exist_ok=True)
output_path = output_directory / "mesh.glb"
scene.export(output_path)
return output_path
class DownloadProgressReporter:
@@ -856,7 +1057,7 @@ def remove_background(args, model_cache=None, cancel_check=None):
images, temporary_directory = prepare_background_input(
args.input,
video_frames,
output_directory.parent,
output_directory,
start_frame=start_frame,
cancel_check=cancel_check,
)
@@ -933,7 +1134,7 @@ def extract_frames(args, cancel_check=None):
images, temporary_directory = prepare_input(
args.input,
getattr(args, "video_frames", 16),
output_directory.parent,
output_directory,
start_frame=getattr(args, "start_frame", 1),
cancel_check=cancel_check,
)
@@ -956,6 +1157,9 @@ def exported_paths(output_directory, export_format):
return sorted((output_directory / "gs_ply").glob("*.ply"))
if export_format == "depth_vis":
return sorted((output_directory / "depth_vis").glob("*.jpg"))
if export_format == "mesh":
result_path = output_directory / "mesh.glb"
return [result_path] if result_path.is_file() else []
if export_format == "mini_npz":
result_path = output_directory / "results.npz"
if result_path.is_file():
@@ -1002,6 +1206,70 @@ def wait_for_exported_paths(
return []
def move_exported_file(
source,
destination,
timeout=10.0,
cancel_check=None,
):
"""Move an async export after Windows releases its file handle."""
source = Path(source)
destination = Path(destination)
deadline = time.monotonic() + timeout
while True:
check_cancelled(cancel_check)
try:
source.replace(destination)
return destination
except OSError as error:
is_file_lock = isinstance(error, PermissionError) or getattr(
error,
"winerror",
None,
) in {32, 33}
if not is_file_lock or time.monotonic() >= deadline:
raise
time.sleep(0.1)
def flatten_exported_paths(
output_directory,
result_paths,
cancel_check=None,
):
"""Move files produced in exporter subdirectories to the job root."""
output_directory = Path(output_directory)
flattened_paths = []
for source in result_paths:
source = Path(source)
check_cancelled(cancel_check)
if source.parent == output_directory:
flattened_paths.append(source)
continue
destination = output_directory / source.name
if destination.exists():
raise RuntimeError(f"Export filename already exists: {destination.name}")
flattened_paths.append(
move_exported_file(
source,
destination,
cancel_check=cancel_check,
)
)
for directory in sorted(
{path.parent for path in map(Path, result_paths)},
key=lambda path: len(path.parts),
reverse=True,
):
while directory != output_directory:
try:
directory.rmdir()
except OSError:
break
directory = directory.parent
return flattened_paths
def reconstruct(args, model_cache=None, cancel_check=None):
write_status(args.status, 0.02, "Starting AI runtime")
check_cancelled(cancel_check)
@@ -1019,7 +1287,7 @@ def reconstruct(args, model_cache=None, cancel_check=None):
images, temporary_directory = prepare_input(
args.input,
args.video_frames,
output_directory.parent,
output_directory,
start_frame=getattr(args, "start_frame", 1),
cancel_check=cancel_check,
)
@@ -1124,12 +1392,22 @@ def reconstruct(args, model_cache=None, cancel_check=None):
background_depth_distance,
)
write_status(args.status, 0.88, f"Exporting {args.export_format}")
export(
prediction,
args.export_format,
str(output_directory),
**export_kwargs,
)
if args.export_format == "mesh":
export_textured_mesh_glb(
prediction,
images,
output_directory,
getattr(args, "conf_thresh_percentile", 40.0),
bool(getattr(args, "mesh_filter_edges", True)),
float(getattr(args, "mesh_max_edge_length", 0.1)),
)
else:
export(
prediction,
args.export_format,
str(output_directory),
**export_kwargs,
)
check_cancelled(cancel_check)
result_paths = wait_for_exported_paths(
@@ -1141,19 +1419,13 @@ def reconstruct(args, model_cache=None, cancel_check=None):
raise RuntimeError(
f"No {args.export_format} files were exported"
)
result_paths = flatten_exported_paths(
output_directory,
result_paths,
cancel_check=cancel_check,
)
if args.export_format == "mini_npz":
nested_path = (
output_directory / "exports" / "mini_npz" / "results.npz"
)
result_path = output_directory / "results.npz"
if nested_path.is_file():
nested_path.replace(result_path)
result_paths = [result_path]
for directory in (nested_path.parent, nested_path.parent.parent):
try:
directory.rmdir()
except OSError:
pass
if reference_depth is not None:
write_npz_reference_depth(result_path, reference_depth)
check_cancelled(cancel_check)

120
tests/test_addon.py
View File

@@ -1279,7 +1279,7 @@ class AddonTest(unittest.TestCase):
self.assertEqual(paths["status"].parent, paths["result"])
self.assertEqual(paths["status"].stem, paths["result"].name)
def test_new_job_paths_uses_type_time_folder_name(self):
def test_new_job_paths_uses_unique_action_object_names(self):
runtime = importlib.import_module("snapform.runtime")
with TemporaryDirectory() as directory:
jobs_root = Path(directory)
@@ -1300,15 +1300,41 @@ class AddonTest(unittest.TestCase):
self.assertRegex(
depth_paths["result"].name,
r"^depth-plane_\d{8}-\d{6}$",
r"^generate-depth-plane_\d{8}-\d{6}-\d{6}$",
)
self.assertTrue(
background_paths["result"].name.startswith(
"remove-background_"
"remove-image-background_"
)
)
self.assertTrue(
frames_paths["result"].name.startswith("depth-plane_")
frames_paths["result"].name.startswith(
"extract-original-frames_"
)
)
def test_generated_inputs_are_moved_into_job_root(self):
runtime = importlib.import_module("snapform.runtime")
with TemporaryDirectory() as directory:
root = Path(directory)
staging = root / "staging"
staging.mkdir()
(staging / "frame.0001.jpg").write_bytes(b"first")
(staging / "frame.0002.jpg").write_bytes(b"second")
job_directory = root / "job"
job_directory.mkdir()
result = runtime.move_input_into_job(staging, job_directory)
self.assertEqual(result, job_directory)
self.assertFalse(staging.exists())
self.assertEqual(
(job_directory / "frame.0001.jpg").read_bytes(),
b"first",
)
self.assertEqual(
(job_directory / "frame.0002.jpg").read_bytes(),
b"second",
)
def test_empty_image_start_frame_honors_image_user(self):
@@ -1468,9 +1494,83 @@ class AddonTest(unittest.TestCase):
self.assertEqual(
identifiers,
["glb", "gs_ply", "depth_vis", "mini_npz"],
["glb", "gs_ply", "depth_vis", "mini_npz", "mesh"],
)
self.assertEqual(export_format["default"], "glb")
self.assertEqual(export_format["name"], "Output Type")
self.assertEqual(
[item[1] for item in export_format["items"]],
[
"Point Cloud (GLB)",
"Gaussian Splats (PLY)",
"Depth Preview (Images)",
"Depth Data (NPZ)",
"Textured Mesh (GLB)",
],
)
def test_mesh_is_a_da3_output_type(self):
settings = SimpleNamespace(
input_path="",
export_format="mesh",
model="BASE",
gs_model="GIANT_1_1",
process_res=504,
process_res_method="upper_bound_resize",
conf_thresh_percentile=40.0,
num_max_points=1_000_000,
show_cameras=True,
gs_views_interval=1,
video_frames=16,
device="CPU",
use_ray_pose=False,
ref_view_strategy="saddle_balanced",
)
operator = self.addon.RunJob()
operator.job_type = self.addon.operators.RECONSTRUCTION_JOB
with TemporaryDirectory() as directory:
input_path = Path(directory) / "input.png"
input_path.touch()
settings.input_path = str(input_path)
paths = {
"log": Path(directory) / "job.log",
"status": Path(directory) / "job.json",
"result": Path(directory) / "result",
}
with patch.object(
self.addon.runtime,
"new_job_paths",
return_value=paths,
), patch.object(
self.addon.runtime,
"runtime_ready",
return_value=True,
), patch.object(
self.addon.runtime,
"model_ready",
return_value=True,
), patch.object(
self.addon.runtime,
"reconstruct_request",
return_value=self.addon.runtime.HostJobRequest(
{"job_id": "test"}
),
) as reconstruct:
operator._prepare_command(None, settings)
self.assertEqual(reconstruct.call_args.args[1], "mesh")
self.assertEqual(reconstruct.call_args.kwargs["model_key"], "BASE")
def test_confidence_threshold_is_a_bounded_percentage(self):
confidence = self.addon.SnapFormSettings.__annotations__[
"conf_thresh_percentile"
]
self.assertEqual(confidence["name"], "Confidence Threshold")
self.assertEqual(confidence["min"], 0.0)
self.assertEqual(confidence["max"], 100.0)
self.assertEqual(confidence["subtype"], "PERCENTAGE")
def test_gs_model_property_only_includes_gaussian_models(self):
gs_model = self.addon.SnapFormSettings.__annotations__[
@@ -1586,9 +1686,17 @@ class AddonTest(unittest.TestCase):
"DEFAULT_CLOSED",
self.addon.DebugPanel.bl_options,
)
for panel in (self.addon.Da3DebugPanel, self.addon.Ben2DebugPanel):
panels = (
self.addon.Da3OutputDebugPanel,
self.addon.Ben2RemovalDebugPanel,
)
for panel in panels:
self.assertEqual(panel.bl_parent_id, self.addon.DebugPanel.bl_idname)
self.assertIn("DEFAULT_CLOSED", panel.bl_options)
self.assertEqual(
[panel.bl_label for panel in panels],
["DA3 Output Test", "BEN2 Removal Test"],
)
def test_host_panel_exposes_shared_host_state_and_controls(self):
self.assertIn(self.addon.HostPanel, self.addon.CLASSES)

158
tests/test_worker.py
View File

@@ -474,6 +474,38 @@ class WorkerTest(unittest.TestCase):
)
self.assertEqual(result, [npz_path])
def test_move_exported_file_retries_windows_file_lock(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source.npz"
destination = root / "destination.npz"
source.touch()
path_type = type(source)
original_replace = path_type.replace
calls = 0
def locked_once(path, target):
nonlocal calls
calls += 1
if calls == 1:
error = PermissionError("file is in use")
error.winerror = 32
raise error
return original_replace(path, target)
with patch.object(path_type, "replace", new=locked_once), patch.object(
self.worker.time,
"sleep",
) as sleep:
result = self.worker.move_exported_file(
source,
destination,
)
self.assertEqual(result, destination)
self.assertTrue(destination.is_file())
sleep.assert_called_once_with(0.1)
def test_rejects_unsupported_input_file(self):
with tempfile.TemporaryDirectory() as directory:
input_path = Path(directory) / "input.txt"
@@ -850,9 +882,10 @@ class WorkerTest(unittest.TestCase):
self.assertEqual(status["progress"], 1.0)
self.assertIn("3 frame", status["message"])
def test_uses_official_da3_export_parameters_for_all_four_formats(self):
def test_uses_da3_export_parameters_for_formats_and_debug_mesh(self):
inference_options = []
export_calls = []
mesh_calls = []
class FakeModel:
@classmethod
@@ -910,6 +943,27 @@ class WorkerTest(unittest.TestCase):
fake_package = ModuleType("depth_anything_3")
fake_package.__path__ = []
def fake_mesh_export(
prediction,
images,
output_directory,
confidence_percentile,
filter_edges,
max_edge_length,
):
mesh_calls.append(
(
prediction,
images,
confidence_percentile,
filter_edges,
max_edge_length,
)
)
output_path = Path(output_directory) / "mesh.glb"
output_path.touch()
return output_path
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
input_path = root / "input.png"
@@ -923,6 +977,10 @@ class WorkerTest(unittest.TestCase):
"depth_anything_3.utils": fake_utils,
"depth_anything_3.utils.export": fake_export_module,
},
), patch.object(
self.worker,
"export_textured_mesh_glb",
side_effect=fake_mesh_export,
):
for export_format in self.worker.EXPORT_FORMATS:
args = SimpleNamespace(
@@ -941,6 +999,8 @@ class WorkerTest(unittest.TestCase):
num_max_points=12345,
show_cameras=0,
gs_views_interval=2,
mesh_filter_edges=True,
mesh_max_edge_length=0.2,
)
self.worker.reconstruct(args)
status = json.loads(
@@ -948,9 +1008,17 @@ class WorkerTest(unittest.TestCase):
)
self.assertEqual(status["progress"], 1.0)
self.assertIn(export_format, status["message"])
result_directory = Path(args.output_dir)
if export_format == "gs_ply":
self.assertTrue((result_directory / "0000.ply").is_file())
self.assertFalse((result_directory / "gs_ply").exists())
if export_format == "depth_vis":
self.assertTrue((result_directory / "0000.jpg").is_file())
self.assertFalse((result_directory / "depth_vis").exists())
self.assertEqual(len(inference_options), 4)
self.assertEqual(len(inference_options), 5)
self.assertEqual(len(export_calls), 4)
self.assertEqual(len(mesh_calls), 1)
for images, options in inference_options:
self.assertEqual(images, [str(input_path)])
self.assertEqual(options["process_res"], 518)
@@ -960,8 +1028,15 @@ class WorkerTest(unittest.TestCase):
)
self.assertTrue(options["use_ray_pose"])
self.assertEqual(options["ref_view_strategy"], "middle")
for _prediction, export_format, _export_dir, options in export_calls:
expected_depth = -1.0 if export_format == "mini_npz" else 2.0
for (
_prediction,
export_format,
_export_dir,
options,
) in export_calls:
expected_depth = (
-1.0 if export_format == "mini_npz" else 2.0
)
np.testing.assert_array_equal(
_prediction.depth,
np.full((1, 2, 2), expected_depth, dtype=np.float32),
@@ -985,9 +1060,14 @@ class WorkerTest(unittest.TestCase):
)
else:
self.assertEqual(options, {})
self.assertEqual(mesh_calls[0][1], [input_path])
self.assertEqual(mesh_calls[0][2:], (35.0, True, 0.2))
infer_gs_values = [options["infer_gs"] for _, options in inference_options]
self.assertEqual(infer_gs_values, [False, True, False, False])
self.assertEqual(
infer_gs_values,
[False, True, False, False, False],
)
def test_reconstruct_copies_frames_for_multi_frame_input(self):
class FakeModel:
@@ -1080,6 +1160,74 @@ class WorkerTest(unittest.TestCase):
status = json.loads(Path(args.status).read_text(encoding="utf-8"))
self.assertEqual(status["progress"], 1.0)
def test_mesh_confidence_uses_percentile(self):
depth = np.ones((1, 2, 2), dtype=np.float32)
confidence = np.array([[[1.0, 2.0], [3.0, 4.0]]])
threshold = self.worker.confidence_percentile_threshold(
depth,
confidence,
50.0,
)
self.assertEqual(threshold, 2.5)
def test_mesh_faces_apply_confidence_threshold_and_face_direction(self):
points = np.array(
[
[[0.0, 0.0, 1.0], [1.0, 0.0, 1.0], [2.0, 0.0, 1.0]],
[[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [2.0, 1.0, 1.0]],
],
dtype=np.float32,
)
depth = np.ones((2, 3), dtype=np.float32)
confidence = np.ones((2, 3), dtype=np.float32)
confidence[:, 0] = 0.0
faces = self.worker.mesh_triangle_faces(
points,
depth,
confidence,
confidence_threshold=0.5,
filter_edges=False,
max_edge_length=0.0,
)
np.testing.assert_array_equal(faces, [[1, 4, 5], [1, 5, 2]])
vertices = points.reshape(-1, 3)[faces[0]]
normal = np.cross(
vertices[1] - vertices[0],
vertices[2] - vertices[0],
)
self.assertLess(normal[2], 0.0)
def test_mesh_percentile_reduces_face_count(self):
depth = np.ones((3, 3), dtype=np.float32)
confidence = np.arange(1, 10, dtype=np.float32).reshape(3, 3)
x, y = np.meshgrid(
np.arange(3, dtype=np.float32),
np.arange(3, dtype=np.float32),
)
points = np.stack((x, y, depth), axis=-1)
face_counts = []
for percentile in (0.0, 50.0, 100.0):
threshold = self.worker.confidence_percentile_threshold(
depth,
confidence,
percentile,
)
faces = self.worker.mesh_triangle_faces(
points,
depth,
confidence,
confidence_threshold=threshold,
filter_edges=False,
max_edge_length=0.0,
)
face_counts.append(len(faces))
self.assertEqual(face_counts, [8, 2, 0])
if __name__ == "__main__":
unittest.main()