diff --git a/README.md b/README.md index eadfaca..59f70d9 100644 --- a/README.md +++ b/README.md @@ -45,8 +45,10 @@ SnapForm 支持视频（mp4、mkv、mov 等）和序列帧，会逐帧处理并 ## 任务结果保存在哪里 -任务结果保存在 `~/.snapform/jobs/` 下，每个任务一个文件夹。视频或序列帧生成的面片会引用这些文件夹 -里的序列帧，请保留对应文件夹，不要随意删除。 +任务结果保存在 `~/.snapform/jobs/` 下，每个任务一个文件夹，名称统一为 +`动作-对象_日期-时间-微秒`。Packed Image 和序列帧的输入副本会直接归档到对应任务根目录， +不再使用公共 `inputs/` 文件夹；导出产物也尽量平铺。视频或序列帧生成的面片会引用任务文件夹里的 +序列帧，请保留对应文件夹，不要随意删除。 ## 常见问题 diff --git a/docs/technical.md b/docs/technical.md index 12d57d0..2d464db 100644 --- a/docs/technical.md +++ b/docs/technical.md @@ -23,13 +23,15 @@ - 视频按顺序截取前若干帧，可设置参与重建的最大帧数 - `Host > Models` 默认折叠，展开后可逐项查看和下载所有可用模型 - Runtime 已就绪时不额外占用面板空间；模型列表每个模型只占一行 -- `Debug` 默认折叠，内部的 DA3 与 BEN2 测试项也分别默认折叠 -- 使用一个导出类型下拉菜单和一个 `Export` 按钮 +- `Debug` 默认折叠，内部的 `DA3 Output Test` 与 `BEN2 Removal Test` 也分别默认折叠 +- 使用一个输出类型下拉菜单和一个 `Generate Output` 按钮 - 只显示当前格式的专属参数；GLB 与 PLY 设置不会同时出现 - Debug 的模型选项按格式过滤；PLY 只提供两个带 Gaussian head 的 Giant 模型 - Debug 与 Convert to Depth Plane 的模型选项直接显示下载状态；未下载模型带 `(Missing)`，已下载模型保持原名 - 分别调用 DA3 的 `glb`、`gs_ply`、`depth_vis` 和 `mini_npz` exporter +- `Textured Mesh (GLB)` 直接使用 DA3 Prediction 中的深度、置信度和相机参数，把相邻像素 + 组成三角面并写入 `mesh.glb`；每个输入帧生成一张独立纹理网格，不做多视图融合 - Image Empty 右键菜单可直接运行 DA3，根据 `mini_npz` 深度生成面片 - Image Empty 右键菜单可直接运行 BEN2，保留原图并换用新建的透明背景图片 - Image Empty 支持视频与序列帧输入：`Convert to Depth Plane` 逐帧估计深度， @@ -39,7 +41,7 @@ - 使用浮点深度贴图、Simple Subdivision、Displace Modifier 和原图 Bump 构建立体面片 - 转换对话框可选择推理模式、处理分辨率和面片网格细分 -- Debug 的 DA3 导出或 BEN2 背景剔除完成后只打开对应 job 文件夹，不向当前 Blender 文件添加任何数据 +- Debug 的 DA3 输出或 BEN2 背景剔除完成后只打开对应 job 文件夹，不修改当前场景 - 完整保留每次任务生成的所有官方文件 - 安装和模型下载使用后台子进程；推理提交给常驻 AI Host，状态栏实时显示阶段、百分比进度条与取消按钮 - 后台任务运行时，Install、Download 和 Export 按钮保持原位置并变灰，任务完成后恢复 @@ -69,10 +71,10 @@ 自行探测回退到 HF 镜像；需要手动指定镜像时可通过 `PIP_INDEX_URL` 环境变量覆盖。 5. 在 3D View 按 `N`，打开 `SnapForm` 页签。 6. 如需其他模型，展开 `Host > Models`，找到模型并点击该行的 `Download`。 -7. 展开默认收起的 `Debug`，选择 `Export Format` 和本次执行使用的模型。 -8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数，然后点击 `Export`。 +7. 展开默认收起的 `Debug`，选择 `Output Type` 和本次执行使用的模型。 +8. 选择一张图片或一个视频文件。视频可设置从头截取的帧数，然后点击 `Generate Output`。 9. 测试背景剔除时，确认 `Host > Models` 中的 `BEN2 Base` 已就绪，然后在 `Debug` 的 - `BEN2 Background Removal` 区域选择单张图片并点击 `Remove Background`。 + `BEN2 Removal Test` 区域选择单张图片并点击 `Remove Background`。 完成后会打开 job 文件夹，其中的 `foreground.png` 带透明 Alpha；状态栏显示 BEN2 推理、模型加载与任务总耗时。当前测试固定关闭较慢的 foreground refinement。 @@ -218,33 +220,52 @@ Image Empty 使用 Packed Image 时，SnapForm 会通过 Blender 写出临时 PN `DA3 Nested Giant Large 1.1`。Small、Base 和 Large 没有 Gaussian head，无法由 DA3 生成 `gs_ply`。 -每个任务在 `~/.snapform/jobs/` 下用独立文件夹包裹（命名格式为 `任务类型_时间号`，例如 -`depth-plane_20260807-153000`），日志、状态与输出产物都放在该文件夹内： +每个任务在 `~/.snapform/jobs/` 下用独立文件夹包裹。文件夹统一采用 +`动作-对象_日期-时间-微秒`，每种任务都有唯一名称，例如 +`generate-depth-plane_20260807-153000-123456`。微秒用于避免同一秒启动的任务共用目录。 + +Blender 为 Packed Image 或序列帧生成的输入副本也归档到对应 job 根目录：单图命名为 +`input.png`，序列帧命名为 `frame.0001.*`。它们以前暂存在 `jobs/inputs/`，现在不再创建这个 +公共中转文件夹。普通磁盘文件和视频仍直接读取原文件，不额外复制。 + +日志、运行中的状态文件、归档输入与输出产物均直接放在 job 根目录。各类任务名称为： + +- `install-runtime` +- `download-model` +- `export-reconstruction` +- `generate-depth-plane` +- `remove-background` +- `remove-image-background` +- `generate-original-depth-plane` +- `extract-original-frames` + +DA3 官方 exporter 产生的 `gs_ply/`、`depth_vis/`、`exports/mini_npz/` 会在完成后自动平铺， +不会保留只含一层产物的子目录： ```text -reconstruct_20260807-153000/ -├── reconstruct_20260807-153000.log -├── reconstruct_20260807-153000.json +export-reconstruction_20260807-153000-123456/ +├── export-reconstruction_20260807-153000-123456.log +├── export-reconstruction_20260807-153000-123456.json └── scene.glb -reconstruct_20260807-153000/ -├── reconstruct_20260807-153000.log -├── reconstruct_20260807-153000.json -└── gs_ply/*.ply +export-reconstruction_20260807-153000-123456/ +├── export-reconstruction_20260807-153000-123456.log +├── export-reconstruction_20260807-153000-123456.json +└── *.ply -reconstruct_20260807-153000/ -├── reconstruct_20260807-153000.log -├── reconstruct_20260807-153000.json -└── depth_vis/*.jpg +export-reconstruction_20260807-153000-123456/ +├── export-reconstruction_20260807-153000-123456.log +├── export-reconstruction_20260807-153000-123456.json +└── *.jpg -reconstruct_20260807-153000/ -├── reconstruct_20260807-153000.log -├── reconstruct_20260807-153000.json +export-reconstruction_20260807-153000-123456/ +├── export-reconstruction_20260807-153000-123456.log +├── export-reconstruction_20260807-153000-123456.json └── results.npz -remove-background_20260807-153000/ -├── remove-background_20260807-153000.log -├── remove-background_20260807-153000.json +remove-background_20260807-153000-123456/ +├── remove-background_20260807-153000-123456.log +├── remove-background_20260807-153000-123456.json ├── foreground.png 单图结果 ├── foreground.0001.png… 视频/序列帧结果 ├── frame.0001.jpg… 视频/序列帧的源帧副本 @@ -256,15 +277,16 @@ CC BY-NC 4.0，仅限非商业用途。 ## 输出格式说明 -`Export Format` 下拉菜单分别把以下原始 `export_format` 传给 DA3： +`Output Type` 提供以下结果： -- GLB：`glb` -- PLY：`gs_ply`，同时传入 `infer_gs=True` -- Depth Images：`depth_vis` -- Mini NPZ：`mini_npz` +- Point Cloud (GLB)：DA3 `glb` +- Gaussian Splats (PLY)：DA3 `gs_ply`，同时传入 `infer_gs=True` +- Depth Preview (Images)：DA3 `depth_vis` +- Depth Data (NPZ)：DA3 `mini_npz` +- Textured Mesh (GLB)：SnapForm 根据 DA3 Prediction 构建并输出 `mesh.glb` -界面使用 DA3 API 的参数名、选项和默认值。GLB 专属设置仅在选择 `GLB` 时显示， -`gs_views_interval` 仅在选择 `PLY (3DGS)` 时显示： +Point Cloud 专属设置仅在对应类型下显示，`gs_views_interval` 仅用于 Gaussian Splats。 +Textured Mesh 使用 0–100% 的 Confidence Threshold 删除置信度最低的一定比例： - `process_res=504` - `process_res_method="upper_bound_resize"` @@ -277,7 +299,7 @@ CC BY-NC 4.0，仅限非商业用途。 DA3 官方 `glb` 是彩色点云与可选相机线框；`gs_ply` 是带 Gaussian 属性的 PLY； `depth_vis` 是原图与着色深度图的并排预览；`mini_npz` 保存深度、置信度和可用的相机参数。 -GLB 任务会关闭其默认附带的深度预览，使四种 job 产物保持分离。 +Point Cloud GLB 任务会关闭其默认附带的深度预览，使五种输出产物保持分离。 官方格式与限制可参阅 [DA3 Python API 文档](https://github.com/ByteDance-Seed/Depth-Anything-3/blob/main/docs/API.md)。 diff --git a/src/snapform/__init__.py b/src/snapform/__init__.py index c77d301..4c81987 100644 --- a/src/snapform/__init__.py +++ b/src/snapform/__init__.py @@ -17,8 +17,8 @@ from .operators import ( close_active_job, ) from .panel import ( - Ben2DebugPanel, - Da3DebugPanel, + Ben2RemovalDebugPanel, + Da3OutputDebugPanel, DebugPanel, HostPanel, ModelsPanel, @@ -42,8 +42,8 @@ CLASSES = ( HostPanel, ModelsPanel, DebugPanel, - Da3DebugPanel, - Ben2DebugPanel, + Da3OutputDebugPanel, + Ben2RemovalDebugPanel, ) diff --git a/src/snapform/operators.py b/src/snapform/operators.py index b658365..9424735 100644 --- a/src/snapform/operators.py +++ b/src/snapform/operators.py @@ -122,6 +122,7 @@ def close_active_job(): shutil.rmtree(temporary_path, ignore_errors=True) else: temporary_path.unlink(missing_ok=True) + temporary_path.parent.rmdir() except OSError: pass _active_job = None @@ -221,6 +222,12 @@ class RunJob(bpy.types.Operator): self.report({"ERROR"}, str(error)) return {"CANCELLED"} + export_format = "" + if self.job_type == RECONSTRUCTION_JOB: + export_format = settings.export_format + elif self.job_type == IMAGE_DEPTH_JOB: + export_format = "mini_npz" + job = ActiveJob( kind=self.job_type, process=None, @@ -228,11 +235,7 @@ class RunJob(bpy.types.Operator): log_path=log_path, status_path=paths["status"], result_directory=paths["result"], - export_format=( - settings.export_format if self.job_type == RECONSTRUCTION_JOB - else "mini_npz" if self.job_type == IMAGE_DEPTH_JOB - else "" - ), + export_format=export_format, source_object_name=self.snapform_source_object_name, depth_data_name=self.snapform_depth_data_name, process_res=self.snapform_process_res, @@ -310,6 +313,21 @@ class RunJob(bpy.types.Operator): def _prepare_command(self, _context, settings): paths = runtime.new_job_paths(self.job_type) + if ( + getattr(self, "snapform_delete_input", False) + and self.job_type in { + IMAGE_DEPTH_JOB, + IMAGE_BACKGROUND_JOB, + ORIGINAL_BACKGROUND_JOB, + ORIGINAL_FRAMES_JOB, + } + ): + archived_input = runtime.move_input_into_job( + self.snapform_input_path, + paths["result"], + ) + self.snapform_input_path = str(archived_input) + self.snapform_delete_input = False background_depth_distance = float( getattr(self, "snapform_background_depth_distance", 0.0) ) @@ -389,13 +407,14 @@ class RunJob(bpy.types.Operator): ), paths, ) - if self.job_type not in {RECONSTRUCTION_JOB, IMAGE_DEPTH_JOB}: + if self.job_type not in { + RECONSTRUCTION_JOB, + IMAGE_DEPTH_JOB, + }: raise RuntimeError(f"Unsupported task type: {self.job_type}") - export_format = ( - "mini_npz" - if self.job_type == IMAGE_DEPTH_JOB - else settings.export_format - ) + export_format = settings.export_format + if self.job_type == IMAGE_DEPTH_JOB: + export_format = "mini_npz" use_ray_pose = None process_res = None model_key = selected_model_key(settings, export_format) @@ -614,7 +633,10 @@ class RunJob(bpy.types.Operator): self._redraw_ui(context, force=True) return {"CANCELLED"} - if job.kind in {RECONSTRUCTION_JOB, BACKGROUND_REMOVAL_JOB}: + if job.kind in { + RECONSTRUCTION_JOB, + BACKGROUND_REMOVAL_JOB, + }: try: open_job_directory(job.result_directory) except Exception as open_error: diff --git a/src/snapform/panel.py b/src/snapform/panel.py index bab7416..2df3ef8 100644 --- a/src/snapform/panel.py +++ b/src/snapform/panel.py @@ -160,9 +160,33 @@ class DebugPanel(bpy.types.Panel): layout.prop(settings, "device") -class Da3DebugPanel(bpy.types.Panel): - bl_label = "DA3 Reconstruction" - bl_idname = "SNAPFORM_PT_debug_da3" +def draw_da3_input_settings(layout, settings, model_property): + layout.prop(settings, model_property) + layout.prop(settings, "input_path") + input_path = bpy.path.abspath(settings.input_path) + if input_path.lower().endswith( + ( + ".mp4", + ".mov", + ".avi", + ".mkv", + ".webm", + ".m4v", + ".mts", + ".mpg", + ".mpeg", + ) + ): + layout.prop(settings, "video_frames") + layout.prop(settings, "process_res") + layout.prop(settings, "process_res_method") + layout.prop(settings, "use_ray_pose") + layout.prop(settings, "ref_view_strategy") + + +class Da3OutputDebugPanel(bpy.types.Panel): + bl_label = "DA3 Output Test" + bl_idname = "SNAPFORM_PT_debug_da3_output" bl_parent_id = DebugPanel.bl_idname bl_space_type = "VIEW_3D" bl_region_type = "UI" @@ -178,28 +202,7 @@ class Da3DebugPanel(bpy.types.Panel): model_property = ( "gs_model" if settings.export_format == "gs_ply" else "model" ) - layout.prop(settings, model_property) - - layout.prop(settings, "input_path") - input_path = bpy.path.abspath(settings.input_path) - if input_path.lower().endswith( - ( - ".mp4", - ".mov", - ".avi", - ".mkv", - ".webm", - ".m4v", - ".mts", - ".mpg", - ".mpeg", - ) - ): - layout.prop(settings, "video_frames") - layout.prop(settings, "process_res") - layout.prop(settings, "process_res_method") - layout.prop(settings, "use_ray_pose") - layout.prop(settings, "ref_view_strategy") + draw_da3_input_settings(layout, settings, model_property) if settings.export_format == "glb": glb_box = layout.box() @@ -211,7 +214,12 @@ class Da3DebugPanel(bpy.types.Panel): ply_box = layout.box() ply_box.label(text="PLY (3DGS) Settings") ply_box.prop(settings, "gs_views_interval") - + elif settings.export_format == "mesh": + mesh_box = layout.box() + mesh_box.label(text="Mesh Settings") + mesh_box.prop(settings, "conf_thresh_percentile") + mesh_box.prop(settings, "mesh_filter_edges") + mesh_box.prop(settings, "mesh_max_edge_length") export_row = layout.row() export_row.enabled = ( active_job() is None @@ -220,15 +228,15 @@ class Da3DebugPanel(bpy.types.Panel): ) operator = export_row.operator( "snapform.run_job", - text="Export", + text="Generate Output", icon="EXPORT", ) operator.job_type = "RECONSTRUCT" -class Ben2DebugPanel(bpy.types.Panel): - bl_label = "BEN2 Background Removal" - bl_idname = "SNAPFORM_PT_debug_ben2" +class Ben2RemovalDebugPanel(bpy.types.Panel): + bl_label = "BEN2 Removal Test" + bl_idname = "SNAPFORM_PT_debug_ben2_removal" bl_parent_id = DebugPanel.bl_idname bl_space_type = "VIEW_3D" bl_region_type = "UI" diff --git a/src/snapform/properties.py b/src/snapform/properties.py index b894676..43c834b 100644 --- a/src/snapform/properties.py +++ b/src/snapform/properties.py @@ -66,8 +66,8 @@ class SnapFormSettings(bpy.types.PropertyGroup): default="upper_bound_resize", ) conf_thresh_percentile: bpy.props.FloatProperty( - name="Conf Thresh Percentile", - description="DA3 GLB confidence threshold percentile", + name="Confidence Threshold", + description="Remove the lowest-confidence percentile of points or mesh faces", default=40.0, min=0.0, max=100.0, @@ -110,24 +110,52 @@ class SnapFormSettings(bpy.types.PropertyGroup): default="saddle_balanced", ) export_format: bpy.props.EnumProperty( - name="Export Format", - description="Official DA3 export format", + name="Output Type", + description="Result generated by the DA3 debug task", items=( - ("glb", "GLB", "Export a colored point cloud as GLB"), - ("gs_ply", "PLY (3DGS)", "Export Gaussian splats as PLY"), + ( + "glb", + "Point Cloud (GLB)", + "Generate a colored point cloud and optional cameras", + ), + ( + "gs_ply", + "Gaussian Splats (PLY)", + "Generate a 3D Gaussian Splatting PLY", + ), ( "depth_vis", - "Depth Images", - "Export image and depth visualization pairs", + "Depth Preview (Images)", + "Generate image and colored-depth preview pairs", ), ( "mini_npz", - "Mini NPZ", - "Export compact depth and camera data", + "Depth Data (NPZ)", + "Generate compact depth, confidence, and camera data", + ), + ( + "mesh", + "Textured Mesh (GLB)", + "Generate one filtered textured depth mesh per input image", ), ), default="glb", ) + mesh_filter_edges: bpy.props.BoolProperty( + name="Filter Depth Edges", + description="Remove faces around sharp depth transitions", + default=True, + ) + mesh_max_edge_length: bpy.props.FloatProperty( + name="Max Edge Length", + description=( + "Remove faces with an edge longer than this scene distance; " + "zero disables the test" + ), + default=0.1, + min=0.0, + soft_max=2.0, + ) gs_views_interval: bpy.props.IntProperty( name="GS Views Interval", description="Export 3D Gaussians every N input views", diff --git a/src/snapform/runtime.py b/src/snapform/runtime.py index 89eb224..9ab383a 100644 --- a/src/snapform/runtime.py +++ b/src/snapform/runtime.py @@ -6,6 +6,7 @@ import shutil import socket import subprocess import sys +import tempfile import threading import time import uuid @@ -34,9 +35,12 @@ DEFAULT_STORAGE_ROOT = Path.home() / ".snapform" JOB_FOLDER_NAMES = { "INSTALL_RUNTIME": "install-runtime", "DOWNLOAD_MODEL": "download-model", - "IMAGE_DEPTH": "depth-plane", - "IMAGE_BACKGROUND": "remove-background", - "ORIGINAL_FRAMES": "depth-plane", + "RECONSTRUCT": "export-reconstruction", + "IMAGE_DEPTH": "generate-depth-plane", + "REMOVE_BACKGROUND": "remove-background", + "IMAGE_BACKGROUND": "remove-image-background", + "ORIGINAL_BACKGROUND": "generate-original-depth-plane", + "ORIGINAL_FRAMES": "extract-original-frames", } @@ -196,15 +200,37 @@ def jobs_directory(): def temporary_input_path(): - path = jobs_directory() / "inputs" - path.mkdir(parents=True, exist_ok=True) - return path / f"{uuid.uuid4().hex}.png" + directory = Path(tempfile.mkdtemp(prefix="snapform-input-")) + return directory / "input.png" def temporary_input_directory(): - path = jobs_directory() / "inputs" / f"{uuid.uuid4().hex}" - path.mkdir(parents=True, exist_ok=True) - return path + return Path(tempfile.mkdtemp(prefix="snapform-input-")) + + +def move_input_into_job(input_path, job_directory): + """Move a Blender-generated input copy into the flat job directory.""" + input_path = Path(input_path) + job_directory = Path(job_directory) + if input_path.is_file(): + destination = job_directory / "input.png" + input_path.replace(destination) + try: + input_path.parent.rmdir() + except OSError: + pass + return destination + if not input_path.is_dir(): + raise ValueError(f"Temporary input does not exist: {input_path}") + for source in input_path.iterdir(): + if not source.is_file(): + raise ValueError(f"Temporary input contains a directory: {source}") + destination = job_directory / source.name + if destination.exists(): + raise ValueError(f"Job input filename already exists: {source.name}") + source.replace(destination) + input_path.rmdir() + return job_directory def job_folder_name(job_type): @@ -213,7 +239,7 @@ def job_folder_name(job_type): def new_job_paths(job_type): - timestamp = datetime.now().strftime("%Y%m%d-%H%M%S") + timestamp = datetime.now().strftime("%Y%m%d-%H%M%S-%f") stem = f"{job_folder_name(job_type)}_{timestamp}" directory = jobs_directory() / stem directory.mkdir(parents=True, exist_ok=True) @@ -824,6 +850,12 @@ def reconstruct_request( "ref_view_strategy": settings.ref_view_strategy, "remove_background_model_dir": "", "background_depth_distance": float(background_depth_distance), + "mesh_filter_edges": bool( + getattr(settings, "mesh_filter_edges", True) + ), + "mesh_max_edge_length": float( + getattr(settings, "mesh_max_edge_length", 0.1) + ), } if remove_background or background_depth_distance > 0.0: parameters["remove_background_model_dir"] = str( diff --git a/src/snapform/worker.py b/src/snapform/worker.py index 66360ae..2fd04f3 100644 --- a/src/snapform/worker.py +++ b/src/snapform/worker.py @@ -24,7 +24,7 @@ VIDEO_SUFFIXES = { ".mpeg", } SCENE_FILENAME = "scene.glb" -EXPORT_FORMATS = ("glb", "gs_ply", "depth_vis", "mini_npz") +EXPORT_FORMATS = ("glb", "gs_ply", "depth_vis", "mini_npz", "mesh") HF_ENDPOINT = "https://huggingface.co" MODELSCOPE_ENDPOINT = "https://modelscope.cn" MODELSCOPE_REVISION = "master" @@ -340,10 +340,211 @@ def copy_frames(images, output_directory): return for index, image_path in enumerate(images): suffix = Path(image_path).suffix.lower() or ".jpg" - shutil.copy2( - image_path, - Path(output_directory) / f"frame.{index + 1:04d}{suffix}", + destination = ( + Path(output_directory) / f"frame.{index + 1:04d}{suffix}" ) + if Path(image_path).resolve() == destination.resolve(): + continue + shutil.copy2(image_path, destination) + + +def unproject_depth_frame(depth, intrinsics, extrinsics): + import numpy as np + + height, width = depth.shape + x, y = np.meshgrid( + np.arange(width, dtype=np.float32), + np.arange(height, dtype=np.float32), + ) + pixels = np.stack((x, y, np.ones_like(x)), axis=-1).reshape(-1, 3) + rays = (np.linalg.inv(intrinsics) @ pixels.T).T + camera_points = rays * depth.reshape(-1, 1) + camera_points = np.column_stack( + (camera_points, np.ones(camera_points.shape[0], dtype=np.float32)) + ) + world_to_camera = np.eye(4, dtype=np.float32) + world_to_camera[: extrinsics.shape[0], :4] = extrinsics + world_points = (np.linalg.inv(world_to_camera) @ camera_points.T).T + return world_points[:, :3].reshape(height, width, 3) + + +def depth_edge_mask(depth, threshold=12.0 / 255.0): + import numpy as np + + padded = np.pad(depth.astype(np.float32), 1, mode="edge") + grad_x = ( + -padded[:-2, :-2] + + padded[:-2, 2:] + - 2.0 * padded[1:-1, :-2] + + 2.0 * padded[1:-1, 2:] + - padded[2:, :-2] + + padded[2:, 2:] + ) + grad_y = ( + -padded[:-2, :-2] + - 2.0 * padded[:-2, 1:-1] + - padded[:-2, 2:] + + padded[2:, :-2] + + 2.0 * padded[2:, 1:-1] + + padded[2:, 2:] + ) + magnitude = np.hypot(grad_x, grad_y) + finite = np.isfinite(magnitude) + if not finite.any(): + return np.ones(depth.shape, dtype=bool) + minimum = float(magnitude[finite].min()) + maximum = float(magnitude[finite].max()) + if maximum <= minimum: + return np.zeros(depth.shape, dtype=bool) + normalized = (magnitude - minimum) / (maximum - minimum) + return normalized >= threshold + + +def confidence_percentile_threshold(depth, confidence, percentile): + import numpy as np + + valid = np.isfinite(depth) & (depth > 0.0) & np.isfinite(confidence) + if not valid.any(): + raise ValueError("DA3 returned no valid depth confidence values") + return float(np.percentile(confidence[valid], percentile)) + + +def mesh_triangle_faces( + points, + depth, + confidence, + confidence_threshold, + filter_edges, + max_edge_length, +): + import numpy as np + + height, width = depth.shape + indices = np.arange(height * width, dtype=np.int32).reshape(height, width) + quads = np.stack( + ( + indices[:-1, :-1], + indices[1:, :-1], + indices[1:, 1:], + indices[:-1, 1:], + ), + axis=-1, + ).reshape(-1, 4) + valid = ( + np.isfinite(points).all(axis=-1) + & np.isfinite(depth) + & (depth > 0.0) + & (confidence >= confidence_threshold) + ) + if filter_edges: + valid &= ~depth_edge_mask(depth) + face_valid = valid.reshape(-1)[quads].all(axis=1) + if max_edge_length > 0.0: + flat_points = points.reshape(-1, 3) + quad_points = flat_points[quads] + next_points = np.roll(quad_points, -1, axis=1) + edge_lengths = np.linalg.norm(quad_points - next_points, axis=-1) + face_valid &= (edge_lengths <= max_edge_length).all(axis=1) + quads = quads[face_valid] + if not len(quads): + return np.empty((0, 3), dtype=np.int32) + return np.concatenate( + (quads[:, (0, 1, 2)], quads[:, (0, 2, 3)]), + axis=0, + ) + + +def export_textured_mesh_glb( + prediction, + image_paths, + output_directory, + confidence_percentile, + filter_edges, + max_edge_length, +): + import numpy as np + import trimesh + from PIL import Image + + required = ("depth", "conf", "intrinsics", "extrinsics") + missing = [name for name in required if getattr(prediction, name) is None] + if missing: + raise ValueError(f"DA3 mesh output is missing: {', '.join(missing)}") + depth = np.asarray(prediction.depth, dtype=np.float32) + confidence = np.asarray(prediction.conf, dtype=np.float32) + threshold = confidence_percentile_threshold( + depth, + confidence, + confidence_percentile, + ) + meshes = [] + referenced_points = [] + for index in range(depth.shape[0]): + points = unproject_depth_frame( + depth[index], + prediction.intrinsics[index], + prediction.extrinsics[index], + ) + faces = mesh_triangle_faces( + points, + depth[index], + confidence[index], + threshold, + filter_edges, + max_edge_length, + ) + if not len(faces): + continue + height, width = depth[index].shape + u = np.linspace(0.0, 1.0, width, dtype=np.float32) + v = np.linspace(1.0, 0.0, height, dtype=np.float32) + uu, vv = np.meshgrid(u, v) + uv = np.stack((uu, vv), axis=-1).reshape(-1, 2) + with Image.open(image_paths[index]) as source_image: + texture_image = source_image.convert("RGB").copy() + visual = trimesh.visual.texture.TextureVisuals( + uv=uv, + image=texture_image, + ) + mesh = trimesh.Trimesh( + vertices=points.reshape(-1, 3), + faces=faces, + visual=visual, + process=False, + ) + mesh.remove_unreferenced_vertices() + meshes.append(mesh) + referenced_points.append(mesh.vertices) + if not meshes: + raise ValueError( + "Mesh filters removed every face; lower Confidence Threshold or disable filters" + ) + + world_to_camera = np.asarray(prediction.extrinsics[0], dtype=np.float64) + if world_to_camera.shape == (3, 4): + homogeneous = np.eye(4, dtype=np.float64) + homogeneous[:3, :4] = world_to_camera + world_to_camera = homogeneous + cv_to_gltf = np.eye(4, dtype=np.float64) + cv_to_gltf[1, 1] = -1.0 + cv_to_gltf[2, 2] = -1.0 + alignment = cv_to_gltf @ world_to_camera + aligned_points = trimesh.transform_points( + np.concatenate(referenced_points, axis=0), + alignment, + ) + center = np.median(aligned_points, axis=0) + alignment[:3, 3] -= center + + scene = trimesh.Scene() + for index, mesh in enumerate(meshes): + mesh.apply_transform(alignment) + scene.add_geometry(mesh, node_name=f"DepthMesh.{index + 1:04d}") + output_directory = Path(output_directory) + output_directory.mkdir(parents=True, exist_ok=True) + output_path = output_directory / "mesh.glb" + scene.export(output_path) + return output_path class DownloadProgressReporter: @@ -856,7 +1057,7 @@ def remove_background(args, model_cache=None, cancel_check=None): images, temporary_directory = prepare_background_input( args.input, video_frames, - output_directory.parent, + output_directory, start_frame=start_frame, cancel_check=cancel_check, ) @@ -933,7 +1134,7 @@ def extract_frames(args, cancel_check=None): images, temporary_directory = prepare_input( args.input, getattr(args, "video_frames", 16), - output_directory.parent, + output_directory, start_frame=getattr(args, "start_frame", 1), cancel_check=cancel_check, ) @@ -956,6 +1157,9 @@ def exported_paths(output_directory, export_format): return sorted((output_directory / "gs_ply").glob("*.ply")) if export_format == "depth_vis": return sorted((output_directory / "depth_vis").glob("*.jpg")) + if export_format == "mesh": + result_path = output_directory / "mesh.glb" + return [result_path] if result_path.is_file() else [] if export_format == "mini_npz": result_path = output_directory / "results.npz" if result_path.is_file(): @@ -1002,6 +1206,70 @@ def wait_for_exported_paths( return [] +def move_exported_file( + source, + destination, + timeout=10.0, + cancel_check=None, +): + """Move an async export after Windows releases its file handle.""" + source = Path(source) + destination = Path(destination) + deadline = time.monotonic() + timeout + while True: + check_cancelled(cancel_check) + try: + source.replace(destination) + return destination + except OSError as error: + is_file_lock = isinstance(error, PermissionError) or getattr( + error, + "winerror", + None, + ) in {32, 33} + if not is_file_lock or time.monotonic() >= deadline: + raise + time.sleep(0.1) + + +def flatten_exported_paths( + output_directory, + result_paths, + cancel_check=None, +): + """Move files produced in exporter subdirectories to the job root.""" + output_directory = Path(output_directory) + flattened_paths = [] + for source in result_paths: + source = Path(source) + check_cancelled(cancel_check) + if source.parent == output_directory: + flattened_paths.append(source) + continue + destination = output_directory / source.name + if destination.exists(): + raise RuntimeError(f"Export filename already exists: {destination.name}") + flattened_paths.append( + move_exported_file( + source, + destination, + cancel_check=cancel_check, + ) + ) + for directory in sorted( + {path.parent for path in map(Path, result_paths)}, + key=lambda path: len(path.parts), + reverse=True, + ): + while directory != output_directory: + try: + directory.rmdir() + except OSError: + break + directory = directory.parent + return flattened_paths + + def reconstruct(args, model_cache=None, cancel_check=None): write_status(args.status, 0.02, "Starting AI runtime") check_cancelled(cancel_check) @@ -1019,7 +1287,7 @@ def reconstruct(args, model_cache=None, cancel_check=None): images, temporary_directory = prepare_input( args.input, args.video_frames, - output_directory.parent, + output_directory, start_frame=getattr(args, "start_frame", 1), cancel_check=cancel_check, ) @@ -1124,12 +1392,22 @@ def reconstruct(args, model_cache=None, cancel_check=None): background_depth_distance, ) write_status(args.status, 0.88, f"Exporting {args.export_format}") - export( - prediction, - args.export_format, - str(output_directory), - **export_kwargs, - ) + if args.export_format == "mesh": + export_textured_mesh_glb( + prediction, + images, + output_directory, + getattr(args, "conf_thresh_percentile", 40.0), + bool(getattr(args, "mesh_filter_edges", True)), + float(getattr(args, "mesh_max_edge_length", 0.1)), + ) + else: + export( + prediction, + args.export_format, + str(output_directory), + **export_kwargs, + ) check_cancelled(cancel_check) result_paths = wait_for_exported_paths( @@ -1141,19 +1419,13 @@ def reconstruct(args, model_cache=None, cancel_check=None): raise RuntimeError( f"No {args.export_format} files were exported" ) + result_paths = flatten_exported_paths( + output_directory, + result_paths, + cancel_check=cancel_check, + ) if args.export_format == "mini_npz": - nested_path = ( - output_directory / "exports" / "mini_npz" / "results.npz" - ) result_path = output_directory / "results.npz" - if nested_path.is_file(): - nested_path.replace(result_path) - result_paths = [result_path] - for directory in (nested_path.parent, nested_path.parent.parent): - try: - directory.rmdir() - except OSError: - pass if reference_depth is not None: write_npz_reference_depth(result_path, reference_depth) check_cancelled(cancel_check) diff --git a/tests/test_addon.py b/tests/test_addon.py index 925762b..ac17a55 100644 --- a/tests/test_addon.py +++ b/tests/test_addon.py @@ -1279,7 +1279,7 @@ class AddonTest(unittest.TestCase): self.assertEqual(paths["status"].parent, paths["result"]) self.assertEqual(paths["status"].stem, paths["result"].name) - def test_new_job_paths_uses_type_time_folder_name(self): + def test_new_job_paths_uses_unique_action_object_names(self): runtime = importlib.import_module("snapform.runtime") with TemporaryDirectory() as directory: jobs_root = Path(directory) @@ -1300,15 +1300,41 @@ class AddonTest(unittest.TestCase): self.assertRegex( depth_paths["result"].name, - r"^depth-plane_\d{8}-\d{6}$", + r"^generate-depth-plane_\d{8}-\d{6}-\d{6}$", ) self.assertTrue( background_paths["result"].name.startswith( - "remove-background_" + "remove-image-background_" ) ) self.assertTrue( - frames_paths["result"].name.startswith("depth-plane_") + frames_paths["result"].name.startswith( + "extract-original-frames_" + ) + ) + + def test_generated_inputs_are_moved_into_job_root(self): + runtime = importlib.import_module("snapform.runtime") + with TemporaryDirectory() as directory: + root = Path(directory) + staging = root / "staging" + staging.mkdir() + (staging / "frame.0001.jpg").write_bytes(b"first") + (staging / "frame.0002.jpg").write_bytes(b"second") + job_directory = root / "job" + job_directory.mkdir() + + result = runtime.move_input_into_job(staging, job_directory) + + self.assertEqual(result, job_directory) + self.assertFalse(staging.exists()) + self.assertEqual( + (job_directory / "frame.0001.jpg").read_bytes(), + b"first", + ) + self.assertEqual( + (job_directory / "frame.0002.jpg").read_bytes(), + b"second", ) def test_empty_image_start_frame_honors_image_user(self): @@ -1468,9 +1494,83 @@ class AddonTest(unittest.TestCase): self.assertEqual( identifiers, - ["glb", "gs_ply", "depth_vis", "mini_npz"], + ["glb", "gs_ply", "depth_vis", "mini_npz", "mesh"], ) self.assertEqual(export_format["default"], "glb") + self.assertEqual(export_format["name"], "Output Type") + self.assertEqual( + [item[1] for item in export_format["items"]], + [ + "Point Cloud (GLB)", + "Gaussian Splats (PLY)", + "Depth Preview (Images)", + "Depth Data (NPZ)", + "Textured Mesh (GLB)", + ], + ) + + def test_mesh_is_a_da3_output_type(self): + settings = SimpleNamespace( + input_path="", + export_format="mesh", + model="BASE", + gs_model="GIANT_1_1", + process_res=504, + process_res_method="upper_bound_resize", + conf_thresh_percentile=40.0, + num_max_points=1_000_000, + show_cameras=True, + gs_views_interval=1, + video_frames=16, + device="CPU", + use_ray_pose=False, + ref_view_strategy="saddle_balanced", + ) + operator = self.addon.RunJob() + operator.job_type = self.addon.operators.RECONSTRUCTION_JOB + + with TemporaryDirectory() as directory: + input_path = Path(directory) / "input.png" + input_path.touch() + settings.input_path = str(input_path) + paths = { + "log": Path(directory) / "job.log", + "status": Path(directory) / "job.json", + "result": Path(directory) / "result", + } + with patch.object( + self.addon.runtime, + "new_job_paths", + return_value=paths, + ), patch.object( + self.addon.runtime, + "runtime_ready", + return_value=True, + ), patch.object( + self.addon.runtime, + "model_ready", + return_value=True, + ), patch.object( + self.addon.runtime, + "reconstruct_request", + return_value=self.addon.runtime.HostJobRequest( + {"job_id": "test"} + ), + ) as reconstruct: + operator._prepare_command(None, settings) + + self.assertEqual(reconstruct.call_args.args[1], "mesh") + self.assertEqual(reconstruct.call_args.kwargs["model_key"], "BASE") + + def test_confidence_threshold_is_a_bounded_percentage(self): + confidence = self.addon.SnapFormSettings.__annotations__[ + "conf_thresh_percentile" + ] + + self.assertEqual(confidence["name"], "Confidence Threshold") + self.assertEqual(confidence["min"], 0.0) + self.assertEqual(confidence["max"], 100.0) + self.assertEqual(confidence["subtype"], "PERCENTAGE") def test_gs_model_property_only_includes_gaussian_models(self): gs_model = self.addon.SnapFormSettings.__annotations__[ @@ -1586,9 +1686,17 @@ class AddonTest(unittest.TestCase): "DEFAULT_CLOSED", self.addon.DebugPanel.bl_options, ) - for panel in (self.addon.Da3DebugPanel, self.addon.Ben2DebugPanel): + panels = ( + self.addon.Da3OutputDebugPanel, + self.addon.Ben2RemovalDebugPanel, + ) + for panel in panels: self.assertEqual(panel.bl_parent_id, self.addon.DebugPanel.bl_idname) self.assertIn("DEFAULT_CLOSED", panel.bl_options) + self.assertEqual( + [panel.bl_label for panel in panels], + ["DA3 Output Test", "BEN2 Removal Test"], + ) def test_host_panel_exposes_shared_host_state_and_controls(self): self.assertIn(self.addon.HostPanel, self.addon.CLASSES) diff --git a/tests/test_worker.py b/tests/test_worker.py index 215ea93..05ab409 100644 --- a/tests/test_worker.py +++ b/tests/test_worker.py @@ -474,6 +474,38 @@ class WorkerTest(unittest.TestCase): ) self.assertEqual(result, [npz_path]) + def test_move_exported_file_retries_windows_file_lock(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = root / "source.npz" + destination = root / "destination.npz" + source.touch() + path_type = type(source) + original_replace = path_type.replace + calls = 0 + + def locked_once(path, target): + nonlocal calls + calls += 1 + if calls == 1: + error = PermissionError("file is in use") + error.winerror = 32 + raise error + return original_replace(path, target) + + with patch.object(path_type, "replace", new=locked_once), patch.object( + self.worker.time, + "sleep", + ) as sleep: + result = self.worker.move_exported_file( + source, + destination, + ) + + self.assertEqual(result, destination) + self.assertTrue(destination.is_file()) + sleep.assert_called_once_with(0.1) + def test_rejects_unsupported_input_file(self): with tempfile.TemporaryDirectory() as directory: input_path = Path(directory) / "input.txt" @@ -850,9 +882,10 @@ class WorkerTest(unittest.TestCase): self.assertEqual(status["progress"], 1.0) self.assertIn("3 frame", status["message"]) - def test_uses_official_da3_export_parameters_for_all_four_formats(self): + def test_uses_da3_export_parameters_for_formats_and_debug_mesh(self): inference_options = [] export_calls = [] + mesh_calls = [] class FakeModel: @classmethod @@ -910,6 +943,27 @@ class WorkerTest(unittest.TestCase): fake_package = ModuleType("depth_anything_3") fake_package.__path__ = [] + def fake_mesh_export( + prediction, + images, + output_directory, + confidence_percentile, + filter_edges, + max_edge_length, + ): + mesh_calls.append( + ( + prediction, + images, + confidence_percentile, + filter_edges, + max_edge_length, + ) + ) + output_path = Path(output_directory) / "mesh.glb" + output_path.touch() + return output_path + with tempfile.TemporaryDirectory() as directory: root = Path(directory) input_path = root / "input.png" @@ -923,6 +977,10 @@ class WorkerTest(unittest.TestCase): "depth_anything_3.utils": fake_utils, "depth_anything_3.utils.export": fake_export_module, }, + ), patch.object( + self.worker, + "export_textured_mesh_glb", + side_effect=fake_mesh_export, ): for export_format in self.worker.EXPORT_FORMATS: args = SimpleNamespace( @@ -941,6 +999,8 @@ class WorkerTest(unittest.TestCase): num_max_points=12345, show_cameras=0, gs_views_interval=2, + mesh_filter_edges=True, + mesh_max_edge_length=0.2, ) self.worker.reconstruct(args) status = json.loads( @@ -948,9 +1008,17 @@ class WorkerTest(unittest.TestCase): ) self.assertEqual(status["progress"], 1.0) self.assertIn(export_format, status["message"]) + result_directory = Path(args.output_dir) + if export_format == "gs_ply": + self.assertTrue((result_directory / "0000.ply").is_file()) + self.assertFalse((result_directory / "gs_ply").exists()) + if export_format == "depth_vis": + self.assertTrue((result_directory / "0000.jpg").is_file()) + self.assertFalse((result_directory / "depth_vis").exists()) - self.assertEqual(len(inference_options), 4) + self.assertEqual(len(inference_options), 5) self.assertEqual(len(export_calls), 4) + self.assertEqual(len(mesh_calls), 1) for images, options in inference_options: self.assertEqual(images, [str(input_path)]) self.assertEqual(options["process_res"], 518) @@ -960,8 +1028,15 @@ class WorkerTest(unittest.TestCase): ) self.assertTrue(options["use_ray_pose"]) self.assertEqual(options["ref_view_strategy"], "middle") - for _prediction, export_format, _export_dir, options in export_calls: - expected_depth = -1.0 if export_format == "mini_npz" else 2.0 + for ( + _prediction, + export_format, + _export_dir, + options, + ) in export_calls: + expected_depth = ( + -1.0 if export_format == "mini_npz" else 2.0 + ) np.testing.assert_array_equal( _prediction.depth, np.full((1, 2, 2), expected_depth, dtype=np.float32), @@ -985,9 +1060,14 @@ class WorkerTest(unittest.TestCase): ) else: self.assertEqual(options, {}) + self.assertEqual(mesh_calls[0][1], [input_path]) + self.assertEqual(mesh_calls[0][2:], (35.0, True, 0.2)) infer_gs_values = [options["infer_gs"] for _, options in inference_options] - self.assertEqual(infer_gs_values, [False, True, False, False]) + self.assertEqual( + infer_gs_values, + [False, True, False, False, False], + ) def test_reconstruct_copies_frames_for_multi_frame_input(self): class FakeModel: @@ -1080,6 +1160,74 @@ class WorkerTest(unittest.TestCase): status = json.loads(Path(args.status).read_text(encoding="utf-8")) self.assertEqual(status["progress"], 1.0) + def test_mesh_confidence_uses_percentile(self): + depth = np.ones((1, 2, 2), dtype=np.float32) + confidence = np.array([[[1.0, 2.0], [3.0, 4.0]]]) + + threshold = self.worker.confidence_percentile_threshold( + depth, + confidence, + 50.0, + ) + + self.assertEqual(threshold, 2.5) + + def test_mesh_faces_apply_confidence_threshold_and_face_direction(self): + points = np.array( + [ + [[0.0, 0.0, 1.0], [1.0, 0.0, 1.0], [2.0, 0.0, 1.0]], + [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [2.0, 1.0, 1.0]], + ], + dtype=np.float32, + ) + depth = np.ones((2, 3), dtype=np.float32) + confidence = np.ones((2, 3), dtype=np.float32) + confidence[:, 0] = 0.0 + + faces = self.worker.mesh_triangle_faces( + points, + depth, + confidence, + confidence_threshold=0.5, + filter_edges=False, + max_edge_length=0.0, + ) + + np.testing.assert_array_equal(faces, [[1, 4, 5], [1, 5, 2]]) + vertices = points.reshape(-1, 3)[faces[0]] + normal = np.cross( + vertices[1] - vertices[0], + vertices[2] - vertices[0], + ) + self.assertLess(normal[2], 0.0) + + def test_mesh_percentile_reduces_face_count(self): + depth = np.ones((3, 3), dtype=np.float32) + confidence = np.arange(1, 10, dtype=np.float32).reshape(3, 3) + x, y = np.meshgrid( + np.arange(3, dtype=np.float32), + np.arange(3, dtype=np.float32), + ) + points = np.stack((x, y, depth), axis=-1) + face_counts = [] + for percentile in (0.0, 50.0, 100.0): + threshold = self.worker.confidence_percentile_threshold( + depth, + confidence, + percentile, + ) + faces = self.worker.mesh_triangle_faces( + points, + depth, + confidence, + confidence_threshold=threshold, + filter_edges=False, + max_edge_length=0.0, + ) + face_counts.append(len(faces)) + + self.assertEqual(face_counts, [8, 2, 0]) + if __name__ == "__main__": unittest.main()