diff --git a/AGENTS.md b/AGENTS.md index 2e7692a..ed9df49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,19 @@ uv lock --refresh - 新增或修改 Blender 类型时同步检查 `src/anyimage/__init__.py` 的注册与逆序注销，并补充对应测试 - 依赖版本、wheel 或 Manifest 改变时同步检查 `pyproject.toml`、`uv.lock`、`blender_manifest.toml` 和打包测试 +## Blender 节点图规范 + +- 不修改节点的 `name`，也不使用 `label` 覆盖节点标题；保留 Blender 为节点类型生成的默认名称和外观，避免读图时无法判断节点原型。用途应通过节点类型、参数、空间布局和必要的少量注释表达 + +- 节点图优先依靠清晰的空间层次表达流程，不用五颜六色的 Frame 做分类；仅在重复区、模拟区等确有包裹语义时少量使用 Frame，并保持配色克制 +- 避免跨越大段节点图的连线；来自 Group Input 的参数可按使用区域放置多个 Group Input，并隐藏未连接的 sockets +- 主处理链尽量保持为一根从左到右连续可见的几何主线，辅助字段和控制分支放在主线上下两侧，就近汇入消费者 +- 避免交错连线、节点重叠和不必要的回折；新增或重建节点组时检查节点坐标唯一性和主要连线方向 +- Repeat、Simulation、For Each Geometry Element 等具有输入/输出“头尾”的成对结构，应先确定完整包裹范围，再放置内部节点，保证头尾顺序、配对关系和区域边界一眼可辨 +- 作为修改器使用的 Geometry Nodes，除确实需要动态字段或其他结构的接口外，输入 socket 的 structure_type 应设为 SINGLE +- 为接口 socket 设置符合业务含义的 `subtype`；例如表示 Blender 场景实际长度的浮点输入使用 `DISTANCE`，明确限制为 `0–1` 的比例使用 `FACTOR`，以 `0–100` 表示的百分比使用 `PERCENTAGE`，角度使用对应 subtype。无上限倍率等不符合范围语义的数值保留普通 Float，不为显示形式强套 subtype +- 调整节点接口或布局后，应补充或更新 Blender 验证，检查 socket 顺序与结构、节点不重叠、Group Input 未使用输出已隐藏，以及几何主线可正常求值 + ## 文档与 Markdown 规范 - `README.md` 面向用户；实现原理放入 `docs/` 对应主题，不在 README 堆叠内部细节 diff --git a/README.md b/README.md index d79df13..820fffb 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,14 @@ Image Perspective 是 deform 工具，不提供 Refine 或反选。它与其他 完整操作说明见 [Image 编辑工具](docs/image-tools.md)。 +### Cutout 基础面与四种形状 + +在 Object Mode 的 3D Viewport 左侧工具栏选择 `Cutout`，沿 Image Empty 绘制后可从四个方向选择 Cutout、Balloon、Depth Balloon 或 Depth Surface；双击会选择完整图片。`Pixel Size` 控制规则三角格点密度。未启用 `Refine Selection` 时，几何只跟随套索，忽略原图 Alpha；启用后才使用 BEN2 Alpha 收紧几何选区。自交轨迹按非零环绕规则填充：同方向的重叠区域保留，只有反方向抵消的内环形成孔洞。未 Refine 的普通选区可以越过图片边框继续生成网格，但必须与原图可见像素相交；启用 Refine 后，框外部分会随最终 AI Alpha 被裁掉。基础面内部使用均匀六角格点，平滑轮廓作为约束边界；小选区会自动缩短边界采样间距，避免在较大的 `Pixel Size` 下被误判为空，同时不增加内部格点密度。Cutout 的 Thickness 以场景距离沿图片平面法线对称加厚，前盖、背面和轮廓侧壁均保持朝外。套索和 Refine Alpha 只决定几何轮廓；四种结果的材质颜色使用当前 Empty Image 的未裁切副本，深度与法线也分析完整图片，因此选区抗锯齿不会再次降低材质边缘 Alpha。若当前图片来自 Upscale，AI 分析会回溯到放大前的原图，颜色仍保留高清版本。未启用 Refine 且关闭 `Normal` 时，Cutout 与 Balloon 会直接在 Blender 内完成遮罩、轮廓提取和建面，不启动 BlendJob。`Normal` 默认关闭；启用后才通过 MoGe-2 生成 normal 纹理，并改用 Job。Depth Surface 直接使用 MoGe-2 原生的 object-space normal；Cutout、Balloon 与 Depth Balloon 会依据成品网格的平滑 TBN 另行烘制像素内容不同的 tangent-space normal。启用 Refine 时，Job 提供 BEN2 修正后的几何 Alpha；轮廓提取与建面仍由 Blender 完成。Depth Balloon 与 Depth Surface 因需要深度分析继续使用 Job，是否额外写入 normal 同样由 `Normal` 控制。 + +完整说明见 [Cutout](docs/cutout.md)。 + +`O Image Depth Surface` 将 `O Smooth` 的 Repeat Zone、双域边界采样和法线权重节点直接展开在主节点图中，并固定按 `Pin Sharp = 1`、`Pin Boundary = 1` 计算，不暴露这两个参数；`Iterations` 默认 `1`、最小可设为 `0`。`Depth Continuity` 默认 `0`，采用最宽松的断层阈值。Thickness 保持正面的深度表面不动，只向背面挤出。 + ### 套索生成 Balloon 选中单张图片的 Image Empty，右键打开 `AnyImage` 子菜单并选择 `Balloon Lasso Tool`。Blender 会激活 3D Viewport 左侧工具栏底部、位于独立 Image Perspective 之后的 Balloon Lasso；Image Mask 工具组与 Add Cube 之间保留一个间隔。Balloon 使用 Poly Build 的绿色图标、Annotate 同款铅笔光标，Tool Header 与状态栏共同提示当前工具状态，不再用黄色外框或强制修改 Shading、Overlay。Blender 的公开 Workspace Tool API 不支持按 Active Object 动态隐藏工具，因此图标在 Object Mode 中保持可见，但只有 Active Object 是 Image Empty 时才能启动 Stroke。 @@ -92,7 +100,7 @@ Image Perspective 是 deform 工具，不提供 Refine 或反选。它与其他 Depth Balloon 使用局部 X 作为深度轴，并创建 ` Base Plane` Single Arrow Empty。Empty 的旋转定义基面朝向，沿自身局部 Z 移动则调整 middle level。默认深度校准为 `修正深度 = (0.5 × 相对基面深度 + 原始 Balloon 最大半厚度) × Depth Strength`；原始最大半厚度来自生成后的固定网格，不受修改器 `Thickness` 影响。这在 `Thickness = 1` 时等价于旧参数的 `Depth Strength = 0.5` 和两倍半厚度基面偏移。节点再以 Smoother Step 中心程度插值修正厚度与修正深度；`Thickness` 只控制厚度分支，`Depth Strength` 只控制深度分支。`Double Sided Depth` 默认关闭，使背面维持稳定厚度。 -MoGe normal 会转换成适配 Balloon UV 的 tangent-space `normal.png`，由专用 `O Balloon Layer` 材质补回网格分辨率未能表达的细节；`Normal Strength` 默认 `0.5`，可在材质节点中调整。Balloon 材质通过 Custom Properties 直接保存源 Image datablock 引用，不依赖材质名或节点组名识别；勾选 `Reuse Material` 时优先复用同一 Image、同一材质类型的标记材质。Depth Balloon 共用一张原图尺寸的 normal，后续套索会把自己的有效区域合并进去，重叠区域采用最新结果。旧材质仍保留节点识别回退。当前只支持静态图片。 +MoGe normal 会转换成适配 Balloon UV 的 tangent-space `normal.png`，由统一的 `O Image Layer` 材质补回网格分辨率未能表达的细节；`Normal Scale` 默认 `1`，`Bump Scale` 默认 `0`，未连接 normal 时使用 `[0.5, 0.5, 1.0, 1.0]` 中性值。Balloon 材质通过 Custom Properties 直接保存源 Image datablock 引用，不依赖材质名或节点组名识别；勾选 `Reuse Material` 时优先复用同一 Image、同一材质类型的标记材质。Depth Balloon 共用一张原图尺寸的 normal，后续套索会把自己的有效区域合并进去，重叠区域采用最新结果。旧材质仍保留节点识别回退。当前只支持静态图片。 ### 去除背景 @@ -116,7 +124,7 @@ Surface 输出流程构建 DA3 与 MoGe-2 源网格：MoGe-2 使用深度断层 修改器完成剔面、快速重拓扑、Relax 与 UV 重投；顶层 `Iterations` 可设为 `0`。开启 Remove Background 时，BEN2 Alpha 只要大于 `0` 就视为前景；三角形三个顶点必须全部为前景，任一顶点 Alpha 为 `0` 都会删除该面并且不再交给 Geometry Nodes；Surface 材质使用生成的颜色图片。匹配相机默认不创建；开启后会作为 Surface 的子级。 -MoGe-2 的像素级 normal 会作为 Object Space normal 贴图接入 `O MoGe2 Surface Layer`，并与原图 +MoGe-2 的像素级 normal 会作为 object-space normal 贴图接入 `O Image Layer`，并与原图 Bump 合并；网格本身不写入 custom normal，因此重拓扑降低顶点数后仍能保留细腻的法线细节。 当前此功能仅支持单张图片。 diff --git a/docs/cutout.md b/docs/cutout.md new file mode 100644 index 0000000..c855d20 --- /dev/null +++ b/docs/cutout.md @@ -0,0 +1,31 @@ +# Cutout + +Cutout 用套索从 Image Empty 中选择范围，再从四个方向选择生成结果：左侧 **Cutout**、右侧 **Balloon**、下方 **Depth Balloon**、上方 **Depth Surface**。双击图片会把完整画面作为选区。 + +## 绘制选区 + +在 Object Mode 的 3D Viewport 左侧工具栏选择 **Cutout**，按住左键绘制，松开后选择形状。绘制时按 `F` 切换反选，右键或 `Esc` 取消当前轨迹。 + +自交轨迹使用非零环绕填充规则。沿相同方向围住的区域即使重叠多次也会保留；只有反方向环绕使计数抵消时才形成孔洞。因此交错部分不会再被当成洞扣除，同时仍能表达中间的反向内环孔洞。普通选区可以越过 Image Empty 边框，框外轨迹仍会生成网格；选区必须与原图中的可见像素相交才有效。反选仍限制在原图范围，避免把框外无限区域截成任意矩形。未启用 **Refine Selection** 时，选区边缘会做轻量抗锯齿；启用后，BEN2 先在同一个几何选区内计算前景 Alpha，再应用反选。 + +## 基础面密度 + +**Pixel Size** 表示基础面相邻采样点大约对应多少源图片像素，可设为 `4`–`32`，默认 `32`。数值越小，规则三角格点越密，后续形状能表达的细节越多，同时网格点数和约束三角化成本也越高。最长边的采样级别限制在 `4`–`1024`。 + +基础面先从最终 Alpha 提取外边界与孔洞，再以接近等边三角形的六角格点填充内部。轮廓按目标间距重采样和平滑，边界点仍作为约束参与 Delaunay，因此内部点保持均匀，外缘不会退化成规则栅格的阶梯锯齿。 + +四种结果都启用平滑着色。Cutout 与 Depth Surface 的修改器也在输出阶段显式设置 Smooth Shading；Balloon 与 Depth Balloon 在创建源网格时设置。 + +Cutout 的基础三角面统一按图片正面逆时针排列，Thickness 沿图片平面法线向前后各偏移一半；背面与轮廓侧壁使用相反绕序，所有封闭面的法线都朝向几何外部。Depth Surface 则保持正面深度表面不动，只沿法线反方向挤出背面。Thickness 是 Blender 场景中的实际距离，因此节点接口使用 Distance 单位显示。 + +Depth Surface 的 `O Image Depth Surface` 按“深度 → 断层剔除 → 向后厚度 → 零厚度清理 → 展开的 O Smooth 结构 → 平滑着色”从左到右排列，使内置平滑与在完整输出后连接 `O Smooth` 获得相同的输入几何。平滑段把 O Essentials 参考资产的 Repeat Zone、两条不同求值域的边界判断、边界采样与法线一致性权重节点直接放入主节点图，每轮重新计算后再移动点，不再嵌套 `O Smooth` 节点组；内部固定按 `Pin Sharp = 1`、`Pin Boundary = 1` 计算，不暴露这两个参数，`Iterations` 默认 `1`、最小可设为 `0`。Depth Continuity 使用与 `O MoGe2 Surface` 相同的百分比阈值曲线，默认 `0`，对应最宽松的断层阈值；提高数值会逐步剔除更多过远的深度跳变面。 + +## 计算说明 + +未启用 **Refine Selection** 时，几何范围只使用抗锯齿套索 mask，原图 Alpha 不参与轮廓；原图 Alpha 只用于确认选区内至少存在可见像素，并继续控制材质透明度。套索或 Refine 产生的 Alpha 只用于几何，不再写进材质颜色贴图：四种结果的 Color 都使用当前 Image Empty 的未裁切 Image 副本，避免选区边缘 Alpha 被材质再次相乘，同时不与源数据块共享可变状态。深度和法线分析同样读取完整图片；若当前图片来自 Upscale，AI 分析读取放大前的原图，材质颜色仍使用当前高清图。Depth Balloon 在整图推理后才用按分析分辨率缩放的套索 mask 限定有效深度。关闭 **Normal** 时，Cutout 与 Balloon 直接在 Blender 内处理，不准备临时输入、不启动 BlendJob；启用 **Normal** 后才提交 Job。MoGe 返回的相机轴法线先按源 Image Empty 的对象坐标编码。Depth Surface 原样保留这张 object-space 贴图；Cutout、Balloon 与 Depth Balloon 则在最终几何节点结果可用后，使用成品网格的 UV、平滑角法线和切线基底重新烘成像素内容不同的 tangent-space 贴图。启用 Refine 时，Job 先计算 BEN2 Alpha 并返回几何 mask，此时几何才从最终 Alpha 提取轮廓。Depth 类型始终通过 Job 取得深度，但也只在启用 **Normal** 时写入 normal。四种形状的网格都在 Blender 主线程构建，不再通过 `contour.json` 传递边界。 + +最终 Alpha 先由 NumPy 批量找出四向边界，只沿实际周长追踪轮廓，并按包含关系把孔洞归入外轮廓。未 Refine 时直接复用套索 mask；只有真实出框时才建立扩展几何画布。启用 Refine 后完全使用原图画布内的最终 AI Alpha，所有框外套索区域都会被裁掉。格点准备使用批量数组生成；点内判断按 Y 范围分桶，只比较当前扫描带可能相交的轮廓边；边界安全距离也只检查每条边附近的少量格点。短轮廓会自动使用不大于周长八分之一的边界采样间距，避免有效小区域因当前 `Pixel Size` 较大而消失；内部格点仍严格使用用户设置的 `Pixel Size`。UV 与封边边界使用 Blender `foreach_*` 和 NumPy 批量写入，不逐元素遍历。Pillow 只会加速占比较小的 Alpha 模糊和缩放，因此 Blender 本地路径保持纯 NumPy，不新增 wheel。最终约束 Delaunay 和网格创建仍随实际点数增长，极密网格应按成品需要使用。 + +Cutout 当前只支持静态图片。**Refine Selection**、Depth 与 normal 分析需要已配置的 AI Environment 和对应模型。 + +Cutout、Balloon、Depth Balloon、Depth Surface 以及普通 Surface 的图片材质共用 `O Image Layer`。节点组始终提供 Normal 输入和 Object Space 开关；没有 normal 纹理时使用 `[0.5, 0.5, 1.0, 1.0]` 中性默认值。这个开关只选择 Blender Normal Map 节点的解码空间，不承担 Object-to-Tangent 转换：三种 tangent-space 结果在接入材质前已经完成像素烘制，Depth Surface 才打开 Object Space 并直接使用 MoGe 原图。 diff --git a/docs/development.md b/docs/development.md index e9c73b2..7e7bc0c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -35,7 +35,7 @@ Blender 交互层管理 UI、RNA 状态、任务参数和结果导入。Runtime Mesh Plane、Depth Plane、Surface、Background 和 Balloon 按功能组织。一个功能的入口、请求和 Blender 回写放在一起；跨功能的图片、材质和参数处理放在 `operators/common.py`、`operators/image.py` 与 `operators/material.py`。 -Server 的模型模块返回 Prediction，几何模块处理数组与网格，输出模块写文件。Blender 数据只在 Operator Response 的主线程阶段创建。 +Server 的模型模块返回 Prediction，几何模块处理数组与网格，输出模块写文件。Blender 数据只在 Operator Response 的主线程阶段创建。Cutout 与 Balloon 在未启用 Refine 和 `Normal` 时不提交 Job：Operator 直接在内存中合成最终 Alpha，并调用 `server/geometry/cutout.py` 的纯数据轮廓函数后创建网格。Refine、Normal 与 Depth Job 只返回最终图片 Alpha 和被选中的 AI 数组或纹理；`normal_analysis` 为假时不请求 MoGe-2 normal 模型，也不写入 `normal.png`。外轮廓、孔洞及基础面始终在 Blender 主线程生成，不使用 `contour.json` 作为中间结果。 扩展级设置统一通过 `preferences.addon_preferences()` 读取，不归属 BlendJob Runtime；Storage Root 路径回调也由 Preferences 模块提供，Runtime 只消费回调结果。扩展包标识直接使用根模块中的 `__package__`，以兼容 Blender 实际的 `bl_ext..anyimage` 命名空间；不得按第一个点截断，否则会错误查找 `bl_ext` 并静默回退默认值。Preferences 的设备枚举和面板数据按需导入，避免循环依赖。 diff --git a/docs/features.md b/docs/features.md index 8a0ac26..1c3cf39 100644 --- a/docs/features.md +++ b/docs/features.md @@ -14,6 +14,7 @@ | Upscale | 在安全输出上限内增加图片像素 | 单图、视频、序列帧 | 三档模型生成的高清图片或序列 | | Image Mask | 用 Box、Lasso、Line 或 Polyline 直接清理图片 | 静态 Image Empty | 原位更新并 Pack 的透明裁切图 | | Image Perspective | 四点透视矫正裁切 | 静态 Image Empty | Pack 的矩形裁切图 | +| Cutout | 套索选择区域并生成四种网格形状 | 静态 Image Empty | 带孔洞与原图 UV 的规则三角基础面 | | Balloon | 沿轮廓绘制鼓起造型 | 静态 Image Empty | 带原图 UV 的封闭网格 | | Debug | 检查模型原始输出 | 模型支持的图片或帧序列 | 保存在 Job 目录中的原始 Prediction | @@ -27,7 +28,11 @@ Depth Plane 与 Surface 可使用 DA3 或 MoGe-2。DA3 提供相对深度和相 `Convert to Mesh Plane` 位于 `Convert to Depth Plane` 上方，不需要 AI Environment。它生成基础四边形，并由 `O Mesh Plane` 修改器实时处理 `Thickness` 与 `Subdivide`；`Thickness` 为 `0` 时保持单面。操作会立即执行；创建后可在 Last Operation 中调整 `Shadeless` 与 `Keep Original`。 -Balloon 作为不依赖 AI 的基础工具单独排列；Mesh Plane、Depth Plane 与 Surface 属于转换组，Remove Background 与 Upscale 由分割线归入图片处理组。Depth Plane、Surface、Remove Background 与 Upscale 四个 AI 功能在缺少 Environment 或默认模型时仍可点击：缺 Environment 时转入 `Set Up AI Environment…`，只缺默认模型时转入 `Download Required Models…`。全部就绪后，菜单底部显示 `AI Environment Settings…`，用于打开 AnyImage 扩展偏好设置。Upscale 菜单项同时显示当前图片分辨率，不弹出属性对话框；模型与输出长边上限统一读取 Preferences 的 `Default Upscale Model` 和 `Maximum Upscale Size`，Operator 不注册 Last Operation 重做面板。高清结果再次执行时从保存的原图重新生成并替换当前版本，不级联处理已有结果；名称始终基于原图，透明图片保留放大后的 Alpha。 +Cutout 复用 Image Lasso 的反选与 Refine 流程，但自交选区使用非零环绕填充：同方向重叠保留，反方向内环形成孔洞。基础面以均匀六角格点填充平滑约束轮廓，`Pixel Size` 控制内部密度，短轮廓会自适应边界采样以免有效小区域消失。未启用 Refine 的 Cutout 与 Balloon 只按套索建面，不把原图 Alpha 计入几何；同时关闭 `Normal` 时在 Blender 内完成遮罩、轮廓和建面，不启动 BlendJob。套索或 Refine Alpha 只作为几何 mask，材质颜色使用当前 Empty Image 的未裁切副本，MoGe 深度与法线始终分析完整图片；若图片来自 Upscale，AI 分析回溯到放大前的原图，避免放大不必要地增加计算量和伪细节。`Normal` 默认关闭，启用后才用 MoGe-2 写入 normal；Depth Surface 保留 MoGe 原生 object-space 像素，Cutout、Balloon 与 Depth Balloon 根据成品网格 TBN 烘成不同的 tangent-space 像素。Depth 类型始终使用 Job 生成深度，但同样由该选项决定是否写入 normal。Refine 启用后由 BEN2 返回原图画布内的最终 Alpha，Blender 再从该 Alpha 建面，框外套索区域不会保留。详细操作与计算边界见 [Cutout](cutout.md)。 + +`O Image Depth Surface` 的平滑阶段直接展开 O Essentials `O Smooth` 的 Repeat Zone 与双域边界判断结构，不嵌套组节点；内部固定按 `Pin Sharp = 1`、`Pin Boundary = 1` 计算，不暴露这两个参数，`Iterations` 默认 `1`、最小可设为 `0`。`Depth Continuity` 默认 `0`；Thickness 固定正面，只向背面挤出。 + +Balloon 作为不依赖 AI 的基础工具单独排列；Mesh Plane、Depth Plane 与 Surface 属于转换组，Remove Background 与 Upscale 由分割线归入图片处理组。Depth Plane、Surface、Remove Background 与 Upscale 四个 AI 功能在缺少 Environment 或默认模型时仍可点击：缺 Environment 时转入 `Set Up AI Environment…`，只缺默认模型时转入 `Download Required Models…`。全部就绪后，菜单底部显示 `AI Environment Settings…`，用于打开 AnyImage 扩展偏好设置。Upscale 菜单项同时显示当前图片分辨率，不弹出属性对话框；模型与输出长边上限统一读取 Preferences 的 `Default Upscale Model` 和 `Maximum Upscale Size`，Operator 不注册 Last Operation 重做面板。高清结果再次执行时从保存的原图重新生成并替换当前版本，不级联处理已有结果；名称始终基于原图，透明图片保留放大后的 Alpha。Depth Plane、Surface 与 Cutout 的深度或法线推理也会沿用该原图引用；高清结果继续用于颜色材质和选区显示。 转换任务完成后，AnyImage 在一个 Undo 步骤中创建网格、图片、材质和修改器。源 Image Empty 是否保留由转换选项决定。 @@ -37,7 +42,7 @@ Image Box、Lasso、Line 与 Polyline 组成 Mask 工具组，并共用 `EditIma Balloon 默认在 Blender 内部生成，无需 Environment 或模型。Tool Settings 提供模式、厚度、采样距离、材质复用、`Contour`、`Depth` 与 MoGe-2 `Depth Model`；AI 未就绪时只禁用这三个控件，并在 Tool Settings 中显示安装按钮。Depth 会把物理距离相对化并稳健拟合初始基面，保留未截断的正负起伏。Base Plane 子 Empty 的旋转定义基面朝向，沿自身局部 Z 移动定义 middle level。 -`O Balloon Shape` 计算 `修正厚度 = abs(原 X) × Thickness`，并将修正深度校准为 `(0.5 × 相对基面深度 + 原始 Balloon 最大半厚度) × Depth Strength`。固定的原始半厚度不乘 `Thickness`，因此 Thickness 与 Depth Strength 分别只控制各自分支。节点以归一化原 X 距离经 Smoother Step 得到中心程度，再插值厚度与深度。`Double Sided Depth` 默认关闭，背面保持 Balloon 厚度。MoGe normal 会输出为原图尺寸的 tangent-space 纹理，并通过 `O Balloon Layer` 的 `Normal Strength`（默认 `0.5`）补偿网格无法表达的细节。 +`O Balloon Shape` 计算 `修正厚度 = abs(原 X) × Thickness`，并将修正深度校准为 `(0.5 × 相对基面深度 + 原始 Balloon 最大半厚度) × Depth Strength`。固定的原始半厚度不乘 `Thickness`，因此 Thickness 与 Depth Strength 分别只控制各自分支。节点以归一化原 X 距离经 Smoother Step 得到中心程度，再插值厚度与深度。`Double Sided Depth` 默认关闭，背面保持 Balloon 厚度。MoGe 相机轴 normal 会在 Blender 评估最终 Balloon 后按平滑 TBN 烘为原图尺寸的 tangent-space 纹理，并通过统一 `O Image Layer` 的 `Normal Scale` 补偿网格无法表达的细节；`Normal Scale` 默认 `1`，`Bump Scale` 默认 `0`，未连接 normal 时使用中性默认值。 新建 Balloon 材质会用 Custom Properties 保存源 Image datablock 和可选 normal Image 的直接引用。`Reuse Material` 优先按这些标记匹配，不依赖材质或节点组名称；普通与 Depth 材质分别复用。Depth Balloon 的共享 normal 会逐次合并新套索的有效区域，避免复用材质时丢失其他套索的法线。 diff --git a/docs/outputs.md b/docs/outputs.md index 279ac47..7bdbd6e 100644 --- a/docs/outputs.md +++ b/docs/outputs.md @@ -11,6 +11,7 @@ | `remove-background` | `image.0001.png` 起 | 带 Alpha 的图片层 | | `upscale-image` | `upscale.0001.png` 起 | 所选高清模型在输出上限内生成的 4× 高清层，并保留源图 Alpha | | `refine-image-mask` | `selection-input.png`、`image.png`、`bounds.json` | BEN2 细化后的局部内容 mask 与原图坐标 bounds | +| `generate-mesh` | 几何 mask `image.0001.png`，按形状输出 `normal.png`、`depth.npz` 或 `field.npz` | 套索或 Refine Alpha 只供 Blender 生成轮廓；材质使用当前 Empty Image 的未裁切副本，Upscale 结果的深度与法线使用放大前原图 | | `analyze-balloon` | `lasso-input.png`，按选项输出 `foreground.png`、`contour.json`、`depth.npz`、`normal.png` | 套索约束输入、BEN2 轮廓与 MoGe-2 深度、法线 | | `debug-da3` | `prediction.npz` | DA3 API 返回 | | `debug-moge2` | `prediction*.npz` | MoGe-2 API 返回 | @@ -21,7 +22,7 @@ DA3 Depth 结果包含 `depth`、`conf`、`extrinsics` 与 `intrinsics`。MoGe-2 Depth 结果包含 `depth`、`normal`、`mask` 与 `intrinsics`。 -Balloon Depth 的 `depth.npz` 包含相对 `depth_signal`、归一化拟合平面 `baseline_vector`、`valid`、`reference_depth`、裁切原点与输入尺寸。Geometry Nodes 将归一化原 X 距离经 Smoother Step 后作为 Thickness 与 Depth 的混合系数。`normal.png` 是铺回原图尺寸的 tangent-space 法线；套索和有效区域外为中性法线。 +Balloon Depth 的 `depth.npz` 包含相对 `depth_signal`、归一化拟合平面 `baseline_vector`、`valid`、`reference_depth`、取样原点与输入尺寸。Cutout 的 Depth Balloon 对完整原图推理，再以套索 mask 限定深度 `valid`；Geometry Nodes 将归一化原 X 距离经 Smoother Step 后作为 Thickness 与 Depth 的混合系数。Job 返回的 `normal.png` 是原图尺寸的 MoGe 相机轴法线中间结果，不按套索清空；只有原图透明或模型无效区域使用中性法线。Blender 创建并评估成品网格后，再用其 UV、平滑角法线与切线基底烘成实际使用的 tangent-space Image。 Surface 的 `mesh.npz` 包含 `vertices`、`faces`、`uv`、`intrinsics`、`image_size` 与 `reference_depth`，以及 Geometry Nodes 使用的网格属性。 @@ -31,4 +32,4 @@ Debug 文件保持模型 API 的数据形态，用于比较推理结果；正式 正式图片层使用 `image.0001.*` 起始的文件名，Upscale 层单独使用 `upscale.0001.png`。导入 Blender 后沿用源 Image 名称，由 Blender 在同名冲突时自动添加 `.001`；不再添加 `.color`。深度动画使用 EXR 序列以保存浮点值。源 Empty 为 Movie 或 Sequence 时，Blender 按序列载入这些文件，即使任务只输出一帧。 -MoGe-2 Surface 的法线单独保存为 `normal.png`。材质在 Blender 端创建，不属于 Job 文件格式。 +MoGe-2 Surface 的原生相机法线单独保存为 `normal.png`，Blender 以 object-space 解码。Depth Surface 同样直接保留该像素编码。Cutout、Balloon 与 Depth Balloon 不会把同一张图直接当作 tangent-space 解码，而是在 Blender 端依据各自最终评估网格的 TBN 重新生成 tangent-space 像素；材质的空间选择只负责匹配已经生成好的贴图。 diff --git a/docs/server.md b/docs/server.md index 0e38830..1fb7e68 100644 --- a/docs/server.md +++ b/docs/server.md @@ -56,7 +56,9 @@ BlendJob 提供 Runtime、HTTP transport、队列、状态和取消协议。AnyI Environment 使用 Python 3.12、ONNX Runtime、NumPy、Pillow、PyAV、trimesh、FastAPI 和 Uvicorn。Windows 使用 DirectML 或 CPU；其他平台根据支持情况使用 CoreML、OpenVINO 或 CPU。 -Balloon 的 `analyze-balloon` Job 会裁切套索输入并清零选区外 RGBA。`Contour` 用 BEN2 回传最大连通轮廓；`Depth` 用 MoGe-2 对有效相机距离稳健拟合二维基面，并把物理距离换算为不受绝对拍摄距离影响、未截断的相对深度信号。Job 同时把 MoGe normal 转换为 tangent-space 纹理，铺回原图尺寸并以中性法线填充无效区域。Blender 主线程负责采样深度属性、创建 Base Plane Empty、组装 `O Balloon Shape` 修改器与 `O Balloon Layer` 材质。 +Balloon 的 `analyze-balloon` Job 会裁切套索输入并清零选区外 RGBA。`Contour` 用 BEN2 回传最大连通轮廓；`Depth` 用 MoGe-2 对有效相机距离稳健拟合二维基面，并把物理距离换算为不受绝对拍摄距离影响、未截断的相对深度信号。Job 同时把 MoGe 相机轴 normal 铺回原图尺寸，并以中性法线填充无效区域。Blender 主线程负责采样深度属性、创建 Base Plane Empty、组装 `O Balloon Shape` 修改器，再根据最终评估网格的 TBN 把相机轴 normal 烘成 tangent-space Image 并接入统一的 `O Image Layer` 材质。 + +Cutout 的 `generate-mesh` 不复用上述裁切图作为 AI 输入。套索或 BEN2 结果仅写成几何 mask；MoGe 深度与法线始终分析完整图片。输入为 Upscale 结果时，请求会额外携带放大前的 `analysis_input`，推理读取该输入，颜色与几何 mask 仍读取当前高清图；Depth Balloon 会先把套索坐标缩放到分析分辨率，再限制有效深度。Blender 材质使用当前 Empty Image 的未裁切副本。 Image Box 与 Lasso 的 `Refine Selection` 使用 `refine-image-mask` Job。它与标准 `remove-background` Job 共用单图 BEN2 函数；Server 在推理前清空 polygon 外的 RGBA，推理后再次应用几何 mask，并返回局部 Alpha 与源图 bounds。Line、Polyline 与 Perspective 不提交 Job。 diff --git a/docs/surface.md b/docs/surface.md index c07f30b..d4e6e59 100644 --- a/docs/surface.md +++ b/docs/surface.md @@ -57,7 +57,7 @@ DA3 使用 `O DA3 Surface`，MoGe-2 使用 `O MoGe2 Surface`。两者采用相 ## 材质与文件 -颜色统一写为 `image.0001.*`，导入时沿用源 Image 名称，由 Blender 自动解决同名冲突。MoGe-2 还会输出 `normal.png`，用于材质法线。材质名称以源 Image datablock 为基名。 +颜色统一写为 `image.0001.*`，导入时沿用源 Image 名称，由 Blender 自动解决同名冲突。MoGe-2 还会输出 `normal.png`，作为 object-space normal 接入统一的 `O Image Layer`；材质会自动打开 Object Space，未提供 normal 的图片材质使用 `[0.5, 0.5, 1.0, 1.0]` 中性默认值。材质名称以源 Image datablock 为基名。 单图结果可以 Pack 到 `.blend`。模型原始结果与 `mesh.npz` 保留在 Job 目录中，便于调试和重新检查。 diff --git a/docs/upscale.md b/docs/upscale.md index 68b9663..ab2bca2 100644 --- a/docs/upscale.md +++ b/docs/upscale.md @@ -20,4 +20,6 @@ Upscale 用 ONNX 模型把 Image Empty 的图片放大 4 倍，不创建网格 对高清结果再次执行 Upscale 时，AnyImage 会从最初的原图重新生成并替换当前结果，不会连续放大已有高清图。 +把高清结果继续转换为 Depth Plane、Surface 或 Cutout 时，颜色材质仍使用当前高清图；深度与法线推理会读取保存的放大前原图，以减少计算量并避免凭空增加不必要的几何细节。 + 如果显存或内存不足，请降低 **Maximum Upscale Size**，或改用 CPU。 diff --git a/mkdocs.yml b/mkdocs.yml index 76e7990..5c407a9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,6 +16,7 @@ nav: - 使用指南: - 选择转换方式: workflows.md - Image 编辑工具: image-tools.md + - Cutout: cutout.md - Mesh Plane: mesh-plane.md - Depth Plane: guide-depth-plane.md - Surface: guide-surface.md diff --git a/src/anyimage/__init__.py b/src/anyimage/__init__.py index e21a589..c4c4623 100644 --- a/src/anyimage/__init__.py +++ b/src/anyimage/__init__.py @@ -2,10 +2,9 @@ import bpy from . import clipboard_image from .clipboard_image import CLASSES as CLIPBOARD_CLASSES - -from .runtime import runtime +from .menu import AnyImageImageMenu, draw_image_context_menu from .operators import CLASSES as OPERATOR_CLASSES -from .operators.balloon import BalloonTool +from .operators.cutout import CutoutTool from .operators.image_edit import ( ImageBoxTool, ImageLassoTool, @@ -13,9 +12,6 @@ from .operators.image_edit import ( ImagePerspectiveTool, ImagePolylineTool, ) -from .preferences import AnyImagePreferences -from .properties import AnyImageSettings -from .menu import AnyImageImageMenu, draw_image_context_menu from .panel import ( Ben2RemovalDebugPanel, Da3OutputDebugPanel, @@ -24,6 +20,9 @@ from .panel import ( ServerPanel, UpscaleDebugPanel, ) +from .preferences import AnyImagePreferences +from .properties import AnyImageSettings +from .runtime import runtime CLASSES = ( @@ -44,11 +43,7 @@ CLASSES = ( def register(): for class_type in CLASSES: bpy.utils.register_class(class_type) - bpy.utils.register_tool( - ImageBoxTool, - separator=True, - group=True, - ) + bpy.utils.register_tool(ImageBoxTool, separator=True, group=True) bpy.utils.register_tool( ImageLassoTool, after={ImageBoxTool.bl_idname}, @@ -64,16 +59,15 @@ def register(): after={ImageLineTool.bl_idname}, separator=False, ) + bpy.utils.register_tool(ImagePerspectiveTool, separator=False) bpy.utils.register_tool( - ImagePerspectiveTool, - separator=False, - ) - bpy.utils.register_tool(BalloonTool, separator=False) - bpy.types.Scene.anyimage_settings = bpy.props.PointerProperty( - type=AnyImageSettings, + CutoutTool, + after={ImagePerspectiveTool.bl_idname}, + separator=True, ) + bpy.types.Scene.anyimage_settings = bpy.props.PointerProperty(type=AnyImageSettings) bpy.types.Image.anyimage_upscale_source = bpy.props.PointerProperty( - type=bpy.types.Image, + type=bpy.types.Image ) bpy.types.VIEW3D_MT_object_context_menu.prepend(draw_image_context_menu) bpy.types.OUTLINER_MT_object.prepend(draw_image_context_menu) @@ -84,7 +78,7 @@ def register(): def unregister(): runtime.unregister() clipboard_image.unregister_keymaps() - bpy.utils.unregister_tool(BalloonTool) + bpy.utils.unregister_tool(CutoutTool) bpy.utils.unregister_tool(ImagePerspectiveTool) bpy.utils.unregister_tool(ImagePolylineTool) bpy.utils.unregister_tool(ImageLineTool) diff --git a/src/anyimage/menu.py b/src/anyimage/menu.py index 446aaa3..1dff836 100644 --- a/src/anyimage/menu.py +++ b/src/anyimage/menu.py @@ -8,28 +8,20 @@ class AnyImageImageMenu(bpy.types.Menu): bl_label = "AnyImage" def draw(self, context): - from .properties import ( - ai_setup_label, - ai_status, - ) + from .properties import ai_setup_label, ai_status layout = self.layout layout.operator_context = "INVOKE_DEFAULT" layout.operator( - "anyimage.convert_to_mesh_plane", - text="Convert to Mesh Plane…", + "anyimage.convert_to_plane", + text="Convert to Plane", icon="MESH_PLANE", ) layout.operator( "anyimage.convert_to_depth_plane", - text="Convert to Depth Plane…", + text="Convert to Depth Plane", icon="MOD_DISPLACE", ) - layout.operator( - "anyimage.convert_to_surface", - text="Convert to Surface…", - icon="MESH_GRID", - ) layout.separator() layout.operator( "anyimage.remove_image_background", @@ -46,7 +38,7 @@ class AnyImageImageMenu(bpy.types.Menu): if status["ready"]: layout.operator( "anyimage.open_ai_environment_settings", - text="AI Environment Settings…", + text="AI Environment Settings", icon="PREFERENCES", ) else: @@ -72,9 +64,5 @@ def upscale_menu_label(context): def draw_image_context_menu(self, context): if not is_image_empty(context.object): return - self.layout.menu( - AnyImageImageMenu.bl_idname, - text="AnyImage", - icon="PLUGIN", - ) + self.layout.menu(AnyImageImageMenu.bl_idname, text="AnyImage", icon="PLUGIN") self.layout.separator() diff --git a/src/anyimage/operators/__init__.py b/src/anyimage/operators/__init__.py index 2e8d2ff..6c3f676 100644 --- a/src/anyimage/operators/__init__.py +++ b/src/anyimage/operators/__init__.py @@ -1,17 +1,12 @@ from .background import ProcessImageBackground, RemoveImageBackground -from .balloon import ( - ActivateBalloonTool, - ExtractBalloon, - AnalyzeBalloon, -) +from .cutout import ActivateCutoutTool, SelectCutoutRegion from .debug import DebugBen2, DebugDa3, DebugMoge2, DebugUpscale from .image_edit import CLASSES as IMAGE_EDIT_CLASSES -from .depth_plane import ( +from .mesh import ( + ConvertCutoutShape, ConvertToDepthPlane, - ExtractOriginalFrames, - GenerateDa3Depth, - GenerateMoge2Depth, - GenerateOriginalBackground, + ConvertToPlane, + GenerateMesh, ) from .models import ( ClearModels, @@ -20,8 +15,6 @@ from .models import ( OpenAIEnvironmentSettings, SetupAIEnvironment, ) -from .mesh_plane import ConvertToMeshPlane -from .surface import ConvertToSurface, GenerateDa3Surface, GenerateMoge2Surface from .upscale import UpscaleImage @@ -34,21 +27,15 @@ CLASSES = ( DebugMoge2, DebugBen2, DebugUpscale, - GenerateDa3Depth, - GenerateMoge2Depth, - GenerateDa3Surface, - GenerateMoge2Surface, + GenerateMesh, + SelectCutoutRegion, + ConvertCutoutShape, + ActivateCutoutTool, ProcessImageBackground, - GenerateOriginalBackground, - ExtractOriginalFrames, ClearModels, *IMAGE_EDIT_CLASSES, - ExtractBalloon, - AnalyzeBalloon, - ActivateBalloonTool, RemoveImageBackground, UpscaleImage, - ConvertToMeshPlane, + ConvertToPlane, ConvertToDepthPlane, - ConvertToSurface, ) diff --git a/src/anyimage/operators/balloon/__init__.py b/src/anyimage/operators/balloon/__init__.py index 30cf68e..b37aca9 100644 --- a/src/anyimage/operators/balloon/__init__.py +++ b/src/anyimage/operators/balloon/__init__.py @@ -1,6 +1 @@ -from .operator import ( - ActivateBalloonTool, - BalloonTool, - ExtractBalloon, - AnalyzeBalloon, -) +"""Balloon profile geometry used by the unified mesh converter.""" diff --git a/src/anyimage/operators/balloon/depth.py b/src/anyimage/operators/balloon/depth.py index 1ca40d0..26bca8b 100644 --- a/src/anyimage/operators/balloon/depth.py +++ b/src/anyimage/operators/balloon/depth.py @@ -4,7 +4,6 @@ from pathlib import Path import bpy - BALLOON_DEPTH_ATTRIBUTE = "o_balloon_depth" BALLOON_SHAPE_NODE_GROUP_NAME = "O Balloon Shape" BALLOON_SHAPE_NODE_GROUP_VERSION = 16 diff --git a/src/anyimage/operators/balloon/geometry.py b/src/anyimage/operators/balloon/geometry.py index 807627e..c093a0b 100644 --- a/src/anyimage/operators/balloon/geometry.py +++ b/src/anyimage/operators/balloon/geometry.py @@ -45,8 +45,8 @@ def resample_closed_polygon(polygon, spacing): return points[indices] + segments[indices] * factors[:, None] -def smooth_lasso_contour(polygon, spacing, iterations=4): - contour = resample_closed_polygon(polygon, spacing * 0.5) +def smooth_lasso_contour(polygon, spacing, iterations=16): + contour = resample_closed_polygon(polygon, spacing * 0.25) original_area = abs(polygon_area(contour)) for _index in range(iterations): import numpy as np @@ -68,81 +68,182 @@ def smooth_lasso_contour(polygon, spacing, iterations=4): return contour +def _contour_edges(contours): + import numpy as np + + starts = [] + ends = [] + for contour in contours: + contour = np.asarray(contour, dtype=np.float64) + following = np.roll(contour, -1, axis=0) + usable = np.abs(following[:, 1] - contour[:, 1]) > 1e-12 + starts.append(contour[usable]) + ends.append(following[usable]) + if not starts: + empty = np.empty((0, 2), dtype=np.float64) + return empty, empty + return np.concatenate(starts), np.concatenate(ends) + + +def _edge_bucket_indices(starts, ends, minimum_y, bucket_height, bucket_count): + import numpy as np + + lower = np.minimum(starts[:, 1], ends[:, 1]) + upper = np.maximum(starts[:, 1], ends[:, 1]) + first = np.floor((lower - minimum_y) / bucket_height).astype(np.int64) + last = np.floor((np.nextafter(upper, lower) - minimum_y) / bucket_height).astype( + np.int64 + ) + first = np.clip(first, 0, bucket_count - 1) + last = np.clip(last, 0, bucket_count - 1) + buckets = [[] for _index in range(bucket_count)] + for edge_index, (start, stop) in enumerate(zip(first, last)): + for bucket_index in range(int(start), int(stop) + 1): + buckets[bucket_index].append(edge_index) + return buckets + + def points_inside_polygon(points, polygon): + return points_inside_contours(points, (polygon,)) + + +def points_inside_contours(points, contours): import numpy as np points = np.asarray(points, dtype=np.float64) - polygon = np.asarray(polygon, dtype=np.float64) inside = np.zeros(len(points), dtype=bool) - previous = np.roll(polygon, 1, axis=0) - for current, prior in zip(polygon, previous): - crosses = (current[1] > points[:, 1]) != ( - prior[1] > points[:, 1] - ) - denominator = prior[1] - current[1] - if abs(denominator) < 1e-12: + if not len(points): + return inside + starts, ends = _contour_edges(contours) + if not len(starts): + return inside + minimum_y = min(float(points[:, 1].min()), float(starts[:, 1].min())) + maximum_y = max(float(points[:, 1].max()), float(starts[:, 1].max())) + typical_span = float(np.median(np.abs(ends[:, 1] - starts[:, 1]))) + bucket_height = max(typical_span, (maximum_y - minimum_y) / 4096.0, 1e-9) + bucket_count = max(1, int(math.ceil((maximum_y - minimum_y) / bucket_height))) + edge_buckets = _edge_bucket_indices( + starts, ends, minimum_y, bucket_height, bucket_count + ) + point_buckets = np.floor( + (points[:, 1] - minimum_y) / bucket_height + ).astype(np.int64) + point_buckets = np.clip(point_buckets, 0, bucket_count - 1) + order = np.argsort(point_buckets, kind="stable") + ordered_buckets = point_buckets[order] + split = np.flatnonzero(np.diff(ordered_buckets)) + 1 + for indices in np.split(order, split): + edge_indices = edge_buckets[int(point_buckets[indices[0]])] + if not edge_indices: continue - crossing_x = ( - (prior[0] - current[0]) - * (points[:, 1] - current[1]) - / denominator - + current[0] + selected_starts = starts[edge_indices] + selected_ends = ends[edge_indices] + sample_y = points[indices, 1, None] + crosses = (selected_starts[None, :, 1] > sample_y) != ( + selected_ends[None, :, 1] > sample_y ) - inside ^= crosses & (points[:, 0] < crossing_x) + crossing_x = selected_starts[None, :, 0] + ( + (sample_y - selected_starts[None, :, 1]) + * (selected_ends[None, :, 0] - selected_starts[None, :, 0]) + / (selected_ends[None, :, 1] - selected_starts[None, :, 1]) + ) + inside[indices] = np.count_nonzero( + crosses & (points[indices, 0, None] < crossing_x), axis=1 + ) % 2 == 1 return inside -def distances_to_polygon(points, polygon): +def uniform_interior_points_for_contours(contours, spacing): import numpy as np - points = np.asarray(points, dtype=np.float64) - polygon = np.asarray(polygon, dtype=np.float64) - squared = np.full(len(points), np.inf, dtype=np.float64) - for start, end in zip(polygon, np.roll(polygon, -1, axis=0)): - segment = end - start - length_squared = float(segment.dot(segment)) - if length_squared <= 1e-16: - continue - factor = np.clip( - (points - start) @ segment / length_squared, - 0.0, - 1.0, - ) - nearest = start + factor[:, None] * segment - squared = np.minimum( - squared, - np.sum((points - nearest) ** 2, axis=1), - ) - return np.sqrt(squared) + contours = [np.asarray(contour, dtype=np.float64) for contour in contours] + boundary_points = np.concatenate(contours, axis=0) + minimum = boundary_points.min(axis=0) + maximum = boundary_points.max(axis=0) + row_spacing = spacing * math.sqrt(3.0) * 0.5 + row_count = int(math.ceil((maximum[1] - minimum[1]) / row_spacing)) + 1 + column_count = int(math.ceil((maximum[0] - minimum[0]) / spacing)) + 1 + rows = np.arange(row_count, dtype=np.float64) + columns = np.arange(column_count, dtype=np.float64) + y = minimum[1] + rows * row_spacing + x = ( + minimum[0] + + columns[None, :] * spacing + + (rows.astype(np.int64) % 2)[:, None] * spacing * 0.5 + ) + x = np.broadcast_to(x, (row_count, column_count)) + points = np.column_stack((x.ravel(), np.repeat(y, column_count))) + inside = points_inside_contours(points, contours).reshape( + row_count, column_count + ) + + clearance = spacing * 0.4 + near_boundary = np.zeros_like(inside) + for contour in contours: + for start, end in zip(contour, np.roll(contour, -1, axis=0)): + segment = end - start + length_squared = float(segment.dot(segment)) + if length_squared <= 1e-16: + continue + first_row = max( + 0, + int( + math.ceil( + (min(start[1], end[1]) - clearance - minimum[1]) + / row_spacing + ) + ), + ) + last_row = min( + row_count - 1, + int( + math.floor( + (max(start[1], end[1]) + clearance - minimum[1]) + / row_spacing + ) + ), + ) + for row in range(first_row, last_row + 1): + offset = spacing * 0.5 if row % 2 else 0.0 + first_column = max( + 0, + int( + math.ceil( + (min(start[0], end[0]) - clearance - minimum[0] - offset) + / spacing + ) + ), + ) + last_column = min( + column_count - 1, + int( + math.floor( + (max(start[0], end[0]) + clearance - minimum[0] - offset) + / spacing + ) + ), + ) + if last_column < first_column: + continue + selected = np.arange(first_column, last_column + 1) + row_points = np.column_stack( + (x[row, selected], np.full(len(selected), y[row])) + ) + factor = np.clip( + (row_points - start) @ segment / length_squared, 0.0, 1.0 + ) + nearest = start + factor[:, None] * segment + close = np.sum((row_points - nearest) ** 2, axis=1) < clearance**2 + near_boundary[row, selected[close]] = True + usable = inside & ~near_boundary + return np.column_stack( + (x[usable], np.broadcast_to(y[:, None], x.shape)[usable]) + ) def uniform_interior_points(contour, spacing): - import numpy as np + return uniform_interior_points_for_contours((contour,), spacing) - contour = np.asarray(contour, dtype=np.float64) - minimum = contour.min(axis=0) - maximum = contour.max(axis=0) - row_spacing = spacing * math.sqrt(3.0) * 0.5 - row_count = int( - math.ceil((maximum[1] - minimum[1]) / row_spacing) - ) + 1 - column_count = int( - math.ceil((maximum[0] - minimum[0]) / spacing) - ) + 1 - candidates = [] - for row in range(row_count): - y = minimum[1] + row * row_spacing - offset = spacing * 0.5 if row % 2 else 0.0 - for column in range(column_count): - candidates.append( - (minimum[0] + offset + column * spacing, y) - ) - points = np.asarray(candidates, dtype=np.float64) - points = points[points_inside_polygon(points, contour)] - if not len(points): - return points - clearance = distances_to_polygon(points, contour) - return points[clearance >= spacing * 0.4] def metric_transform(world_metric=None): @@ -163,8 +264,8 @@ def metric_transform(world_metric=None): ) from error -def prepare_planar_points( - polygon, +def prepare_planar_contours( + polygons, image_size, local_bounds, spacing=POINT_SPACING, @@ -172,41 +273,177 @@ def prepare_planar_points( ): import numpy as np + if spacing <= 0.0: + raise ValueError("Balloon point spacing must be greater than zero") image_width, image_height = image_size x_min, x_max, y_min, y_max = local_bounds local_width = x_max - x_min local_height = y_max - y_min - pixels = np.asarray(polygon, dtype=np.float64) - local = np.empty_like(pixels) - local[:, 0] = x_min + local_width * pixels[:, 0] / image_width - local[:, 1] = y_max - local_height * pixels[:, 1] / image_height - if float(np.ptp(local, axis=0).max()) <= 0.0: - raise ValueError("The lasso does not overlap the source image") - if spacing <= 0.0: - raise ValueError("Balloon point spacing must be greater than zero") + transform = metric_transform(world_metric) + contours = [] + for polygon in polygons: + pixels = np.asarray(polygon, dtype=np.float64) + if pixels.ndim != 2 or pixels.shape[0] < 3 or pixels.shape[1] != 2: + continue + local = np.empty_like(pixels) + local[:, 0] = x_min + local_width * pixels[:, 0] / image_width + local[:, 1] = y_max - local_height * pixels[:, 1] / image_height + planar = local @ transform + perimeter = float( + np.linalg.norm(np.roll(planar, -1, axis=0) - planar, axis=1).sum() + ) + if perimeter <= 1e-12: + continue + contour_spacing = min(spacing, perimeter / 8.0) + contours.append(smooth_lasso_contour(planar, contour_spacing)) + if not contours: + raise ValueError("The Cutout has no region large enough for this resolution") + interior = uniform_interior_points_for_contours(contours, spacing) + return contours, interior, spacing - planar = local @ metric_transform(world_metric) - contour = smooth_lasso_contour(planar, spacing) - interior = uniform_interior_points(contour, spacing) +def prepare_planar_regions( + regions, + image_size, + local_bounds, + spacing=POINT_SPACING, + world_metric=None, +): + prepared = [] + errors = [] + for region in regions: + try: + contours, interior, _spacing = prepare_planar_contours( + region, + image_size, + local_bounds, + spacing=spacing, + world_metric=world_metric, + ) + except ValueError as error: + errors.append(error) + continue + prepared.append((contours, interior)) + if not prepared: + if errors: + raise errors[0] + raise ValueError("The Cutout has no usable region") + return prepared, spacing + + + +def prepare_planar_points( + polygon, + image_size, + local_bounds, + spacing=POINT_SPACING, + world_metric=None, +): + contours, interior, spacing = prepare_planar_contours( + (polygon,), + image_size, + local_bounds, + spacing=spacing, + world_metric=world_metric, + ) + contour = contours[0] return contour, interior, spacing -def boundary_vertices_from_faces(vertex_count, faces): - from collections import Counter +def triangulate_planar_contours(contours, interior, spacing): + import numpy as np + from mathutils import Vector + from mathutils.geometry import delaunay_2d_cdt + + input_points = np.concatenate((*contours, interior), axis=0) + boundary_edges = [] + offset = 0 + for contour in contours: + count = len(contour) + boundary_edges.extend( + (offset + index, offset + (index + 1) % count) + for index in range(count) + ) + offset += count + result = delaunay_2d_cdt( + [Vector(point) for point in input_points], + boundary_edges, + [], + 0, + max(spacing * 1e-6, 1e-9), + False, + ) + planar_points = np.asarray( + [tuple(point) for point in result[0]], + dtype=np.float64, + ) + faces = np.asarray( + [tuple(face) for face in result[2] if len(face) == 3], + dtype=np.int32, + ) + if len(faces): + centers = planar_points[faces].mean(axis=1) + faces = faces[points_inside_contours(centers, contours)] + if not len(faces): + raise ValueError("Unable to triangulate the Cutout regions") + used = np.unique(faces) + if len(used) != len(planar_points): + remap = np.full(len(planar_points), -1, dtype=np.int32) + remap[used] = np.arange(len(used), dtype=np.int32) + planar_points = planar_points[used] + faces = remap[faces] + boundary = boundary_vertices_from_faces(len(planar_points), faces) + planar_faces = [tuple(int(value) for value in face) for face in faces] + return planar_points, planar_faces, boundary + + +def boundary_vertices_from_faces(vertex_count, faces): import numpy as np - edge_users = Counter() - for face in faces: - for start, end in zip(face, face[1:] + face[:1]): - edge_users[tuple(sorted((start, end)))] += 1 + faces = np.asarray(faces, dtype=np.int32) boundary = np.zeros(vertex_count, dtype=bool) - for (start, end), users in edge_users.items(): - if users == 1: - boundary[start] = True - boundary[end] = True + if not len(faces): + return boundary + edges = np.concatenate( + [ + faces[:, (index, (index + 1) % faces.shape[1])] + for index in range(faces.shape[1]) + ], + axis=0, + ) + edges.sort(axis=1) + unique_edges, users = np.unique(edges, axis=0, return_counts=True) + boundary[unique_edges[users == 1].ravel()] = True return boundary +def triangulate_planar_regions(regions, spacing): + import numpy as np + + points = [] + faces = [] + boundaries = [] + offset = 0 + for contours, interior in regions: + region_points, region_faces, region_boundary = triangulate_planar_contours( + contours, + interior, + spacing, + ) + points.append(region_points) + faces.extend( + tuple(offset + value for value in face) for face in region_faces + ) + boundaries.append(region_boundary) + offset += len(region_points) + if not points: + raise ValueError("Unable to triangulate the Cutout regions") + return ( + np.concatenate(points, axis=0), + faces, + np.concatenate(boundaries), + ) + + def balloon_heights( algorithm, planar_points, @@ -283,7 +520,7 @@ def build_closed_geometry( return vertices, faces, np.asarray(vertex_uv, dtype=np.float32) def triangulate_balloon( - polygon, + regions, image_size, local_bounds, spacing=POINT_SPACING, @@ -292,46 +529,21 @@ def triangulate_balloon( inflation=INFLATION, algorithm="POISSON", ): - import numpy as np - from mathutils import Vector - from mathutils.geometry import delaunay_2d_cdt - - contour, interior, spacing = prepare_planar_points( - polygon, + planar_regions, spacing = prepare_planar_regions( + regions, image_size, local_bounds, spacing=spacing, world_metric=world_metric, ) - input_points = np.concatenate((contour, interior), axis=0) - boundary_count = len(contour) - boundary_edges = [ - (index, (index + 1) % boundary_count) - for index in range(boundary_count) - ] - result = delaunay_2d_cdt( - [Vector(point) for point in input_points], - boundary_edges, - [list(range(boundary_count))], - 1, - max(spacing * 1e-6, 1e-9), - ) - planar_points = np.asarray( - [tuple(point) for point in result[0]], - dtype=np.float64, - ) - planar_faces = [tuple(face) for face in result[2] if len(face) == 3] - if not planar_faces: - raise ValueError("Unable to triangulate the lasso region") - boundary_mask = boundary_vertices_from_faces( - len(planar_points), - planar_faces, + planar_points, planar_faces, boundary_mask = triangulate_planar_regions( + planar_regions, spacing ) heights = balloon_heights( algorithm, planar_points, planar_faces, - contour, + planar_regions[0][0][0], boundary_mask, inflation, ) diff --git a/src/anyimage/operators/balloon/operator.py b/src/anyimage/operators/balloon/operator.py index 1ef43c4..5c5e859 100644 --- a/src/anyimage/operators/balloon/operator.py +++ b/src/anyimage/operators/balloon/operator.py @@ -3,25 +3,31 @@ import math import bpy -from .depth import ( - add_balloon_shape_modifier, - assign_balloon_shape_attributes, - balloon_depth_fields, - create_balloon_base_plane, - load_balloon_depth_data, -) -from .geometry import INFLATION, POINT_SPACING, triangulate_balloon from ...properties import ( BEN2_MODEL_KEY, ai_ready, ai_setup_label, ai_status, +) +from ...properties import ( balloon_contour_analysis_property as contour_analysis_property, +) +from ...properties import ( balloon_depth_analysis_property as depth_analysis_property, +) +from ...properties import ( balloon_depth_model_property as depth_model_property, +) +from ...properties import ( balloon_mode_property as mode_property, +) +from ...properties import ( balloon_resolution_property as resolution_property, +) +from ...properties import ( balloon_reuse_material_property as reuse_material_property, +) +from ...properties import ( balloon_thickness_property as thickness_property, ) from ...runtime import JobOperatorBase @@ -44,12 +50,19 @@ from ..image import ( video_total_frames, ) from ..material import ( - BALLOON_MATERIAL_NODE_GROUP_NAME, - TEXTURE_MATERIAL_NODE_GROUP_NAME, + IMAGE_MATERIAL_NODE_GROUP_NAME, create_material, ) +from ..normal_map import bake_tangent_normal_image from ..surface import assign_vertex_uv - +from .depth import ( + add_balloon_shape_modifier, + assign_balloon_shape_attributes, + balloon_depth_fields, + create_balloon_base_plane, + load_balloon_depth_data, +) +from .geometry import INFLATION, POINT_SPACING, triangulate_balloon MIN_PATH_DISTANCE = 2.0 BALLOON_MATERIAL_PROPERTY = "anyimage_balloon_material" @@ -151,7 +164,6 @@ def data_name_matches(name, base_name): return suffix != name and suffix.isdigit() - def marked_balloon_material(source_image, normal_required=False): return next( ( @@ -160,9 +172,7 @@ def marked_balloon_material(source_image, normal_required=False): if callable(getattr(material, "get", None)) and bool(material.get(BALLOON_MATERIAL_PROPERTY)) and material.get(BALLOON_SOURCE_IMAGE_PROPERTY) is source_image - and ( - material.get(BALLOON_NORMAL_IMAGE_PROPERTY) is not None - ) + and (material.get(BALLOON_NORMAL_IMAGE_PROPERTY) is not None) == normal_required ), None, @@ -227,6 +237,7 @@ BALLOON_AXIS_TO_SOURCE = ( (0.0, 0.0, 0.0, 1.0), ) + def orient_balloon_vertices(vertices): """Rotate legacy XY/Z Balloon coordinates into a YZ/-X frame.""" return [(-z, -x, y) for x, y, z in vertices] @@ -237,14 +248,16 @@ def orient_balloon_world_matrix(source_matrix): return source_matrix @ Matrix(BALLOON_AXIS_TO_SOURCE) + def orient_balloon_base_plane(local_plane): slope_x, slope_y, offset = local_plane return (slope_x, -slope_y, -offset) + def extract_balloon_object( context, source_object, - polygon, + regions, matrix_world, algorithm, thickness=1.0, @@ -253,8 +266,10 @@ def extract_balloon_object( reuse_material=True, depth_data=None, normal_path=None, + material_image=None, + bake_normal_to_tangent=False, ): - image = source_object.data + image = material_image or source_object.data image_size = tuple(image.size) local_bounds = plane_bounds(source_object) world_metric, world_z_scale = world_plane_metric(matrix_world) @@ -263,7 +278,7 @@ def extract_balloon_object( resolution, ) vertices, faces, vertex_uv = triangulate_balloon( - polygon, + regions, image_size, local_bounds, spacing=point_spacing, @@ -315,7 +330,9 @@ def extract_balloon_object( finally: bpy.data.images.remove(update_normal) if material is None: - color_image = reusable_color_image(color_name) if reuse_material else None + color_image = material_image + if color_image is None and reuse_material: + color_image = reusable_color_image(color_name) if color_image is None: color_image = image.copy() color_image.name = color_name @@ -330,12 +347,8 @@ def extract_balloon_object( material = create_material( material_name, color_image, - 1.0, - surface_group_name=( - BALLOON_MATERIAL_NODE_GROUP_NAME - if normal_image is not None - else TEXTURE_MATERIAL_NODE_GROUP_NAME - ), + 0.0, + surface_group_name=IMAGE_MATERIAL_NODE_GROUP_NAME, normal_image=normal_image, scene=context.scene, ) @@ -369,7 +382,18 @@ def extract_balloon_object( vertices, local_base_plane, ) - add_balloon_shape_modifier(balloon, thickness, base_plane) + add_balloon_shape_modifier( + balloon, + thickness, + base_plane, + ) + if bake_normal_to_tangent and normal_image is not None: + bake_tangent_normal_image( + context, + source_object, + balloon, + normal_image, + ) if depth_data is not None: balloon["anyimage_balloon_depth_analysis"] = True balloon["anyimage_reference_depth"] = depth_data["reference_depth"] @@ -625,7 +649,7 @@ class ExtractBalloon(bpy.types.Operator): balloon = extract_balloon_object( context, source_object, - polygon, + ((polygon,),), matrix_world, self.mode, thickness=self.thickness, @@ -710,7 +734,7 @@ class AnalyzeBalloon(JobOperatorBase, bpy.types.Operator): balloon = extract_balloon_object( context, source_object, - polygon, + ((polygon,),), matrix_world, self.mode, thickness=self.thickness, @@ -719,6 +743,7 @@ class AnalyzeBalloon(JobOperatorBase, bpy.types.Operator): reuse_material=self.reuse_material, depth_data=depth_data, normal_path=(result.file("normal") if self.depth_analysis else None), + bake_normal_to_tangent=self.depth_analysis, ) balloon["anyimage_resolution_m"] = self.resolution if self.contour_analysis: diff --git a/src/anyimage/operators/cutout.py b/src/anyimage/operators/cutout.py new file mode 100644 index 0000000..f812d9e --- /dev/null +++ b/src/anyimage/operators/cutout.py @@ -0,0 +1,255 @@ +import json + +import bpy + +from ..properties import ( + image_edit_refine_selection_property as refine_selection_property, +) +from .balloon.operator import hidden_string_property, serialize_matrix +from .image import is_image_empty, video_total_frames +from .image_edit.common import ( + ImageEditWarning, + apply_image_mask_settings, + draw_image_mask_settings, + polygon_bounds, + report_image_edit_exception, + resolve_image_edit_click, +) +from .image_edit.mask import ( + ImageMaskGesture, + gesture_property, + selected_image_rgba, +) +from .mesh import BALLOON, CUTOUT, DEPTH_BALLOON, DEPTH_SURFACE + +CUTOUT_DETAIL_MIN = 4 +CUTOUT_DETAIL_MAX = 1024 + + +def cutout_mesh_detail(image_size, pixel_size): + longest_edge = max(int(value) for value in image_size) + detail = round(longest_edge / max(int(pixel_size), 1)) + return max(CUTOUT_DETAIL_MIN, min(CUTOUT_DETAIL_MAX, detail)) + + +def _pie_button( + pie, + shape, + label, + icon, + source_name, + selection_data, + refine_selection, + normal_analysis, + invert, + mesh_detail, +): + operator = pie.operator( + "anyimage.convert_cutout_shape", + text=label, + icon=icon, + ) + operator.source_object_name = source_name + operator.selection_data = selection_data + operator.shape = shape + operator.refine_selection = refine_selection + operator.normal_analysis = normal_analysis + operator.invert = invert + operator.mesh_detail = mesh_detail + + +def _open_shape_pie( + context, + event, + source_object, + polygon, + refine_selection=False, + invert=False, +): + source_name = source_object.name + selection_data = json.dumps(polygon, separators=(",", ":")) + settings = context.scene.anyimage_settings + pixel_size = settings.cutout_pixel_size + normal_analysis = bool(getattr(settings, "cutout_normal_analysis", False)) + mesh_detail = cutout_mesh_detail(source_object.data.size, pixel_size) + + def draw(menu, _context): + pie = menu.layout.menu_pie() + _pie_button( + pie, + CUTOUT, + "Cutout", + "MESH_GRID", + source_name, + selection_data, + refine_selection, + normal_analysis, + invert, + mesh_detail, + ) + _pie_button( + pie, + BALLOON, + "Balloon", + "MESH_UVSPHERE", + source_name, + selection_data, + refine_selection, + normal_analysis, + invert, + mesh_detail, + ) + _pie_button( + pie, + DEPTH_BALLOON, + "Depth Balloon", + "META_BALL", + source_name, + selection_data, + refine_selection, + normal_analysis, + invert, + mesh_detail, + ) + _pie_button( + pie, + DEPTH_SURFACE, + "Depth Surface", + "SURFACE_NCURVE", + source_name, + selection_data, + refine_selection, + normal_analysis, + invert, + mesh_detail, + ) + + context.window_manager.popup_menu_pie( + event, + draw, + title="Cutout Shape", + icon="MESH_DATA", + ) + + +class SelectCutoutRegion(ImageMaskGesture, bpy.types.Operator): + bl_idname = "anyimage.select_cutout_region" + bl_label = "Select Cutout Region" + bl_description = "Lasso an Image Empty region, then choose its mesh shape" + bl_options = {"INTERNAL"} + active_tool_id = "anyimage.cutout_lasso" + + source_object_name: hidden_string_property() + source_matrix_data: hidden_string_property() + selection_data: hidden_string_property() + refine_selection: refine_selection_property() + invert: bpy.props.BoolProperty(default=False, options={"SKIP_SAVE"}) + gesture: gesture_property() + full_image: bpy.props.BoolProperty(default=False, options={"HIDDEN", "SKIP_SAVE"}) + + _pie_event = None + + def _set_status(self, context): + invert = "On" if self.invert else "Off" + context.workspace.status_text_set( + f"Cutout Lasso: drag and release for shapes; " + f"F toggles Invert ({invert}); RMB or Esc cancels" + ) + + def invoke(self, context, event): + self._pie_event = event + if not self.full_image and event.value != "DOUBLE_CLICK": + return super().invoke(context, event) + source_object, should_edit = resolve_image_edit_click( + context, event, self.report + ) + if not should_edit: + return {"FINISHED"} if source_object is not None else {"CANCELLED"} + if source_object is None: + return {"CANCELLED"} + if video_total_frames(source_object.data) > 1: + self.report({"WARNING"}, "Image tools only support a still image") + return {"CANCELLED"} + apply_image_mask_settings(self, context) + self.invert = False + self.source_object_name = source_object.name + self.source_matrix_data = serialize_matrix(source_object.matrix_world) + width, height = (int(value) for value in source_object.data.size) + self.selection_data = json.dumps( + ((0, 0), (width, 0), (width, height), (0, height)), + separators=(",", ":"), + ) + return self.execute(context) + + def modal(self, context, event): + self._pie_event = event + return super().modal(context, event) + + def execute(self, context): + source_object = bpy.data.objects.get(self.source_object_name) + if not is_image_empty(source_object): + self.report({"ERROR"}, "The source Image Empty is no longer available") + return {"CANCELLED"} + try: + polygon = json.loads(self.selection_data) + polygon_bounds(polygon) + selected_image_rgba( + source_object, polygon, self.invert, "NONZERO" + ) + except (ImageEditWarning, json.JSONDecodeError, TypeError, ValueError) as error: + report_image_edit_exception(self, error) + return {"CANCELLED"} + _open_shape_pie( + context, + self._pie_event, + source_object, + polygon, + self.refine_selection, + self.invert, + ) + return {"FINISHED"} + + +class ActivateCutoutTool(bpy.types.Operator): + bl_idname = "anyimage.activate_cutout_tool" + bl_label = "Cutout Tool" + bl_description = "Activate the Cutout Lasso tool" + bl_options = {"INTERNAL"} + + @classmethod + def poll(cls, context): + return context.area is not None and context.area.type == "VIEW_3D" + + def execute(self, _context): + return bpy.ops.wm.tool_set_by_id( + name=CutoutTool.bl_idname, + space_type="VIEW_3D", + ) + + +class CutoutTool(bpy.types.WorkSpaceTool): + bl_space_type = "VIEW_3D" + bl_context_mode = "OBJECT" + bl_idname = "anyimage.cutout_lasso" + bl_label = "Cutout" + bl_description = "Lasso an Image Empty and choose a mesh shape" + bl_icon = "ops.sculpt.lasso_mask" + bl_operator = SelectCutoutRegion.bl_idname + bl_keymap = ( + ( + SelectCutoutRegion.bl_idname, + {"type": "LEFTMOUSE", "value": "DOUBLE_CLICK"}, + {"properties": [("gesture", "LASSO"), ("full_image", True)]}, + ), + ( + SelectCutoutRegion.bl_idname, + {"type": "LEFTMOUSE", "value": "PRESS"}, + {"properties": [("gesture", "LASSO")]}, + ), + ) + + def draw_settings(context, layout, _tool): + settings = context.scene.anyimage_settings + layout.prop(settings, "cutout_pixel_size") + layout.prop(settings, "cutout_normal_analysis") + draw_image_mask_settings(context, layout) diff --git a/src/anyimage/operators/depth_plane/__init__.py b/src/anyimage/operators/depth_plane/__init__.py index 8aea1ed..cf871b4 100644 --- a/src/anyimage/operators/depth_plane/__init__.py +++ b/src/anyimage/operators/depth_plane/__init__.py @@ -1,7 +1 @@ -from .operator import ( - ConvertToDepthPlane, - ExtractOriginalFrames, - GenerateDa3Depth, - GenerateMoge2Depth, - GenerateOriginalBackground, -) +"""Depth-plane asset builders retained for asset validation.""" diff --git a/src/anyimage/operators/depth_plane/operator.py b/src/anyimage/operators/depth_plane/operator.py index 20f8537..09cc5d9 100644 --- a/src/anyimage/operators/depth_plane/operator.py +++ b/src/anyimage/operators/depth_plane/operator.py @@ -17,10 +17,12 @@ from ..common import ( ) from .build import build_depth_plane, build_image_depth_plane, build_moge2_depth_plane from ..image import ( + analysis_source_image, default_extract_frames, depth_data_name, draw_process_resolution, is_image_empty, + prepare_image_analysis_inputs, prepare_image_input, process_resolution_scale_property, remove_temporary_input, @@ -88,7 +90,11 @@ class ConvertToDepthPlane(bpy.types.Operator): layout = self.layout layout.prop(self, "model") if self.model in {"SMALL", "BASE"}: - draw_process_resolution(layout, self, context.object.data) + draw_process_resolution( + layout, + self, + analysis_source_image(context.object.data), + ) elif self.model.startswith("MOGE2_"): layout.prop(self, "resolution_level") layout.prop(self, "remove_background") @@ -131,7 +137,12 @@ class ConvertToDepthPlane(bpy.types.Operator): if self.model.startswith("MOGE2_"): try: - source_path, delete_input = prepare_image_input( + ( + source_path, + delete_input, + analysis_path, + delete_analysis_input, + ) = prepare_image_analysis_inputs( source_object.data, context.scene, ) @@ -140,11 +151,13 @@ class ConvertToDepthPlane(bpy.types.Operator): return {"CANCELLED"} result = bpy.ops.anyimage.generate_moge2_depth( input_path=str(source_path), + analysis_input_path=str(analysis_path), source_object_name=source_object.name, resolution_level=self.resolution_level, model=self.model, delete_source=not self.keep_original, delete_input=delete_input, + delete_analysis_input=delete_analysis_input, shadeless=self.shadeless, remove_background=self.remove_background, ) @@ -152,9 +165,11 @@ class ConvertToDepthPlane(bpy.types.Operator): return {"FINISHED"} if delete_input: remove_temporary_input(source_path) + if delete_analysis_input: + remove_temporary_input(analysis_path) return {"CANCELLED"} - image = source_object.data + image = analysis_source_image(source_object.data) try: process_res = scaled_process_resolution( image, @@ -164,8 +179,13 @@ class ConvertToDepthPlane(bpy.types.Operator): self.report({"ERROR"}, str(error)) return {"CANCELLED"} try: - source_path, delete_input = prepare_image_input( - image, + ( + source_path, + delete_input, + analysis_path, + delete_analysis_input, + ) = prepare_image_analysis_inputs( + source_object.data, context.scene, ) except RuntimeError as error: @@ -174,18 +194,22 @@ class ConvertToDepthPlane(bpy.types.Operator): result = bpy.ops.anyimage.generate_da3_depth( input_path=str(source_path), + analysis_input_path=str(analysis_path), source_object_name=source_object.name, depth_data_name=depth_data_name(source_object), model=self.model, process_res=process_res, delete_source=not self.keep_original, delete_input=delete_input, + delete_analysis_input=delete_analysis_input, shadeless=self.shadeless, remove_background=self.remove_background, ) if "CANCELLED" in result: if delete_input: remove_temporary_input(source_path) + if delete_analysis_input: + remove_temporary_input(analysis_path) return {"CANCELLED"} return {"FINISHED"} @@ -310,12 +334,14 @@ class GenerateDa3Depth(JobOperatorBase, bpy.types.Operator): starting_message = "Starting depth estimation" input_path: bpy.props.StringProperty(options=HIDDEN) + analysis_input_path: bpy.props.StringProperty(options=HIDDEN) source_object_name: bpy.props.StringProperty(options=HIDDEN) depth_data_name: bpy.props.StringProperty(options=HIDDEN) model: bpy.props.StringProperty(default="BASE", options=HIDDEN) process_res: bpy.props.IntProperty(default=1024, min=1, options=HIDDEN) delete_source: bpy.props.BoolProperty(default=True, options=HIDDEN) delete_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_analysis_input: bpy.props.BoolProperty(default=False, options=HIDDEN) shadeless: bpy.props.BoolProperty(default=False, options=HIDDEN) remove_background: bpy.props.BoolProperty(default=False, options=HIDDEN) @@ -325,7 +351,7 @@ class GenerateDa3Depth(JobOperatorBase, bpy.types.Operator): if self.remove_background: require_model(BEN2_MODEL_KEY) settings = context.scene.anyimage_settings - return { + parameters = { **da3_parameters( settings, input_path(self.input_path), @@ -338,6 +364,9 @@ class GenerateDa3Depth(JobOperatorBase, bpy.types.Operator): BEN2_MODEL_KEY if self.remove_background else "" ), } + analysis_value = getattr(self, "analysis_input_path", "") or self.input_path + parameters["analysis_input"] = str(input_path(analysis_value)) + return parameters def response(self, context, result): build_da3_depth_response( @@ -353,6 +382,10 @@ class GenerateDa3Depth(JobOperatorBase, bpy.types.Operator): def cleanup(self): cleanup_temporary_input(self.input_path, self.delete_input) + cleanup_temporary_input( + getattr(self, "analysis_input_path", ""), + getattr(self, "delete_analysis_input", False), + ) class GenerateMoge2Depth(JobOperatorBase, bpy.types.Operator): @@ -362,11 +395,13 @@ class GenerateMoge2Depth(JobOperatorBase, bpy.types.Operator): starting_message = "Starting MoGe-2 inference" input_path: bpy.props.StringProperty(options=HIDDEN) + analysis_input_path: bpy.props.StringProperty(options=HIDDEN) source_object_name: bpy.props.StringProperty(options=HIDDEN) model: bpy.props.StringProperty(default=DEFAULT_MOGE2_MODEL_KEY, options=HIDDEN) resolution_level: bpy.props.IntProperty(default=5, min=0, max=9, options=HIDDEN) delete_source: bpy.props.BoolProperty(default=True, options=HIDDEN) delete_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_analysis_input: bpy.props.BoolProperty(default=False, options=HIDDEN) shadeless: bpy.props.BoolProperty(default=False, options=HIDDEN) remove_background: bpy.props.BoolProperty(default=False, options=HIDDEN) @@ -376,7 +411,7 @@ class GenerateMoge2Depth(JobOperatorBase, bpy.types.Operator): if self.remove_background: require_model(BEN2_MODEL_KEY) settings = context.scene.anyimage_settings - return { + parameters = { **moge2_parameters( settings, input_path(self.input_path), @@ -388,6 +423,9 @@ class GenerateMoge2Depth(JobOperatorBase, bpy.types.Operator): BEN2_MODEL_KEY if self.remove_background else "" ), } + analysis_value = getattr(self, "analysis_input_path", "") or self.input_path + parameters["analysis_input"] = str(input_path(analysis_value)) + return parameters def response(self, context, result): build_moge2_depth_response( @@ -402,6 +440,10 @@ class GenerateMoge2Depth(JobOperatorBase, bpy.types.Operator): def cleanup(self): cleanup_temporary_input(self.input_path, self.delete_input) + cleanup_temporary_input( + getattr(self, "analysis_input_path", ""), + getattr(self, "delete_analysis_input", False), + ) class GenerateOriginalBackground(JobOperatorBase, bpy.types.Operator): diff --git a/src/anyimage/operators/image.py b/src/anyimage/operators/image.py index 6c51437..fc8eb6b 100644 --- a/src/anyimage/operators/image.py +++ b/src/anyimage/operators/image.py @@ -279,6 +279,42 @@ def prepare_image_input(image, scene): return source_path, False +def analysis_source_image(image): + """Return the oldest available source behind an upscaled image.""" + current = image + visited = set() + while current is not None and id(current) not in visited: + visited.add(id(current)) + source = getattr(current, "anyimage_upscale_source", None) + if source is None or source is current: + break + current = source + return current + + +def prepare_image_analysis_inputs(image, scene): + """Prepare display and analysis inputs, reusing one path when possible.""" + source_path, delete_input = prepare_image_input(image, scene) + analysis_image = analysis_source_image(image) + if analysis_image is image: + return source_path, delete_input, source_path, False + try: + analysis_path, delete_analysis_input = prepare_image_input( + analysis_image, + scene, + ) + except Exception: + if delete_input: + remove_temporary_input(source_path) + raise + return ( + source_path, + delete_input, + analysis_path, + delete_analysis_input, + ) + + def configure_image_user( image_user, frame_count=None, diff --git a/src/anyimage/operators/image_edit/mask.py b/src/anyimage/operators/image_edit/mask.py index 6a2f329..808fba8 100644 --- a/src/anyimage/operators/image_edit/mask.py +++ b/src/anyimage/operators/image_edit/mask.py @@ -24,13 +24,13 @@ from ..image import ( video_total_frames, ) from .common import ( + ImageEditWarning, active_view3d_tool_id, apply_image_mask_settings, crop_pixels_to_alpha, draw_dashed_line, draw_image_mask_operator_settings, draw_polygon_fill, - ImageEditWarning, image_edit_poll, image_mask_refine_request, polygon_bounds, @@ -39,7 +39,6 @@ from .common import ( resolve_image_edit_click, ) - GESTURE_TOOL_IDS = { "BOX": "anyimage.image_box", "LASSO": "anyimage.image_lasso", @@ -59,30 +58,55 @@ def points_inside_polygon(points, polygon): previous = polygon[-1] for current in polygon: crosses = (current[1] > points[:, 1]) != (previous[1] > points[:, 1]) - edge_x = ( - (previous[0] - current[0]) - * (points[:, 1] - current[1]) - / (previous[1] - current[1] + 1e-300) - + current[0] - ) + edge_x = (previous[0] - current[0]) * (points[:, 1] - current[1]) / ( + previous[1] - current[1] + 1e-300 + ) + current[0] inside ^= crosses & (points[:, 0] < edge_x) previous = current return inside -def polygon_mask(image_size, polygon): +def polygon_mask(image_size, polygon, fill_rule="EVEN_ODD"): import numpy as np width, height = (int(value) for value in image_size) if width <= 0 or height <= 0: raise ValueError("The source image has no readable dimensions") - polygon_bounds(polygon) - y, x = np.indices((height, width), dtype=np.float64) - centers = np.column_stack((x.ravel() + 0.5, y.ravel() + 0.5)) - return points_inside_polygon(centers, polygon).reshape((height, width)).astype( - np.float32 - ) + if fill_rule not in {"EVEN_ODD", "NONZERO"}: + raise ValueError(f"Unknown polygon fill rule: {fill_rule}") + left, top, right, bottom = polygon_bounds(polygon) + left = max(left, 0) + top = max(top, 0) + right = min(right, width) + bottom = min(bottom, height) + mask = np.zeros((height, width), dtype=bool) + if right <= left or bottom <= top: + return mask + points = np.asarray(polygon, dtype=np.float64) + x = np.arange(left, right, dtype=np.float64) + 0.5 + for row_start in range(top, bottom, 128): + row_end = min(row_start + 128, bottom) + y = np.arange(row_start, row_end, dtype=np.float64) + 0.5 + state = np.zeros( + (len(y), len(x)), + dtype=bool if fill_rule == "EVEN_ODD" else np.int16, + ) + previous = points[-1] + for current in points: + crosses = (current[1] > y) != (previous[1] > y) + edge_x = (previous[0] - current[0]) * (y - current[1]) / ( + previous[1] - current[1] + 1e-300 + ) + current[0] + selected = crosses[:, None] & (x[None, :] < edge_x[:, None]) + if fill_rule == "EVEN_ODD": + state ^= selected + else: + winding = 1 if current[1] > previous[1] else -1 + state += selected * winding + previous = current + mask[row_start:row_end, left:right] = state != 0 + return mask def refined_mask_from_pixels(mask_pixels, mask_size, bounds, source_size): import numpy as np @@ -90,9 +114,7 @@ def refined_mask_from_pixels(mask_pixels, mask_size, bounds, source_size): mask_width, mask_height = (int(value) for value in mask_size) source_width, source_height = (int(value) for value in source_size) local = np.flipud( - np.asarray(mask_pixels, dtype=np.float32).reshape( - (mask_height, mask_width, 4) - ) + np.asarray(mask_pixels, dtype=np.float32).reshape((mask_height, mask_width, 4)) )[:, :, 3] result = np.zeros((source_height, source_width), dtype=np.float32) left, top, right, bottom = (int(value) for value in bounds) @@ -116,7 +138,7 @@ def edit_image_mask_pixels(source_pixels, source_size, selection_mask, invert=Fa width, height = (int(value) for value in source_size) rgba = np.flipud( np.asarray(source_pixels, dtype=np.float32).reshape((height, width, 4)) - ).copy() + ) mask = np.asarray(selection_mask, dtype=np.float32) if mask.shape != (height, width): raise ValueError("The refined selection does not match the source image") @@ -146,6 +168,25 @@ def edit_image_object(source_object, selection_mask, invert=False): ) +def selected_image_rgba( + source_object, polygon, invert=False, fill_rule="EVEN_ODD" +): + import numpy as np + + image = source_object.data + width, height = (int(value) for value in image.size) + pixels = np.empty(len(image.pixels), dtype=np.float32) + image.pixels.foreach_get(pixels) + rgba = np.flipud(pixels.reshape((height, width, 4))) + selection = polygon_mask((width, height), polygon, fill_rule) + if invert: + selection = ~selection + rgba[:, :, 3] *= selection + if not np.any(rgba[:, :, 3] > 0.0): + raise ImageEditWarning("The selection contains no visible image pixels") + return rgba + + def gesture_property(): return bpy.props.EnumProperty( items=( @@ -191,9 +232,9 @@ def line_half_plane_polygon(start, end, rectangle_size): vertices = [(0.0, 0.0), (width, 0.0), (width, height), (0.0, height)] def side(point): - return (end[0] - start[0]) * (point[1] - start[1]) - ( - end[1] - start[1] - ) * (point[0] - start[0]) + return (end[0] - start[0]) * (point[1] - start[1]) - (end[1] - start[1]) * ( + point[0] - start[0] + ) result = [] previous = vertices[-1] @@ -222,24 +263,8 @@ def image_line_half_plane_polygon(line, image_size): return line_half_plane_polygon(line[1], line[0], image_size) -class EditImageMask(bpy.types.Operator): - bl_idname = "anyimage.edit_image_mask" - bl_label = "Edit Image Mask" - bl_description = "Edit and tightly crop an Image Empty with a gesture" - bl_options = {"REGISTER", "UNDO"} - - source_object_name: hidden_string_property() - source_matrix_data: hidden_string_property() - selection_data: hidden_string_property() - refine_selection: refine_selection_property() - invert: bpy.props.BoolProperty( - name="Invert", - description="Keep the opposite side of the refined gesture mask", - default=False, - options={"SKIP_SAVE"}, - ) - gesture: gesture_property() - +class ImageMaskGesture: + active_tool_id = None _path = None _cursor = None _handle = None @@ -277,9 +302,7 @@ class EditImageMask(bpy.types.Operator): if len(polygon) >= 3: draw_polygon_fill( polygon, - (1.0, 0.25, 0.2, 0.16) - if self.invert - else (1.0, 1.0, 1.0, 0.16), + (1.0, 0.25, 0.2, 0.16) if self.invert else (1.0, 1.0, 1.0, 0.16), ) if self.gesture == "LINE": draw_dashed_line((self._path[0], self._cursor)) @@ -407,7 +430,8 @@ class EditImageMask(bpy.types.Operator): def modal(self, context, event): active_tool = active_view3d_tool_id(context) - if active_tool is not None and active_tool != GESTURE_TOOL_IDS[self.gesture]: + expected_tool = self.active_tool_id or GESTURE_TOOL_IDS[self.gesture] + if active_tool is not None and active_tool != expected_tool: self._finish(context) return {"CANCELLED"} if event.type in {"ESC", "RIGHTMOUSE"} and event.value == "PRESS": @@ -423,7 +447,8 @@ class EditImageMask(bpy.types.Operator): if event.type == "MOUSEMOVE": point = (float(event.mouse_region_x), float(event.mouse_region_y)) if self.gesture == "LASSO": - self._path.append(point) + if math.dist(point, self._path[-1]) >= 2.0: + self._path.append(point) self._cursor = point elif self.gesture == "LINE": self._cursor = snapped_line_endpoint(self._path[0], point, event.ctrl) @@ -434,7 +459,8 @@ class EditImageMask(bpy.types.Operator): if event.type == "LEFTMOUSE" and event.value == "RELEASE": point = (float(event.mouse_region_x), float(event.mouse_region_y)) if self.gesture == "LASSO": - self._path.append(point) + if math.dist(point, self._path[-1]) >= 2.0: + self._path.append(point) self._cursor = point elif self.gesture == "LINE": self._cursor = snapped_line_endpoint(self._path[0], point, event.ctrl) @@ -525,6 +551,25 @@ class EditImageMask(bpy.types.Operator): return {"FINISHED"} +class EditImageMask(ImageMaskGesture, bpy.types.Operator): + bl_idname = "anyimage.edit_image_mask" + bl_label = "Edit Image Mask" + bl_description = "Edit and tightly crop an Image Empty with a gesture" + bl_options = {"REGISTER", "UNDO"} + + source_object_name: hidden_string_property() + source_matrix_data: hidden_string_property() + selection_data: hidden_string_property() + refine_selection: refine_selection_property() + invert: bpy.props.BoolProperty( + name="Invert", + description="Keep the opposite side of the refined gesture mask", + default=False, + options={"SKIP_SAVE"}, + ) + gesture: gesture_property() + + class RefineImageMask(JobOperatorBase, bpy.types.Operator): bl_idname = "anyimage.refine_image_mask" bl_label = "Refine Image Mask" diff --git a/src/anyimage/operators/material.py b/src/anyimage/operators/material.py index a1a230d..1ea4687 100644 --- a/src/anyimage/operators/material.py +++ b/src/anyimage/operators/material.py @@ -3,6 +3,7 @@ from pathlib import Path import bpy from .image import configure_image_user, is_animated_image + DISPLACEMENT_STRENGTH = 1.0 DISPLACEMENT_MIDLEVEL = 0.0 ORIGINAL_DISPLACEMENT_STRENGTH = 1.0 @@ -15,10 +16,12 @@ NODE_ASSET_NAME = "O_AnyImage.blend" MATERIAL_DISPLACEMENT_ATTRIBUTE = "o_material_displacement" BACKGROUND_DEPTH_ATTRIBUTE = "o_background_depth" DEPTH_SCALE_ATTRIBUTE = "o_depth_scale" -SURFACE_MATERIAL_NODE_GROUP_NAME = "O DA3 Surface Layer" -TEXTURE_MATERIAL_NODE_GROUP_NAME = "O Texture Layer" -MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME = "O MoGe2 Surface Layer" -BALLOON_MATERIAL_NODE_GROUP_NAME = "O Balloon Layer" +IMAGE_MATERIAL_NODE_GROUP_NAME = "O Image Layer" +IMAGE_MATERIAL_NODE_GROUP_VERSION = 6 +SURFACE_MATERIAL_NODE_GROUP_NAME = IMAGE_MATERIAL_NODE_GROUP_NAME +TEXTURE_MATERIAL_NODE_GROUP_NAME = IMAGE_MATERIAL_NODE_GROUP_NAME +MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME = IMAGE_MATERIAL_NODE_GROUP_NAME +BALLOON_MATERIAL_NODE_GROUP_NAME = IMAGE_MATERIAL_NODE_GROUP_NAME DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME = "O DA3 Depth Plane Layer" MOGE_MATERIAL_NODE_GROUP_NAME = "O MoGe2 Depth Plane Layer" SHADELESS_NODE_GROUP_NAME = "O Shadeless" @@ -30,6 +33,7 @@ VIEW_TRANSFORM_COLOR_SPACES = ( ("Khronos PBR Neutral", "Khronos PBR Neutral sRGB"), ) + def material_color_space_name(scene): view_settings = getattr(scene, "view_settings", None) view_transform = getattr(view_settings, "view_transform", "") @@ -46,6 +50,18 @@ def set_material_color_space(image, scene=None): settings = image.colorspace_settings preferred = material_color_space_name(scene) original = settings.name + pixels = None + if ( + preferred != original + and ( + getattr(image, "source", None) == "GENERATED" + or getattr(image, "is_dirty", False) + ) + ): + import numpy as np + + pixels = np.empty(len(image.pixels), dtype=np.float32) + image.pixels.foreach_get(pixels) for color_space in dict.fromkeys((preferred, "sRGB", original)): if not color_space: continue @@ -54,11 +70,13 @@ def set_material_color_space(image, scene=None): except (TypeError, ValueError): continue if settings.name == color_space: + if pixels is not None: + image.pixels.foreach_set(pixels) + image.update() return color_space return settings.name - def moge_material_node_group(): existing = bpy.data.node_groups.get(MOGE_MATERIAL_NODE_GROUP_NAME) if existing is not None: @@ -235,7 +253,6 @@ def create_moge_material( return material - def material_node_group(depth_plane=False, surface_group_name=None): group_name = ( DEPTH_PLANE_MATERIAL_NODE_GROUP_NAME @@ -243,22 +260,38 @@ def material_node_group(depth_plane=False, surface_group_name=None): else surface_group_name or SURFACE_MATERIAL_NODE_GROUP_NAME ) existing = bpy.data.node_groups.get(group_name) - if existing is not None: + rebuild_image_layer = ( + existing is not None + and group_name == IMAGE_MATERIAL_NODE_GROUP_NAME + and existing.get("anyimage_image_layer_version") + != IMAGE_MATERIAL_NODE_GROUP_VERSION + ) + if existing is not None and not rebuild_image_layer: + for item in existing.interface.items_tree: + if ( + item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal" + ): + item.default_value = (0.5, 0.5, 1.0, 1.0) return existing - group = bpy.data.node_groups.new(group_name, "ShaderNodeTree") + if existing is None: + group = bpy.data.node_groups.new(group_name, "ShaderNodeTree") + else: + group = existing + group.nodes.clear() + group.interface.clear() interface = group.interface input_sockets = [ ("Color", "NodeSocketColor"), ("Alpha", "NodeSocketFloat"), ("Bump Scale", "NodeSocketFloat"), ] - normal_group = group_name in { - MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME, - BALLOON_MATERIAL_NODE_GROUP_NAME, - } + normal_group = not depth_plane if normal_group: input_sockets.insert(2, ("Normal", "NodeSocketColor")) - input_sockets.insert(3, ("Normal Strength", "NodeSocketFloat")) + input_sockets.insert(3, ("Normal Scale", "NodeSocketFloat")) + input_sockets.insert(4, ("Object Space", "NodeSocketBool")) if depth_plane: input_sockets.insert(2, ("Depth", "NodeSocketFloat")) for name, socket_type in input_sockets: @@ -273,15 +306,17 @@ def material_node_group(depth_plane=False, surface_group_name=None): socket.max_value = 1.0 socket.subtype = "FACTOR" elif name == "Bump Scale": - socket.default_value = 1.0 + socket.default_value = 0.0 socket.min_value = -10000.0 socket.max_value = 10000.0 - elif name == "Normal Strength": - socket.default_value = ( - 0.5 if group_name == BALLOON_MATERIAL_NODE_GROUP_NAME else 1.0 - ) + elif name == "Normal": + socket.default_value = (0.5, 0.5, 1.0, 1.0) + elif name == "Normal Scale": + socket.default_value = 1.0 socket.min_value = 0.0 socket.max_value = 10.0 + elif name == "Object Space": + socket.default_value = False output_sockets = [ ("Color", "NodeSocketColor"), ("Roughness", "NodeSocketFloat"), @@ -311,21 +346,27 @@ def material_node_group(depth_plane=False, surface_group_name=None): roughness_curve.mapping.update() roughness_curve.location = (-250.0, 350.0) bump = nodes.new("ShaderNodeBump") - bump.location = (-250.0, 0.0) + bump.location = (100.0, 0.0) if normal_group: - normal_map = nodes.new("ShaderNodeNormalMap") - normal_map.space = ( - "TANGENT" - if group_name == BALLOON_MATERIAL_NODE_GROUP_NAME - else "OBJECT" - ) - normal_map.location = (-500.0, -150.0) - links.new(group_input.outputs["Normal"], normal_map.inputs["Color"]) - links.new( - group_input.outputs["Normal Strength"], - normal_map.inputs["Strength"], - ) - links.new(normal_map.outputs["Normal"], bump.inputs["Normal"]) + tangent_normal = nodes.new("ShaderNodeNormalMap") + tangent_normal.space = "TANGENT" + tangent_normal.location = (-500.0, -100.0) + object_normal = nodes.new("ShaderNodeNormalMap") + object_normal.space = "OBJECT" + object_normal.location = (-500.0, -350.0) + choose_normal = nodes.new("ShaderNodeMix") + choose_normal.data_type = "VECTOR" + choose_normal.location = (-150.0, -150.0) + for normal_map in (tangent_normal, object_normal): + links.new(group_input.outputs["Normal"], normal_map.inputs["Color"]) + links.new( + group_input.outputs["Normal Scale"], + normal_map.inputs["Strength"], + ) + links.new(group_input.outputs["Object Space"], choose_normal.inputs[0]) + links.new(tangent_normal.outputs["Normal"], choose_normal.inputs[4]) + links.new(object_normal.outputs["Normal"], choose_normal.inputs[5]) + links.new(choose_normal.outputs["Result"], bump.inputs["Normal"]) displacement = None processed_alpha = group_input.outputs["Alpha"] @@ -408,6 +449,8 @@ def material_node_group(depth_plane=False, surface_group_name=None): else f"Prepare {group_name} material outputs." ) group.use_fake_user = True + if group_name == IMAGE_MATERIAL_NODE_GROUP_NAME: + group["anyimage_image_layer_version"] = IMAGE_MATERIAL_NODE_GROUP_VERSION return group @@ -428,6 +471,7 @@ def create_material( use_cyclic=None, surface_group_name=None, normal_image=None, + normal_space="TANGENT", scene=None, ): material = bpy.data.materials.new(name) @@ -452,14 +496,10 @@ def create_material( output.location = (850.0, 350.0) extracted_frames = frame_count if frame_count > 1 else None color_frames = ( - color_frame_duration - if color_frame_duration is not None - else extracted_frames + color_frame_duration if color_frame_duration is not None else extracted_frames ) depth_frames = ( - depth_frame_duration - if depth_frame_duration is not None - else extracted_frames + depth_frame_duration if depth_frame_duration is not None else extracted_frames ) material_frames = color_frames material_frame_offset = color_frame_offset @@ -481,10 +521,12 @@ def create_material( is_depth_plane, surface_group_name=surface_group_name, ) - material_inputs.name = material_inputs.node_tree.name - material_inputs.label = material_inputs.name material_inputs.location = (-300.0, 350.0) material_inputs.inputs["Bump Scale"].default_value = depth_scale + if "Object Space" in material_inputs.inputs: + if normal_space not in {"TANGENT", "OBJECT"}: + raise ValueError(f"Unsupported normal space: {normal_space}") + material_inputs.inputs["Object Space"].default_value = normal_space == "OBJECT" links.new(color_texture.outputs["Color"], material_inputs.inputs["Color"]) links.new(alpha, material_inputs.inputs["Alpha"]) if normal_image is not None and "Normal" in material_inputs.inputs: @@ -608,6 +650,16 @@ def node_asset_path(): def load_asset_node_group(group_name): existing = bpy.data.node_groups.get(group_name) if existing is not None: + for item in existing.interface.items_tree: + if ( + item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal" + ): + item.default_value = (0.5, 0.5, 1.0, 1.0) + for node in existing.nodes: + if node.bl_idname == "ShaderNodeNormalMap": + node.space = "TANGENT" return existing asset_path = node_asset_path() if not asset_path.is_file(): @@ -648,6 +700,7 @@ def load_shadeless_node_group(): def load_depth_plane_node_group(group_name=DEPTH_PLANE_NODE_GROUP_NAME): return load_asset_node_group(group_name) + def load_mesh_plane_node_group(): return load_asset_node_group(MESH_PLANE_NODE_GROUP_NAME) @@ -691,18 +744,13 @@ def get_modifier_input(modifier, identifier): def modifier_input_driver_path(modifier, identifier): """Return the driver data path for a modifier input socket.""" if modifier_input_slots(modifier) is not None: - return ( - f'modifiers["{modifier.name}"]' - f".properties.inputs.{identifier}.value" - ) + return f'modifiers["{modifier.name}"].properties.inputs.{identifier}.value' return f'modifiers["{modifier.name}"]["{identifier}"]' def drive_modifier_input(owner_object, modifier, identifier, expression="frame"): """Add a scripted driver to a modifier input so it follows the scene frame.""" - driver = owner_object.driver_add( - modifier_input_driver_path(modifier, identifier) - ) + driver = owner_object.driver_add(modifier_input_driver_path(modifier, identifier)) driver.driver.type = "SCRIPTED" driver.driver.expression = expression return driver diff --git a/src/anyimage/operators/mesh.py b/src/anyimage/operators/mesh.py new file mode 100644 index 0000000..01a4904 --- /dev/null +++ b/src/anyimage/operators/mesh.py @@ -0,0 +1,1691 @@ +import json +import bpy + +from ..preferences import addon_preferences +from ..properties import BEN2_MODEL_KEY, DEFAULT_MOGE2_MODEL_KEY +from ..runtime import JobOperatorBase +from ..server.geometry.cutout import alpha_regions +from .balloon.depth import load_balloon_depth_data +from .balloon.geometry import ( + metric_transform, + prepare_planar_regions, + triangulate_planar_regions, +) +from .balloon.operator import extract_balloon_object +from .common import ( + HIDDEN, + cleanup_temporary_input, + da3_parameters, + input_path, + invoke_ai_setup_if_needed, + moge2_parameters, + production_device, + push_undo, + require_environment, + require_model, + source_start_frame, +) +from .image import ( + analysis_source_image, + create_color_image, + image_base_name, + image_user_settings, + is_image_empty, + material_data_name, + plane_bounds, + prepare_image_analysis_inputs, + remove_temporary_input, + scaled_process_resolution, + video_total_frames, +) +from .image_edit.common import ( + ImageEditWarning, + polygon_bounds, + report_image_edit_exception, +) +from .image_edit.mask import polygon_mask +from .material import ( + IMAGE_MATERIAL_NODE_GROUP_NAME, + create_material, + load_mesh_plane_node_group, + modifier_input_identifier, + set_modifier_input, +) +from .normal_map import bake_tangent_normal_image + +PLANE = "PLANE" +DEPTH_PLANE = "DEPTH_PLANE" +CUTOUT = "CUTOUT" +BALLOON = "BALLOON" +DEPTH_BALLOON = "DEPTH_BALLOON" +DEPTH_SURFACE = "DEPTH_SURFACE" +CUTOUT_SHAPES = {CUTOUT, BALLOON, DEPTH_BALLOON, DEPTH_SURFACE} +LOCAL_CUTOUT_SHAPES = {CUTOUT, BALLOON} +DEPTH_SHAPES = {DEPTH_PLANE, DEPTH_BALLOON, DEPTH_SURFACE} +PROFILE_ATTRIBUTE = "o_profile_offset" +DEPTH_ATTRIBUTE = "o_depth_offset" +FRONT_FACE_ATTRIBUTE = "o_front_face" +DEPTH_DISCONTINUITY_ATTRIBUTE = "o_depth_discontinuity" +DEPTH_PROTECTION_ATTRIBUTE = "o_depth_protection" +BOUNDARY_ATTRIBUTE = "o_boundary" +NODE_GROUP_VERSION = 13 +DEPTH_EDGE_THRESHOLD = 12.0 / 255.0 + + +def _plane_domain(bounds, detail, world_metric=None): + import numpy as np + + x_min, x_max, y_min, y_max = bounds + vertices = np.asarray( + ( + (x_min, y_min, 0.0), + (x_max, y_min, 0.0), + (x_max, y_max, 0.0), + (x_min, y_max, 0.0), + ), + dtype=np.float32, + ) + + faces = np.asarray(((0, 1, 2, 3),), dtype=np.int32) + uv = np.asarray(((0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 1.0))) + boundary = np.ones(4, dtype=bool) + planar = vertices[:, :2] @ metric_transform(world_metric) + return vertices, faces, uv, boundary, planar + + +def _depth_field_image(source_object, anchored): + import numpy as np + + values = np.asarray(anchored, dtype=np.float32)[..., 2] + height, width = values.shape + pixels = np.empty((height, width, 4), dtype=np.float32) + pixels[:, :, :3] = values[:, :, None] + pixels[:, :, 3] = 1.0 + image = bpy.data.images.new( + f"{image_base_name(source_object.data)}.depth", + width=width, + height=height, + alpha=True, + float_buffer=True, + ) + image.colorspace_settings.name = "Non-Color" + image.pixels.foreach_set(np.flipud(pixels).ravel()) + image.pack() + return image + + +def _cutout_domain(regions_pixels, image_size, bounds, detail, world_metric): + import numpy as np + + x_min, x_max, y_min, y_max = bounds + spacing = max(x_max - x_min, y_max - y_min) / max(int(detail), 1) + regions, spacing = prepare_planar_regions( + regions_pixels, + image_size, + bounds, + spacing=spacing, + world_metric=world_metric, + ) + planar, planar_faces, boundary = triangulate_planar_regions( + regions, spacing + ) + faces = np.asarray(planar_faces, dtype=np.int32) + local = planar @ np.linalg.inv(metric_transform(world_metric)) + vertices = np.column_stack((local, np.zeros(len(local), dtype=np.float64))) + uv = np.column_stack( + ( + (local[:, 0] - x_min) / (x_max - x_min), + (local[:, 1] - y_min) / (y_max - y_min), + ) + ).astype(np.float32) + return vertices, faces, uv, boundary, planar + + +def _world_plane_metric(matrix_world): + from mathutils import Vector + + linear = matrix_world.to_3x3() + x = linear @ Vector((1.0, 0.0, 0.0)) + y = linear @ Vector((0.0, 1.0, 0.0)) + z = linear @ Vector((0.0, 0.0, 1.0)) + return ((x.dot(x), x.dot(y)), (x.dot(y), y.dot(y))), z.length + + + +def _load_field(path): + import numpy as np + + with np.load(path, allow_pickle=False) as archive: + points = np.asarray(archive["points"], dtype=np.float32) + intrinsics = np.asarray(archive["intrinsics"], dtype=np.float32) + valid = np.asarray(archive["valid"], dtype=bool) + image_size = tuple(int(value) for value in archive["image_size"]) + reference_depth = float(archive["reference_depth"]) + if points.ndim != 3 or points.shape[2] != 3 or valid.shape != points.shape[:2]: + raise ValueError("The mesh point field is invalid") + if intrinsics.shape != (3, 3) or not reference_depth > 0.0: + raise ValueError("The mesh camera field is invalid") + return points, intrinsics, valid, image_size, reference_depth + + +def _anchored_point_field(source_object, field): + import numpy as np + + points, intrinsics, valid, image_size, reference_depth = field + width, height = image_size + x_min, x_max, y_min, y_max = plane_bounds(source_object) + plane_width = x_max - x_min + plane_height = y_max - y_min + fx, fy = float(intrinsics[0, 0]), float(intrinsics[1, 1]) + cx, cy = float(intrinsics[0, 2]), float(intrinsics[1, 2]) + if fx <= 0.0 or fy <= 0.0: + raise ValueError("Camera intrinsics contain an invalid focal length") + scale_x = plane_width * fx / (max(width - 1, 1) * reference_depth) + scale_y = plane_height * fy / (max(height - 1, 1) * reference_depth) + scale_z = float(np.sqrt(scale_x * scale_y)) + camera_x = x_min + plane_width * cx / max(width - 1, 1) + camera_y = y_max - plane_height * cy / max(height - 1, 1) + anchored = np.empty_like(points, dtype=np.float32) + anchored[..., 0] = camera_x + points[..., 0] * scale_x + anchored[..., 1] = camera_y - points[..., 1] * scale_y + anchored[..., 2] = (reference_depth - points[..., 2]) * scale_z + return anchored, valid, points[..., 2] + + +def _sample_field(values, valid, uv): + import numpy as np + + height, width = valid.shape + x = np.clip(uv[:, 0], 0.0, 1.0) * max(width - 1, 0) + y = (1.0 - np.clip(uv[:, 1], 0.0, 1.0)) * max(height - 1, 0) + x0 = np.floor(x).astype(np.int32) + y0 = np.floor(y).astype(np.int32) + x1 = np.minimum(x0 + 1, width - 1) + y1 = np.minimum(y0 + 1, height - 1) + weights = np.stack( + ( + (x1 - x) * (y1 - y), + (x - x0) * (y1 - y), + (x1 - x) * (y - y0), + (x - x0) * (y - y0), + ), + axis=1, + ) + samples = np.stack( + (values[y0, x0], values[y0, x1], values[y1, x0], values[y1, x1]), axis=1 + ) + usable = np.stack( + (valid[y0, x0], valid[y0, x1], valid[y1, x0], valid[y1, x1]), axis=1 + ) + weights *= usable + total = weights.sum(axis=1) + vector_field = values.ndim == 3 + result = np.zeros( + (len(uv), values.shape[-1]) if vector_field else len(uv), + dtype=np.float32, + ) + selected = total > 1e-8 + if vector_field: + result[selected] = (samples[selected] * weights[selected, :, None]).sum( + axis=1 + ) / total[selected, None] + else: + result[selected] = (samples[selected] * weights[selected]).sum(axis=1) / total[ + selected + ] + if not selected.all(): + nearest_x = np.rint(x[~selected]).astype(np.int32) + nearest_y = np.rint(y[~selected]).astype(np.int32) + result[~selected] = values[nearest_y, nearest_x] + return result + + +def _depth_discontinuity_values(faces, depth): + import numpy as np + + triangle_depth = np.asarray(depth, dtype=np.float32)[faces] + valid = np.isfinite(triangle_depth).all(axis=1) & (triangle_depth > 0.0).all(axis=1) + scale = np.maximum(np.min(np.abs(triangle_depth), axis=1), 1e-6) + difference = np.max(triangle_depth, axis=1) - np.min(triangle_depth, axis=1) + result = np.full(len(faces), float("inf"), dtype=np.float32) + result[valid] = difference[valid] / scale[valid] + return result + + +def _depth_discontinuity_map(depth, valid): + import numpy as np + + depth = np.asarray(depth, dtype=np.float32) + valid = np.asarray(valid, dtype=bool) & np.isfinite(depth) & (depth > 0.0) + padded_depth = np.pad(depth, 1, mode="edge") + padded_valid = np.pad(valid, 1, mode="edge") + windows = np.lib.stride_tricks.sliding_window_view(padded_depth, (3, 3)) + usable = np.lib.stride_tricks.sliding_window_view(padded_valid, (3, 3)) + minimum = np.min(np.where(usable, windows, np.inf), axis=(-2, -1)) + maximum = np.max(np.where(usable, windows, -np.inf), axis=(-2, -1)) + result = (maximum - minimum) / np.maximum(np.abs(depth), 1e-6) + result[~usable.all(axis=(-2, -1))] = 1.0 + return result.astype(np.float32) + + +def _vertex_normals(vertices, faces): + import numpy as np + + triangles = vertices[faces] + normals = np.cross( + triangles[:, 1] - triangles[:, 0], triangles[:, 2] - triangles[:, 0] + ) + result = np.zeros_like(vertices, dtype=np.float64) + for corner in range(3): + np.add.at(result, faces[:, corner], normals) + lengths = np.linalg.norm(result, axis=1) + result[lengths > 1e-10] /= lengths[lengths > 1e-10, None] + result[lengths <= 1e-10] = (0.0, 0.0, 1.0) + return result + + +def _counter_clockwise_faces(points, faces): + import numpy as np + + points = np.asarray(points, dtype=np.float64) + result = np.asarray(faces, dtype=np.int32).copy() + polygons = points[result] + following = np.roll(polygons, -1, axis=1) + signed_area = np.sum( + polygons[..., 0] * following[..., 1] + - polygons[..., 1] * following[..., 0], + axis=1, + ) + result[signed_area < 0.0] = result[signed_area < 0.0, ::-1] + return result + + +def _boundary_edges(faces): + import numpy as np + + faces = np.asarray(faces, dtype=np.int32) + directed = np.concatenate( + (faces[:, (0, 1)], faces[:, (1, 2)], faces[:, (2, 0)]), axis=0 + ) + keys = np.sort(directed, axis=1) + _unique, inverse, users = np.unique( + keys, axis=0, return_inverse=True, return_counts=True + ) + return directed[users[inverse] == 1] + + +def _layered_geometry(domain, shape, field=None): + import numpy as np + + vertices, faces, uv, boundary, planar = domain + faces = _counter_clockwise_faces(planar, faces) + depth = np.zeros_like(vertices, dtype=np.float64) + discontinuity = np.zeros(len(faces), dtype=np.float32) + vertex_discontinuity = np.zeros(len(vertices), dtype=np.float32) + carrier = vertices + if field is not None: + anchored, valid, camera_depth = field + target = _sample_field(anchored, valid, uv) + if shape == DEPTH_SURFACE: + vertex_discontinuity = _sample_field( + _depth_discontinuity_map(camera_depth, valid), valid, uv + ) + discontinuity = np.max(vertex_discontinuity[faces], axis=1) + depth = target - vertices + carrier = target + elif shape == DEPTH_PLANE: + depth[:, 2] = target[:, 2] + + normals = _vertex_normals(carrier, faces) + front = np.arange(len(vertices), dtype=np.int32) + back = front + len(vertices) + output_vertices = np.concatenate((vertices, vertices), axis=0) + output_uv = np.concatenate((uv, uv), axis=0) + if shape == DEPTH_SURFACE: + profile_offset = np.concatenate((np.zeros_like(normals), -normals), axis=0) + depth_offset = np.concatenate((depth, depth), axis=0) + else: + profile_offset = np.concatenate((normals * 0.5, -normals * 0.5), axis=0) + depth_offset = np.concatenate((depth, np.zeros_like(depth)), axis=0) + + output_faces = [tuple(int(value) for value in face) for face in faces] + output_faces.extend( + tuple(int(back[value]) for value in reversed(face)) for face in faces + ) + output_faces.extend( + (int(front[start]), int(back[start]), int(back[end]), int(front[end])) + for start, end in _boundary_edges(faces) + ) + front_faces = [True] * len(faces) + [False] * (len(output_faces) - len(faces)) + output_boundary = np.concatenate((boundary, boundary), axis=0) + depth_protection = np.concatenate( + (vertex_discontinuity, vertex_discontinuity), axis=0 + ) + face_discontinuity = np.concatenate( + (discontinuity, discontinuity, np.zeros(len(output_faces) - len(faces) * 2)) + ) + return ( + output_vertices, + output_faces, + output_uv, + profile_offset, + depth_offset, + front_faces, + output_boundary, + depth_protection, + face_discontinuity, + ) + + +def _assign_uv(mesh, uv): + import numpy as np + + for layer in list(mesh.uv_layers): + mesh.uv_layers.remove(layer) + uv_layer = mesh.uv_layers.new(name="UVMap") + loop_vertices = np.empty(len(mesh.loops), dtype=np.int32) + mesh.loops.foreach_get("vertex_index", loop_vertices) + loop_uv = np.asarray(uv, dtype=np.float32)[loop_vertices] + uv_layer.data.foreach_set("uv", loop_uv.ravel()) + + +def _assign_vector_attribute(mesh, name, values): + import numpy as np + + attribute = mesh.attributes.new(name=name, type="FLOAT_VECTOR", domain="POINT") + attribute.data.foreach_set("vector", np.asarray(values, dtype=np.float32).ravel()) + + +def _assign_bool_point_attribute(mesh, name, values): + import numpy as np + + attribute = mesh.attributes.new(name=name, type="BOOLEAN", domain="POINT") + attribute.data.foreach_set("value", np.asarray(values, dtype=bool)) + + +def _assign_float_point_attribute(mesh, name, values): + import numpy as np + + attribute = mesh.attributes.new(name=name, type="FLOAT", domain="POINT") + attribute.data.foreach_set("value", np.asarray(values, dtype=np.float32)) + + +def _assign_face_attribute(mesh, name, values): + import numpy as np + + attribute = mesh.attributes.new(name=name, type="BOOLEAN", domain="FACE") + attribute.data.foreach_set("value", np.asarray(values, dtype=bool)) + + +def _assign_float_face_attribute(mesh, name, values): + import numpy as np + + attribute = mesh.attributes.new(name=name, type="FLOAT", domain="FACE") + attribute.data.foreach_set("value", np.asarray(values, dtype=np.float32)) + + +def _interface_socket(group, name, in_out, socket_type): + socket = group.interface.new_socket( + name=name, + in_out=in_out, + socket_type=socket_type, + ) + if in_out == "INPUT" and hasattr(socket, "structure_type"): + socket.structure_type = "SINGLE" + return socket + + +def _group_input(nodes, location, visible_names): + node = nodes.new("NodeGroupInput") + node.location = location + visible = set(visible_names) + for output in node.outputs: + output.hide = output.name not in visible + return node + + +def _edge_boundary_field(nodes, links, location): + x, y = location + vertex_neighbors = nodes.new("GeometryNodeInputMeshVertexNeighbors") + vertex_neighbors.location = (x, y + 150.0) + loose_vertex = nodes.new("FunctionNodeCompare") + loose_vertex.data_type = "INT" + loose_vertex.operation = "EQUAL" + loose_vertex_a = next( + socket for socket in loose_vertex.inputs if socket.identifier == "A_INT" + ) + loose_vertex_b = next( + socket for socket in loose_vertex.inputs if socket.identifier == "B_INT" + ) + loose_vertex_b.default_value = 1 + loose_vertex.location = (x + 200.0, y + 150.0) + links.new(vertex_neighbors.outputs["Vertex Count"], loose_vertex_a) + + edge_neighbors = nodes.new("GeometryNodeInputMeshEdgeNeighbors") + edge_neighbors.location = (x, y - 150.0) + boundary_edge = nodes.new("FunctionNodeCompare") + boundary_edge.data_type = "INT" + boundary_edge.operation = "EQUAL" + boundary_edge_a = next( + socket for socket in boundary_edge.inputs if socket.identifier == "A_INT" + ) + boundary_edge_b = next( + socket for socket in boundary_edge.inputs if socket.identifier == "B_INT" + ) + boundary_edge_b.default_value = 1 + boundary_edge.location = (x + 200.0, y - 150.0) + links.new(edge_neighbors.outputs["Face Count"], boundary_edge_a) + edge_domain = nodes.new("GeometryNodeFieldOnDomain") + edge_domain.data_type = "BOOLEAN" + edge_domain.domain = "EDGE" + edge_domain.location = (x + 400.0, y - 150.0) + links.new(boundary_edge.outputs["Result"], edge_domain.inputs["Value"]) + + is_boundary = nodes.new("FunctionNodeBooleanMath") + is_boundary.operation = "OR" + is_boundary.location = (x + 600.0, y) + links.new(edge_domain.outputs["Value"], is_boundary.inputs[0]) + links.new(loose_vertex.outputs["Result"], is_boundary.inputs[1]) + return is_boundary.outputs["Boolean"] + + +def _expand_smooth(group, geometry, x_offset=0.0): + nodes = group.nodes + links = group.links + existing_nodes = set(nodes) + iteration_controls = _group_input(nodes, (550.0, 1150.0), {"Iterations"}) + weight_controls = _group_input(nodes, (2300.0, 1450.0), {"Smooth"}) + repeat_output = nodes.new("GeometryNodeRepeatOutput") + repeat_output.location = (3050.0, 550.0) + repeat_input = nodes.new("GeometryNodeRepeatInput") + repeat_input.pair_with_output(repeat_output) + repeat_input.location = (550.0, 550.0) + links.new(iteration_controls.outputs["Iterations"], repeat_input.inputs["Iterations"]) + links.new(geometry, repeat_input.inputs["Geometry"]) + + boundary_field = _edge_boundary_field(nodes, links, (750.0, -350.0)) + boundary_points = nodes.new("GeometryNodeFieldOnDomain") + boundary_points.data_type = "BOOLEAN" + boundary_points.domain = "POINT" + boundary_points.location = (1550.0, -350.0) + links.new(boundary_field, boundary_points.inputs["Value"]) + + boundary_geometry = nodes.new("GeometryNodeSeparateGeometry") + boundary_geometry.domain = "POINT" + boundary_geometry.location = (1750.0, -150.0) + links.new(repeat_input.outputs["Geometry"], boundary_geometry.inputs["Geometry"]) + links.new(boundary_points.outputs["Value"], boundary_geometry.inputs["Selection"]) + + position = nodes.new("GeometryNodeInputPosition") + position.location = (750.0, 850.0) + smooth_all = nodes.new("GeometryNodeBlurAttribute") + smooth_all.data_type = "FLOAT_VECTOR" + smooth_all.inputs["Iterations"].default_value = 1 + smooth_all.inputs["Weight"].default_value = 1.0 + smooth_all.location = (950.0, 850.0) + links.new(position.outputs["Position"], smooth_all.inputs["Value"]) + + boundary_position = nodes.new("GeometryNodeInputPosition") + boundary_position.location = (1350.0, 250.0) + smooth_boundary = nodes.new("GeometryNodeBlurAttribute") + smooth_boundary.data_type = "FLOAT_VECTOR" + smooth_boundary.inputs["Iterations"].default_value = 1 + smooth_boundary.inputs["Weight"].default_value = 0.5 + smooth_boundary.location = (1550.0, 250.0) + links.new(boundary_position.outputs["Position"], smooth_boundary.inputs["Value"]) + nearest_boundary = nodes.new("GeometryNodeSampleNearest") + nearest_boundary.domain = "POINT" + nearest_boundary.location = (1800.0, 150.0) + links.new(boundary_geometry.outputs["Selection"], nearest_boundary.inputs["Geometry"]) + sample_boundary = nodes.new("GeometryNodeSampleIndex") + sample_boundary.data_type = "FLOAT_VECTOR" + sample_boundary.domain = "POINT" + sample_boundary.clamp = False + sample_boundary.location = (2050.0, 250.0) + links.new(boundary_geometry.outputs["Selection"], sample_boundary.inputs["Geometry"]) + links.new(smooth_boundary.outputs["Value"], sample_boundary.inputs["Value"]) + links.new(nearest_boundary.outputs["Index"], sample_boundary.inputs["Index"]) + + choose_boundary = nodes.new("GeometryNodeSwitch") + choose_boundary.input_type = "VECTOR" + choose_boundary.location = (2300.0, 350.0) + direct_boundary = _edge_boundary_field(nodes, links, (750.0, 450.0)) + links.new(direct_boundary, choose_boundary.inputs["Switch"]) + links.new(smooth_all.outputs["Value"], choose_boundary.inputs["False"]) + links.new(sample_boundary.outputs["Value"], choose_boundary.inputs["True"]) + boundary_mix = nodes.new("ShaderNodeMix") + boundary_mix.data_type = "VECTOR" + boundary_mix.inputs[0].default_value = 1.0 + boundary_mix.location = (2550.0, 400.0) + links.new(smooth_all.outputs["Value"], boundary_mix.inputs[4]) + links.new(choose_boundary.outputs["Output"], boundary_mix.inputs[5]) + + normal = nodes.new("GeometryNodeInputNormal") + normal.location = (1350.0, 1050.0) + smooth_normal = nodes.new("GeometryNodeBlurAttribute") + smooth_normal.data_type = "FLOAT_VECTOR" + smooth_normal.inputs["Iterations"].default_value = 10 + smooth_normal.inputs["Weight"].default_value = 1.0 + smooth_normal.location = (1550.0, 1050.0) + links.new(normal.outputs["Normal"], smooth_normal.inputs["Value"]) + normal_coherence = nodes.new("ShaderNodeVectorMath") + normal_coherence.operation = "LENGTH" + normal_coherence.location = (1800.0, 1050.0) + links.new(smooth_normal.outputs["Value"], normal_coherence.inputs[0]) + sharp_switch = nodes.new("GeometryNodeSwitch") + sharp_switch.input_type = "FLOAT" + sharp_switch.inputs["Switch"].default_value = True + sharp_switch.inputs["False"].default_value = 1.0 + sharp_switch.location = (2050.0, 1050.0) + links.new(normal_coherence.outputs["Value"], sharp_switch.inputs["True"]) + smooth_weight = nodes.new("ShaderNodeMath") + smooth_weight.operation = "MULTIPLY" + smooth_weight.location = (2300.0, 1050.0) + links.new(weight_controls.outputs["Smooth"], smooth_weight.inputs[0]) + links.new(sharp_switch.outputs["Output"], smooth_weight.inputs[1]) + + target_position = nodes.new("GeometryNodeSwitch") + target_position.input_type = "VECTOR" + target_position.inputs["Switch"].default_value = True + target_position.location = (2550.0, 700.0) + links.new(smooth_all.outputs["Value"], target_position.inputs["False"]) + links.new(boundary_mix.outputs["Result"], target_position.inputs["True"]) + current_position = nodes.new("GeometryNodeInputPosition") + current_position.location = (2300.0, 850.0) + mix_position = nodes.new("ShaderNodeMix") + mix_position.data_type = "VECTOR" + mix_position.location = (2800.0, 850.0) + links.new(smooth_weight.outputs[0], mix_position.inputs[0]) + links.new(current_position.outputs["Position"], mix_position.inputs[4]) + links.new(target_position.outputs["Output"], mix_position.inputs[5]) + set_position = nodes.new("GeometryNodeSetPosition") + set_position.location = (2800.0, 550.0) + links.new(repeat_input.outputs["Geometry"], set_position.inputs["Geometry"]) + links.new(mix_position.outputs["Result"], set_position.inputs["Position"]) + links.new(set_position.outputs["Geometry"], repeat_output.inputs["Geometry"]) + for node in set(nodes) - existing_nodes: + node.location.x += x_offset + return repeat_output.outputs["Geometry"] + + +def _depth_plane_node_group(): + group_name = "O Image Depth Plane" + group = bpy.data.node_groups.get(group_name) + if group is not None and group.get("anyimage_mesh_version") == NODE_GROUP_VERSION: + return group + if group is None: + group = bpy.data.node_groups.new(group_name, "GeometryNodeTree") + else: + group.nodes.clear() + group.interface.clear() + group.is_modifier = True + _interface_socket(group, "Geometry", "INPUT", "NodeSocketGeometry") + subdivide = _interface_socket(group, "Subdivide", "INPUT", "NodeSocketInt") + subdivide.default_value = 7 + subdivide.min_value = 0 + subdivide.max_value = 10 + thickness = _interface_socket(group, "Thickness", "INPUT", "NodeSocketFloat") + thickness.default_value = 0.1 + thickness.min_value = 0.0 + thickness.subtype = "DISTANCE" + depth_amount = _interface_socket(group, "Depth Amount", "INPUT", "NodeSocketFloat") + depth_amount.default_value = 1.0 + depth_amount.min_value = 0.0 + depth_image = _interface_socket(group, "Depth Image", "INPUT", "NodeSocketImage") + depth_image.hide_in_modifier = True + material = _interface_socket(group, "Material", "INPUT", "NodeSocketMaterial") + material.hide_in_modifier = True + _interface_socket(group, "Geometry", "OUTPUT", "NodeSocketGeometry") + + nodes = group.nodes + links = group.links + geometry_input = _group_input( + nodes, + (-1850.0, 450.0), + {"Geometry", "Subdivide", "Material"}, + ) + shape_input = _group_input( + nodes, + (-1250.0, 850.0), + {"Thickness", "Depth Amount"}, + ) + image_input = _group_input(nodes, (-1050.0, 50.0), {"Depth Image"}) + cleanup_input = _group_input(nodes, (650.0, -200.0), {"Thickness"}) + group_output = nodes.new("NodeGroupOutput") + group_output.location = (1600.0, 450.0) + mesh_plane = nodes.new("GeometryNodeGroup") + mesh_plane.node_tree = load_mesh_plane_node_group() + mesh_plane.inputs["Thickness"].default_value = 0.0 + mesh_plane.location = (-1550.0, 450.0) + links.new(geometry_input.outputs["Geometry"], mesh_plane.inputs["Geometry"]) + links.new(geometry_input.outputs["Subdivide"], mesh_plane.inputs["Subdivide"]) + links.new(geometry_input.outputs["Material"], mesh_plane.inputs["Material"]) + + extrude = nodes.new("GeometryNodeExtrudeMesh") + extrude.mode = "FACES" + extrude.inputs["Offset"].default_value = (0.0, 0.0, -1.0) + extrude.inputs["Individual"].default_value = False + extrude.location = (-1200.0, 450.0) + links.new(mesh_plane.outputs["Geometry"], extrude.inputs["Mesh"]) + links.new(shape_input.outputs["Thickness"], extrude.inputs["Offset Scale"]) + geometry = extrude.outputs["Mesh"] + + uv = nodes.new("GeometryNodeInputNamedAttribute") + uv.data_type = "FLOAT_VECTOR" + uv.inputs["Name"].default_value = "UVMap" + uv.location = (-1000.0, 50.0) + texture = nodes.new("GeometryNodeImageTexture") + texture.location = (-800.0, 50.0) + links.new(image_input.outputs["Depth Image"], texture.inputs["Image"]) + links.new(uv.outputs["Attribute"], texture.inputs["Vector"]) + separate = nodes.new("ShaderNodeSeparateXYZ") + separate.location = (-600.0, 50.0) + links.new(texture.outputs["Color"], separate.inputs["Vector"]) + raw_depth = nodes.new("ShaderNodeMath") + raw_depth.operation = "MULTIPLY" + raw_depth.location = (-350.0, 150.0) + links.new(separate.outputs["X"], raw_depth.inputs[0]) + links.new(shape_input.outputs["Depth Amount"], raw_depth.inputs[1]) + inward_limit = nodes.new("ShaderNodeMath") + inward_limit.operation = "MULTIPLY" + inward_limit.inputs[1].default_value = -0.98 + inward_limit.location = (-350.0, -50.0) + links.new(shape_input.outputs["Thickness"], inward_limit.inputs[0]) + clamp = nodes.new("ShaderNodeMath") + clamp.operation = "MAXIMUM" + clamp.location = (-100.0, 50.0) + links.new(raw_depth.outputs[0], clamp.inputs[0]) + links.new(inward_limit.outputs[0], clamp.inputs[1]) + has_thickness = nodes.new("ShaderNodeMath") + has_thickness.operation = "GREATER_THAN" + has_thickness.inputs[1].default_value = 1e-6 + has_thickness.location = (-350.0, -250.0) + links.new(shape_input.outputs["Thickness"], has_thickness.inputs[0]) + choose_depth = nodes.new("GeometryNodeSwitch") + choose_depth.input_type = "FLOAT" + choose_depth.location = (150.0, 50.0) + links.new(has_thickness.outputs[0], choose_depth.inputs["Switch"]) + links.new(raw_depth.outputs[0], choose_depth.inputs["False"]) + links.new(clamp.outputs[0], choose_depth.inputs["True"]) + combine = nodes.new("ShaderNodeCombineXYZ") + combine.location = (350.0, 50.0) + links.new(choose_depth.outputs["Output"], combine.inputs["Z"]) + front = nodes.new("FunctionNodeBooleanMath") + front.operation = "NOT" + front.location = (150.0, 300.0) + links.new(extrude.outputs["Top"], front.inputs[0]) + set_position = nodes.new("GeometryNodeSetPosition") + set_position.location = (550.0, 450.0) + links.new(geometry, set_position.inputs["Geometry"]) + links.new(front.outputs[0], set_position.inputs["Selection"]) + links.new(combine.outputs["Vector"], set_position.inputs["Offset"]) + geometry = set_position.outputs["Geometry"] + + zero_thickness = nodes.new("ShaderNodeMath") + zero_thickness.operation = "LESS_THAN" + zero_thickness.inputs[1].default_value = 1e-6 + zero_thickness.location = (750.0, -150.0) + links.new(cleanup_input.outputs["Thickness"], zero_thickness.inputs[0]) + back_or_side = nodes.new("FunctionNodeBooleanMath") + back_or_side.operation = "OR" + back_or_side.location = (750.0, 150.0) + links.new(extrude.outputs["Top"], back_or_side.inputs[0]) + links.new(extrude.outputs["Side"], back_or_side.inputs[1]) + delete_selection = nodes.new("FunctionNodeBooleanMath") + delete_selection.operation = "AND" + delete_selection.location = (950.0, 50.0) + links.new(zero_thickness.outputs[0], delete_selection.inputs[0]) + links.new(back_or_side.outputs[0], delete_selection.inputs[1]) + delete_geometry = nodes.new("GeometryNodeDeleteGeometry") + delete_geometry.domain = "FACE" + delete_geometry.location = (1100.0, 450.0) + links.new(geometry, delete_geometry.inputs["Geometry"]) + links.new(delete_selection.outputs[0], delete_geometry.inputs["Selection"]) + shade_smooth = nodes.new("GeometryNodeSetShadeSmooth") + shade_smooth.domain = "FACE" + shade_smooth.inputs["Shade Smooth"].default_value = True + shade_smooth.location = (1350.0, 450.0) + links.new(delete_geometry.outputs["Geometry"], shade_smooth.inputs["Geometry"]) + links.new(shade_smooth.outputs["Geometry"], group_output.inputs["Geometry"]) + group["anyimage_mesh_version"] = NODE_GROUP_VERSION + group.use_fake_user = True + return group + + +def _mesh_node_group(shape): + if shape == PLANE: + return load_mesh_plane_node_group() + if shape == DEPTH_PLANE: + return _depth_plane_node_group() + group_name = { + CUTOUT: "O Image Cutout", + DEPTH_SURFACE: "O Image Depth Surface", + }[shape] + group = bpy.data.node_groups.get(group_name) + if group is not None and group.get("anyimage_mesh_version") == NODE_GROUP_VERSION: + return group + if group is None: + group = bpy.data.node_groups.new(group_name, "GeometryNodeTree") + else: + group.nodes.clear() + group.interface.clear() + group.is_modifier = True + _interface_socket(group, "Geometry", "INPUT", "NodeSocketGeometry") + thickness = _interface_socket(group, "Thickness", "INPUT", "NodeSocketFloat") + thickness.default_value = 0.0 + thickness.min_value = 0.0 + thickness.subtype = "DISTANCE" + if shape == DEPTH_SURFACE: + depth_amount = _interface_socket( + group, "Depth Scale", "INPUT", "NodeSocketFloat" + ) + depth_amount.default_value = 1.0 + depth_amount.min_value = 0.0 + iterations = _interface_socket( + group, "Iterations", "INPUT", "NodeSocketInt" + ) + iterations.default_value = 1 + iterations.min_value = 0 + iterations.max_value = 20 + smooth = _interface_socket(group, "Smooth", "INPUT", "NodeSocketFloat") + smooth.default_value = 1.0 + smooth.min_value = 0.0 + smooth.max_value = 1.0 + smooth.subtype = "FACTOR" + continuity = _interface_socket( + group, "Depth Continuity", "INPUT", "NodeSocketFloat" + ) + continuity.default_value = 0.0 + continuity.min_value = 0.0 + continuity.max_value = 100.0 + continuity.subtype = "PERCENTAGE" + _interface_socket(group, "Geometry", "OUTPUT", "NodeSocketGeometry") + + nodes = group.nodes + links = group.links + shape_input = _group_input( + nodes, + (-1900.0, 250.0), + {"Geometry", "Thickness", "Depth Scale"}, + ) + group_output = nodes.new("NodeGroupOutput") + + profile_attribute = nodes.new("GeometryNodeInputNamedAttribute") + profile_attribute.data_type = "FLOAT_VECTOR" + profile_attribute.inputs["Name"].default_value = PROFILE_ATTRIBUTE + profile_attribute.location = (-1900.0, 650.0) + profile_scale = nodes.new("ShaderNodeVectorMath") + profile_scale.operation = "SCALE" + profile_scale.location = (-1650.0, 650.0) + links.new(profile_attribute.outputs["Attribute"], profile_scale.inputs[0]) + links.new(shape_input.outputs["Thickness"], profile_scale.inputs[3]) + offset = profile_scale.outputs["Vector"] + + if shape == DEPTH_SURFACE: + depth_attribute = nodes.new("GeometryNodeInputNamedAttribute") + depth_attribute.data_type = "FLOAT_VECTOR" + depth_attribute.inputs["Name"].default_value = DEPTH_ATTRIBUTE + depth_attribute.location = (-1900.0, 400.0) + depth_scale = nodes.new("ShaderNodeVectorMath") + depth_scale.operation = "SCALE" + depth_scale.location = (-1650.0, 400.0) + links.new(depth_attribute.outputs["Attribute"], depth_scale.inputs[0]) + links.new(shape_input.outputs["Depth Scale"], depth_scale.inputs[3]) + offset = depth_scale.outputs["Vector"] + + set_position = nodes.new("GeometryNodeSetPosition") + set_position.location = (-1150.0, 550.0) + links.new(shape_input.outputs["Geometry"], set_position.inputs["Geometry"]) + links.new(offset, set_position.inputs["Offset"]) + geometry = set_position.outputs["Geometry"] + + if shape == DEPTH_SURFACE: + culling_input = _group_input( + nodes, + (-1050.0, -400.0), + {"Depth Continuity"}, + ) + discontinuity = nodes.new("GeometryNodeInputNamedAttribute") + discontinuity.data_type = "FLOAT" + discontinuity.inputs["Name"].default_value = DEPTH_DISCONTINUITY_ATTRIBUTE + discontinuity.location = (-1150.0, 50.0) + continuity_scale = nodes.new("ShaderNodeMath") + continuity_scale.operation = "MULTIPLY" + continuity_scale.inputs[1].default_value = 0.01 + continuity_scale.location = (-900.0, -250.0) + inverse_continuity = nodes.new("ShaderNodeMath") + inverse_continuity.operation = "SUBTRACT" + inverse_continuity.inputs[0].default_value = 0.5 + inverse_continuity.location = (-700.0, -250.0) + exponent_scale = nodes.new("ShaderNodeMath") + exponent_scale.operation = "MULTIPLY" + exponent_scale.inputs[1].default_value = 10.0 + exponent_scale.location = (-500.0, -250.0) + threshold_power = nodes.new("ShaderNodeMath") + threshold_power.operation = "POWER" + threshold_power.inputs[0].default_value = 2.0 + threshold_power.location = (-300.0, -250.0) + threshold_scale = nodes.new("ShaderNodeMath") + threshold_scale.operation = "MULTIPLY" + threshold_scale.inputs[1].default_value = 0.04 + threshold_scale.location = (-100.0, -250.0) + depth_jump = nodes.new("ShaderNodeMath") + depth_jump.operation = "GREATER_THAN" + depth_jump.location = (100.0, 50.0) + delete_depth = nodes.new("GeometryNodeDeleteGeometry") + delete_depth.domain = "FACE" + delete_depth.location = (350.0, 550.0) + links.new( + culling_input.outputs["Depth Continuity"], + continuity_scale.inputs[0], + ) + links.new(continuity_scale.outputs[0], inverse_continuity.inputs[1]) + links.new(inverse_continuity.outputs[0], exponent_scale.inputs[0]) + links.new(exponent_scale.outputs[0], threshold_power.inputs[1]) + links.new(threshold_power.outputs[0], threshold_scale.inputs[0]) + links.new(discontinuity.outputs["Attribute"], depth_jump.inputs[0]) + links.new(threshold_scale.outputs[0], depth_jump.inputs[1]) + links.new(geometry, delete_depth.inputs["Geometry"]) + links.new(depth_jump.outputs[0], delete_depth.inputs["Selection"]) + geometry = delete_depth.outputs["Geometry"] + + apply_thickness = nodes.new("GeometryNodeSetPosition") + apply_thickness.location = (550.0, 550.0) + links.new(geometry, apply_thickness.inputs["Geometry"]) + links.new(profile_scale.outputs["Vector"], apply_thickness.inputs["Offset"]) + geometry = apply_thickness.outputs["Geometry"] + + front_face = nodes.new("GeometryNodeInputNamedAttribute") + front_face.data_type = "BOOLEAN" + front_face.inputs["Name"].default_value = FRONT_FACE_ATTRIBUTE + not_front = nodes.new("FunctionNodeBooleanMath") + not_front.operation = "NOT" + zero_thickness = nodes.new("ShaderNodeMath") + zero_thickness.operation = "LESS_THAN" + zero_thickness.inputs[1].default_value = 1e-6 + delete_selection = nodes.new("FunctionNodeBooleanMath") + delete_selection.operation = "AND" + delete_geometry = nodes.new("GeometryNodeDeleteGeometry") + delete_geometry.domain = "FACE" + finish_x = 800.0 if shape == DEPTH_SURFACE else -750.0 + finish_input = _group_input( + nodes, + (finish_x + 150.0, -250.0), + {"Thickness"}, + ) + front_face.location = (finish_x, 100.0) + not_front.location = (finish_x + 200.0, 100.0) + zero_thickness.location = (finish_x + 200.0, -100.0) + delete_selection.location = (finish_x + 400.0, 0.0) + delete_geometry.location = (finish_x + 650.0, 550.0) + links.new(front_face.outputs["Attribute"], not_front.inputs[0]) + links.new(finish_input.outputs["Thickness"], zero_thickness.inputs[0]) + links.new(not_front.outputs[0], delete_selection.inputs[0]) + links.new(zero_thickness.outputs[0], delete_selection.inputs[1]) + links.new(geometry, delete_geometry.inputs["Geometry"]) + links.new(delete_selection.outputs[0], delete_geometry.inputs["Selection"]) + + geometry = delete_geometry.outputs["Geometry"] + if shape == DEPTH_SURFACE: + geometry = _expand_smooth(group, geometry, x_offset=2250.0) + shade_x = 5550.0 + else: + shade_x = finish_x + 900.0 + + shade_smooth = nodes.new("GeometryNodeSetShadeSmooth") + shade_smooth.domain = "FACE" + shade_smooth.inputs["Shade Smooth"].default_value = True + shade_smooth.location = (shade_x, 550.0) + group_output.location = (shade_x + 250.0, 550.0) + links.new(geometry, shade_smooth.inputs["Geometry"]) + links.new(shade_smooth.outputs["Geometry"], group_output.inputs["Geometry"]) + group.color_tag = "GEOMETRY" + group.description = ( + "Cull depth discontinuities, extrude thickness backward, then smooth inline." + if shape == DEPTH_SURFACE + else "Apply cutout thickness and smooth shading." + ) + group["anyimage_mesh_version"] = NODE_GROUP_VERSION + group.use_fake_user = True + return group + + +def _unique_color_name(source_object, current=None): + base = image_base_name(source_object.data) + index = 0 + while True: + number = "" if index == 0 else f".{index:03d}" + candidate = f"{base}{number}.color" + existing = bpy.data.images.get(candidate) + if existing is None or existing is current: + return candidate + index += 1 + + +def _copy_color_image(source_object): + source = source_object.data + image = source.copy() + if getattr(source, "source", None) == "GENERATED" or getattr( + source, + "is_dirty", + False, + ): + import numpy as np + + pixels = np.empty(len(source.pixels), dtype=np.float32) + source.pixels.foreach_get(pixels) + image.pixels.foreach_set(pixels) + image.update() + image.name = _unique_color_name(source_object, image) + return image + + +def _image_rgba(image): + import numpy as np + + width, height = (int(value) for value in image.size) + pixels = np.empty(len(image.pixels), dtype=np.float32) + image.pixels.foreach_get(pixels) + return np.flipud(pixels.reshape((height, width, 4))) + + +def _cutout_regions(alpha, mesh_detail): + contour_resolution = max(512, min(int(mesh_detail) * 8, 1024)) + return alpha_regions(alpha, max_resolution=contour_resolution) + + +def _antialias_selection(selection): + import numpy as np + + values = np.asarray(selection, dtype=np.float32) + kernel = np.asarray((0.029, 0.235, 0.472, 0.235, 0.029), dtype=np.float32) + radius = len(kernel) // 2 + horizontal = np.zeros_like(values) + padded = np.pad(values, ((0, 0), (radius, radius)), mode="edge") + for offset, weight in enumerate(kernel): + horizontal += padded[:, offset : offset + values.shape[1]] * weight + result = np.zeros_like(values) + padded = np.pad(horizontal, ((radius, radius), (0, 0)), mode="edge") + for offset, weight in enumerate(kernel): + result += padded[offset : offset + values.shape[0]] * weight + return result + + +def _geometry_cutout_regions( + alpha, + polygon, + mesh_detail, + invert=False, + refine_selection=False, + selection=None, +): + import numpy as np + + values = np.asarray(alpha, dtype=np.float32) + if values.ndim != 2: + raise ValueError("Cutout Alpha must be two-dimensional") + source_height, source_width = values.shape + source_size = (source_width, source_height) + if refine_selection: + regions, _image_size = _cutout_regions(values, mesh_detail) + return regions, source_size + if invert: + geometry_alpha = selection + if geometry_alpha is None: + geometry_alpha = 1.0 - _antialias_selection( + polygon_mask(source_size, polygon, "NONZERO") + ) + regions, _image_size = _cutout_regions(geometry_alpha, mesh_detail) + return regions, source_size + + left, top, right, bottom = polygon_bounds(polygon) + inside_source = ( + left >= 0 + and top >= 0 + and right <= source_width + and bottom <= source_height + ) + if inside_source: + geometry_alpha = values if refine_selection else selection + if geometry_alpha is None: + geometry_alpha = _antialias_selection( + polygon_mask(source_size, polygon, "NONZERO") + ) + regions, _image_size = _cutout_regions(geometry_alpha, mesh_detail) + return regions, source_size + + canvas_size = (right - left, bottom - top) + relative = np.asarray(polygon, dtype=np.float64) - (left, top) + geometry_alpha = _antialias_selection( + polygon_mask(canvas_size, relative, "NONZERO") + ) + regions, _canvas_size = _cutout_regions(geometry_alpha, mesh_detail) + return ( + [ + [ + [[x + left, y + top] for x, y in contour] + for contour in region + ] + for region in regions + ], + source_size, + ) + +def _local_cutout_rgba(source_object, polygon, invert=False): + import numpy as np + + rgba = _image_rgba(source_object.data).copy() + height, width = rgba.shape[:2] + selection = _antialias_selection( + polygon_mask((width, height), polygon, "NONZERO") + ) + if invert: + selection = 1.0 - selection + rgba[:, :, 3] *= selection + rgba[rgba[:, :, 3] <= 0.0, :3] = 0.0 + if not np.any(rgba[:, :, 3] > 0.0): + raise ImageEditWarning("The selection contains no visible image pixels") + return rgba, selection + + +def _rename_color_image(image, source_object): + image.name = _unique_color_name(source_object, image) + return image + + +def _new_material( + context, + source_object, + color_image, + normal_image=None, + normal_space="TANGENT", +): + return create_material( + material_data_name(source_object), + color_image, + 0.0, + surface_group_name=IMAGE_MATERIAL_NODE_GROUP_NAME, + normal_image=normal_image, + normal_space=normal_space, + scene=context.scene, + ) + + +def _build_mesh_object( + context, + source_object, + shape, + detail, + material, + field=None, + regions=None, + image_size=None, + keep_original=False, +): + import numpy as np + + bounds = plane_bounds(source_object) + world_metric, _world_z_scale = _world_plane_metric(source_object.matrix_world) + domain = ( + _plane_domain(bounds, detail, world_metric) + if shape in {PLANE, DEPTH_PLANE} + else _cutout_domain(regions, image_size, bounds, detail, world_metric) + ) + anchored = _anchored_point_field(source_object, field) if field else None + depth_image = None + face_discontinuity = None + if shape in {PLANE, DEPTH_PLANE}: + vertices, faces, uv, boundary, _planar = domain + if shape == DEPTH_PLANE: + depth_image = _depth_field_image(source_object, anchored[0]) + else: + ( + vertices, + faces, + uv, + profile, + depth, + front_faces, + boundary, + depth_protection, + face_discontinuity, + ) = _layered_geometry(domain, shape, anchored) + source_name = source_object.name + mesh = bpy.data.meshes.new(source_name) + mesh.from_pydata(np.asarray(vertices).tolist(), [], faces) + for polygon in mesh.polygons: + polygon.use_smooth = True + _assign_uv(mesh, uv) + if shape not in {PLANE, DEPTH_PLANE}: + _assign_vector_attribute(mesh, PROFILE_ATTRIBUTE, profile) + _assign_vector_attribute(mesh, DEPTH_ATTRIBUTE, depth) + _assign_face_attribute(mesh, FRONT_FACE_ATTRIBUTE, front_faces) + _assign_bool_point_attribute(mesh, BOUNDARY_ATTRIBUTE, boundary) + _assign_float_point_attribute( + mesh, DEPTH_PROTECTION_ATTRIBUTE, depth_protection + ) + if face_discontinuity is not None: + _assign_float_face_attribute( + mesh, DEPTH_DISCONTINUITY_ATTRIBUTE, face_discontinuity + ) + mesh.materials.append(material) + mesh.update() + mesh_object = bpy.data.objects.new(source_name, mesh) + context.collection.objects.link(mesh_object) + mesh_object.matrix_world = source_object.matrix_world.copy() + mesh_object["anyimage_mesh_shape"] = shape + node_group = _mesh_node_group(shape) + modifier = mesh_object.modifiers.new(name=node_group.name, type="NODES") + modifier.node_group = node_group + modifier.show_group_selector = False + if shape in {PLANE, DEPTH_PLANE}: + subdivision = max(0, min(10, int(detail))) + set_modifier_input( + modifier, + modifier_input_identifier(node_group, "Subdivide"), + subdivision, + ) + set_modifier_input( + modifier, + modifier_input_identifier(node_group, "Material"), + material, + ) + thickness = 0.1 if shape == DEPTH_PLANE else 0.0 + set_modifier_input( + modifier, + modifier_input_identifier(node_group, "Thickness"), + thickness, + ) + if shape in {DEPTH_PLANE, DEPTH_SURFACE}: + set_modifier_input( + modifier, + modifier_input_identifier( + node_group, + "Depth Scale" if shape == DEPTH_SURFACE else "Depth Amount", + ), + 1.0, + ) + if depth_image is not None: + set_modifier_input( + modifier, + modifier_input_identifier(node_group, "Depth Image"), + depth_image, + ) + for selected in context.selected_objects: + selected.select_set(False) + mesh_object.select_set(True) + context.view_layer.objects.active = mesh_object + if not keep_original: + bpy.data.objects.remove(source_object, do_unlink=True) + mesh_object.name = source_name + return mesh_object + + +def _load_normal_image(path, source_object): + image = bpy.data.images.load(str(path), check_existing=False) + image.name = f"{image_base_name(source_object.data)}.normal" + image.colorspace_settings.name = "Non-Color" + image.pack() + return image + + +def _balloon_resolution(source_object, detail): + import math + + x_min, x_max, y_min, y_max = plane_bounds(source_object) + metric, _world_z_scale = _world_plane_metric(source_object.matrix_world) + width = (x_max - x_min) * math.sqrt(max(metric[0][0], 0.0)) + height = (y_max - y_min) * math.sqrt(max(metric[1][1], 0.0)) + return max(width, height) / max(int(detail), 8) + + +def _build_local_cutout( + operator, + context, + source_object, + shape, + selection_data, + invert=False, +): + if video_total_frames(source_object.data) != 1: + raise ValueError("Mesh conversion currently supports still images") + polygon = json.loads(selection_data) if selection_data else None + if not isinstance(polygon, list) or len(polygon) < 3: + raise ValueError("The Cutout selection is invalid") + rgba, selection = _local_cutout_rgba(source_object, polygon, invert) + regions, image_size = _geometry_cutout_regions( + rgba[:, :, 3], + polygon, + operator.mesh_detail, + invert, + refine_selection=False, + selection=selection, + ) + color_image = None + material = None + try: + color_image = _copy_color_image(source_object) + if shape == BALLOON: + result_object = extract_balloon_object( + context, + source_object, + regions, + source_object.matrix_world.copy(), + "POISSON", + thickness=1.0, + resolution=_balloon_resolution(source_object, operator.mesh_detail), + select_result=True, + reuse_material=False, + depth_data=None, + normal_path=None, + material_image=color_image, + ) + result_object["anyimage_mesh_shape"] = shape + else: + material = _new_material(context, source_object, color_image) + _build_mesh_object( + context, + source_object, + shape, + operator.mesh_detail, + material, + regions=regions, + image_size=image_size, + keep_original=True, + ) + except Exception: + if material is not None and material.users == 0: + bpy.data.materials.remove(material) + if color_image is not None and color_image.users == 0: + bpy.data.images.remove(color_image) + raise + push_undo(f"Convert to {shape.replace('_', ' ').title()}") + +def _build_response(context, result, operator): + source_object = bpy.data.objects.get(operator.source_object_name) + if not is_image_empty(source_object): + raise RuntimeError("The source Image Empty no longer exists") + regions = image_size = None + if operator.shape in CUTOUT_SHAPES: + geometry_image = create_color_image( + source_object, + result.directory, + assign_to_source=False, + ) + try: + rgba = _image_rgba(geometry_image) + finally: + if geometry_image is not source_object.data: + bpy.data.images.remove(geometry_image) + polygon = json.loads(operator.selection_data) + regions, image_size = _geometry_cutout_regions( + rgba[:, :, 3], + polygon, + operator.mesh_detail, + operator.invert, + refine_selection=operator.refine_selection, + ) + color_image = _copy_color_image(source_object) + else: + color_image = _rename_color_image( + create_color_image( + source_object, + result.directory, + assign_to_source=False, + ), + source_object, + ) + normal_path = result.directory / "normal.png" + + if operator.shape in {BALLOON, DEPTH_BALLOON}: + depth_data = ( + load_balloon_depth_data(result.file("depth")) + if operator.shape == DEPTH_BALLOON + else None + ) + balloon = extract_balloon_object( + context, + source_object, + regions, + source_object.matrix_world.copy(), + "POISSON", + thickness=1.0, + resolution=_balloon_resolution(source_object, operator.mesh_detail), + select_result=True, + reuse_material=False, + depth_data=depth_data, + normal_path=normal_path if normal_path.is_file() else None, + material_image=color_image, + bake_normal_to_tangent=normal_path.is_file(), + ) + balloon["anyimage_mesh_shape"] = operator.shape + else: + normal_image = ( + _load_normal_image(normal_path, source_object) + if normal_path.is_file() + else None + ) + material = _new_material( + context, + source_object, + color_image, + normal_image, + normal_space=( + "OBJECT" + if operator.shape == DEPTH_SURFACE and normal_image is not None + else "TANGENT" + ), + ) + field = ( + _load_field(result.file("field")) + if operator.shape in DEPTH_SHAPES + else None + ) + mesh_object = _build_mesh_object( + context, + source_object, + operator.shape, + operator.mesh_detail, + material, + field=field, + regions=regions, + image_size=image_size, + keep_original=operator.shape in CUTOUT_SHAPES, + ) + if normal_image is not None and operator.shape == CUTOUT: + bake_tangent_normal_image( + context, + source_object, + mesh_object, + normal_image, + ) + push_undo(f"Convert to {operator.shape.replace('_', ' ').title()}") + + +def _depth_settings(context, shape): + preferences = addon_preferences(context) + model = getattr(preferences, "mesh_depth_model", DEFAULT_MOGE2_MODEL_KEY) + if shape == DEPTH_BALLOON: + model = DEFAULT_MOGE2_MODEL_KEY + process_factor = float(getattr(preferences, "mesh_process_res_factor", 100.0)) + resolution_level = int(getattr(preferences, "mesh_resolution_level", 5)) + return model, process_factor, resolution_level + + +class GenerateMesh(JobOperatorBase, bpy.types.Operator): + bl_idname = "anyimage.generate_mesh" + bl_label = "Generate Image Mesh" + job_type = "generate-mesh" + starting_message = "Preparing image mesh" + + input_path: bpy.props.StringProperty(options=HIDDEN) + analysis_input_path: bpy.props.StringProperty(options=HIDDEN) + source_object_name: bpy.props.StringProperty(options=HIDDEN) + shape: bpy.props.StringProperty(options=HIDDEN) + selection_data: bpy.props.StringProperty(default="", options=HIDDEN) + mesh_detail: bpy.props.IntProperty(default=128, min=4, max=1024, options=HIDDEN) + model: bpy.props.StringProperty(default=DEFAULT_MOGE2_MODEL_KEY, options=HIDDEN) + process_res: bpy.props.IntProperty(default=1008, min=1, options=HIDDEN) + resolution_level: bpy.props.IntProperty(default=5, min=0, max=9, options=HIDDEN) + normal_model: bpy.props.StringProperty( + default=DEFAULT_MOGE2_MODEL_KEY, options=HIDDEN + ) + normal_resolution_level: bpy.props.IntProperty( + default=5, min=0, max=9, options=HIDDEN + ) + normal_analysis: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_analysis_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + refine_selection: bpy.props.BoolProperty(default=False, options=HIDDEN) + invert: bpy.props.BoolProperty(default=False, options=HIDDEN) + + def request(self, context): + require_environment() + if self.shape in DEPTH_SHAPES: + require_model(self.model) + if self.normal_analysis and self.shape in CUTOUT_SHAPES and ( + self.shape not in DEPTH_SHAPES or self.normal_model != self.model + ): + require_model(self.normal_model) + if self.refine_selection: + require_model(BEN2_MODEL_KEY) + polygon = json.loads(self.selection_data) if self.selection_data else None + if self.shape in CUTOUT_SHAPES and (polygon is None or len(polygon) < 3): + raise ValueError("The Cutout selection is invalid") + analysis_value = getattr(self, "analysis_input_path", "") or self.input_path + parameters = { + "input": str(input_path(self.input_path)), + "analysis_input": str(input_path(analysis_value)), + "shape": self.shape, + "polygon": polygon, + "mesh_detail": self.mesh_detail, + "device": production_device().lower(), + "video_frames": 1, + "start_frame": source_start_frame(self.source_object_name), + "refine_selection": self.refine_selection, + "normal_analysis": self.normal_analysis, + "invert": self.invert, + } + if self.normal_analysis and self.shape in CUTOUT_SHAPES: + parameters.update( + normal_model=self.normal_model, + normal_resolution_level=self.normal_resolution_level, + ) + if self.refine_selection: + parameters["refine_model"] = BEN2_MODEL_KEY + if self.shape not in DEPTH_SHAPES: + return parameters + settings = context.scene.anyimage_settings + if self.model.startswith("MOGE2_"): + parameters.update( + moge2_parameters( + settings, + input_path(self.input_path), + self.model, + self.resolution_level, + source_start_frame(self.source_object_name), + ) + ) + else: + parameters.update( + da3_parameters( + settings, + input_path(self.input_path), + self.model, + self.process_res, + source_start_frame(self.source_object_name), + ) + ) + parameters.update(shape=self.shape, polygon=polygon) + return parameters + + def response(self, context, result): + _build_response(context, result, self) + return f"{self.shape.replace('_', ' ').title()} is ready" + + def cleanup(self): + cleanup_temporary_input(self.input_path, self.delete_input) + cleanup_temporary_input( + getattr(self, "analysis_input_path", ""), + getattr(self, "delete_analysis_input", False), + ) + + +def _start_job( + operator, + context, + shape, + selection_data="", + refine_selection=False, + normal_analysis=False, + invert=False, +): + source_object = context.object + if video_total_frames(source_object.data) != 1: + operator.report({"ERROR"}, "Mesh conversion currently supports still images") + return {"CANCELLED"} + setup_result = invoke_ai_setup_if_needed() + if setup_result is not None: + return setup_result + try: + ( + source_path, + delete_input, + analysis_path, + delete_analysis_input, + ) = prepare_image_analysis_inputs( + source_object.data, context.scene + ) + model, process_factor, resolution_level = _depth_settings(context, shape) + analysis_image = analysis_source_image(source_object.data) + process_res = ( + scaled_process_resolution(analysis_image, process_factor) + if shape in DEPTH_SHAPES and not model.startswith("MOGE2_") + else 1 + ) + except (RuntimeError, ValueError) as error: + operator.report({"ERROR"}, str(error)) + return {"CANCELLED"} + result = bpy.ops.anyimage.generate_mesh( + input_path=str(source_path), + analysis_input_path=str(analysis_path), + source_object_name=source_object.name, + shape=shape, + selection_data=selection_data, + mesh_detail=operator.mesh_detail, + model=model, + process_res=process_res, + resolution_level=resolution_level, + normal_model=DEFAULT_MOGE2_MODEL_KEY, + normal_resolution_level=resolution_level, + normal_analysis=normal_analysis, + delete_input=delete_input, + delete_analysis_input=delete_analysis_input, + refine_selection=refine_selection, + invert=invert, + ) + if "CANCELLED" in result: + if delete_input: + remove_temporary_input(source_path) + if delete_analysis_input: + remove_temporary_input(analysis_path) + return {"CANCELLED"} if "CANCELLED" in result else {"FINISHED"} + + +class ConvertToPlane(bpy.types.Operator): + bl_idname = "anyimage.convert_to_plane" + bl_label = "Convert to Plane" + bl_description = "Convert the complete Image Empty into a rectangular plane" + bl_options = {"REGISTER", "UNDO"} + + mesh_detail: bpy.props.IntProperty( + name="Subdivide", + default=0, + min=0, + max=10, + ) + + @classmethod + def poll(cls, context): + return is_image_empty(context.object) + + def execute(self, context): + source_object = context.object + color_image = _copy_color_image(source_object) + frame_start, frame_offset, frame_duration, use_cyclic = image_user_settings( + source_object + ) + material = create_material( + material_data_name(source_object), + color_image, + 1.0, + frame_start=frame_start, + color_frame_offset=frame_offset, + color_frame_duration=frame_duration, + use_cyclic=use_cyclic, + scene=context.scene, + ) + _build_mesh_object( + context, + source_object, + PLANE, + self.mesh_detail, + material, + ) + return {"FINISHED"} + + +class ConvertToDepthPlane(bpy.types.Operator): + bl_idname = "anyimage.convert_to_depth_plane" + bl_label = "Convert to Depth Plane" + bl_description = "Convert the complete Image Empty into a front-projected relief" + bl_options = {"REGISTER", "UNDO"} + + mesh_detail: bpy.props.IntProperty( + name="Subdivide", + default=0, + min=0, + max=10, + ) + + @classmethod + def poll(cls, context): + from ..runtime import runtime + + return is_image_empty(context.object) and not runtime.server_busy() + + def invoke(self, context, _event): + return self.execute(context) + + def execute(self, context): + return _start_job(self, context, DEPTH_PLANE) + + +class ConvertCutoutShape(bpy.types.Operator): + bl_idname = "anyimage.convert_cutout_shape" + bl_label = "Convert Cutout Shape" + bl_description = "Convert the saved Cutout selection into the chosen shape" + bl_options = {"UNDO"} + + source_object_name: bpy.props.StringProperty(options=HIDDEN) + selection_data: bpy.props.StringProperty(options=HIDDEN) + shape: bpy.props.StringProperty(options=HIDDEN) + mesh_detail: bpy.props.IntProperty(default=128, min=4, max=1024, options=HIDDEN) + refine_selection: bpy.props.BoolProperty(default=False, options=HIDDEN) + normal_analysis: bpy.props.BoolProperty(default=False, options=HIDDEN) + invert: bpy.props.BoolProperty(default=False, options=HIDDEN) + + def execute(self, context): + source_object = bpy.data.objects.get(self.source_object_name) + if not is_image_empty(source_object): + self.report({"ERROR"}, "The source Image Empty is no longer available") + return {"CANCELLED"} + context.view_layer.objects.active = source_object + source_object.select_set(True) + if ( + self.shape in LOCAL_CUTOUT_SHAPES + and not self.refine_selection + and not self.normal_analysis + ): + try: + _build_local_cutout( + self, + context, + source_object, + self.shape, + self.selection_data, + self.invert, + ) + except Exception as error: + report_image_edit_exception(self, error) + return {"CANCELLED"} + return {"FINISHED"} + return _start_job( + self, + context, + self.shape, + self.selection_data, + self.refine_selection, + self.normal_analysis, + self.invert, + ) diff --git a/src/anyimage/operators/normal_map.py b/src/anyimage/operators/normal_map.py new file mode 100644 index 0000000..97a13de --- /dev/null +++ b/src/anyimage/operators/normal_map.py @@ -0,0 +1,158 @@ +def _normalized(values, fallback): + import numpy as np + + values = np.asarray(values, dtype=np.float64) + lengths = np.linalg.norm(values, axis=-1) + result = np.empty_like(values) + usable = lengths > 1e-10 + result[usable] = values[usable] / lengths[usable, None] + result[~usable] = fallback + return result + + +def rasterize_tangent_normal( + object_normal, + triangle_uv, + triangle_normal, + triangle_tangent, + triangle_sign, + return_coverage=False, +): + """Convert object-space pixels through a smooth per-corner UV tangent frame.""" + import numpy as np + + object_normal = _normalized(object_normal, (0.0, 0.0, 1.0)) + height, width = object_normal.shape[:2] + output = np.zeros((height, width, 3), dtype=np.float32) + output[..., 2] = 1.0 + coverage = np.zeros((height, width), dtype=bool) + scale = np.asarray((max(width - 1, 1), max(height - 1, 1))) + + for uv, corner_normal, corner_tangent, corner_sign in zip( + triangle_uv, + triangle_normal, + triangle_tangent, + triangle_sign, + strict=True, + ): + pixel = np.asarray(uv, dtype=np.float64) * scale + x_min = max(int(np.ceil(np.min(pixel[:, 0]))), 0) + x_max = min(int(np.floor(np.max(pixel[:, 0]))), width - 1) + y_min = max(int(np.ceil(np.min(pixel[:, 1]))), 0) + y_max = min(int(np.floor(np.max(pixel[:, 1]))), height - 1) + if x_min > x_max or y_min > y_max: + continue + + a, b, c = pixel + denominator = (b[1] - c[1]) * (a[0] - c[0]) + ( + c[0] - b[0] + ) * (a[1] - c[1]) + if abs(denominator) <= 1e-12: + continue + grid_x, grid_y = np.meshgrid( + np.arange(x_min, x_max + 1, dtype=np.float64), + np.arange(y_min, y_max + 1, dtype=np.float64), + ) + first = ( + (b[1] - c[1]) * (grid_x - c[0]) + + (c[0] - b[0]) * (grid_y - c[1]) + ) / denominator + second = ( + (c[1] - a[1]) * (grid_x - c[0]) + + (a[0] - c[0]) * (grid_y - c[1]) + ) / denominator + third = 1.0 - first - second + inside = (first >= -1e-6) & (second >= -1e-6) & (third >= -1e-6) + if not inside.any(): + continue + + weights = np.stack((first[inside], second[inside], third[inside]), axis=-1) + normal = _normalized(weights @ corner_normal, (0.0, 0.0, 1.0)) + tangent = weights @ corner_tangent + tangent -= normal * np.sum(tangent * normal, axis=-1, keepdims=True) + tangent = _normalized(tangent, (1.0, 0.0, 0.0)) + sign = np.where(weights @ corner_sign >= 0.0, 1.0, -1.0) + bitangent = np.cross(normal, tangent) * sign[:, None] + + rows = grid_y[inside].astype(np.int32) + columns = grid_x[inside].astype(np.int32) + source = object_normal[rows, columns] + output[rows, columns, 0] = np.sum(source * tangent, axis=-1) + output[rows, columns, 1] = np.sum(source * bitangent, axis=-1) + output[rows, columns, 2] = np.sum(source * normal, axis=-1) + coverage[rows, columns] = True + + output[coverage] = _normalized(output[coverage], (0.0, 0.0, 1.0)) + return (output, coverage) if return_coverage else output + + +def bake_tangent_normal_image(context, source_object, target_object, image): + """Bake a MoGe object-space image against the evaluated target mesh TBN.""" + import numpy as np + + width, height = (int(value) for value in image.size) + rgba = np.empty(width * height * 4, dtype=np.float32) + image.pixels.foreach_get(rgba) + rgba = rgba.reshape((height, width, 4)) + source_normal = rgba[..., :3].astype(np.float64) * 2.0 - 1.0 + + source_to_target = ( + target_object.matrix_world.inverted().to_3x3() + @ source_object.matrix_world.to_3x3() + ) + normal_matrix = source_to_target.inverted().transposed() + transform = np.asarray(normal_matrix, dtype=np.float64) + target_normal = source_normal @ transform.T + target_normal = _normalized(target_normal, (0.0, 0.0, 1.0)) + front = _normalized( + np.asarray((0.0, 0.0, 1.0), dtype=np.float64)[None, :] @ transform.T, + (0.0, 0.0, 1.0), + )[0] + + context.view_layer.update() + depsgraph = context.evaluated_depsgraph_get() + evaluated = target_object.evaluated_get(depsgraph) + mesh = evaluated.to_mesh(preserve_all_data_layers=True, depsgraph=depsgraph) + try: + uv_layer = mesh.uv_layers.active + if uv_layer is None: + raise ValueError("The target mesh has no active UV map") + mesh.calc_loop_triangles() + mesh.calc_tangents(uvmap=uv_layer.name) + loop_count = len(mesh.loops) + uv = np.empty((loop_count, 2), dtype=np.float64) + uv_layer.data.foreach_get("uv", uv.ravel()) + tangent = np.empty((loop_count, 3), dtype=np.float64) + sign = np.empty(loop_count, dtype=np.float64) + mesh.loops.foreach_get("tangent", tangent.ravel()) + mesh.loops.foreach_get("bitangent_sign", sign) + normal = np.asarray( + [tuple(loop.normal) for loop in mesh.loops], + dtype=np.float64, + ) + triangles = np.asarray( + [tuple(triangle.loops) for triangle in mesh.loop_triangles], + dtype=np.int32, + ) + triangle_normal = np.asarray( + [tuple(triangle.normal) for triangle in mesh.loop_triangles], + dtype=np.float64, + ) + front_facing = triangle_normal @ front > 1e-6 + triangles = triangles[front_facing] + tangent_normal, coverage = rasterize_tangent_normal( + target_normal, + uv[triangles], + normal[triangles], + tangent[triangles], + sign[triangles], + return_coverage=True, + ) + finally: + evaluated.to_mesh_clear() + + rgba[coverage, :3] = tangent_normal[coverage] * 0.5 + 0.5 + image.pixels.foreach_set(rgba.ravel()) + image.update() + image.pack() + return image diff --git a/src/anyimage/operators/surface.py b/src/anyimage/operators/surface.py index 6a8f90e..95c72e2 100644 --- a/src/anyimage/operators/surface.py +++ b/src/anyimage/operators/surface.py @@ -18,12 +18,13 @@ from .common import ( ) from .image import ( + analysis_source_image, draw_process_resolution, is_image_empty, material_data_name, normal_data_name, plane_bounds, - prepare_image_input, + prepare_image_analysis_inputs, process_resolution_scale_property, remove_temporary_input, scaled_process_resolution, @@ -474,6 +475,7 @@ def build_image_mesh( MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME if moge2 else None ), normal_image=normal_image, + normal_space="OBJECT" if moge2 and normal_image is not None else "TANGENT", scene=context.scene, ) mesh.materials.append(material) @@ -568,7 +570,11 @@ class ConvertToSurface(bpy.types.Operator): if self.model.startswith("MOGE2_"): layout.prop(self, "resolution_level") else: - draw_process_resolution(layout, self, context.object.data) + draw_process_resolution( + layout, + self, + analysis_source_image(context.object.data), + ) layout.prop(self, "create_camera") layout.prop(self, "remove_background") layout.prop(self, "shadeless") @@ -581,13 +587,19 @@ class ConvertToSurface(bpy.types.Operator): self.report({"ERROR"}, "Convert to Surface currently supports still images") return {"CANCELLED"} try: + analysis_image = analysis_source_image(source_object.data) process_res = 1 if not self.model.startswith("MOGE2_"): process_res = scaled_process_resolution( - source_object.data, + analysis_image, self.process_res_factor, ) - source_path, delete_input = prepare_image_input( + ( + source_path, + delete_input, + analysis_path, + delete_analysis_input, + ) = prepare_image_analysis_inputs( source_object.data, context.scene, ) @@ -596,10 +608,12 @@ class ConvertToSurface(bpy.types.Operator): return {"CANCELLED"} common = { "input_path": str(source_path), + "analysis_input_path": str(analysis_path), "source_object_name": source_object.name, "model": self.model, "delete_source": not self.keep_original, "delete_input": delete_input, + "delete_analysis_input": delete_analysis_input, "create_camera": self.create_camera, "shadeless": self.shadeless, "remove_background": self.remove_background, @@ -617,6 +631,8 @@ class ConvertToSurface(bpy.types.Operator): if "CANCELLED" in result: if delete_input: remove_temporary_input(source_path) + if delete_analysis_input: + remove_temporary_input(analysis_path) return {"CANCELLED"} return {"FINISHED"} @@ -669,11 +685,13 @@ class GenerateDa3Surface(JobOperatorBase, bpy.types.Operator): starting_message = "Starting perspective surface generation" input_path: bpy.props.StringProperty(options=HIDDEN) + analysis_input_path: bpy.props.StringProperty(options=HIDDEN) source_object_name: bpy.props.StringProperty(options=HIDDEN) model: bpy.props.StringProperty(default="BASE", options=HIDDEN) process_res: bpy.props.IntProperty(default=1024, min=1, options=HIDDEN) delete_source: bpy.props.BoolProperty(default=True, options=HIDDEN) delete_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_analysis_input: bpy.props.BoolProperty(default=False, options=HIDDEN) create_camera: bpy.props.BoolProperty(default=False, options=HIDDEN) shadeless: bpy.props.BoolProperty(default=False, options=HIDDEN) remove_background: bpy.props.BoolProperty(default=False, options=HIDDEN) @@ -684,7 +702,7 @@ class GenerateDa3Surface(JobOperatorBase, bpy.types.Operator): if self.remove_background: require_model(BEN2_MODEL_KEY) settings = context.scene.anyimage_settings - return { + parameters = { **da3_parameters( settings, input_path(self.input_path), @@ -696,6 +714,9 @@ class GenerateDa3Surface(JobOperatorBase, bpy.types.Operator): BEN2_MODEL_KEY if self.remove_background else "" ), } + analysis_value = getattr(self, "analysis_input_path", "") or self.input_path + parameters["analysis_input"] = str(input_path(analysis_value)) + return parameters def response(self, context, result): build_surface_response( @@ -711,6 +732,10 @@ class GenerateDa3Surface(JobOperatorBase, bpy.types.Operator): def cleanup(self): cleanup_temporary_input(self.input_path, self.delete_input) + cleanup_temporary_input( + getattr(self, "analysis_input_path", ""), + getattr(self, "delete_analysis_input", False), + ) class GenerateMoge2Surface(JobOperatorBase, bpy.types.Operator): @@ -720,11 +745,13 @@ class GenerateMoge2Surface(JobOperatorBase, bpy.types.Operator): starting_message = "Starting MoGe-2 inference" input_path: bpy.props.StringProperty(options=HIDDEN) + analysis_input_path: bpy.props.StringProperty(options=HIDDEN) source_object_name: bpy.props.StringProperty(options=HIDDEN) model: bpy.props.StringProperty(default=DEFAULT_MOGE2_MODEL_KEY, options=HIDDEN) resolution_level: bpy.props.IntProperty(default=5, min=0, max=9, options=HIDDEN) delete_source: bpy.props.BoolProperty(default=True, options=HIDDEN) delete_input: bpy.props.BoolProperty(default=False, options=HIDDEN) + delete_analysis_input: bpy.props.BoolProperty(default=False, options=HIDDEN) create_camera: bpy.props.BoolProperty(default=False, options=HIDDEN) shadeless: bpy.props.BoolProperty(default=False, options=HIDDEN) remove_background: bpy.props.BoolProperty(default=False, options=HIDDEN) @@ -735,7 +762,7 @@ class GenerateMoge2Surface(JobOperatorBase, bpy.types.Operator): if self.remove_background: require_model(BEN2_MODEL_KEY) settings = context.scene.anyimage_settings - return { + parameters = { **moge2_parameters( settings, input_path(self.input_path), @@ -747,6 +774,9 @@ class GenerateMoge2Surface(JobOperatorBase, bpy.types.Operator): BEN2_MODEL_KEY if self.remove_background else "" ), } + analysis_value = getattr(self, "analysis_input_path", "") or self.input_path + parameters["analysis_input"] = str(input_path(analysis_value)) + return parameters def response(self, context, result): build_surface_response( @@ -762,3 +792,7 @@ class GenerateMoge2Surface(JobOperatorBase, bpy.types.Operator): def cleanup(self): cleanup_temporary_input(self.input_path, self.delete_input) + cleanup_temporary_input( + getattr(self, "analysis_input_path", ""), + getattr(self, "delete_analysis_input", False), + ) diff --git a/src/anyimage/preferences.py b/src/anyimage/preferences.py index 4b06c03..acdf0c2 100644 --- a/src/anyimage/preferences.py +++ b/src/anyimage/preferences.py @@ -2,9 +2,12 @@ from pathlib import Path import bpy -from .server.model_catalog import DEFAULT_UPSCALE_MODEL_KEY, UPSCALE_MODELS - +from .server.model_catalog import ( + DEFAULT_UPSCALE_MODEL_KEY, + UPSCALE_MODELS, +) +DEFAULT_MOGE2_MODEL_KEY = "MOGE2_VITS_NORMAL" ADDON_PACKAGE = __package__ DEFAULT_STORAGE_ROOT = Path.home() / ".anyimage" @@ -18,9 +21,7 @@ UPSCALE_MODEL_ITEMS = tuple( def addon_preferences(context=None): preference_context = ( - context - if getattr(context, "preferences", None) is not None - else bpy.context + context if getattr(context, "preferences", None) is not None else bpy.context ) addon = preference_context.preferences.addons.get(ADDON_PACKAGE) return getattr(addon, "preferences", None) @@ -39,6 +40,12 @@ def device_items(owner, context): return device_enum_items(owner, context) +def mesh_depth_model_items(owner, context): + from .properties import surface_model_enum_items + + return surface_model_enum_items(owner, context) + + class AnyImagePreferences(bpy.types.AddonPreferences): bl_idname = ADDON_PACKAGE @@ -78,6 +85,29 @@ class AnyImagePreferences(bpy.types.AddonPreferences): step=256, subtype="PIXEL", ) + mesh_depth_model: bpy.props.EnumProperty( + name="Mesh Depth Model", + description="Depth model used by Depth Plane, Depth Balloon and Depth Surface", + items=mesh_depth_model_items, + default=0, + ) + mesh_process_res_factor: bpy.props.FloatProperty( + name="DA3 Resolution Scale", + description="Source image scale used for Depth Anything 3 mesh conversion", + default=100.0, + min=1.0, + max=100.0, + step=100, + precision=0, + subtype="PERCENTAGE", + ) + mesh_resolution_level: bpy.props.IntProperty( + name="MoGe Resolution Level", + description="MoGe-2 inference resolution used for mesh conversion", + default=5, + min=0, + max=9, + ) def draw(self, _context): from .properties import ai_setup_label, ai_status, model_catalog @@ -88,6 +118,14 @@ class AnyImagePreferences(bpy.types.AddonPreferences): layout.prop(self, "device") layout.prop(self, "upscale_model") layout.prop(self, "upscale_max_output_size") + layout.separator(type="LINE") + layout.label(text="Mesh Conversion") + layout.prop(self, "mesh_depth_model") + mesh_depth_model = getattr(self, "mesh_depth_model", DEFAULT_MOGE2_MODEL_KEY) + if mesh_depth_model.startswith("MOGE2_"): + layout.prop(self, "mesh_resolution_level") + else: + layout.prop(self, "mesh_process_res_factor") layout.separator() status = ai_status() layout.label(text=f"Environment: {runtime.environment_directory()}") diff --git a/src/anyimage/properties.py b/src/anyimage/properties.py index 612d17b..d76f7d0 100644 --- a/src/anyimage/properties.py +++ b/src/anyimage/properties.py @@ -14,7 +14,6 @@ from .preferences import ( from .runtime import runtime from .server import model_catalog as shared_model_catalog - BEN2_MODEL_KEY = "BEN2_BASE" REALESRGAN_GENERAL_WDN_MODEL_KEY = shared_model_catalog.DEFAULT_UPSCALE_MODEL_KEY SAFMN_MODEL_KEY = shared_model_catalog.SAFMN_MODEL.key @@ -121,13 +120,9 @@ def required_models_ready(): def ai_status(): environment_ready = runtime.environment_ready() - ready_models = { - model["key"] for model in model_catalog() if model.get("ready") - } + ready_models = {model["key"] for model in model_catalog() if model.get("ready")} missing_models = tuple( - model_key - for model_key in REQUIRED_MODEL_KEYS - if model_key not in ready_models + model_key for model_key in REQUIRED_MODEL_KEYS if model_key not in ready_models ) return { "environment_ready": environment_ready, @@ -201,6 +196,7 @@ def balloon_depth_model_enum_items(_owner, _context): models.sort(key=lambda model: model.get("key") != DEFAULT_MOGE2_MODEL_KEY) return _model_items(models) + def depth_plane_model_enum_items(_owner, _context): models = [ model @@ -329,6 +325,26 @@ def image_edit_refine_selection_property(): ) +def cutout_normal_analysis_property(): + return bpy.props.BoolProperty( + name="Normal", + description="Use MoGe-2 to add a normal texture for the selected shape", + default=False, + ) + +def cutout_pixel_size_property(): + return bpy.props.IntProperty( + name="Pixel Size", + description=( + "Approximate source image pixels per mesh step; smaller values " + "create denser geometry" + ), + default=32, + min=4, + max=32, + ) + + def balloon_depth_analysis_property(): return bpy.props.BoolProperty( name="Depth", @@ -345,6 +361,7 @@ def balloon_depth_model_property(): default=0, ) + def balloon_radial_zoom_property(scale): return bpy.props.FloatVectorProperty( size=2, @@ -355,6 +372,8 @@ def balloon_radial_zoom_property(scale): class AnyImageSettings(bpy.types.PropertyGroup): + cutout_pixel_size: cutout_pixel_size_property() + cutout_normal_analysis: cutout_normal_analysis_property() image_edit_refine_selection: image_edit_refine_selection_property() balloon_mode: balloon_mode_property() balloon_thickness: balloon_thickness_property() diff --git a/src/anyimage/server/app.py b/src/anyimage/server/app.py index ea4c6cf..382c142 100644 --- a/src/anyimage/server/app.py +++ b/src/anyimage/server/app.py @@ -2,16 +2,14 @@ from blendjob import JobServer -from .jobs import generate_depth as _generate_depth -from .jobs import generate_surface as _generate_surface from .jobs.background import run as _remove_background -from .jobs.balloon import run as _analyze_balloon from .jobs.debug_ben2 import run as _debug_ben2 from .jobs.debug_da3 import run as _debug_da3 from .jobs.debug_moge2 import run as _debug_moge2 from .jobs.debug_upscale import run as _debug_upscale from .jobs.frames import run as _extract_frames from .jobs.image_mask import run as _refine_image_mask +from .jobs.mesh import run as _generate_mesh from .jobs.upscale import run as _upscale_image from .model_manager import ModelManager @@ -36,11 +34,6 @@ def download_required_models(context, parameters): return manager.download_missing(context, parameters["models"]) -@server.job("analyze-balloon") -def analyze_balloon(context, parameters): - return _analyze_balloon(context, parameters) - - @server.job("refine-image-mask") def refine_image_mask(context, parameters): return _refine_image_mask(context, parameters) @@ -61,14 +54,9 @@ def extract_frames(context, parameters): return _extract_frames(context, parameters) -@server.job("generate-depth") -def generate_depth(context, parameters): - return _generate_depth(context, parameters) - - -@server.job("generate-surface") -def generate_surface(context, parameters): - return _generate_surface(context, parameters) +@server.job("generate-mesh") +def generate_mesh(context, parameters): + return _generate_mesh(context, parameters) @server.job("debug-da3") diff --git a/src/anyimage/server/geometry/cutout.py b/src/anyimage/server/geometry/cutout.py new file mode 100644 index 0000000..f0d1cd7 --- /dev/null +++ b/src/anyimage/server/geometry/cutout.py @@ -0,0 +1,211 @@ +def _choose_boundary_edge(start, direction, candidates): + right = (-direction[1], direction[0]) + left = (direction[1], -direction[0]) + back = (-direction[0], -direction[1]) + preference = {right: 0, direction: 1, left: 2, back: 3} + return min( + candidates, + key=lambda end: preference.get( + (end[0] - start[0], end[1] - start[1]), + 4, + ), + ) + + +def _simplify_axis_aligned_contour(points): + simplified = [] + for index, current in enumerate(points): + previous = points[index - 1] + following = points[(index + 1) % len(points)] + first = (current[0] - previous[0], current[1] - previous[1]) + second = (following[0] - current[0], following[1] - current[1]) + if first[0] * second[1] - first[1] * second[0] != 0: + simplified.append(current) + return simplified + + +def trace_mask_contours(mask): + import numpy as np + + mask = np.asarray(mask, dtype=bool) + if mask.ndim != 2: + raise ValueError("Cutout Alpha must be two-dimensional") + height, width = mask.shape + edges = {} + + def add_edge(start, end): + edges.setdefault(start, []).append(end) + + top = mask.copy() + top[1:] &= ~mask[:-1] + for y, x in np.argwhere(top): + add_edge((int(x), int(y)), (int(x) + 1, int(y))) + + right = mask.copy() + right[:, :-1] &= ~mask[:, 1:] + for y, x in np.argwhere(right): + add_edge((int(x) + 1, int(y)), (int(x) + 1, int(y) + 1)) + + bottom = mask.copy() + bottom[:-1] &= ~mask[1:] + for y, x in np.argwhere(bottom): + add_edge((int(x) + 1, int(y) + 1), (int(x), int(y) + 1)) + + left = mask.copy() + left[:, 1:] &= ~mask[:, :-1] + for y, x in np.argwhere(left): + add_edge((int(x), int(y) + 1), (int(x), int(y))) + if not edges: + raise ValueError("The Cutout region has no usable boundary") + + contours = [] + edge_count = sum(len(values) for values in edges.values()) + while edges: + start = min(edges, key=lambda point: (point[1], point[0])) + current = start + direction = (1, 0) + contour = [] + for _index in range(edge_count + 1): + contour.append(current) + candidates = edges.get(current) + if not candidates: + raise ValueError("The Cutout boundary is open") + following = _choose_boundary_edge(current, direction, candidates) + candidates.remove(following) + if not candidates: + del edges[current] + direction = (following[0] - current[0], following[1] - current[1]) + current = following + if current == start: + break + else: + raise ValueError("The Cutout boundary could not be closed") + contour = _simplify_axis_aligned_contour(contour) + if len(contour) >= 3: + contours.append(contour) + if not contours: + raise ValueError("The Cutout region is too small") + return contours + + +def _signed_contour_area(contour): + return 0.5 * sum( + start[0] * end[1] - end[0] * start[1] + for start, end in zip(contour, contour[1:] + contour[:1]) + ) + + +def _point_inside_contour(point, contour): + inside = False + previous = contour[-1] + for current in contour: + if (current[1] > point[1]) != (previous[1] > point[1]): + crossing_x = current[0] + ( + (point[1] - current[1]) + * (previous[0] - current[0]) + / (previous[1] - current[1]) + ) + inside ^= point[0] < crossing_x + previous = current + return inside + + +def mask_regions(mask): + contours = trace_mask_contours(mask) + outer = [contour for contour in contours if _signed_contour_area(contour) > 0.0] + holes = [contour for contour in contours if _signed_contour_area(contour) < 0.0] + if not outer: + raise ValueError("The Cutout region has no outer boundary") + regions = [[contour] for contour in outer] + outer_areas = [abs(_signed_contour_area(contour)) for contour in outer] + for hole in holes: + start = hole[0] + end = hole[1] + edge_x = end[0] - start[0] + edge_y = end[1] - start[1] + length = max((edge_x * edge_x + edge_y * edge_y) ** 0.5, 1e-12) + sample = ( + (start[0] + end[0]) * 0.5 - edge_y * 0.25 / length, + (start[1] + end[1]) * 0.5 + edge_x * 0.25 / length, + ) + owners = [ + index + for index, contour in enumerate(outer) + if _point_inside_contour(sample, contour) + ] + if not owners: + raise ValueError("The Cutout hole has no enclosing region") + owner = min(owners, key=outer_areas.__getitem__) + regions[owner].append(hole) + return regions + + +def _resize_alpha(values, size): + import numpy as np + + target_width, target_height = (int(value) for value in size) + source_height, source_width = values.shape + if (target_width, target_height) == (source_width, source_height): + return values + x = (np.arange(target_width, dtype=np.float32) + 0.5) * ( + source_width / target_width + ) - 0.5 + y = (np.arange(target_height, dtype=np.float32) + 0.5) * ( + source_height / target_height + ) - 0.5 + x0_unclipped = np.floor(x).astype(np.int32) + y0_unclipped = np.floor(y).astype(np.int32) + x_weight = x - x0_unclipped + y_weight = y - y0_unclipped + x0 = np.clip(x0_unclipped, 0, source_width - 1) + y0 = np.clip(y0_unclipped, 0, source_height - 1) + x1 = np.clip(x0_unclipped + 1, 0, source_width - 1) + y1 = np.clip(y0_unclipped + 1, 0, source_height - 1) + horizontal = ( + values[:, x0] * (1.0 - x_weight)[None, :] + + values[:, x1] * x_weight[None, :] + ) + return ( + horizontal[y0] * (1.0 - y_weight)[:, None] + + horizontal[y1] * y_weight[:, None] + ) + + +def alpha_regions(alpha, max_resolution=512, threshold=0.5): + import numpy as np + + values = np.asarray(alpha) + if values.ndim != 2: + raise ValueError("Cutout Alpha must be two-dimensional") + if np.issubdtype(values.dtype, np.floating): + values = np.clip(values, 0.0, 1.0) * 255.0 + else: + values = np.clip(values, 0, 255).astype(np.float32) + source_size = (values.shape[1], values.shape[0]) + scale = min(1.0, float(max_resolution) / max(source_size)) + sampled_size = tuple(max(1, round(value * scale)) for value in source_size) + if sampled_size != source_size: + values = _resize_alpha(values, sampled_size) + mask = values > round(255.0 * threshold) + regions = mask_regions(mask) + scale_x = source_size[0] / sampled_size[0] + scale_y = source_size[1] / sampled_size[1] + return ( + [ + [ + [[float(x) * scale_x, float(y) * scale_y] for x, y in contour] + for contour in region + ] + for region in regions + ], + source_size, + ) + + +def image_alpha_regions(path, max_resolution=512, threshold=0.5): + import numpy as np + from PIL import Image + + with Image.open(path) as source: + alpha = np.asarray(source.convert("RGBA").getchannel("A"), dtype=np.uint8) + return alpha_regions(alpha, max_resolution=max_resolution, threshold=threshold) diff --git a/src/anyimage/server/jobs/__init__.py b/src/anyimage/server/jobs/__init__.py index 9a97649..c0f8ffb 100644 --- a/src/anyimage/server/jobs/__init__.py +++ b/src/anyimage/server/jobs/__init__.py @@ -1,5 +1 @@ -from .depth import run as generate_depth -from .surface import run as generate_surface - - -__all__ = ("generate_depth", "generate_surface") +"""Job implementations registered by :mod:`anyimage.server.app`.""" diff --git a/src/anyimage/server/jobs/balloon.py b/src/anyimage/server/jobs/balloon.py index a164453..6019fe8 100644 --- a/src/anyimage/server/jobs/balloon.py +++ b/src/anyimage/server/jobs/balloon.py @@ -5,15 +5,18 @@ from pathlib import Path from ..models import moge2 from ..models.ben2 import infer_foreground +from .image_mask import polygon_selection_mask ALPHA_THRESHOLD = 0.5 -def create_lasso_input(input_path, polygon, output_path): +def create_lasso_input( + input_path, polygon, output_path, fill_rule="EVEN_ODD" +): """Crop a lasso and clear every RGBA channel outside its polygon.""" import numpy as np - from PIL import Image, ImageDraw + from PIL import Image points = np.asarray(polygon, dtype=np.float64) if points.ndim != 2 or points.shape[0] < 3 or points.shape[1] != 2: @@ -33,12 +36,7 @@ def create_lasso_input(input_path, polygon, output_path): relative = points - np.asarray((left, top), dtype=np.float64) cropped = source.crop((left, top, right, bottom)) - selection = Image.new("L", cropped.size, 0) - ImageDraw.Draw(selection).polygon( - [tuple(point) for point in relative], - fill=255, - ) - selection_mask = np.asarray(selection, dtype=np.uint8) > 0 + selection_mask = polygon_selection_mask(cropped.size, relative, fill_rule) pixels = np.asarray(cropped, dtype=np.uint8).copy() pixels[~selection_mask] = 0 masked = Image.fromarray(pixels, mode="RGBA") @@ -285,7 +283,7 @@ def write_depth_analysis( return Path(output_path) -def write_balloon_normal_texture( +def write_camera_normal_texture( normal, valid, origin, @@ -293,7 +291,7 @@ def write_balloon_normal_texture( source_size, output_path, ): - """Write MoGe camera normals as a full-image tangent-space texture.""" + """Encode MoGe camera-axis normals into a full-image RGB texture.""" import numpy as np from PIL import Image @@ -412,7 +410,7 @@ def run(context, parameters): import numpy as np with np.load(depth_path, allow_pickle=False) as data: - normal_path = write_balloon_normal_texture( + normal_path = write_camera_normal_texture( prediction.normal[0], data["valid"], origin, diff --git a/src/anyimage/server/jobs/common.py b/src/anyimage/server/jobs/common.py index b658c1d..6a74f5b 100644 --- a/src/anyimage/server/jobs/common.py +++ b/src/anyimage/server/jobs/common.py @@ -24,3 +24,14 @@ def input_frames(context, parameters): finally: if temporary_directory is not None: shutil.rmtree(temporary_directory, ignore_errors=True) + + +@contextmanager +def analysis_frames(context, parameters, fallback_images): + analysis_input = parameters.get("analysis_input") + if not analysis_input or analysis_input == parameters["input"]: + yield fallback_images + return + analysis_parameters = dict(parameters, input=analysis_input) + with input_frames(context, analysis_parameters) as images: + yield images diff --git a/src/anyimage/server/jobs/depth.py b/src/anyimage/server/jobs/depth.py index 547c18f..ac3c770 100644 --- a/src/anyimage/server/jobs/depth.py +++ b/src/anyimage/server/jobs/depth.py @@ -1,7 +1,7 @@ from ..media import copy_color_images from ..models import da3, geometry_adapter, moge2 from ..outputs.depth import write_da3_depth, write_moge_depth -from .common import frame_progress_message, input_frames +from .common import analysis_frames, frame_progress_message, input_frames from .foreground import write_optional @@ -11,31 +11,32 @@ def run(context, parameters): model_manager = context.resource("model_manager") adapter = geometry_adapter(parameters["model"]) with input_frames(context, parameters) as images: - if adapter is da3: - prediction = adapter.infer( - model_manager, - parameters, - images, - progress_callback=lambda index, total: context.progress( - 0.15 + 0.65 * index / total, - frame_progress_message("Estimating DA3 depth", index, total), - ), - cancel_check=context.check_cancelled, - ) - write_da3_depth(prediction, context.directory / "results.npz") - else: - prediction = adapter.infer( - model_manager, - parameters, - images, - include_points=False, - progress_callback=lambda index, total: context.progress( - 0.15 + 0.65 * index / total, - frame_progress_message("Estimating MoGe-2 depth", index, total), - ), - cancel_check=context.check_cancelled, - ) - write_moge_depth(context.directory, prediction) + with analysis_frames(context, parameters, images) as model_images: + if adapter is da3: + prediction = adapter.infer( + model_manager, + parameters, + model_images, + progress_callback=lambda index, total: context.progress( + 0.15 + 0.65 * index / total, + frame_progress_message("Estimating DA3 depth", index, total), + ), + cancel_check=context.check_cancelled, + ) + write_da3_depth(prediction, context.directory / "results.npz") + else: + prediction = adapter.infer( + model_manager, + parameters, + model_images, + include_points=False, + progress_callback=lambda index, total: context.progress( + 0.15 + 0.65 * index / total, + frame_progress_message("Estimating MoGe-2 depth", index, total), + ), + cancel_check=context.check_cancelled, + ) + write_moge_depth(context.directory, prediction) color_paths = write_optional( context, parameters, diff --git a/src/anyimage/server/jobs/image_mask.py b/src/anyimage/server/jobs/image_mask.py index 946c5e0..750e619 100644 --- a/src/anyimage/server/jobs/image_mask.py +++ b/src/anyimage/server/jobs/image_mask.py @@ -22,10 +22,68 @@ def polygon_bounds(polygon): return left, top, right, bottom -def create_selection_canvas(input_path, polygon): +def polygon_selection_mask(image_size, polygon, fill_rule="EVEN_ODD"): + """Rasterize a polygon with an explicit fill rule.""" + import numpy as np + + width, height = (int(value) for value in image_size) + points = np.asarray(polygon, dtype=np.float64) + if width <= 0 or height <= 0: + raise ValueError("The selection canvas has no readable dimensions") + if points.ndim != 2 or points.shape[0] < 3 or points.shape[1] != 2: + raise ValueError("The selection does not define a valid image region") + if not np.isfinite(points).all(): + raise ValueError("The selection contains invalid image coordinates") + if fill_rule not in {"EVEN_ODD", "NONZERO"}: + raise ValueError(f"Unknown polygon fill rule: {fill_rule}") + + scanlines = np.arange(height, dtype=np.float64) + 0.5 + crossings = [[] for _index in range(height)] + for start, end in zip(points, np.roll(points, -1, axis=0)): + selected = np.flatnonzero( + (start[1] > scanlines) != (end[1] > scanlines) + ) + if not len(selected): + continue + crossing_x = start[0] + ( + (scanlines[selected] - start[1]) + * (end[0] - start[0]) + / (end[1] - start[1]) + ) + winding = 1 if end[1] > start[1] else -1 + for row, value in zip(selected, crossing_x): + crossings[int(row)].append((float(value), winding)) + + mask = np.zeros((height, width), dtype=bool) + for row, values in enumerate(crossings): + values.sort() + state = 0 + index = 0 + while index < len(values): + crossing_x = values[index][0] + winding = 0 + count = 0 + while index < len(values) and values[index][0] == crossing_x: + winding += values[index][1] + count += 1 + index += 1 + state = ( + state + winding + if fill_rule == "NONZERO" + else state ^ (count % 2) + ) + if index >= len(values) or state == 0: + continue + following_x = values[index][0] + first = max(0, int(math.ceil(crossing_x - 0.5))) + last = min(width, int(math.floor(following_x - 0.5)) + 1) + mask[row, first:last] = True + return mask + +def create_selection_canvas(input_path, polygon, fill_rule="EVEN_ODD"): """Create an unbounded selection canvas with transparent black outside.""" import numpy as np - from PIL import Image, ImageDraw + from PIL import Image with Image.open(input_path) as opened_image: source = opened_image.convert("RGBA") @@ -35,9 +93,7 @@ def create_selection_canvas(input_path, polygon): canvas = Image.new("RGBA", (right - left, bottom - top), (0, 0, 0, 0)) canvas.paste(source, (-left, -top)) relative = [(float(x) - left, float(y) - top) for x, y in polygon] - selection = Image.new("L", canvas.size, 0) - ImageDraw.Draw(selection).polygon(relative, fill=255) - selection_mask = np.asarray(selection, dtype=np.uint8) > 0 + selection_mask = polygon_selection_mask(canvas.size, relative, fill_rule) pixels = np.asarray(canvas, dtype=np.uint8).copy() pixels[~selection_mask] = 0 return Image.fromarray(pixels, mode="RGBA"), selection_mask, ( diff --git a/src/anyimage/server/jobs/mesh.py b/src/anyimage/server/jobs/mesh.py new file mode 100644 index 0000000..d39a4ee --- /dev/null +++ b/src/anyimage/server/jobs/mesh.py @@ -0,0 +1,303 @@ +from ..media import copy_color_images +from ..models import geometry_adapter, moge2 +from .background import remove_image_background +from .balloon import ( + write_camera_normal_texture, + write_depth_analysis, +) +from .common import analysis_frames, input_frames +from .image_mask import ( + apply_selection_mask, + create_selection_canvas, + polygon_selection_mask, +) + + +def _write_field(frame, output_path): + import numpy as np + + depth = np.asarray(frame.depth, dtype=np.float32) + points = np.asarray(frame.points, dtype=np.float32) + intrinsics = np.asarray(frame.intrinsics, dtype=np.float32) + valid = np.asarray(frame.valid_mask, dtype=bool) + if depth.ndim != 2 or points.shape != (*depth.shape, 3): + raise ValueError("The depth model returned an invalid point field") + valid &= np.isfinite(depth) & (depth > 0.0) + valid &= np.isfinite(points).all(axis=-1) + if not valid.any(): + raise ValueError("The depth model returned no valid geometry") + reference_values = depth[valid] + if frame.model_mask is not None: + model_mask = np.asarray(frame.model_mask, dtype=bool) + masked = depth[valid & model_mask] + if len(masked): + reference_values = masked + np.savez_compressed( + output_path, + points=points, + depth=depth, + intrinsics=intrinsics, + valid=valid, + image_size=np.asarray((depth.shape[1], depth.shape[0]), dtype=np.int32), + reference_depth=np.asarray(np.median(reference_values), dtype=np.float32), + ) + + +def _write_masked_color(input_path, polygon, output_path, invert=False): + import numpy as np + from PIL import Image, ImageChops, ImageFilter, ImageOps + + with Image.open(input_path) as opened: + source = opened.convert("RGBA") + selection = Image.fromarray( + polygon_selection_mask(source.size, polygon, "NONZERO").astype(np.uint8) * 255, + mode="L", + ).filter(ImageFilter.GaussianBlur(radius=0.75)) + if invert: + selection = ImageOps.invert(selection) + alpha = ImageChops.multiply(source.getchannel("A"), selection) + pixels = np.asarray(source, dtype=np.uint8).copy() + pixels[:, :, 3] = np.asarray(alpha, dtype=np.uint8) + pixels[pixels[:, :, 3] == 0, :3] = 0 + Image.fromarray(pixels, mode="RGBA").save( + output_path, + format="PNG", + compress_level=1, + ) + return output_path + + +def _write_refined_color(context, parameters, input_path, polygon, output_path): + from PIL import Image, ImageChops + + canvas, selection_mask, bounds = create_selection_canvas( + input_path, polygon, "NONZERO" + ) + selection_path = context.directory / "selection-input.png" + canvas.save(selection_path, format="PNG", compress_level=1) + model_manager = context.resource("model_manager") + foreground, _load_ms, _inference_ms = remove_image_background( + selection_path, + model_manager.directory(parameters["refine_model"]), + parameters["device"], + model_manager, + context.check_cancelled, + ) + refined = apply_selection_mask(foreground, selection_mask) + with Image.open(input_path) as opened: + source = opened.convert("RGBA") + if parameters.get("invert"): + mask = Image.new("L", source.size, 0) + mask.paste(refined.getchannel("A"), bounds[:2]) + source.putalpha( + ImageChops.multiply(source.getchannel("A"), ImageChops.invert(mask)) + ) + output = source + else: + output = Image.new("RGBA", source.size, (0, 0, 0, 0)) + output.paste(refined, bounds[:2]) + output.save(output_path, format="PNG", compress_level=1) + return output_path + + +def _write_camera_normal_from_prediction(prediction, color_path, output_path): + import numpy as np + from PIL import Image + + normal = np.asarray(prediction.normal[0], dtype=np.float32) + with Image.open(color_path) as source: + alpha = source.convert("RGBA").getchannel("A") + source_size = source.size + valid = ( + np.asarray( + alpha.resize((normal.shape[1], normal.shape[0]), Image.Resampling.BILINEAR) + ) + > 127 + ) + return write_camera_normal_texture( + normal, + valid, + (0, 0), + source_size, + source_size, + output_path, + ) + + +def _write_camera_normal(context, parameters, color_path): + normal_parameters = dict(parameters) + normal_parameters.update( + model=parameters["normal_model"], + resolution_level=parameters["normal_resolution_level"], + ) + prediction = moge2.infer( + context.resource("model_manager"), + normal_parameters, + [color_path], + include_points=False, + cancel_check=context.check_cancelled, + ) + return _write_camera_normal_from_prediction( + prediction, + color_path, + context.directory / "normal.png", + ) + + +def _write_balloon_depth( + context, + parameters, + input_path, + polygon, + selection_size=None, + include_normal=False, +): + from PIL import Image + + with Image.open(input_path) as source: + source_size = source.size + selection_size = selection_size or source_size + scale_x = source_size[0] / selection_size[0] + scale_y = source_size[1] / selection_size[1] + analysis_polygon = tuple( + (point[0] * scale_x, point[1] * scale_y) for point in polygon + ) + selection_mask = polygon_selection_mask( + source_size, + analysis_polygon, + "NONZERO", + ) + prediction = moge2.infer( + context.resource("model_manager"), + parameters, + [input_path], + include_points=False, + cancel_check=context.check_cancelled, + ) + depth_path = write_depth_analysis( + prediction, + selection_mask, + (0, 0), + source_size, + context.directory / "depth.npz", + ) + normal_path = None + if include_normal: + normal_path = _write_camera_normal_from_prediction( + prediction, + input_path, + context.directory / "normal.png", + ) + return depth_path, normal_path + + +def _write_surface_depth( + context, parameters, color_path, include_normal=False +): + model_manager = context.resource("model_manager") + adapter = geometry_adapter(parameters["model"]) + if adapter is moge2: + prediction = adapter.infer( + model_manager, + parameters, + [color_path], + include_points=True, + cancel_check=context.check_cancelled, + ) + else: + prediction = adapter.infer( + model_manager, + parameters, + [color_path], + cancel_check=context.check_cancelled, + ) + context.check_cancelled() + field_path = context.directory / "field.npz" + _write_field(adapter.surface_frame(prediction), field_path) + normal_path = None + if include_normal and adapter is moge2: + normal_path = _write_camera_normal_from_prediction( + prediction, color_path, context.directory / "normal.png" + ) + return field_path, normal_path + + +def run(context, parameters): + context.progress(0.02, "Preparing mesh input") + context.directory.mkdir(parents=True, exist_ok=True) + with input_frames(context, parameters) as images: + if len(images) != 1: + raise ValueError("Empty Image to Mesh currently supports one image") + shape = parameters["shape"] + polygon = parameters.get("polygon") + if polygon: + if parameters.get("refine_selection"): + color_path = _write_refined_color( + context, + parameters, + images[0], + polygon, + context.directory / "image.0001.png", + ) + else: + color_path = _write_masked_color( + images[0], + polygon, + context.directory / "image.0001.png", + parameters.get("invert", False), + ) + else: + copy_color_images(images, context.directory) + color_path = min(context.directory.glob("image.0001.*")) + result = {} + normal_analysis = bool(parameters.get("normal_analysis")) + with analysis_frames(context, parameters, images) as model_images: + if len(model_images) != 1: + raise ValueError("Empty Image to Mesh currently supports one image") + analysis_path = model_images[0] + + if shape in {"CUTOUT", "BALLOON"} and normal_analysis: + context.progress(0.4, "Estimating surface normal") + result["normal"] = _write_camera_normal( + context, parameters, analysis_path + ).name + elif shape == "DEPTH_BALLOON": + from PIL import Image + + with Image.open(images[0]) as selection_source: + selection_size = selection_source.size + context.progress(0.4, "Estimating Balloon depth") + depth_path, normal_path = _write_balloon_depth( + context, + parameters, + analysis_path, + polygon, + selection_size, + include_normal=normal_analysis, + ) + result["depth"] = depth_path.name + if normal_path is not None: + result["normal"] = normal_path.name + elif shape in {"DEPTH_PLANE", "DEPTH_SURFACE"}: + context.progress(0.4, "Estimating image depth") + field_path, normal_path = _write_surface_depth( + context, + parameters, + analysis_path, + include_normal=normal_analysis, + ) + result["field"] = field_path.name + if ( + normal_analysis + and normal_path is None + and shape == "DEPTH_SURFACE" + ): + normal_path = _write_camera_normal( + context, parameters, analysis_path + ) + if normal_path is not None: + result["normal"] = normal_path.name + + context.check_cancelled() + context.progress(1.0, "Mesh data is ready") + return result diff --git a/src/anyimage/server/jobs/surface.py b/src/anyimage/server/jobs/surface.py index 0da8a4b..7035e21 100644 --- a/src/anyimage/server/jobs/surface.py +++ b/src/anyimage/server/jobs/surface.py @@ -2,7 +2,7 @@ from ..geometry import build_surface, foreground_mesh_mask from ..media import copy_color_images from ..models import geometry_adapter, moge2 from ..outputs.surface import write_normal_texture, write_surface_archive -from .common import frame_progress_message, input_frames +from .common import analysis_frames, frame_progress_message, input_frames from .foreground import write_optional def run(context, parameters): @@ -14,29 +14,32 @@ def run(context, parameters): if len(images) != 1: raise ValueError("Convert to Surface currently supports one image") context.progress(0.12, "Estimating surface geometry") - if adapter is moge2: - prediction = adapter.infer( - model_manager, - parameters, - images, - include_points=True, - progress_callback=lambda index, total: context.progress( - 0.12 + 0.58 * index / total, - frame_progress_message("Estimating MoGe-2 geometry", index, total), - ), - cancel_check=context.check_cancelled, - ) - else: - prediction = adapter.infer( - model_manager, - parameters, - images, - progress_callback=lambda index, total: context.progress( - 0.12 + 0.58 * index / total, - frame_progress_message("Estimating DA3 geometry", index, total), - ), - cancel_check=context.check_cancelled, - ) + with analysis_frames(context, parameters, images) as model_images: + if len(model_images) != 1: + raise ValueError("Convert to Surface currently supports one image") + if adapter is moge2: + prediction = adapter.infer( + model_manager, + parameters, + model_images, + include_points=True, + progress_callback=lambda index, total: context.progress( + 0.12 + 0.58 * index / total, + frame_progress_message("Estimating MoGe-2 geometry", index, total), + ), + cancel_check=context.check_cancelled, + ) + else: + prediction = adapter.infer( + model_manager, + parameters, + model_images, + progress_callback=lambda index, total: context.progress( + 0.12 + 0.58 * index / total, + frame_progress_message("Estimating DA3 geometry", index, total), + ), + cancel_check=context.check_cancelled, + ) context.check_cancelled() frame = adapter.surface_frame(prediction) color_paths = write_optional( diff --git a/src/anyimage/server/model_manager.py b/src/anyimage/server/model_manager.py index 1e994c7..3463bf1 100644 --- a/src/anyimage/server/model_manager.py +++ b/src/anyimage/server/model_manager.py @@ -25,7 +25,7 @@ class ModelSessionCache: key = (str(Path(model_dir).resolve()), requested_device, "onnx") if self.ben2_model is not None and self.ben2_key == key: return self.ben2_model, 0.0 - self.close() + self.release_ben2() return self._load_ben2(model_dir, requested_device, key) def _load_ben2(self, model_dir, requested_device, key): @@ -43,7 +43,7 @@ class ModelSessionCache: key = (str(Path(model_dir).resolve()), requested_device, "onnx") if self.da3_model is not None and self.da3_key == key: return self.da3_model - self.close() + self.release_geometry() return self._load_da3(model_dir, requested_device, key) def _load_da3(self, model_dir, requested_device, key): @@ -60,7 +60,7 @@ class ModelSessionCache: key = (str(Path(model_dir).resolve()), requested_device, "onnx") if self.moge_model is not None and self.moge_key == key: return self.moge_model - self.close() + self.release_geometry() model = create_session(model_dir, requested_device) self.moge_model = model self.moge_key = key @@ -72,7 +72,7 @@ class ModelSessionCache: key = (str(Path(model_dir).resolve()), requested_device, "onnx") if self.upscale_model is not None and self.upscale_key == key: return self.upscale_model - self.close() + self.release_upscale() model = create_session(model_dir, requested_device) self.upscale_model = model self.upscale_key = key @@ -83,12 +83,9 @@ class ModelSessionCache: self.ben2_key = None self._release_runtime_cache() - def release_da3(self): + def release_geometry(self): self.da3_model = None self.da3_key = None - self._release_runtime_cache() - - def release_moge(self): self.moge_model = None self.moge_key = None self._release_runtime_cache() @@ -110,15 +107,11 @@ class ModelSessionCache: self._release_runtime_cache() def snapshot(self): + geometry_key = self.da3_key or self.moge_key return { "ben2": Path(self.ben2_key[0]).name if self.ben2_key else "", - "da3": Path(self.da3_key[0]).name if self.da3_key else "", - "moge2": Path(self.moge_key[0]).name if self.moge_key else "", - "upscale": ( - Path(self.upscale_key[0]).name - if self.upscale_key - else "" - ), + "geometry": Path(geometry_key[0]).name if geometry_key else "", + "upscale": (Path(self.upscale_key[0]).name if self.upscale_key else ""), } def _release_runtime_cache(self): @@ -176,6 +169,7 @@ class ModelManager: total = len(missing) for index, model_key in enumerate(missing): + def report(value, message, *, offset=index): context.progress((offset + value) / total, message) diff --git a/tests/test_addon.py b/tests/test_addon.py index a99b682..0489e24 100644 --- a/tests/test_addon.py +++ b/tests/test_addon.py @@ -8,7 +8,6 @@ from pathlib import Path from types import ModuleType, SimpleNamespace from unittest.mock import patch - PROJECT_ROOT = Path(__file__).resolve().parents[1] SOURCE_ROOT = PROJECT_ROOT / "src" @@ -94,27 +93,18 @@ class AddonTest(unittest.TestCase): cls.depth_build = importlib.import_module( "anyimage.operators.depth_plane.build" ) - cls.mesh_plane = importlib.import_module( - "anyimage.operators.mesh_plane" - ) + cls.mesh_plane = importlib.import_module("anyimage.operators.mesh_plane") + cls.mesh = importlib.import_module("anyimage.operators.mesh") cls.image_background = importlib.import_module("anyimage.operators.background") cls.image_upscale = importlib.import_module("anyimage.operators.upscale") cls.image_mesh = importlib.import_module("anyimage.operators.surface") cls.image_edit_common = importlib.import_module( "anyimage.operators.image_edit.common" ) - cls.image_mask = importlib.import_module( - "anyimage.operators.image_edit.mask" - ) - cls.image_box = importlib.import_module( - "anyimage.operators.image_edit.box" - ) - cls.image_lasso = importlib.import_module( - "anyimage.operators.image_edit.lasso" - ) - cls.image_line = importlib.import_module( - "anyimage.operators.image_edit.line" - ) + cls.image_mask = importlib.import_module("anyimage.operators.image_edit.mask") + cls.image_box = importlib.import_module("anyimage.operators.image_edit.box") + cls.image_lasso = importlib.import_module("anyimage.operators.image_edit.lasso") + cls.image_line = importlib.import_module("anyimage.operators.image_edit.line") cls.image_polyline = importlib.import_module( "anyimage.operators.image_edit.polyline" ) @@ -251,8 +241,8 @@ class AddonTest(unittest.TestCase): self.assertFalse(name.startswith("anyimage_"), (operator, name)) def test_concrete_job_operator_is_the_blender_operator(self): - operator = self.addon.operators.GenerateMoge2Depth - self.assertEqual(operator.bl_idname, "anyimage.generate_moge2_depth") + operator = self.addon.operators.GenerateMesh + self.assertEqual(operator.bl_idname, "anyimage.generate_mesh") self.assertIn(self.job_runtime.JobOperatorBase, operator.__mro__) self.assertIn(self.fake_bpy.types.Operator, operator.__mro__) @@ -265,15 +255,15 @@ class AddonTest(unittest.TestCase): device="CPU", video_frames=3, ) - context = SimpleNamespace( - scene=SimpleNamespace(anyimage_settings=settings) - ) + context = SimpleNamespace(scene=SimpleNamespace(anyimage_settings=settings)) for model_key in self.addon.properties.shared_model_catalog.UPSCALE_MODELS: operator = self.debug.DebugUpscale() operator.model = model_key - with self.subTest(model=model_key), patch.object( - self.debug, "require_environment" - ), patch.object(self.debug, "require_model") as require_model: + with ( + self.subTest(model=model_key), + patch.object(self.debug, "require_environment"), + patch.object(self.debug, "require_model") as require_model, + ): request = operator.request(context) require_model.assert_called_once_with(model_key) self.assertEqual(request["model"], model_key) @@ -618,18 +608,23 @@ class AddonTest(unittest.TestCase): ): status = properties.ai_status() self.assertFalse(status["ready"]) - self.assertEqual(properties.ai_setup_label(status), "Set Up AI Environment…") + self.assertEqual( + properties.ai_setup_label(status), "Set Up AI Environment…" + ) def test_setup_ai_environment_selects_the_needed_install_step(self): calls = [] ops = SimpleNamespace( anyimage=SimpleNamespace( - install_environment=lambda *_args: calls.append("official") - or {"FINISHED"}, - install_environment_mirror=lambda *_args: calls.append("mirror") - or {"FINISHED"}, - download_required_models=lambda *_args: calls.append("models") - or {"FINISHED"}, + install_environment=lambda *_args: ( + calls.append("official") or {"FINISHED"} + ), + install_environment_mirror=lambda *_args: ( + calls.append("mirror") or {"FINISHED"} + ), + download_required_models=lambda *_args: ( + calls.append("models") or {"FINISHED"} + ), ) ) operator = self.model_operators.SetupAIEnvironment() @@ -696,12 +691,20 @@ class AddonTest(unittest.TestCase): preferences = preference_module.AnyImagePreferences() preferences.layout = Layout() - catalog = tuple({ - "key": key, "label": key, "family": "upscale", - "enabled": True, "ready": False, - } for key in ( - "REALESRGAN_GENERAL_WDN_X4V3", "SAFMN_REAL_X4", "REALESRGAN_X4PLUS", - )) + catalog = tuple( + { + "key": key, + "label": key, + "family": "upscale", + "enabled": True, + "ready": False, + } + for key in ( + "REALESRGAN_GENERAL_WDN_X4V3", + "SAFMN_REAL_X4", + "REALESRGAN_X4PLUS", + ) + ) with ( patch.object( self.addon.properties, @@ -728,19 +731,21 @@ class AddonTest(unittest.TestCase): self.assertIn("upscale_model", properties) self.assertIn("upscale_max_output_size", properties) self.assertEqual( - preference_module.AnyImagePreferences.__annotations__[ - "upscale_model" - ]["name"], + preference_module.AnyImagePreferences.__annotations__["upscale_model"][ + "name" + ], "Default Upscale Model", ) - def test_mesh_plane_operator_is_registered_with_material_options(self): - operator = self.mesh_plane.ConvertToMeshPlane + def test_semantic_mesh_operators_are_registered_without_dialog_options(self): + operator = self.mesh.ConvertToPlane self.assertIn(operator, self.addon.CLASSES) - self.assertEqual(operator.bl_idname, "anyimage.convert_to_mesh_plane") - self.assertFalse(operator.__annotations__["shadeless"]["default"]) - self.assertFalse(operator.__annotations__["keep_original"]["default"]) - self.assertIn("invoke", operator.__dict__) + self.assertIn(self.mesh.ConvertToDepthPlane, self.addon.CLASSES) + self.assertIn(self.mesh.ConvertCutoutShape, self.addon.CLASSES) + self.assertEqual(operator.bl_idname, "anyimage.convert_to_plane") + self.assertNotIn("shadeless", operator.__annotations__) + self.assertNotIn("keep_original", operator.__annotations__) + self.assertNotIn("invoke", operator.__dict__) self.assertNotIn("draw", operator.__dict__) self.assertIn("REGISTER", operator.bl_options) self.assertIn("UNDO", operator.bl_options) @@ -758,9 +763,246 @@ class AddonTest(unittest.TestCase): ) plain_mesh = SimpleNamespace(type="MESH", modifiers=[]) self.assertTrue(operator.poll(SimpleNamespace(object=image_empty))) - self.assertTrue(operator.poll(SimpleNamespace(object=mesh_plane))) + self.assertFalse(operator.poll(SimpleNamespace(object=mesh_plane))) self.assertFalse(operator.poll(SimpleNamespace(object=plain_mesh))) + def test_plane_depth_off_accepts_animated_image_without_server(self): + operator = SimpleNamespace( + depth_mode="OFF", + remove_background=False, + shadeless=False, + profile="SURFACE", + mesh_detail=64, + thickness=0.1, + inflation=0.1, + algorithm="POISSON", + keep_original=False, + ) + source = SimpleNamespace(data=SimpleNamespace(name="Sequence")) + context = SimpleNamespace(object=source, scene=object()) + + with ( + patch.object(self.mesh, "video_total_frames", return_value=24), + patch.object( + self.mesh, + "image_user_settings", + return_value=(1, 0, 24, True), + ), + patch.object(self.mesh, "create_material", return_value="Material"), + patch.object(self.mesh, "_copy_color_image", return_value="Color"), + patch.object(self.mesh, "_build_mesh_object") as build_mesh, + patch.object( + self.mesh, + "invoke_ai_setup_if_needed", + side_effect=AssertionError("local conversion must not start AI setup"), + ), + ): + result = self.mesh.ConvertToPlane.execute(operator, context) + + self.assertEqual(result, {"FINISHED"}) + build_mesh.assert_called_once() + + def test_cutout_and_balloon_without_refine_build_locally(self): + source = SimpleNamespace(select_set=lambda _selected: None) + context = SimpleNamespace( + view_layer=SimpleNamespace(objects=SimpleNamespace(active=None)) + ) + + for shape in (self.mesh.CUTOUT, self.mesh.BALLOON): + with self.subTest(shape=shape): + operator = SimpleNamespace( + source_object_name="Source", + selection_data="[[0,0],[10,0],[0,10]]", + shape=shape, + mesh_detail=64, + refine_selection=False, + normal_analysis=False, + invert=True, + report=lambda *_args: None, + ) + with ( + patch.object( + self.mesh.bpy.data.objects, + "get", + return_value=source, + ), + patch.object(self.mesh, "is_image_empty", return_value=True), + patch.object(self.mesh, "_build_local_cutout") as build_local, + patch.object( + self.mesh, + "_start_job", + side_effect=AssertionError("local Cutout must not start a Job"), + ), + ): + result = self.mesh.ConvertCutoutShape.execute(operator, context) + + self.assertEqual(result, {"FINISHED"}) + build_local.assert_called_once_with( + operator, + context, + source, + shape, + operator.selection_data, + True, + ) + + def test_refined_cutout_and_balloon_submit_blendjob(self): + source = SimpleNamespace(select_set=lambda _selected: None) + context = SimpleNamespace( + view_layer=SimpleNamespace(objects=SimpleNamespace(active=None)) + ) + + for shape in (self.mesh.CUTOUT, self.mesh.BALLOON): + with self.subTest(shape=shape): + operator = SimpleNamespace( + source_object_name="Source", + selection_data="[[0,0],[10,0],[0,10]]", + shape=shape, + mesh_detail=64, + refine_selection=True, + normal_analysis=False, + invert=False, + report=lambda *_args: None, + ) + with ( + patch.object( + self.mesh.bpy.data.objects, + "get", + return_value=source, + ), + patch.object(self.mesh, "is_image_empty", return_value=True), + patch.object( + self.mesh, + "_build_local_cutout", + side_effect=AssertionError("Refine must run through the Job"), + ), + patch.object( + self.mesh, "_start_job", return_value={"FINISHED"} + ) as start_job, + ): + result = self.mesh.ConvertCutoutShape.execute(operator, context) + + self.assertEqual(result, {"FINISHED"}) + start_job.assert_called_once_with( + operator, + context, + shape, + operator.selection_data, + True, + False, + False, + ) + + def test_plain_balloon_job_requests_moge_normal(self): + source = SimpleNamespace( + name="Source", + data=SimpleNamespace(size=(2048, 1024)), + ) + context = SimpleNamespace(object=source, scene=SimpleNamespace()) + operator = SimpleNamespace(mesh_detail=64, report=lambda *_args: None) + + with ( + patch.object(self.mesh, "video_total_frames", return_value=1), + patch.object( + self.mesh, "invoke_ai_setup_if_needed", return_value=None + ) as setup, + patch.object( + self.mesh, + "prepare_image_analysis_inputs", + return_value=( + Path("input.png"), + False, + Path("input.png"), + False, + ), + ), + patch.object( + self.mesh, + "_depth_settings", + return_value=("MOGE2_VITL", 100.0, 5), + ), + patch.object( + self.mesh.bpy.ops.anyimage, + "generate_mesh", + return_value={"FINISHED"}, + create=True, + ) as generate_mesh, + ): + result = self.mesh._start_job( + operator, context, self.mesh.BALLOON, normal_analysis=True + ) + + self.assertEqual(result, {"FINISHED"}) + setup.assert_called_once_with() + self.assertEqual(generate_mesh.call_args.kwargs["shape"], self.mesh.BALLOON) + self.assertEqual(generate_mesh.call_args.kwargs["mesh_detail"], 64) + self.assertEqual( + generate_mesh.call_args.kwargs["normal_model"], + self.addon.properties.DEFAULT_MOGE2_MODEL_KEY, + ) + self.assertEqual(generate_mesh.call_args.kwargs["normal_resolution_level"], 5) + + def test_all_cutout_shapes_request_moge_normal_when_enabled(self): + context = SimpleNamespace( + scene=SimpleNamespace(anyimage_settings=SimpleNamespace(video_frames=1)) + ) + for shape in self.mesh.CUTOUT_SHAPES: + with self.subTest(shape=shape): + operator = self.mesh.GenerateMesh() + operator.input_path = __file__ + operator.source_object_name = "" + operator.shape = shape + operator.selection_data = "[[0,0],[10,0],[0,10]]" + operator.model = self.addon.properties.DEFAULT_MOGE2_MODEL_KEY + operator.mesh_detail = 64 + operator.process_res = 1 + operator.resolution_level = 5 + operator.normal_model = self.addon.properties.DEFAULT_MOGE2_MODEL_KEY + operator.normal_resolution_level = 5 + operator.normal_analysis = True + operator.refine_selection = False + operator.invert = False + with ( + patch.object(self.mesh, "require_environment"), + patch.object(self.mesh, "require_model") as require_model, + patch.object(self.mesh, "production_device", return_value="CPU"), + patch.object(self.mesh, "source_start_frame", return_value=1), + ): + parameters = operator.request(context) + + require_model.assert_called_once_with( + self.addon.properties.DEFAULT_MOGE2_MODEL_KEY + ) + self.assertEqual( + parameters["normal_model"], + self.addon.properties.DEFAULT_MOGE2_MODEL_KEY, + ) + self.assertEqual(parameters["normal_resolution_level"], 5) + self.assertEqual(parameters["mesh_detail"], 64) + + def test_cutout_job_skips_moge_normal_model_when_disabled(self): + operator = self.mesh.GenerateMesh() + operator.input_path = __file__ + operator.source_object_name = "" + operator.shape = self.mesh.CUTOUT + operator.selection_data = "[[0,0],[10,0],[0,10]]" + operator.mesh_detail = 64 + operator.normal_analysis = False + operator.refine_selection = False + operator.invert = False + + with ( + patch.object(self.mesh, "require_environment"), + patch.object(self.mesh, "require_model") as require_model, + patch.object(self.mesh, "production_device", return_value="CPU"), + patch.object(self.mesh, "source_start_frame", return_value=1), + ): + parameters = operator.request(SimpleNamespace(scene=SimpleNamespace())) + + require_model.assert_not_called() + self.assertFalse(parameters["normal_analysis"]) + self.assertNotIn("normal_model", parameters) + self.assertNotIn("normal_resolution_level", parameters) def test_mesh_plane_redo_keeps_the_original_world_matrix(self): operator = self.mesh_plane.ConvertToMeshPlane() operator.shadeless = False @@ -818,9 +1060,7 @@ class AddonTest(unittest.TestCase): import inspect mesh_source = inspect.getsource(self.mesh_plane.create_mesh_plane_mesh) - processing_source = inspect.getsource( - self.mesh_plane.add_mesh_plane_processing - ) + processing_source = inspect.getsource(self.mesh_plane.add_mesh_plane_processing) self.assertIn("((0, 1, 2, 3),)", mesh_source) self.assertIn('("Subdivide", max(0, int(subdivisions)))', processing_source) self.assertIn('("Thickness", max(0.0, float(thickness)))', processing_source) @@ -865,9 +1105,8 @@ class AddonTest(unittest.TestCase): self.assertEqual( calls, [ - ("anyimage.convert_to_mesh_plane", True), + ("anyimage.convert_to_plane", True), ("anyimage.convert_to_depth_plane", True), - ("anyimage.convert_to_surface", True), ("separator", True), ("anyimage.remove_image_background", True), ("anyimage.upscale_image", True), @@ -884,6 +1123,7 @@ class AddonTest(unittest.TestCase): labels["anyimage.upscale_image"], "Upscale (640 × 480)", ) + def test_image_menu_opens_environment_settings_when_ai_is_ready(self): calls = [] @@ -933,9 +1173,7 @@ class AddonTest(unittest.TestCase): operator.model = "SAFMN_REAL_X4" operator.max_output_size = 6144 context = SimpleNamespace( - scene=SimpleNamespace( - anyimage_settings=SimpleNamespace(video_frames=7) - ) + scene=SimpleNamespace(anyimage_settings=SimpleNamespace(video_frames=7)) ) with ( patch.object(self.image_upscale, "require_environment"), @@ -943,12 +1181,8 @@ class AddonTest(unittest.TestCase): patch.object( self.image_upscale, "input_path", return_value=Path("input.png") ), - patch.object( - self.image_upscale, "production_device", return_value="CUDA" - ), - patch.object( - self.image_upscale, "source_start_frame", return_value=4 - ), + patch.object(self.image_upscale, "production_device", return_value="CUDA"), + patch.object(self.image_upscale, "source_start_frame", return_value=4), ): request = operator.request(context) @@ -979,9 +1213,7 @@ class AddonTest(unittest.TestCase): ) context = SimpleNamespace( object=source_object, - scene=SimpleNamespace( - anyimage_settings=SimpleNamespace(video_frames=1) - ), + scene=SimpleNamespace(anyimage_settings=SimpleNamespace(video_frames=1)), ) operator = self.image_upscale.UpscaleImage() with ( @@ -1109,6 +1341,22 @@ class AddonTest(unittest.TestCase): ): operator.response(None, SimpleNamespace(directory=Path("result"))) + def test_depth_analysis_follows_the_oldest_upscale_source(self): + original = SimpleNamespace(name="Original") + first_upscale = SimpleNamespace( + name="First", + anyimage_upscale_source=original, + ) + second_upscale = SimpleNamespace( + name="Second", + anyimage_upscale_source=first_upscale, + ) + + self.assertIs( + self.image_data.analysis_source_image(second_upscale), + original, + ) + def test_repeated_upscale_uses_original_image_and_reports_warning(self): original_image = SimpleNamespace(name="Original", source="FILE") current_image = SimpleNamespace( @@ -1123,9 +1371,7 @@ class AddonTest(unittest.TestCase): ) context = SimpleNamespace( object=source_object, - scene=SimpleNamespace( - anyimage_settings=SimpleNamespace(video_frames=1) - ), + scene=SimpleNamespace(anyimage_settings=SimpleNamespace(video_frames=1)), ) reports = [] operator = self.image_upscale.UpscaleImage() @@ -1287,14 +1533,53 @@ class AddonTest(unittest.TestCase): self.assertGreater(self.balloon_geometry.polygon_area(contour), 0.0) self.assertLess(float(lengths.std() / lengths.mean()), 0.15) + def test_balloon_contour_smooths_pixel_stairs_before_coarse_resampling(self): + import numpy as np + + staircase = [(0.0, 0.0)] + for value in range(16): + staircase.append((value + 1.0, float(value))) + staircase.append((value + 1.0, value + 1.0)) + staircase.append((0.0, 16.0)) + smooth = self.balloon_geometry.smooth_lasso_contour(staircase, 2.0) + + def diagonal_noise(points): + points = np.asarray(points) + selected = ( + (points[:, 0] > 2.0) + & (points[:, 0] < 14.0) + & (np.abs(points[:, 1] - points[:, 0]) < 2.0) + ) + return float(np.std(points[selected, 1] - points[selected, 0])) + + self.assertLess(diagonal_noise(smooth), 0.05) + + def test_cutout_polygon_mask_keeps_nonzero_winding_overlap(self): + polygon = ( + (1, 1), + (7, 1), + (7, 7), + (1, 7), + (1, 1), + (4, 1), + (10, 1), + (10, 7), + (4, 7), + (4, 1), + (1, 1), + ) + + even_odd = self.image_mask.polygon_mask((12, 9), polygon) + nonzero = self.image_mask.polygon_mask((12, 9), polygon, "NONZERO") + + self.assertFalse(even_odd[3, 5]) + self.assertTrue(nonzero[3, 5]) def test_image_mask_preserves_source_rgb_outside_the_selection(self): import numpy as np top_down = np.full((3, 3, 4), (0.2, 0.4, 0.6, 1.0), dtype=np.float32) source = np.flipud(top_down).ravel() - selection = self.image_mask.polygon_mask( - (3, 3), ((0, 0), (3, 0), (0, 3)) - ) + selection = self.image_mask.polygon_mask((3, 3), ((0, 0), (3, 0), (0, 3))) pixels, bounds = self.image_mask.edit_image_mask_pixels( source, (3, 3), @@ -1558,9 +1843,7 @@ class AddonTest(unittest.TestCase): self.assertNotIn(tool, self.addon.CLASSES) self.assertEqual(tool.bl_operator, operator.bl_idname) self.assertEqual(tool.bl_icon, icon) - self.assertEqual( - tool.bl_keymap[0][2]["properties"], [("gesture", gesture)] - ) + self.assertEqual(tool.bl_keymap[0][2]["properties"], [("gesture", gesture)]) self.assertNotIn("bl_cursor", tool.__dict__) self.assertIn("draw_settings", self.image_box.ImageBoxTool.__dict__) self.assertIn("draw_settings", self.image_lasso.ImageLassoTool.__dict__) @@ -1574,10 +1857,16 @@ class AddonTest(unittest.TestCase): self.assertNotIn("padding", operator.__annotations__) settings = self.addon.properties.AnyImageSettings.__annotations__ self.assertFalse(settings["image_edit_refine_selection"]["default"]) + self.assertEqual(settings["cutout_pixel_size"]["default"], 32) + self.assertEqual(settings["cutout_pixel_size"]["min"], 4) + self.assertEqual(settings["cutout_pixel_size"]["max"], 32) + self.assertFalse(settings["cutout_normal_analysis"]["default"]) self.assertNotIn("image_extract_continuous", settings) self.assertNotIn("image_extract_padding", settings) self.assertNotIn("image_extract_remove_background", settings) - source_picker = inspect.getsource(self.image_edit_common.resolve_image_edit_click) + source_picker = inspect.getsource( + self.image_edit_common.resolve_image_edit_click + ) self.assertIn("bpy.ops.view3d.select(", source_picker) self.assertIn("Click an Image Empty", source_picker) self.assertIn("_restore_selection(", source_picker) @@ -1585,6 +1874,11 @@ class AddonTest(unittest.TestCase): self.assertIn('event.type == "F"', modal_source) self.assertIn("self.invert = not self.invert", modal_source) + self.assertIn("math.dist(point, self._path[-1]) >= 2.0", modal_source) + mask_source = inspect.getsource(self.image_mask.polygon_mask) + self.assertNotIn("np.indices", mask_source) + self.assertIn("range(top, bottom, 128)", mask_source) + def test_image_line_and_polyline_disable_refine_selection(self): settings = SimpleNamespace(image_edit_refine_selection=True) context = SimpleNamespace(scene=SimpleNamespace(anyimage_settings=settings)) @@ -1614,9 +1908,7 @@ class AddonTest(unittest.TestCase): def test_image_area_click_is_cancelled_without_a_warning(self): import inspect - completion_source = inspect.getsource( - self.image_mask.EditImageMask._complete - ) + completion_source = inspect.getsource(self.image_mask.EditImageMask._complete) self.assertIn("if len(path) < 3", completion_source) self.assertIn('return {"CANCELLED"}', completion_source) @@ -1625,12 +1917,8 @@ class AddonTest(unittest.TestCase): def test_image_polyline_closes_from_start_or_enter(self): import inspect - modal_source = inspect.getsource( - self.image_mask.EditImageMask._modal_polyline - ) - draw_source = inspect.getsource( - self.image_mask.EditImageMask._draw_selection - ) + modal_source = inspect.getsource(self.image_mask.EditImageMask._modal_polyline) + draw_source = inspect.getsource(self.image_mask.EditImageMask._draw_selection) self.assertIn("return self._confirm_polyline(context)", modal_source) self.assertNotIn("DOUBLE_CLICK", modal_source) self.assertIn('event.type in {"RET", "NUMPAD_ENTER"}', modal_source) @@ -1671,10 +1959,8 @@ class AddonTest(unittest.TestCase): clicked.name: clicked, } try: - source, should_edit = ( - self.image_edit_common.resolve_image_edit_click( - context, event, lambda *_args: None - ) + source, should_edit = self.image_edit_common.resolve_image_edit_click( + context, event, lambda *_args: None ) finally: self.fake_bpy.data.objects = previous_objects @@ -1718,10 +2004,8 @@ class AddonTest(unittest.TestCase): ) self.fake_bpy.data.objects = {active.name: active, clicked.name: clicked} try: - source, should_edit = ( - self.image_edit_common.resolve_image_edit_click( - context, event, lambda *_args: None - ) + source, should_edit = self.image_edit_common.resolve_image_edit_click( + context, event, lambda *_args: None ) finally: self.fake_bpy.data.objects = previous_objects @@ -1781,6 +2065,7 @@ class AddonTest(unittest.TestCase): def test_image_perspective_corner_order_is_click_order_independent(self): import itertools + import numpy as np corners = ((10, 20), (80, 15), (90, 70), (5, 75)) @@ -1794,15 +2079,9 @@ class AddonTest(unittest.TestCase): def test_image_perspective_detects_source_image_overlap(self): overlaps = self.image_perspective.perspective_quad_overlaps_image - self.assertFalse( - overlaps(((20, 20), (30, 20), (30, 30), (20, 30)), (10, 10)) - ) - self.assertTrue( - overlaps(((5, 5), (15, 5), (15, 15), (5, 15)), (10, 10)) - ) - self.assertTrue( - overlaps(((-5, -5), (15, -5), (15, 15), (-5, 15)), (10, 10)) - ) + self.assertFalse(overlaps(((20, 20), (30, 20), (30, 30), (20, 30)), (10, 10))) + self.assertTrue(overlaps(((5, 5), (15, 5), (15, 15), (5, 15)), (10, 10))) + self.assertTrue(overlaps(((-5, -5), (15, -5), (15, 15), (-5, 15)), (10, 10))) def test_image_perspective_builds_polyline_mask_fill_triangles(self): self.assertEqual( @@ -1825,9 +2104,7 @@ class AddonTest(unittest.TestCase): 1.0, ) - self.assertLessEqual( - width * height, self.image_perspective.MAX_OUTPUT_PIXELS - ) + self.assertLessEqual(width * height, self.image_perspective.MAX_OUTPUT_PIXELS) def test_image_perspective_mouse_adjusts_preview_aspect_exponentially(self): adjust = self.image_perspective.aspect_ratio_from_mouse @@ -1866,9 +2143,7 @@ class AddonTest(unittest.TestCase): def remove(*_args, **_options): raise ReferenceError("StructRNA of type Image has been removed") - self.fake_bpy.data = SimpleNamespace( - images=SimpleNamespace(remove=remove) - ) + self.fake_bpy.data = SimpleNamespace(images=SimpleNamespace(remove=remove)) operator._remove_preview() self.assertIsNone(operator._preview_image) @@ -1961,18 +2236,12 @@ class AddonTest(unittest.TestCase): endpoint = self.image_mask.snapped_line_endpoint((0, 0), (10, 1), True) self.assertAlmostEqual(math.degrees(math.atan2(endpoint[1], endpoint[0])), 5.0) - preview = self.image_mask.line_half_plane_polygon( - (0, 5), (10, 5), (10, 10) - ) + preview = self.image_mask.line_half_plane_polygon((0, 5), (10, 5), (10, 10)) image = self.image_mask.image_line_half_plane_polygon( ((0, 5), (10, 5)), (10, 10) ) - self.assertEqual( - self.image_edit_common.polygon_bounds(preview), (0, 5, 10, 10) - ) - self.assertEqual( - self.image_edit_common.polygon_bounds(image), (0, 0, 10, 5) - ) + self.assertEqual(self.image_edit_common.polygon_bounds(preview), (0, 5, 10, 10)) + self.assertEqual(self.image_edit_common.polygon_bounds(image), (0, 0, 10, 5)) def test_balloon_interior_points_have_uniform_spacing(self): import numpy as np @@ -2087,6 +2356,64 @@ class AddonTest(unittest.TestCase): self.assertGreater(wide_height, narrow_height) + def test_balloon_keeps_overlapping_regions_disconnected(self): + import numpy as np + + regions = ( + ( + ((20, 0), (90, 0), (90, 70), (20, 70)), + ((50, 20), (50, 50), (70, 50), (70, 20)), + ), + ( + ((0, 20), (40, 20), (40, 45), (0, 45)), + ), + ) + vertices, faces, uv = self.balloon_geometry.triangulate_balloon( + regions, + (100, 100), + (0.0, 1.0, 0.0, 1.0), + spacing=0.1, + world_metric=((1.0, 0.0), (0.0, 1.0)), + ) + centers = uv[np.asarray(faces)].mean(axis=1) + in_hole = ( + (centers[:, 0] > 0.55) + & (centers[:, 0] < 0.65) + & (centers[:, 1] > 0.57) + & (centers[:, 1] < 0.73) + ) + in_left_region = ( + (centers[:, 0] > 0.02) + & (centers[:, 0] < 0.12) + & (centers[:, 1] > 0.55) + & (centers[:, 1] < 0.8) + ) + heights = np.asarray(vertices)[:, 2] + + self.assertFalse(in_hole.any()) + self.assertTrue(in_left_region.any()) + self.assertGreater(float(heights.max()), 0.0) + self.assertLess(float(heights.min()), 0.0) + adjacency = {} + for face in faces: + for vertex in face: + adjacency.setdefault(int(vertex), set()) + for start, end in zip(face, np.roll(face, -1)): + adjacency[int(start)].add(int(end)) + adjacency[int(end)].add(int(start)) + remaining = set(adjacency) + component_count = 0 + while remaining: + component_count += 1 + pending = [remaining.pop()] + while pending: + vertex = pending.pop() + connected = adjacency[vertex] & remaining + remaining.difference_update(connected) + pending.extend(connected) + + self.assertEqual(component_count, 2) + def test_poisson_balloon_supports_resolution_002(self): import numpy as np @@ -2228,19 +2555,16 @@ class AddonTest(unittest.TestCase): self.assertTrue(edges) self.assertTrue(all(users == 2 for users in edges.values())) - def test_balloon_tool_operators_are_registered(self): - self.assertIn(self.balloon.ExtractBalloon, self.addon.CLASSES) - self.assertIn(self.balloon.ActivateBalloonTool, self.addon.CLASSES) + def test_balloon_tool_operators_are_not_registered(self): + self.assertNotIn(self.balloon.ExtractBalloon, self.addon.CLASSES) + self.assertNotIn(self.balloon.ActivateBalloonTool, self.addon.CLASSES) self.assertNotIn(self.balloon.BalloonTool, self.addon.CLASSES) - self.assertLess( - self.addon.CLASSES.index(self.balloon.ExtractBalloon), - self.addon.CLASSES.index(self.balloon.ActivateBalloonTool), - ) + self.assertIn(self.mesh.GenerateMesh, self.addon.CLASSES) def test_balloon_ai_analysis_is_registered_and_forwarded(self): import inspect - self.assertIn(self.balloon.AnalyzeBalloon, self.addon.CLASSES) + self.assertNotIn(self.balloon.AnalyzeBalloon, self.addon.CLASSES) operator = self.balloon.ExtractBalloon self.assertIn("contour_analysis", operator.__annotations__) self.assertIn("depth_analysis", operator.__annotations__) @@ -2333,6 +2657,7 @@ class AddonTest(unittest.TestCase): def test_balloon_depth_fields_and_shape_nodes(self): import inspect + import numpy as np depth_data = { @@ -2379,8 +2704,12 @@ class AddonTest(unittest.TestCase): controller_source = inspect.getsource( self.balloon_depth.create_balloon_base_plane ) + modifier_source = inspect.getsource( + self.balloon_depth.add_balloon_shape_modifier + ) self.assertIn('"Thickness"', source) self.assertIn('"Depth Strength"', source) + self.assertIn('"Depth Strength": 1.0', modifier_source) self.assertNotIn('"Far Clamp"', source) self.assertIn('"Double Sided Depth"', source) self.assertIn('"Base Plane"', source) @@ -2399,7 +2728,7 @@ class AddonTest(unittest.TestCase): self.assertNotIn("o_balloon_depth_mix", source) self.assertIn('front.operation = "LESS_THAN"', source) self.assertIn('depth_residual.operation = "ADD"', source) - self.assertIn('depth_calibration.inputs[1].default_value = 0.5', source) + self.assertIn("depth_calibration.inputs[1].default_value = 0.5", source) self.assertIn('corrected_depth.operation = "ADD"', source) self.assertIn( 'profile_statistics.outputs["Max"], corrected_depth.inputs[1]', @@ -2410,7 +2739,7 @@ class AddonTest(unittest.TestCase): self.assertIn('double_choice.outputs["Output"], combine.inputs["X"]', source) self.assertIn('depth_mix = nodes.new("ShaderNodeMix")', source) self.assertIn('depth_mix.data_type = "FLOAT"', source) - self.assertIn('depth_mix.clamp_factor = True', source) + self.assertIn("depth_mix.clamp_factor = True", source) self.assertIn('base_magnitude.outputs[0], depth_mix.inputs["A"]', source) self.assertIn('depth_scale.outputs[0], depth_mix.inputs["B"]', source) self.assertNotIn("profile_inverse", source) @@ -2438,7 +2767,6 @@ class AddonTest(unittest.TestCase): "o_balloon_depth", ) - def test_balloon_redo_properties_use_mode_and_meter_resolution(self): import inspect @@ -2595,11 +2923,11 @@ class AddonTest(unittest.TestCase): self.assertIn("color_image = image.copy()", source) self.assertIn("color_image.name = color_name", source) self.assertIn( - "material_name,\n color_image,\n 1.0,", + "material_name,\n color_image,\n 0.0,", source, ) self.assertIn( - "else TEXTURE_MATERIAL_NODE_GROUP_NAME", + "surface_group_name=IMAGE_MATERIAL_NODE_GROUP_NAME", source, ) self.assertIn("owns_material", source) @@ -2608,14 +2936,13 @@ class AddonTest(unittest.TestCase): self.assertIn("assign_balloon_shape_attributes(", source) self.assertIn("inflation=INFLATION", source) self.assertIn("create_balloon_base_plane(", source) - self.assertIn( - "add_balloon_shape_modifier(balloon, thickness, base_plane)", source - ) - self.assertIn("BALLOON_MATERIAL_NODE_GROUP_NAME", source) + self.assertIn("add_balloon_shape_modifier(", source) + self.assertNotIn("depth_strength", source) + self.assertIn("IMAGE_MATERIAL_NODE_GROUP_NAME", source) self.assertIn("normal_path=", source) self.assertEqual( self.image_depth.TEXTURE_MATERIAL_NODE_GROUP_NAME, - "O Texture Layer", + "O Image Layer", ) def test_balloon_material_reuse_uses_explicit_source_markers(self): @@ -2858,9 +3185,12 @@ class AddonTest(unittest.TestCase): }, ), ( - self.balloon.BalloonTool, + self.addon.CutoutTool, { - "separator": False, + "after": { + self.image_perspective.ImagePerspectiveTool.bl_idname + }, + "separator": True, }, ), ], @@ -2875,9 +3205,7 @@ class AddonTest(unittest.TestCase): self.outliner_menu_draws[0], self.addon.draw_image_context_menu, ) - self.assertTrue( - hasattr(self.fake_bpy.types.Image, "anyimage_upscale_source") - ) + self.assertTrue(hasattr(self.fake_bpy.types.Image, "anyimage_upscale_source")) self.addon.unregister() self.assertEqual(self.registered, []) @@ -2885,9 +3213,7 @@ class AddonTest(unittest.TestCase): self.assertEqual(self.status_bar_draws, []) self.assertEqual(self.context_menu_draws, [existing_context_draw]) self.assertEqual(self.outliner_menu_draws, [existing_outliner_draw]) - self.assertFalse( - hasattr(self.fake_bpy.types.Image, "anyimage_upscale_source") - ) + self.assertFalse(hasattr(self.fake_bpy.types.Image, "anyimage_upscale_source")) def test_blender_image_exports_do_not_use_scene_render_color_management(self): source_root = PROJECT_ROOT / "src" / "anyimage" @@ -2922,7 +3248,7 @@ class AddonTest(unittest.TestCase): self.assertEqual(calls[1], ("operator", "anyimage.cancel_job")) def test_moge_depth_request_is_written_on_the_operator(self): - operator = self.addon.operators.GenerateMoge2Depth() + operator = self.depth_operator.GenerateMoge2Depth() operator.input_path = __file__ operator.source_object_name = "" operator.model = "MOGE2_VITS_NORMAL" @@ -2943,7 +3269,7 @@ class AddonTest(unittest.TestCase): self.assertNotIn("job_type", parameters) def test_moge_depth_response_calls_an_ordinary_function(self): - operator = self.addon.operators.GenerateMoge2Depth() + operator = self.depth_operator.GenerateMoge2Depth() operator.source_object_name = "Reference" operator.remove_background = False operator.delete_source = True @@ -2990,8 +3316,13 @@ class AddonTest(unittest.TestCase): with patch.object( self.depth_operator, - "prepare_image_input", - return_value=(Path(__file__), False), + "prepare_image_analysis_inputs", + return_value=( + Path(__file__), + False, + Path(__file__), + False, + ), ): self.assertEqual(operator.execute(context), {"FINISHED"}) diff --git a/tests/test_cutout.py b/tests/test_cutout.py new file mode 100644 index 0000000..a8c5297 --- /dev/null +++ b/tests/test_cutout.py @@ -0,0 +1,136 @@ +import json +import sys +from pathlib import Path +from types import MethodType, SimpleNamespace + +import bpy + +SOURCE_ROOT = Path(__file__).parents[1] / "src" +sys.path.insert(0, str(SOURCE_ROOT)) + +from anyimage.operators import cutout, mesh +from anyimage.operators.image_edit.mask import ImageMaskGesture + + +class Pie: + def __init__(self): + self.calls = [] + + def operator(self, identifier, **options): + operator = SimpleNamespace() + self.calls.append((identifier, options, operator)) + return operator + + +class WindowManager: + def __init__(self): + self.pie = Pie() + + def popup_menu_pie(self, _event, draw, **_options): + menu = SimpleNamespace(layout=SimpleNamespace(menu_pie=lambda: self.pie)) + draw(menu, None) + + +def cutout_context(window_manager, pixel_size=32, normal_analysis=False): + settings = SimpleNamespace( + cutout_pixel_size=pixel_size, + cutout_normal_analysis=normal_analysis, + ) + return SimpleNamespace( + window_manager=window_manager, + scene=SimpleNamespace(anyimage_settings=settings), + ) + + +def cutout_source(name="Source", size=(2048, 1024)): + return SimpleNamespace(name=name, data=SimpleNamespace(size=size)) + + +def test_cutout_pie_uses_four_semantic_shapes_in_cardinal_order(): + window_manager = WindowManager() + context = cutout_context(window_manager) + source = cutout_source() + + cutout._open_shape_pie(context, object(), source, ((0, 0), (10, 0), (0, 10))) + + assert [call[2].shape for call in window_manager.pie.calls] == [ + mesh.CUTOUT, + mesh.BALLOON, + mesh.DEPTH_BALLOON, + mesh.DEPTH_SURFACE, + ] + assert all( + call[0] == "anyimage.convert_cutout_shape" for call in window_manager.pie.calls + ) + assert not any(call[2].refine_selection for call in window_manager.pie.calls) + assert not any(call[2].normal_analysis for call in window_manager.pie.calls) + assert not any(call[2].invert for call in window_manager.pie.calls) + assert all(call[2].mesh_detail == 64 for call in window_manager.pie.calls) + + +def test_cutout_pie_passes_image_lasso_options_to_every_shape(): + window_manager = WindowManager() + context = cutout_context( + window_manager, pixel_size=16, normal_analysis=True + ) + source = cutout_source(size=(4096, 2048)) + + cutout._open_shape_pie( + context, + object(), + source, + ((0, 0), (10, 0), (0, 10)), + refine_selection=True, + invert=True, + ) + + assert all(call[2].refine_selection for call in window_manager.pie.calls) + assert all(call[2].normal_analysis for call in window_manager.pie.calls) + assert all(call[2].invert for call in window_manager.pie.calls) + assert all(call[2].mesh_detail == 256 for call in window_manager.pie.calls) + + +def test_cutout_mesh_detail_tracks_the_source_long_edge_and_clamps(): + assert cutout.cutout_mesh_detail((4096, 2048), 32) == 128 + assert cutout.cutout_mesh_detail((16, 8), 4) == 4 + assert cutout.cutout_mesh_detail((65536, 32768), 4) == 1024 + + +def test_cutout_reuses_image_lasso_gesture_and_double_click_selects_all(monkeypatch): + bpy.ops.wm.read_factory_settings(use_empty=True) + image = bpy.data.images.new("Source", width=320, height=180) + source = bpy.data.objects.new("Source", None) + source.empty_display_type = "IMAGE" + source.data = image + bpy.context.collection.objects.link(source) + window_manager = WindowManager() + context = cutout_context(window_manager) + + monkeypatch.setattr( + cutout, "resolve_image_edit_click", lambda *_args: (source, True) + ) + monkeypatch.setattr(cutout, "video_total_frames", lambda _image: 1) + monkeypatch.setattr(cutout, "apply_image_mask_settings", lambda *_args: None) + monkeypatch.setattr(cutout, "selected_image_rgba", lambda *_args: object()) + + operator = SimpleNamespace( + full_image=True, + refine_selection=False, + report=lambda *_args: None, + ) + operator.execute = MethodType(cutout.SelectCutoutRegion.execute, operator) + result = cutout.SelectCutoutRegion.invoke( + operator, context, SimpleNamespace(value="DOUBLE_CLICK") + ) + + assert issubclass(cutout.SelectCutoutRegion, ImageMaskGesture) + assert result == {"FINISHED"} + selection = json.loads(window_manager.pie.calls[0][2].selection_data) + assert selection == [[0, 0], [320, 0], [320, 180], [0, 180]] + assert window_manager.pie.calls[0][2].mesh_detail == 10 + assert not hasattr(cutout, "pending_selection") + + +def test_cutout_tool_prioritizes_double_click_before_lasso_press(): + assert cutout.CutoutTool.bl_keymap[0][1]["value"] == "DOUBLE_CLICK" + assert cutout.CutoutTool.bl_keymap[1][1]["value"] == "PRESS" diff --git a/tests/test_mesh.py b/tests/test_mesh.py new file mode 100644 index 0000000..0eac52f --- /dev/null +++ b/tests/test_mesh.py @@ -0,0 +1,322 @@ +import builtins +import sys +import tempfile +from pathlib import Path +from unittest.mock import patch + +import numpy as np +from PIL import Image + +SOURCE_ROOT = Path(__file__).parents[1] / "src" +sys.path.insert(0, str(SOURCE_ROOT)) + +from anyimage.operators import mesh +from anyimage.operators.normal_map import rasterize_tangent_normal +from anyimage.server.geometry.cutout import alpha_regions, image_alpha_regions + + +def mesh_component_count(faces): + adjacency = {} + for face in faces: + for vertex in face: + adjacency.setdefault(int(vertex), set()) + for start, end in zip(face, np.roll(face, -1)): + adjacency[int(start)].add(int(end)) + adjacency[int(end)].add(int(start)) + remaining = set(adjacency) + count = 0 + while remaining: + count += 1 + pending = [remaining.pop()] + while pending: + vertex = pending.pop() + connected = adjacency[vertex] & remaining + remaining.difference_update(connected) + pending.extend(connected) + return count + + +def test_tangent_normal_pixels_use_the_final_surface_frame(): + object_normal = np.zeros((3, 3, 3), dtype=np.float32) + object_normal[..., 2] = 1.0 + triangle_uv = np.asarray((((0.0, 0.0), (1.0, 0.0), (0.0, 1.0)),)) + triangle_normal = np.asarray((((1.0, 0.0, 0.0),) * 3,)) + triangle_tangent = np.asarray((((0.0, 1.0, 0.0),) * 3,)) + triangle_sign = np.ones((1, 3), dtype=np.float32) + + tangent = rasterize_tangent_normal( + object_normal, + triangle_uv, + triangle_normal, + triangle_tangent, + triangle_sign, + ) + + assert np.allclose(tangent[0, 0], (0.0, 1.0, 0.0)) + assert not np.allclose(tangent[0, 0], object_normal[0, 0]) + assert np.allclose(tangent[2, 2], (0.0, 0.0, 1.0)) + + +def test_plane_source_topology_is_one_quad_at_every_detail_level(): + low = mesh._plane_domain((-2.0, 2.0, -1.0, 1.0), 1) + high = mesh._plane_domain((-2.0, 2.0, -1.0, 1.0), 512) + + assert low[0].shape == high[0].shape == (4, 3) + assert low[1].shape == high[1].shape == (1, 4) + + +def test_cutout_front_back_and_side_faces_point_outward(): + vertices = np.asarray( + ((0.0, 0.0, 0.0), (0.0, 1.0, 0.0), (1.0, 0.0, 0.0)), + dtype=np.float64, + ) + faces = np.asarray(((0, 1, 2),), dtype=np.int32) + uv = vertices[:, :2].astype(np.float32) + boundary = np.ones(3, dtype=bool) + layered = mesh._layered_geometry( + (vertices, faces, uv, boundary, vertices[:, :2]), + mesh.CUTOUT, + ) + positioned = layered[0] + layered[3] + output_faces = layered[1] + + def face_normal(face): + points = positioned[np.asarray(face[:3], dtype=np.int32)] + return np.cross(points[1] - points[0], points[2] - points[0]) + + assert face_normal(output_faces[0])[2] > 0.0 + assert face_normal(output_faces[1])[2] < 0.0 + center = vertices[:, :2].mean(axis=0) + for face in output_faces[2:]: + normal = face_normal(face) + side_center = positioned[np.asarray(face, dtype=np.int32), :2].mean(axis=0) + assert np.dot(normal[:2], side_center - center) > 0.0 + + +def test_depth_surface_moves_both_sides_with_camera_projection_and_extrudes_back(): + domain = mesh._plane_domain((-1.0, 1.0, -1.0, 1.0), 4) + field = np.zeros((2, 2, 3), dtype=np.float32) + field[..., 2] = 1.0 + valid = np.ones((2, 2), dtype=bool) + camera_depth = np.ones((2, 2), dtype=np.float32) + + depth_surface = mesh._layered_geometry( + domain, mesh.DEPTH_SURFACE, (field, valid, camera_depth) + ) + base_count = len(domain[0]) + + assert np.allclose(depth_surface[4][:base_count, 2], 1.0) + assert np.allclose(depth_surface[4][base_count:, 2], 1.0) + assert np.allclose(depth_surface[3][:base_count], 0.0) + assert np.allclose(depth_surface[3][base_count:, 2], -1.0) + assert all(depth_surface[5][: len(domain[1])]) + assert not any(depth_surface[5][len(domain[1]) :]) + + +def test_depth_surface_records_depth_discontinuities_for_geometry_nodes(): + faces = np.asarray(((0, 1, 2), (1, 2, 3)), dtype=np.int32) + depth = np.asarray((1.0, 1.01, 1.02, 2.0), dtype=np.float32) + + discontinuity = mesh._depth_discontinuity_values(faces, depth) + + assert discontinuity[0] < mesh.DEPTH_EDGE_THRESHOLD + assert discontinuity[1] > mesh.DEPTH_EDGE_THRESHOLD + + +def test_depth_surface_discontinuity_map_uses_local_three_by_three_range(): + depth = np.ones((5, 5), dtype=np.float32) + depth[:, 3:] = 2.0 + valid = np.ones_like(depth, dtype=bool) + + discontinuity = mesh._depth_discontinuity_map(depth, valid) + + assert discontinuity[2, 0] < mesh.DEPTH_EDGE_THRESHOLD + assert discontinuity[2, 2] > mesh.DEPTH_EDGE_THRESHOLD + + +def test_cutout_regions_group_holes_with_their_connected_component(): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "cutout.png" + pixels = np.zeros((120, 200, 4), dtype=np.uint8) + pixels[20:100, 50:160] = (20, 40, 60, 255) + pixels[45:75, 80:120] = 0 + pixels[5:15, 5:15] = (20, 40, 60, 255) + Image.fromarray(pixels, mode="RGBA").save(path) + + regions, size = image_alpha_regions(path, max_resolution=200) + + assert size == (200, 120) + assert sorted(len(region) for region in regions) == [1, 2] + contours = [contour for region in regions for contour in region] + bounds = [ + (*np.min(contour, axis=0), *np.max(contour, axis=0)) + for contour in contours + ] + assert (50.0, 20.0, 160.0, 100.0) in bounds + assert (80.0, 45.0, 120.0, 75.0) in bounds + assert (5.0, 5.0, 15.0, 15.0) in bounds + + +def test_alpha_regions_accepts_blender_float_alpha_with_a_hole(): + alpha = np.zeros((120, 200), dtype=np.float32) + alpha[20:100, 50:160] = 1.0 + alpha[45:75, 80:120] = 0.0 + + regions, size = alpha_regions(alpha, max_resolution=200) + + assert size == (200, 120) + assert len(regions) == 1 + assert len(regions[0]) == 2 + + +def test_blender_cutout_alpha_pipeline_does_not_import_pillow(): + original_import = builtins.__import__ + + def without_pillow(name, *args, **kwargs): + if name == "PIL" or name.startswith("PIL."): + raise ModuleNotFoundError("Pillow is unavailable in Blender") + return original_import(name, *args, **kwargs) + + selection = np.zeros((32, 48), dtype=bool) + selection[4:28, 6:42] = True + with patch("builtins.__import__", side_effect=without_pillow): + antialiased = mesh._antialias_selection(selection) + regions, size = alpha_regions(antialiased, max_resolution=24) + + assert size == (48, 32) + assert regions + assert np.any((antialiased > 0.0) & (antialiased < 1.0)) + +def test_cutout_geometry_ignores_source_alpha_without_refine(): + alpha = np.zeros((80, 120), dtype=np.float32) + alpha[10:70, 10:110] = 1.0 + alpha[30:50, 45:75] = 0.0 + polygon = ((10, 10), (110, 10), (110, 70), (10, 70)) + + regions, size = mesh._geometry_cutout_regions(alpha, polygon, 64) + + assert size == (120, 80) + assert len(regions) == 1 + assert len(regions[0]) == 1 + + +def test_cutout_geometry_uses_final_alpha_with_refine(): + alpha = np.zeros((80, 120), dtype=np.float32) + alpha[10:70, 10:110] = 1.0 + alpha[30:50, 45:75] = 0.0 + polygon = ((10, 10), (110, 10), (110, 70), (10, 70)) + + regions, size = mesh._geometry_cutout_regions( + alpha, polygon, 64, refine_selection=True + ) + + assert size == (120, 80) + assert len(regions) == 1 + assert len(regions[0]) == 2 + +def test_cutout_geometry_can_extend_beyond_the_source_image(): + alpha = np.zeros((60, 100), dtype=np.float32) + alpha[10:50] = 1.0 + polygon = ((-20, 10), (120, 10), (120, 50), (-20, 50)) + + regions, size = mesh._geometry_cutout_regions(alpha, polygon, 64) + + points = np.concatenate( + [np.asarray(contour) for region in regions for contour in region] + ) + assert size == (100, 60) + assert points[:, 0].min() <= -19.0 + assert points[:, 0].max() >= 119.0 + +def test_refined_cutout_geometry_discards_out_of_frame_selection(): + alpha = np.zeros((60, 100), dtype=np.float32) + alpha[10:50] = 1.0 + polygon = ((-20, 10), (120, 10), (120, 50), (-20, 50)) + + regions, size = mesh._geometry_cutout_regions( + alpha, polygon, 64, refine_selection=True + ) + + points = np.concatenate( + [np.asarray(contour) for region in regions for contour in region] + ) + assert size == (100, 60) + assert points[:, 0].min() >= 0.0 + assert points[:, 0].max() <= 100.0 + +def test_blender_cutout_contour_resolution_tracks_mesh_detail(): + expected = ([(((0, 0), (1, 0), (0, 1)),)], (1, 1)) + alpha = np.ones((2, 2), dtype=np.float32) + with patch.object(mesh, "alpha_regions", return_value=expected) as trace: + assert mesh._cutout_regions(alpha, 32) == expected + assert mesh._cutout_regions(alpha, 256) == expected + + assert trace.call_args_list[0].kwargs["max_resolution"] == 512 + assert trace.call_args_list[1].kwargs["max_resolution"] == 1024 + +def test_cutout_domain_adapts_boundary_spacing_for_a_small_valid_region(): + regions = ( + (((2046, 2046), (2050, 2046), (2050, 2050), (2046, 2050)),), + ) + + domain = mesh._cutout_domain( + regions, + (4096, 4096), + (-1.0, 1.0, -1.0, 1.0), + 4, + ((1.0, 0.0), (0.0, 1.0)), + ) + + assert len(domain[0]) >= 3 + assert len(domain[1]) >= 1 + + +def test_cutout_domain_builds_a_full_image_at_minimum_detail(): + regions = ((((0, 0), (4096, 0), (4096, 4096), (0, 4096)),),) + + domain = mesh._cutout_domain( + regions, + (4096, 4096), + (-1.0, 1.0, -1.0, 1.0), + 4, + ((1.0, 0.0), (0.0, 1.0)), + ) + + assert len(domain[0]) >= 4 + assert len(domain[1]) >= 2 + +def test_cutout_domain_keeps_overlapping_regions_disconnected(): + regions = ( + ( + ((20, 0), (100, 0), (100, 80), (20, 80)), + ((50, 20), (50, 60), (70, 60), (70, 20)), + ), + ( + ((0, 20), (40, 20), (40, 50), (0, 50)), + ), + ) + domain = mesh._cutout_domain( + regions, + (120, 80), + (0.0, 1.0, 0.0, 1.0), + 40, + ((1.0, 0.0), (0.0, 1.0)), + ) + + centers = domain[2][domain[1]].mean(axis=1) + in_hole = ( + (centers[:, 0] > 0.45) + & (centers[:, 0] < 0.55) + & (centers[:, 1] > 0.35) + & (centers[:, 1] < 0.65) + ) + in_left_region = ( + (centers[:, 0] > 0.02) + & (centers[:, 0] < 0.12) + & (centers[:, 1] > 0.4) + & (centers[:, 1] < 0.75) + ) + + assert not in_hole.any() + assert in_left_region.any() + assert mesh_component_count(domain[1]) == 2 diff --git a/tests/test_mesh_blender.py b/tests/test_mesh_blender.py new file mode 100644 index 0000000..a5d245e --- /dev/null +++ b/tests/test_mesh_blender.py @@ -0,0 +1,562 @@ +import sys +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import bpy +import numpy as np +from mathutils import Matrix + +SOURCE_ROOT = Path(__file__).parents[1] / "src" +sys.path.insert(0, str(SOURCE_ROOT)) + +from anyimage.operators import material, mesh +from anyimage.operators.normal_map import bake_tangent_normal_image + + +def _inputs(group): + return [ + item.name + for item in group.interface.items_tree + if item.item_type == "SOCKET" and item.in_out == "INPUT" + ] + + +def _group_node(group, node_tree): + return next( + node + for node in node_tree.nodes + if node.bl_idname == "ShaderNodeGroup" and node.node_tree == group + ) + + +def test_tangent_bake_changes_pixels_using_evaluated_mesh_tbn(): + bpy.ops.wm.read_factory_settings(use_empty=True) + source = bpy.data.objects.new("Source", None) + bpy.context.collection.objects.link(source) + target_mesh = bpy.data.meshes.new("Target") + target_mesh.from_pydata( + ((0.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0)), + (), + ((0, 1, 2),), + ) + target_mesh.polygons[0].use_smooth = True + mesh._assign_uv( + target_mesh, + np.asarray(((0.0, 0.0), (1.0, 0.0), (0.0, 1.0))), + ) + target = bpy.data.objects.new("Target", target_mesh) + bpy.context.collection.objects.link(target) + target.matrix_world = Matrix.Rotation(np.radians(-90.0), 4, "Y") + image = bpy.data.images.new("Object Normal", width=3, height=3, alpha=True) + pixels = np.ones((3, 3, 4), dtype=np.float32) + pixels[..., :3] = (1.0, 0.5, 0.5) + image.pixels.foreach_set(pixels.ravel()) + + bake_tangent_normal_image(bpy.context, source, target, image) + + baked = np.empty(3 * 3 * 4, dtype=np.float32) + image.pixels.foreach_get(baked) + baked = baked.reshape((3, 3, 4)) + assert not np.allclose(baked[0, 0, :3], pixels[0, 0, :3]) + + +def test_shape_node_groups_expose_only_shape_parameters(): + bpy.ops.wm.read_factory_settings(use_empty=True) + + plane = mesh._mesh_node_group(mesh.PLANE) + depth_plane = mesh._mesh_node_group(mesh.DEPTH_PLANE) + cutout = mesh._mesh_node_group(mesh.CUTOUT) + depth_surface = mesh._mesh_node_group(mesh.DEPTH_SURFACE) + + assert plane.name == "O Mesh Plane" + assert _inputs(plane) == ["Geometry", "Subdivide", "Thickness", "Material"] + assert _inputs(cutout) == ["Geometry", "Thickness"] + assert _inputs(depth_plane) == [ + "Geometry", + "Subdivide", + "Thickness", + "Depth Amount", + "Depth Image", + "Material", + ] + assert _inputs(depth_surface) == [ + "Geometry", + "Thickness", + "Depth Scale", + "Iterations", + "Smooth", + "Depth Continuity", + ] + for group in (depth_plane, cutout, depth_surface): + thickness = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" and item.name == "Thickness" + ) + assert thickness.subtype == "DISTANCE" + for group, name in ( + (depth_plane, "Depth Amount"), + (depth_surface, "Depth Scale"), + ): + depth_amount = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" and item.name == name + ) + assert depth_amount.subtype == "NONE" + assert depth_amount.min_value == 0.0 + assert depth_amount.max_value > 1e20 + assert any( + node.bl_idname == "ShaderNodeMath" and node.operation == "MAXIMUM" + for node in depth_plane.nodes + ) + assert any(node.bl_idname == "GeometryNodeMeshGrid" for node in plane.nodes) + assert any( + node.bl_idname == "GeometryNodeGroup" and node.node_tree == plane + for node in depth_plane.nodes + ) + assert any( + node.bl_idname == "GeometryNodeDeleteGeometry" for node in depth_plane.nodes + ) + smooth = next( + item + for item in depth_surface.interface.items_tree + if item.item_type == "SOCKET" and item.name == "Smooth" + ) + assert smooth.default_value == 1.0 + iterations = next( + item + for item in depth_surface.interface.items_tree + if item.item_type == "SOCKET" and item.name == "Iterations" + ) + assert iterations.default_value == 1 + assert iterations.min_value == 0 + continuity = next( + item + for item in depth_surface.interface.items_tree + if item.item_type == "SOCKET" and item.name == "Depth Continuity" + ) + assert continuity.default_value == 0.0 + assert not any(node.bl_idname == "GeometryNodeGroup" for node in depth_surface.nodes) + assert any( + node.bl_idname == "GeometryNodeRepeatInput" for node in depth_surface.nodes + ) + assert any( + node.bl_idname == "GeometryNodeRepeatOutput" for node in depth_surface.nodes + ) + assert any( + node.bl_idname == "GeometryNodeBlurAttribute" for node in depth_surface.nodes + ) + assert any( + node.bl_idname == "GeometryNodeInputMeshEdgeNeighbors" + for node in depth_surface.nodes + ) + assert sum( + node.bl_idname == "GeometryNodeInputMeshEdgeNeighbors" + for node in depth_surface.nodes + ) == 2 + assert sum( + node.bl_idname == "GeometryNodeInputMeshVertexNeighbors" + for node in depth_surface.nodes + ) == 2 + boundary_compares = [ + node + for node in depth_surface.nodes + if node.bl_idname == "FunctionNodeCompare" + and node.data_type == "INT" + and node.operation == "EQUAL" + ] + assert len(boundary_compares) == 4 + for compare in boundary_compares: + compare_a = next( + socket for socket in compare.inputs if socket.identifier == "A_INT" + ) + compare_b = next( + socket for socket in compare.inputs if socket.identifier == "B_INT" + ) + assert compare_a.is_linked + assert compare_b.default_value == 1 + fixed_pin_switches = [ + node + for node in depth_surface.nodes + if node.bl_idname == "GeometryNodeSwitch" + and not node.inputs["Switch"].is_linked + ] + assert len(fixed_pin_switches) == 2 + assert all(node.inputs["Switch"].default_value for node in fixed_pin_switches) + assert any( + node.bl_idname == "ShaderNodeMix" + and not node.inputs[0].is_linked + and node.inputs[0].default_value == 1.0 + for node in depth_surface.nodes + ) + assert any( + node.bl_idname == "GeometryNodeInputNamedAttribute" + and node.inputs["Name"].default_value == mesh.DEPTH_DISCONTINUITY_ATTRIBUTE + for node in depth_surface.nodes + ) + for group in (depth_plane, cutout, depth_surface): + input_sockets = [ + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" and item.in_out == "INPUT" + ] + assert all(item.structure_type == "SINGLE" for item in input_sockets) + for group_input in ( + node for node in group.nodes if node.bl_idname == "NodeGroupInput" + ): + assert all( + output.is_linked or output.hide + for output in group_input.outputs + if output.name + ) + geometry_links = [ + link + for link in group.links + if link.from_socket.type == "GEOMETRY" + and link.to_socket.type == "GEOMETRY" + ] + assert all( + link.from_node.location.x <= link.to_node.location.x + for link in geometry_links + ) + assert all(not node.label for node in group.nodes) + node_locations = {tuple(node.location) for node in depth_surface.nodes} + assert len(node_locations) == len(depth_surface.nodes) + depth_plane_locations = {tuple(node.location) for node in depth_plane.nodes} + assert len(depth_plane_locations) == len(depth_plane.nodes) + assert any( + node.bl_idname == "GeometryNodeSetShadeSmooth" + and node.inputs["Shade Smooth"].default_value + for node in depth_surface.nodes + ) + + +def test_image_layer_switches_between_tangent_and_object_space_normal(): + bpy.ops.wm.read_factory_settings(use_empty=True) + + assert material.SURFACE_MATERIAL_NODE_GROUP_NAME == "O Image Layer" + assert ( + material.MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME + == material.BALLOON_MATERIAL_NODE_GROUP_NAME + == material.IMAGE_MATERIAL_NODE_GROUP_NAME + ) + group = material.material_node_group( + surface_group_name=material.IMAGE_MATERIAL_NODE_GROUP_NAME + ) + normal = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal" + ) + assert tuple(normal.default_value) == (0.5, 0.5, 1.0, 1.0) + bump_scale = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Bump Scale" + ) + assert bump_scale.default_value == 0.0 + normal_scale = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal Scale" + ) + assert normal_scale.default_value == 1.0 + object_space = next( + item + for item in group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Object Space" + ) + assert object_space.default_value is False + normal_maps = { + node.space + for node in group.nodes + if node.bl_idname == "ShaderNodeNormalMap" + } + assert normal_maps == {"TANGENT", "OBJECT"} + assert all( + not node.label + for node in group.nodes + if node.bl_idname in {"ShaderNodeNormalMap", "ShaderNodeMix"} + ) + + color = bpy.data.images.new("Color", width=2, height=2) + normal_image = bpy.data.images.new("Normal", width=2, height=2) + object_material = material.create_material( + "Object Normal", + color, + 1.0, + surface_group_name=material.IMAGE_MATERIAL_NODE_GROUP_NAME, + normal_image=normal_image, + normal_space="OBJECT", + ) + layer = _group_node(group, object_material.node_tree) + assert layer.inputs["Object Space"].default_value is True + + tangent_material = material.create_material( + "Tangent Normal", + color, + 1.0, + surface_group_name=material.IMAGE_MATERIAL_NODE_GROUP_NAME, + normal_image=normal_image, + ) + layer = _group_node(group, tangent_material.node_tree) + assert layer.inputs["Object Space"].default_value is False + reused = material.material_node_group( + surface_group_name=material.IMAGE_MATERIAL_NODE_GROUP_NAME + ) + assert { + node.space + for node in reused.nodes + if node.bl_idname == "ShaderNodeNormalMap" + } == {"TANGENT", "OBJECT"} + + +def test_build_plane_keeps_one_quad_and_generates_topology_in_geometry_nodes(): + bpy.ops.wm.read_factory_settings(use_empty=True) + image = bpy.data.images.new("Source", width=4096, height=2048) + source = bpy.data.objects.new("Source", None) + source.empty_display_type = "IMAGE" + source.data = image + source.empty_display_size = 2.0 + bpy.context.collection.objects.link(source) + material = bpy.data.materials.new("Source.material") + + result = mesh._build_mesh_object( + bpy.context, + source, + mesh.PLANE, + 2, + material, + keep_original=True, + ) + + assert result["anyimage_mesh_shape"] == mesh.PLANE + assert len(result.data.vertices) == 4 + assert len(result.data.polygons) == 1 + assert all(polygon.use_smooth for polygon in result.data.polygons) + assert mesh.PROFILE_ATTRIBUTE not in result.data.attributes + assert mesh.DEPTH_ATTRIBUTE not in result.data.attributes + assert mesh.FRONT_FACE_ATTRIBUTE not in result.data.attributes + assert result.modifiers[0].node_group.name == "O Mesh Plane" + evaluated = result.evaluated_get(bpy.context.evaluated_depsgraph_get()) + evaluated_mesh = evaluated.to_mesh() + try: + assert len(evaluated_mesh.polygons) > 0 + finally: + evaluated.to_mesh_clear() + +def test_depth_surface_protected_smooth_evaluates(): + bpy.ops.wm.read_factory_settings(use_empty=True) + image = bpy.data.images.new("Surface", width=64, height=64) + source = bpy.data.objects.new("Surface", None) + source.empty_display_type = "IMAGE" + source.data = image + source.empty_display_size = 2.0 + bpy.context.collection.objects.link(source) + surface_material = bpy.data.materials.new("Surface.material") + + size = 8 + depth = np.ones((size, size), dtype=np.float32) + x, y = np.meshgrid(np.arange(size), np.arange(size)) + intrinsics = np.asarray( + ((8.0, 0.0, 3.5), (0.0, 8.0, 3.5), (0.0, 0.0, 1.0)), + dtype=np.float32, + ) + points = np.empty((size, size, 3), dtype=np.float32) + points[..., 0] = (x - intrinsics[0, 2]) / intrinsics[0, 0] * depth + points[..., 1] = (y - intrinsics[1, 2]) / intrinsics[1, 1] * depth + points[..., 2] = depth + field = ( + points, + intrinsics, + np.ones((size, size), dtype=bool), + (size, size), + 1.0, + ) + + result = mesh._build_mesh_object( + bpy.context, + source, + mesh.DEPTH_SURFACE, + 8, + surface_material, + field=field, + regions=((((0, 0), (64, 0), (64, 64), (0, 64)),),), + image_size=(64, 64), + keep_original=True, + ) + + assert mesh.BOUNDARY_ATTRIBUTE in result.data.attributes + assert mesh.DEPTH_PROTECTION_ATTRIBUTE in result.data.attributes + modifier = result.modifiers[0] + mesh.set_modifier_input( + modifier, + mesh.modifier_input_identifier(modifier.node_group, "Smooth"), + 0.0, + ) + evaluated = result.evaluated_get(bpy.context.evaluated_depsgraph_get()) + baseline_mesh = evaluated.to_mesh() + try: + baseline_z = np.asarray([vertex.co.z for vertex in baseline_mesh.vertices]) + baseline_min = baseline_z.min() + baseline_max = baseline_z.max() + finally: + evaluated.to_mesh_clear() + + mesh.set_modifier_input( + modifier, + mesh.modifier_input_identifier(modifier.node_group, "Thickness"), + 1.0, + ) + result.update_tag(refresh={"DATA"}) + bpy.context.view_layer.update() + evaluated = result.evaluated_get(bpy.context.evaluated_depsgraph_get()) + evaluated_mesh = evaluated.to_mesh() + try: + assert len(evaluated_mesh.polygons) > 0 + z = np.asarray([vertex.co.z for vertex in evaluated_mesh.vertices]) + assert np.isclose(z.max(), baseline_max, atol=1e-5) + assert np.isclose(z.min(), baseline_min - 1.0, atol=1e-5) + finally: + evaluated.to_mesh_clear() + + mesh.set_modifier_input( + modifier, + mesh.modifier_input_identifier(modifier.node_group, "Iterations"), + 4, + ) + mesh.set_modifier_input( + modifier, + mesh.modifier_input_identifier(modifier.node_group, "Smooth"), + 1.0, + ) + result.update_tag(refresh={"DATA"}) + bpy.context.view_layer.update() + evaluated = result.evaluated_get(bpy.context.evaluated_depsgraph_get()) + internal_mesh = evaluated.to_mesh() + try: + internal_positions = np.asarray( + [tuple(vertex.co) for vertex in internal_mesh.vertices] + ) + finally: + evaluated.to_mesh_clear() + + mesh.set_modifier_input( + modifier, + mesh.modifier_input_identifier(modifier.node_group, "Iterations"), + 0, + ) + external_group = bpy.data.node_groups.new("External O Smooth", "GeometryNodeTree") + external_group.is_modifier = True + mesh._interface_socket( + external_group, "Geometry", "OUTPUT", "NodeSocketGeometry" + ) + mesh._interface_socket( + external_group, "Geometry", "INPUT", "NodeSocketGeometry" + ) + external_iterations = mesh._interface_socket( + external_group, "Iterations", "INPUT", "NodeSocketInt" + ) + external_iterations.default_value = 4 + external_iterations.min_value = 0 + external_smooth = mesh._interface_socket( + external_group, "Smooth", "INPUT", "NodeSocketFloat" + ) + external_smooth.default_value = 1.0 + external_input = external_group.nodes.new("NodeGroupInput") + external_output = external_group.nodes.new("NodeGroupOutput") + external_geometry = mesh._expand_smooth( + external_group, external_input.outputs["Geometry"] + ) + external_group.links.new( + external_geometry, external_output.inputs["Geometry"] + ) + external_modifier = result.modifiers.new("External O Smooth", "NODES") + external_modifier.node_group = external_group + result.update_tag(refresh={"DATA"}) + bpy.context.view_layer.update() + evaluated = result.evaluated_get(bpy.context.evaluated_depsgraph_get()) + external_mesh = evaluated.to_mesh() + try: + external_positions = np.asarray( + [tuple(vertex.co) for vertex in external_mesh.vertices] + ) + finally: + evaluated.to_mesh_clear() + assert internal_positions.shape == external_positions.shape + assert np.allclose(internal_positions, external_positions, atol=1e-6) + + +def test_converted_color_images_are_new_and_keep_color_as_terminal_suffix(): + bpy.ops.wm.read_factory_settings(use_empty=True) + image = bpy.data.images.new("Poster", width=16, height=16) + source = bpy.data.objects.new("Poster", None) + source.empty_display_type = "IMAGE" + source.data = image + bpy.context.collection.objects.link(source) + + first = mesh._copy_color_image(source) + second = mesh._copy_color_image(source) + + assert first is not image + assert second is not image + assert first is not second + assert first.name.endswith(".color") + assert second.name.endswith(".color") + + +def test_local_cutout_builds_from_blender_image_pixels_without_a_job(): + bpy.ops.wm.read_factory_settings(use_empty=True) + image = bpy.data.images.new("Local Cutout", width=64, height=48, alpha=True) + pixels = np.ones((48, 64, 4), dtype=np.float32) + image.pixels.foreach_set(pixels.ravel()) + source = bpy.data.objects.new("Local Cutout", None) + source.empty_display_type = "IMAGE" + source.data = image + bpy.context.collection.objects.link(source) + source.select_set(True) + bpy.context.view_layer.objects.active = source + selection = "[[-8,8],[72,8],[72,40],[-8,40]]" + source_bounds = mesh.plane_bounds(source) + + with patch.object(mesh, "push_undo") as undo: + mesh._build_local_cutout( + SimpleNamespace(mesh_detail=16), + bpy.context, + source, + mesh.CUTOUT, + selection, + ) + + result = bpy.context.view_layer.objects.active + assert result is not source + assert result.type == "MESH" + assert result["anyimage_mesh_shape"] == mesh.CUTOUT + assert len(result.data.vertices) > 4 + assert len(result.data.polygons) > 1 + assert all(polygon.use_smooth for polygon in result.data.polygons) + vertex_x = [vertex.co.x for vertex in result.data.vertices] + assert min(vertex_x) < source_bounds[0] + assert max(vertex_x) > source_bounds[1] + assert bpy.data.objects.get(source.name) is source + assert result.data.materials[0].node_tree is not None + color_texture = next( + node + for node in result.data.materials[0].node_tree.nodes + if node.bl_idname == "ShaderNodeTexImage" + ) + assert color_texture.image is not image + copied = np.empty(48 * 64 * 4, dtype=np.float32) + color_texture.image.pixels.foreach_get(copied) + assert np.allclose(copied.reshape((48, 64, 4)), pixels) + undo.assert_called_once_with("Convert to Cutout") diff --git a/tests/test_server.py b/tests/test_server.py index 6313e5f..d95a7e8 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -8,7 +8,6 @@ from pathlib import Path from types import ModuleType, SimpleNamespace from unittest.mock import patch - PROJECT_ROOT = Path(__file__).resolve().parents[1] @@ -42,6 +41,8 @@ class ServerTest(unittest.TestCase): @classmethod def setUpClass(cls): cls.server_module = load_server_module() + importlib.import_module("server.jobs.balloon") + importlib.import_module("server.jobs.mesh") from blendjob import JobServer from server.model_manager import ModelManager, ModelSessionCache @@ -50,6 +51,7 @@ class ServerTest(unittest.TestCase): cls.ModelManager, cls.ModelSessionCache = ModelManager, ModelSessionCache cls.server = cls.server_module.server cls.app_module = sys.modules["server.app"] + def setUp(self): self.jobs_temp = tempfile.TemporaryDirectory() self.addCleanup(self.jobs_temp.cleanup) @@ -89,8 +91,7 @@ class ServerTest(unittest.TestCase): { "loaded": { "ben2": "", - "da3": "", - "moge2": "", + "geometry": "", "upscale": "", } }, @@ -99,9 +100,7 @@ class ServerTest(unittest.TestCase): self.assertNotIn("model_catalog", sys.modules) def create_cache(self): - cache = self.ModelSessionCache.__new__( - self.ModelSessionCache - ) + cache = self.ModelSessionCache.__new__(self.ModelSessionCache) cache.ben2_model = None cache.ben2_key = None cache.da3_model = None @@ -114,9 +113,7 @@ class ServerTest(unittest.TestCase): def create_server(self): source = self.server_module.server - server = self.JobServer( - "AnyImage Job Server", storage_root=self.storage_root - ) + server = self.JobServer("AnyImage Job Server", storage_root=self.storage_root) server.handlers.update(source.handlers) server.resource_factories.update(source.resource_factories) server.bind(self.storage_root) @@ -168,14 +165,19 @@ class ServerTest(unittest.TestCase): self.assertIs(first, second) self.assertEqual(loads, [("model", "cuda")]) - self.assertEqual(cache.snapshot()["da3"], "model") + self.assertEqual(cache.snapshot()["geometry"], "model") - def test_model_cache_keeps_only_the_most_recent_family(self): + def test_model_cache_keeps_three_categories(self): cache = self.create_cache() - cache.ben2_model = object() + ben2 = object() + old_moge = object() + upscale = object() + cache.ben2_model = ben2 cache.ben2_key = ("ben2", "cuda") - cache.moge_model = object() - cache.moge_key = ("moge", "cuda") + cache.moge_model = old_moge + cache.moge_key = ("old-moge", "cuda") + cache.upscale_model = upscale + cache.upscale_key = ("upscale", "cuda") device = SimpleNamespace(type="cuda") da3 = object() @@ -187,18 +189,21 @@ class ServerTest(unittest.TestCase): with patch.object(cache, "_load_da3", side_effect=load_da3): self.assertIs(cache.get_da3("da3", device), da3) + self.assertIs(cache.ben2_model, ben2) self.assertIsNone(cache.moge_model) - self.assertIsNone(cache.ben2_model) - - moge = object() + self.assertIs(cache.upscale_model, upscale) + self.assertEqual(cache.snapshot()["geometry"], "da3") + new_moge = object() fake_moge = ModuleType("server.models.onnx_moge2") - fake_moge.create_session = lambda _model_dir, _device: moge + fake_moge.create_session = lambda _model_dir, _device: new_moge with patch.dict(sys.modules, {"server.models.onnx_moge2": fake_moge}): - self.assertIs(cache.get_moge2("moge", "cuda"), moge) + self.assertIs(cache.get_moge2("new-moge", "cuda"), new_moge) + self.assertIs(cache.ben2_model, ben2) self.assertIsNone(cache.da3_model) - self.assertIsNone(cache.ben2_model) + self.assertIs(cache.upscale_model, upscale) + self.assertEqual(cache.snapshot()["geometry"], "new-moge") def test_upscale_model_is_reused_for_matching_runtime_key(self): loads = [] @@ -254,13 +259,11 @@ class ServerTest(unittest.TestCase): { "download-model", "download-required-models", - "analyze-balloon", "refine-image-mask", "remove-background", "upscale-image", "extract-frames", - "generate-depth", - "generate-surface", + "generate-mesh", "debug-da3", "debug-moge2", "debug-ben2", @@ -438,6 +441,54 @@ class ServerTest(unittest.TestCase): self.assertEqual(tuple(result[0, 0]), (20, 40, 60, 64)) self.assertEqual(tuple(result[3, 3]), (0, 0, 0, 0)) + def test_polygon_selection_mask_uses_even_odd_fill_for_self_intersection(self): + image_mask = sys.modules["server.jobs.image_mask"] + polygon = ( + (1, 1), + (9, 1), + (9, 9), + (1, 9), + (1, 1), + (3, 3), + (3, 7), + (7, 7), + (7, 3), + (3, 3), + (1, 1), + ) + + mask = image_mask.polygon_selection_mask((10, 10), polygon) + + self.assertTrue(mask[2, 2]) + self.assertFalse(mask[4, 4]) + self.assertEqual(int(mask.sum()), 48) + + def test_polygon_selection_mask_nonzero_keeps_same_direction_overlap(self): + image_mask = sys.modules["server.jobs.image_mask"] + polygon = ( + (1, 1), + (7, 1), + (7, 7), + (1, 7), + (1, 1), + (4, 1), + (10, 1), + (10, 7), + (4, 7), + (4, 1), + (1, 1), + ) + + even_odd = image_mask.polygon_selection_mask((12, 9), polygon) + nonzero = image_mask.polygon_selection_mask( + (12, 9), polygon, "NONZERO" + ) + + self.assertTrue(even_odd[3, 2]) + self.assertFalse(even_odd[3, 5]) + self.assertTrue(nonzero[3, 2]) + self.assertTrue(nonzero[3, 5]) + self.assertTrue(nonzero[3, 8]) def test_image_mask_canvas_clears_rgba_outside_the_selection(self): import numpy as np from PIL import Image @@ -475,6 +526,288 @@ class ServerTest(unittest.TestCase): self.assertEqual(result.shape, (2, 2, 4)) self.assertTrue(np.all(result[:, :, 3] == 255)) + def test_mesh_refine_selection_keeps_source_size_and_selection_alpha(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "source.png" + pixels = np.full((6, 8, 4), (20, 40, 60, 255), dtype=np.uint8) + Image.fromarray(pixels, mode="RGBA").save(source_path) + job_directory = self.storage_root / "job" + job_directory.mkdir() + context = SimpleNamespace( + directory=job_directory, + resource=lambda _name: SimpleNamespace( + directory=lambda _model: Path("model") + ), + check_cancelled=lambda: None, + ) + + def passthrough(path, *_args): + return Image.open(path).convert("RGBA"), 0.0, 0.0 + + output_path = job_directory / "image.0001.png" + with patch.object(mesh_job, "remove_image_background", side_effect=passthrough): + mesh_job._write_refined_color( + context, + {"refine_model": "BEN2_BASE", "device": "cpu"}, + source_path, + ((1, 1), (6, 1), (1, 5)), + output_path, + ) + result = np.asarray(Image.open(output_path).convert("RGBA")) + + self.assertEqual(result.shape, (6, 8, 4)) + self.assertEqual(tuple(result[1, 1]), (20, 40, 60, 255)) + self.assertEqual(tuple(result[5, 7]), (0, 0, 0, 0)) + + def test_mesh_job_keeps_original_ai_input_separate_from_geometry_mask(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "mesh-source.png" + pixels = np.full((6, 8, 4), (20, 40, 60, 255), dtype=np.uint8) + Image.fromarray(pixels, mode="RGBA").save(source_path) + job_directory = self.storage_root / "mesh-job" + context = SimpleNamespace( + directory=job_directory, + progress=lambda *_args: None, + check_cancelled=lambda: None, + resource=lambda _name: object(), + ) + normal = np.zeros((1, 6, 8, 3), dtype=np.float32) + normal[..., 0] = 1.0 + prediction = SimpleNamespace(normal=normal) + received = [] + received_images = [] + + def infer(_manager, parameters, images, **_kwargs): + received.append(parameters) + received_images.extend(images) + return prediction + + with patch.object(mesh_job.moge2, "infer", side_effect=infer): + result = mesh_job.run( + context, + { + "input": str(source_path), + "shape": "BALLOON", + "polygon": ((2, 1), (7, 1), (7, 5), (2, 5)), + "mesh_detail": 64, + "invert": False, + "device": "cpu", + "normal_model": "MOGE2_VITS_NORMAL", + "normal_resolution_level": 5, + "normal_analysis": True, + }, + ) + + color_path = job_directory / "image.0001.png" + color_pixels = np.asarray(Image.open(color_path).convert("RGBA")) + normal_pixels = np.asarray( + Image.open(job_directory / result["normal"]).convert("RGB") + ) + self.assertEqual(result, {"normal": "normal.png"}) + self.assertTrue(color_path.is_file()) + self.assertFalse((job_directory / "contour.json").exists()) + self.assertEqual(received[0]["model"], "MOGE2_VITS_NORMAL") + self.assertEqual(received_images, [source_path]) + self.assertEqual(tuple(color_pixels[3, 4, :3]), (20, 40, 60)) + self.assertGreater(int(color_pixels[3, 4, 3]), 240) + self.assertLess(int(color_pixels[0, 0, 3]), 5) + self.assertEqual(tuple(normal_pixels[0, 0]), (255, 127, 127)) + self.assertEqual(tuple(normal_pixels[3, 4]), (255, 127, 127)) + + def test_mesh_job_skips_moge_normal_when_option_is_disabled(self): + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "mesh-no-normal-source.png" + Image.new("RGBA", (8, 6), (20, 40, 60, 255)).save(source_path) + job_directory = self.storage_root / "mesh-no-normal-job" + context = SimpleNamespace( + directory=job_directory, + progress=lambda *_args: None, + check_cancelled=lambda: None, + resource=lambda _name: object(), + ) + + with patch.object(mesh_job, "_write_camera_normal") as write_normal: + result = mesh_job.run( + context, + { + "input": str(source_path), + "shape": "BALLOON", + "polygon": ((0, 0), (8, 0), (8, 6), (0, 6)), + "mesh_detail": 64, + "normal_analysis": False, + "invert": False, + "device": "cpu", + }, + ) + + self.assertEqual(result, {}) + write_normal.assert_not_called() + self.assertFalse((job_directory / "normal.png").exists()) + + def test_mesh_depth_balloon_analyzes_full_source_then_applies_selection(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "mesh-depth-balloon-source.png" + Image.new("RGBA", (6, 6), (20, 40, 60, 255)).save(source_path) + job_directory = self.storage_root / "mesh-depth-balloon-job" + job_directory.mkdir() + context = SimpleNamespace( + directory=job_directory, + check_cancelled=lambda: None, + resource=lambda _name: object(), + ) + received = [] + + def infer(_manager, _parameters, images, **_kwargs): + received.extend(images) + depth = np.linspace(1.0, 2.0, 36, dtype=np.float32).reshape(6, 6) + normal = np.zeros((1, 6, 6, 3), dtype=np.float32) + normal[..., 0] = 1.0 + return SimpleNamespace( + depth=depth[None], + mask=np.ones((1, 6, 6), dtype=bool), + normal=normal, + ) + + with patch.object(mesh_job.moge2, "infer", side_effect=infer): + depth_path, normal_path = mesh_job._write_balloon_depth( + context, + {"device": "cpu"}, + source_path, + ((2, 2), (10, 2), (2, 10)), + selection_size=(12, 12), + include_normal=True, + ) + + self.assertEqual(received, [source_path]) + with np.load(depth_path, allow_pickle=False) as data: + self.assertEqual(tuple(data["origin"]), (0, 0)) + self.assertEqual(tuple(data["crop_size"]), (6, 6)) + self.assertFalse(bool(data["valid"][0, 0])) + self.assertTrue(bool(data["valid"][2, 2])) + normal = np.asarray(Image.open(normal_path).convert("RGB")) + self.assertEqual(tuple(normal[0, 0]), (255, 127, 127)) + self.assertEqual(tuple(normal[2, 2]), (255, 127, 127)) + + def test_mesh_job_uses_upscale_source_only_for_analysis(self): + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + display_path = self.storage_root / "upscaled.png" + analysis_path = self.storage_root / "original.png" + Image.new("RGBA", (12, 8), (200, 20, 40, 255)).save(display_path) + Image.new("RGBA", (3, 2), (10, 80, 160, 255)).save(analysis_path) + job_directory = self.storage_root / "mesh-upscale-source-job" + context = SimpleNamespace( + directory=job_directory, + progress=lambda *_args: None, + check_cancelled=lambda: None, + ) + field_path = job_directory / "field.npz" + + with patch.object( + mesh_job, + "_write_surface_depth", + return_value=(field_path, None), + ) as write_surface: + result = mesh_job.run( + context, + { + "input": str(display_path), + "analysis_input": str(analysis_path), + "shape": "DEPTH_SURFACE", + "normal_analysis": False, + "video_frames": 1, + }, + ) + + self.assertEqual(result, {"field": "field.npz"}) + self.assertEqual(write_surface.call_args.args[2], analysis_path) + with Image.open(job_directory / "image.0001.png") as color: + self.assertEqual(color.size, (12, 8)) + + def test_mesh_masked_color_keeps_same_direction_overlap(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "opaque-overlap.png" + output_path = self.storage_root / "cutout-overlap.png" + Image.new("RGBA", (12, 9), (20, 40, 60, 255)).save(source_path) + polygon = ( + (1, 1), + (7, 1), + (7, 7), + (1, 7), + (1, 1), + (4, 1), + (10, 1), + (10, 7), + (4, 7), + (4, 1), + (1, 1), + ) + + mesh_job._write_masked_color(source_path, polygon, output_path) + alpha = np.asarray(Image.open(output_path).convert("RGBA"))[:, :, 3] + + self.assertGreater(int(alpha[3, 2]), 200) + self.assertGreater(int(alpha[3, 5]), 200) + self.assertGreater(int(alpha[3, 8]), 200) + def test_mesh_masked_color_preserves_nonzero_winding_hole(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "opaque-hole.png" + output_path = self.storage_root / "cutout-hole.png" + Image.new("RGBA", (10, 10), (20, 40, 60, 255)).save(source_path) + polygon = ( + (1, 1), + (9, 1), + (9, 9), + (1, 9), + (1, 1), + (3, 3), + (3, 7), + (7, 7), + (7, 3), + (3, 3), + (1, 1), + ) + + mesh_job._write_masked_color(source_path, polygon, output_path) + alpha = np.asarray(Image.open(output_path).convert("RGBA"))[:, :, 3] + + self.assertGreater(int(alpha[2, 2]), 200) + self.assertLess(int(alpha[4, 4]), 32) + + def test_mesh_masked_color_antialiases_the_cutout_edge(self): + import numpy as np + from PIL import Image + + mesh_job = sys.modules["server.jobs.mesh"] + source_path = self.storage_root / "opaque.png" + output_path = self.storage_root / "cutout.png" + Image.new("RGBA", (16, 16), (20, 40, 60, 255)).save(source_path) + + mesh_job._write_masked_color( + source_path, + ((1, 1), (14, 1), (1, 14)), + output_path, + ) + alpha = np.asarray(Image.open(output_path).convert("RGBA"))[:, :, 3] + def test_balloon_job_sends_the_masked_crop_to_ben2(self): import numpy as np from PIL import Image @@ -587,6 +920,7 @@ class ServerTest(unittest.TestCase): np.testing.assert_allclose(farther[3], original[3], atol=2e-4) np.testing.assert_allclose(rescaled[0], original[0], atol=2e-4) np.testing.assert_allclose(rescaled[3], original[3], atol=2e-4) + def test_balloon_depth_job_sends_masked_crop_only_to_moge(self): import numpy as np from PIL import Image diff --git a/tools/validate_balloon_asset.py b/tools/validate_balloon_asset.py index 45a59cf..7e0248c 100644 --- a/tools/validate_balloon_asset.py +++ b/tools/validate_balloon_asset.py @@ -99,13 +99,14 @@ def main(): material_group = material_node_group( surface_group_name=BALLOON_MATERIAL_NODE_GROUP_NAME, ) - normal_map = next( - node + normal_maps = { + node.space for node in material_group.nodes if node.bl_idname == "ShaderNodeNormalMap" - ) - assert normal_map.space == "TANGENT" - assert abs(input_socket(material_group, "Normal Strength").default_value - 0.5) < 1e-6 + } + assert normal_maps == {"TANGENT", "OBJECT"} + assert input_socket(material_group, "Object Space").default_value is False + assert input_socket(material_group, "Normal Scale").default_value == 1.0 source_image = bpy.data.images.new("Reuse Source", 2, 1) shared_normal = bpy.data.images.new("Shared Normal", 2, 1) @@ -115,8 +116,6 @@ def main(): material = bpy.data.materials.new("Unrelated Material Name") mark_balloon_material(material, source_image, shared_normal) assert reusable_balloon_material( - "unused", - "unused", source_image, normal_required=True, ) is material diff --git a/tools/validate_depth_plane_asset.py b/tools/validate_depth_plane_asset.py index cf1708e..c5ae357 100644 --- a/tools/validate_depth_plane_asset.py +++ b/tools/validate_depth_plane_asset.py @@ -285,14 +285,37 @@ def main(): and node.node_tree == surface_group ] assert len(surface_group_nodes) == 1 - assert surface_group_nodes[0].name == "O DA3 Surface Layer" + assert surface_group_nodes[0].name == "O Image Layer" surface_inputs = [ item.name for item in surface_group.interface.items_tree if item.item_type == "SOCKET" and item.in_out == "INPUT" ] - assert surface_inputs == ["Color", "Alpha", "Bump Scale"] + assert surface_inputs == [ + "Color", + "Alpha", + "Normal", + "Normal Scale", + "Object Space", + "Bump Scale", + ] + normal_input = next( + item + for item in surface_group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal" + ) + assert tuple(normal_input.default_value) == (0.5, 0.5, 1.0, 1.0) + normal_scale = next( + item + for item in surface_group.interface.items_tree + if item.item_type == "SOCKET" + and item.in_out == "INPUT" + and item.name == "Normal Scale" + ) + assert normal_scale.default_value == 1.0 surface_outputs = [ item.name for item in surface_group.interface.items_tree diff --git a/tools/validate_moge_surface.py b/tools/validate_moge_surface.py index 5a644dc..9ec4c56 100644 --- a/tools/validate_moge_surface.py +++ b/tools/validate_moge_surface.py @@ -5,7 +5,6 @@ from pathlib import Path import bpy import numpy as np - PROJECT_ROOT = Path(__file__).parents[1] sys.path.insert(0, str(PROJECT_ROOT / "src")) sys.path.insert( @@ -43,7 +42,7 @@ def write_test_normal_texture(normal, output_path): def main(): bpy.ops.wm.read_factory_settings(use_empty=True) - arguments = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else [] + arguments = sys.argv[sys.argv.index("--") + 1 :] if "--" in sys.argv else [] if arguments: validate_real_result(Path(arguments[0]), Path(arguments[1])) return @@ -142,14 +141,14 @@ def main(): depth_edge_socket = next( item for item in surface.modifiers[0].node_group.interface.items_tree - if getattr(item, "in_out", None) == "INPUT" - and item.name == "Depth Continuity" + if getattr(item, "in_out", None) == "INPUT" and item.name == "Depth Continuity" ) assert depth_edge_socket.default_value == 50.0 assert depth_edge_socket.subtype == "PERCENTAGE" assert depth_edge_socket.structure_type == "SINGLE" culling_panel = next( - item for item in surface.modifiers[0].node_group.interface.items_tree + item + for item in surface.modifiers[0].node_group.interface.items_tree if item.item_type == "PANEL" and item.name == "Culling" ) assert [item.name for item in culling_panel.interface_items] == [ @@ -157,7 +156,8 @@ def main(): "Mask", ] input_sockets = [ - item for item in surface.modifiers[0].node_group.interface.items_tree + item + for item in surface.modifiers[0].node_group.interface.items_tree if item.item_type == "SOCKET" and item.in_out == "INPUT" ] assert all(item.structure_type == "SINGLE" for item in input_sockets) @@ -179,22 +179,34 @@ def main(): assert MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME in bpy.data.node_groups material_group = bpy.data.node_groups[MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME] assert any(node.bl_idname == "ShaderNodeBump" for node in material_group.nodes) - normal_map = next( - node for node in material_group.nodes - if node.bl_idname == "ShaderNodeNormalMap" - ) - assert normal_map.space == "OBJECT" + normal_maps = [ + node for node in material_group.nodes if node.bl_idname == "ShaderNodeNormalMap" + ] + assert {node.space for node in normal_maps} == {"TANGENT", "OBJECT"} normal_strength = next( - item for item in material_group.interface.items_tree - if getattr(item, "in_out", None) == "INPUT" - and item.name == "Normal Strength" + item + for item in material_group.interface.items_tree + if getattr(item, "in_out", None) == "INPUT" and item.name == "Normal Scale" ) assert normal_strength.default_value == 1.0 assert normal_strength.min_value == 0.0 - assert normal_map.inputs["Strength"].is_linked + normal_input = next( + item + for item in material_group.interface.items_tree + if getattr(item, "in_out", None) == "INPUT" and item.name == "Normal" + ) + assert tuple(normal_input.default_value) == (0.5, 0.5, 1.0, 1.0) + assert all(node.inputs["Strength"].is_linked for node in normal_maps) + layer = next( + node + for node in surface.data.materials[0].node_tree.nodes + if node.bl_idname == "ShaderNodeGroup" and node.node_tree == material_group + ) + assert layer.inputs["Object Space"].default_value is True assert not surface.data.has_custom_normals color_texture = next( - node for node in surface.data.materials[0].node_tree.nodes + node + for node in surface.data.materials[0].node_tree.nodes if node.bl_idname == "ShaderNodeTexImage" and node.image == foreground ) assert color_texture.image.name == "MoGe Surface Test.color" @@ -202,7 +214,8 @@ def main(): bpy.context.scene ) normal_texture = next( - node for node in surface.data.materials[0].node_tree.nodes + node + for node in surface.data.materials[0].node_tree.nodes if node.bl_idname == "ShaderNodeTexImage" and node.image.colorspace_settings.name == "Non-Color" ) @@ -243,12 +256,17 @@ def validate_real_result(results_path, image_path): assert not surface.data.has_custom_normals material_group = bpy.data.node_groups[MOGE_SURFACE_MATERIAL_NODE_GROUP_NAME] assert any(node.bl_idname == "ShaderNodeBump" for node in material_group.nodes) - normal_map = next( - node for node in material_group.nodes - if node.bl_idname == "ShaderNodeNormalMap" + normal_maps = [ + node for node in material_group.nodes if node.bl_idname == "ShaderNodeNormalMap" + ] + assert {node.space for node in normal_maps} == {"TANGENT", "OBJECT"} + assert all(node.inputs["Strength"].is_linked for node in normal_maps) + layer = next( + node + for node in surface.data.materials[0].node_tree.nodes + if node.bl_idname == "ShaderNodeGroup" and node.node_tree == material_group ) - assert normal_map.space == "OBJECT" - assert normal_map.inputs["Strength"].is_linked + assert layer.inputs["Object Space"].default_value is True attribute_names = set(surface.data.attributes.keys()) assert {"o_depth", "o_mask"}.issubset(attribute_names) assert "o_depth_discontinuity" in attribute_names