Bug in clipping: index is out of bounds for dimension with size 0

[Algomorph]

🐛 Bugs / Unexpected behaviors

Rasterizing a torus throws a bug in clipping if there are no clipped faces.

Instructions To Reproduce the Issue:

Run this code (very similar to #631, but with subtle differences):

import numpy as np
import pytorch3d.utils
from pytorch3d.renderer.cameras import PerspectiveCameras
from pytorch3d.renderer.mesh import MeshRasterizer, RasterizationSettings
import torch

intrinsic_matrix = np.array([[575.548, 0.,     323.172, 0.],
                             [0.,      577.46, 236.417, 0.],
                             [0.,      0.,     1.,      0.],
                             [0.,      0., 0., 1.        ]])
intrinsic_matrix_torch = torch.from_numpy(intrinsic_matrix).cuda().unsqueeze(0)

torch_device = torch.device("cuda:0")
meshes_torch3d = pytorch3d.utils.torus(1.0, 20.0, 32, 16, device=torch_device)
image_size = (480, 640)

camera_rotation = torch.eye(3, dtype=torch.float32, device=torch_device)[None]  # (1, 3, 3)
camera_translation = torch.zeros(1, 3, dtype=torch.float32, device=torch_device)  # (1, 3)

cameras = PerspectiveCameras(device=torch_device,
                             R=camera_rotation,
                             T=camera_translation,
                             K=intrinsic_matrix_torch,
                             image_size=[image_size])

rasterization_settings = RasterizationSettings(image_size=image_size, cull_backfaces=True, cull_to_frustum=True, z_clip_value=4.0)
rasterizer = MeshRasterizer(cameras, raster_settings=rasterization_settings)
fragments = rasterizer.forward(meshes_torch3d)

Output

This raises the following error (disregard line numbers w/ respect to above code, early stacks are unimportant):

File "/home/algomorph/Workbench/NeuralTracking/pipeline/rendering_test.py", line 45, in main
    fragments = rasterizer.forward(meshes_torch3d)
  File "/home/algomorph/.local/lib/python3.8/site-packages/pytorch3d/renderer/mesh/rasterizer.py", line 164, in forward
    pix_to_face, zbuf, bary_coords, dists = rasterize_meshes(
  File "/home/algomorph/.local/lib/python3.8/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 246, in rasterize_meshes
    outputs = convert_clipped_rasterization_to_original_faces(
  File "/home/algomorph/.local/lib/python3.8/site-packages/pytorch3d/renderer/mesh/clip.py", line 652, in convert_clipped_rasterization_to_original_faces
    faces_clipped_to_unclipped_idx[pix_to_face_clipped],
IndexError: index is out of bounds for dimension with size 0

Version Info

Version info:
pytorch3d built from commit cc08c6b (from this this morning)
pytorch3d.version : '0.4.0'
torch.version: '1.7.0a0+57bffc3' ( == 1.7.1 release, built from source)
torch.version.cuda: '11.1'
Python: 3.8.5
OS: Ubuntu 20.04

[Algomorph]

I was able to work around the bug by changing:

pytorch3d/pytorch3d/renderer/mesh/clip.py

Lines 642 to 643 in 24ee279

	if faces_clipped_to_unclipped_idx is None:
	return pix_to_face_clipped, bary_coords_clipped

to

    if faces_clipped_to_unclipped_idx is None or\
        (len(faces_clipped_to_unclipped_idx.shape) == 1 and faces_clipped_to_unclipped_idx.shape[0] == 0):
        return pix_to_face_clipped, bary_coords_clipped

However, I'm not sure if this fix is logically sound.

[HectorAnadon]

I am having the same issue when using look_at_rotation instead of look_at_view_transform.

[wufeim]

I had the same issue when I changed the at parameter in look_at_rotation to a non-zero position. @Algomorph's solution seems to fix my issue.

[nikhilaravi]

@Algomorph the key differences I see compared to #631 are that you have changed the torus initialization options and the z clip value. Have you changed anything else?

[Algomorph]

@nikhilaravi : beside the z-clip value, I only changed the dimensions of the torus, i.e.:

meshes_torch3d = pytorch3d.utils.torus(1.0, 20.0, 32, 16, device=torch_device)

[nikhilaravi]

This issue should now be fixed by c75ca04.