Dataloaders Tutorial image-grid and visualization questions

[albertotono]

I have 3 questions for the in https://pytorch3d.org/tutorials/dataloaders_ShapeNetCore_R2N2 tutorial,

1. dict numpy
   A.

r2n2_renderings = r2n2_dataset[10,[1,2]]
image_grid(r2n2_renderings.numpy(), rows=1, cols=2, rgb=True)

it returns

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-55-166b469a0921> in <module>
      1 r2n2_renderings = r2n2_dataset[10,[1,2]]
----> 2 image_grid(r2n2_renderings.numpy(), rows=1, cols=2, rgb=True)

**AttributeError: 'dict' object has no attribute 'numpy'**

I am able to visualize if I do an extra step, r2n2_renderings = r2n2_renderings["images"]

r2n2_renderings = r2n2_dataset[10]
r2n2_renderings = r2n2_renderings["images"]
r2n2_renderings = r2n2_renderings[1]
plt.imshow( r2n2_renderings)

the same with the batch.

batch_renderings = r2n2_batch["images"] # (N, V, H, W, 3), and in this case V is 1.
batch_renderings = batch_renderings[0]
plt.imshow( batch_renderings.squeeze())

B.
and at the end for visualizing the voxels

vox_render = render_cubified_voxels(r2n2_model["voxels"], device=device)
image_grid(vox_render.cpu().numpy(), rows=1, cols=2, rgb=True)

I got this

vox_render = render_cubified_voxels(r2n2_model["voxels"], device=device)

image_grid(vox_render.cpu().numpy(), rows=1, cols=2, rgb=True)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-66-fd759f855679> in <module>
----> 1 vox_render = render_cubified_voxels(r2n2_model["voxels"], device=device)
      2 image_grid(vox_render.cpu().numpy(), rows=1, cols=2, rgb=True)

KeyError: 'voxels'

maybe they are correlated

2. Visualization confirmation

I would like to share also some images to make sure they are correct.
If so why are they rendered in this way?

3. How to obtain all the meshes without texture?

Now it takes from shapenet_base.py that if it doesn't have a texture, it will return white. But what if I want to return all images without texture?!

I tried to modify

shapenet_dataset = ShapeNetCore(SHAPENET_PATH)
and r2n2_dataset = R2N2("train", SHAPENET_PATH, R2N2_PATH, SPLITS_PATH, return_voxels=True) 

to

shapenet_dataset = ShapeNetCore(SHAPENET_PATH, load_textures=False)
and r2n2_dataset = R2N2("train", SHAPENET_PATH, R2N2_PATH, SPLITS_PATH, return_voxels=True, load_textures=False)

but I got this error later

UnboundLocalError: local variable 'textures' referenced before assignment

when I run this

shapenet_model = shapenet_dataset[6]
print("This model belongs to the category " + shapenet_model["synset_id"] + ".")
print("This model has model id " + shapenet_model["model_id"] + ".")
print("This model is a " + shapenet_model["label"])
model_verts, model_faces = shapenet_model["verts"], shapenet_model["faces"]

and the texture displays in their colors not white.

Thanks again for the help and availability in advance

[apugoneappu]

For the 3rd query, the following change to the _load_mesh function in pytorch3d/datasets/shapenet_base.py seems to work to get the meshes without the texture:

def _load_mesh(self, model_path) -> Tuple:
	verts, faces, aux = load_obj(
		model_path,
		create_texture_atlas=self.load_textures,
		load_textures=self.load_textures,
		texture_atlas_size=self.texture_resolution,
	)
+	textures = None
	if self.load_textures:
		textures = aux.texture_atlas
		# Some meshes don't have textures. In this case
		# create a white texture map
-		if textures is None:
-			textures = verts.new_ones(
-				faces.verts_idx.shape[0],
-				self.texture_resolution,
-				self.texture_resolution,
-				3,
-			)
+	if textures is None:
+		textures = verts.new_ones(
+			faces.verts_idx.shape[0],
+			self.texture_resolution,
+			self.texture_resolution,
+			3,
+		)
	return verts, faces.verts_idx, textures

Unfortunately, I'm not sure if this would break other things :(

[wangsen1312]

@albertotono Any update for the visualization questions? This kind of results seems wrong. I have encounter the similar problem.

[YefanZhou]

I have encountered a similar problem with the "Visualization confirmation". Images rendered by the R2N2 dataset (second row) render function look weird compared with original images (first row). The image size of the rasterization setting is 137.

[Zxlan]

@albertotono Hello, I also want to learn this tutorial https://pytorch3d.org/tutorials/dataloaders_ShapeNetCore_R2N2, can you share your dataset, I can't download it online, thank you very much.

[xinyuwang9906]

for 1B, you need to change the voxels_rel_path to the r2n2 voxel folder name you downloaded for the r2n2 dataset when you initiate the R2N2 instance like this

r2n2_dataset = R2N2("train", SHAPENET_PATH, R2N2_PATH, SPLITS_PATH, return_voxels=True,voxels_rel_path = "ShapeNetVox32")