fix: handle alphaToCoverage in shadow map generation

[mattverse]

Related issue: #30462

Description
When materials use alphaToCoverage shadow map used to fail to respect the transparency. This occurs because shadow generation relies on alphaTest to discard fragments, but alphaToCoverage bypasses alphaTest and uses multisampling instead. As a result, shadows render opaque even when the material appears transparent.

This PR fixes this by adding a check if original material uses alphaToCoverage then setting the shadow material's alphatest manually.

Before:

After this change has been applied:

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	336.08 78.29	336.13 78.3	+47 B +12 B
WebGPU	525.17 146.28	525.17 146.28	+0 B +0 B
WebGPU Nodes	524.63 146.18	524.63 146.18	+0 B +0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	465.15 112.18	465.2 112.19	+47 B +12 B
WebGPU	597.97 162.56	597.97 162.56	+0 B +0 B
WebGPU Nodes	553.1 152.04	553.1 152.04	+0 B +0 B

[Mugen87]

Why did you introduce this new if statement?

[mattverse]

Previously, when a user provided customDepthMaterial, the original code overwrote properties like alphaMap or alphaTest of the customMaterial without this if statement

[Mugen87]

Do you mind testing this version of getDepthMaterial()? Only two lines need to be changed and the codesandbox works:

function getDepthMaterial( object, material, light, type ) {

	let result = null;

	const customMaterial = ( light.isPointLight === true ) ? object.customDistanceMaterial : object.customDepthMaterial;

	if ( customMaterial !== undefined ) {

		result = customMaterial;

	} else {

		result = ( light.isPointLight === true ) ? _distanceMaterial : _depthMaterial;

		if ( ( renderer.localClippingEnabled && material.clipShadows === true && Array.isArray( material.clippingPlanes ) && material.clippingPlanes.length !== 0 ) ||
			( material.displacementMap && material.displacementScale !== 0 ) ||
			( material.alphaMap && material.alphaTest > 0 ) ||
			( material.map && material.alphaTest > 0 ) ||
			( material.alphaToCoverage === true ) ) {

			// in this case we need a unique material instance reflecting the
			// appropriate state

			const keyA = result.uuid, keyB = material.uuid;

			let materialsForVariant = _materialCache[ keyA ];

			if ( materialsForVariant === undefined ) {

				materialsForVariant = {};
				_materialCache[ keyA ] = materialsForVariant;

			}

			let cachedMaterial = materialsForVariant[ keyB ];

			if ( cachedMaterial === undefined ) {

				cachedMaterial = result.clone();
				materialsForVariant[ keyB ] = cachedMaterial;
				material.addEventListener( 'dispose', onMaterialDispose );

			}

			result = cachedMaterial;

		}

	}

	result.visible = material.visible;
	result.wireframe = material.wireframe;

	if ( type === VSMShadowMap ) {

		result.side = ( material.shadowSide !== null ) ? material.shadowSide : material.side;

	} else {

		result.side = ( material.shadowSide !== null ) ? material.shadowSide : shadowSide[ material.side ];

	}

	result.alphaMap = material.alphaMap;
	result.alphaTest = ( material.alphaToCoverage === true ) ? 0.5 : material.alphaTest;
	result.map = material.map;

	result.clipShadows = material.clipShadows;
	result.clippingPlanes = material.clippingPlanes;
	result.clipIntersection = material.clipIntersection;

	result.displacementMap = material.displacementMap;
	result.displacementScale = material.displacementScale;
	result.displacementBias = material.displacementBias;

	result.wireframeLinewidth = material.wireframeLinewidth;
	result.linewidth = material.linewidth;

	if ( light.isPointLight === true && result.isMeshDistanceMaterial === true ) {

		const materialProperties = renderer.properties.get( result );
		materialProperties.light = light;

	}

	return result;

}

[Mugen87]

To further explain the issue: alphaToCoverage requires a MSAA context which is not available when rendering shadow maps. The idea is to "emulate" the transparency by setting alphaTest to a fixed value. The resulting shadows should be a good approximation.

[Mugen87]

Do you mind sharing a live example that demonstrates the issue? That would make it easier to verify the suggested change.

[mattverse]

@Mugen87 This is a good example from the original author of the issue https://codesandbox.io/p/sandbox/exciting-water-hz8k92

[Mugen87]

Closing in favor of #30871.