Skip to content

Lip Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation #3003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rittik9 opened this issue Mar 11, 2025 · 6 comments · May be fixed by #3090
Open

Lip Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation #3003

rittik9 opened this issue Mar 11, 2025 · 6 comments · May be fixed by #3090
Assignees
@rittik9
Labels
enhancement New feature or request New metric
Milestone
future

Comments

@rittik9
Copy link
Contributor

rittik9 commented Mar 11, 2025

🚀 Feature

I would like to propose incorporating an essential evaluation metric for 3D talking heads into the TorchMetrics library: Lip Vertex Error (LVE) . This metric is widely used in speech-driven facial animation research to assess lip synchronization accuracy respectively.

Motivation

Current TorchMetrics offerings lack dedicated metrics for evaluating 3D talking heads, particularly in assessing the quality of lip synchronization . I think this metric also fits in multimodal folder of this library.

Pitch

The Lip Vertex Error (LVE) metric evaluates the quality of lip synchronization in 3D facial animations by measuring the maximum Euclidean distance (L2 error) between corresponding lip vertices of the generated and ground truth meshes for each frame. This metric assesses how accurately the animated lip movements align with the intended speech input.

References

Paper: MeshTalk

Additional context

I would like to open a PR for the same.
@rittik9 rittik9 added the enhancement New feature or request label Mar 11, 2025
@rittik9 rittik9 changed the title Proposal to Add Lip Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation Mar 11, 2025
@rittik9 rittik9 changed the title Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation Lip Vertex Error (LVE) Metrics for 3D Talking Heads Evaluation Mar 11, 2025
@SkafteNicki SkafteNicki added this to the future milestone Mar 12, 2025
@SkafteNicki
Copy link
Member

@rittik9 sound good to me, feel free to open a PR !

@hugleecool
Copy link

when ,very need,

@rittik9
Copy link
Contributor Author

rittik9 commented Apr 5, 2025

Hi @hugleecool I'm occupied with some other stuff at this moment, So I'm planning this for mid April. But if you are willing to open a pr, please go ahead, I'll be happy to help :)

@hugleecool
Copy link

hugleecool commented Apr 5, 2025 via email

@SkafteNicki
Copy link
Member

@rittik9 I am wrong for thinking the metric is simply calculated as:

def lip_vertex_error(vertices_pred: torch.Tensor, vertices_gt: torch.Tensor, mouth_map: list[int]) -> torch.Tensor:
    """
    Computes Lip Vertex Error (LVE) as used in CodeTalker.

    Args:
        vertices_pred (Tensor): (T, V, 3) predicted vertices
        vertices_gt (Tensor): (T, V, 3) ground truth vertices
        mouth_map (list[int]): list of mouth vertex indices

    Returns:
        Tensor: scalar LVE value
    """
    # Difference only for mouth vertices
    diff = vertices_gt[:, mouth_map, :] - vertices_pred[:, mouth_map, :]  # (T, M, 3)
    sq_dist = torch.sum(diff ** 2, dim=-1)  # (T, M)
    max_per_frame = torch.max(sq_dist, dim=1).values  # (T,)
    return torch.mean(max_per_frame)

if so I can probably do it fairly easy within the next couple of days

@rittik9
Copy link
Contributor Author

rittik9 commented Apr 7, 2025 •
edited
Loading

@SkafteNicki Sure pls....
You can use this as a reference or this one

This was referenced May 11, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rittik9 rittik9

Labels
enhancement New feature or request New metric
Projects
None yet
Milestone
future
Development

Successfully merging a pull request may close this issue.

New Metric: Lip Vertex Error rittik9/torchmetrics
3 participants
@hugleecool @SkafteNicki @rittik9