Skip to content

inference extension not correctly scrape pod metrics #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Kuromesi opened this issue Feb 19, 2025 · 0 comments · Fixed by #366
Closed

inference extension not correctly scrape pod metrics #365

Kuromesi opened this issue Feb 19, 2025 · 0 comments · Fixed by #366
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Kuromesi
Copy link
Contributor

What happened:
Currently inference extension request url := fmt.Sprintf("http://%s/metrics", existing.Address) for metrics, however, in the latest version, pod address is the pod ip and the port is not assigned to the url, so inference extension can not correctly scrape the pod metrics since it always send requests to port 80.

new := &PodMetrics{
		Pod: Pod{
			NamespacedName: types.NamespacedName{
				Name:      pod.Name,
				Namespace: pod.Namespace,
			},
			Address: pod.Status.PodIP,
		},
		Metrics: Metrics{
			ActiveModels: make(map[string]int),
		},
	}

And also, the scrape time is refreshMetricsInterval = flag.Duration("refreshMetricsInterval", 50*time.Millisecond, "interval to refresh metrics via polling pods"). So once the metrics fail to be scraped, the resulting logs can become quite extensive.

Image

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Inference extension version (use git describe --tags --dirty --always):
  • Cloud provider or hardware configuration:
  • Install tools:
  • Others:
@Kuromesi Kuromesi added the kind/bug Categorizes issue or PR as related to a bug. label Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant