Skip to content

Add e2e tests using Python / Typescript clients #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 of 12 tasks
sd2k opened this issue Mar 13, 2025 · 2 comments
Open
5 of 12 tasks

Add e2e tests using Python / Typescript clients #43

sd2k opened this issue Mar 13, 2025 · 2 comments
Assignees

Comments

@sd2k
Copy link
Collaborator

sd2k commented Mar 13, 2025

Most clients are likely to be using the Python or Typescript SDKs and they're official so will be closely aligned with the spec. We should add some e2e tests to ensure that an example application (a test 'host') can connect and list/use tools, esp with some custom headers.

We could also use this for evaluation using LangEval (related to LangWatch), by having some standard questions, running them against multiple models with MCP tools enabled, and asserting that we obtain the correct answers within X tokens or API calls.

TODO:

  • Require contributor approval before running actions on PRs
  • Create a new cloud instance for tests
  • Create a tests subdirectory
  • Create a Python package in there
  • Add langeval
  • Add tests
  • Run in CI

Workflows to test:

  • [docker-compose] what are the most recent log lines from Grafana?
  • [docker-compose] what dashboard should I look at to see container CPU?
  • [cloud]: who is on call for team X?
  • [cloud]: what incidents are active?
  • (add more)
@sd2k sd2k assigned sd2k and ioanarm Mar 24, 2025
@ioanarm
Copy link
Contributor

ioanarm commented Apr 7, 2025

First iteration of this in here

@sd2k
Copy link
Collaborator Author

sd2k commented Apr 14, 2025

It might be nice to also test SSE as well as stdio (probably not worth parametrizing all our tests, but maybe a single smoke test to make sure both work would be good).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants