[Bug]: calculate audio transcription cost #7601

liuhu · 2025-01-07T10:03:47Z

What happened?

1. custom_pricing not working for stt model.

The model config:

model_list:
  - model_name: groq/whisper
    litellm_params:
      model: groq/whisper-large-v3
      api_key: os.environ/GROQ_API_KEY
      input_cost_per_second: 0.00185
      output_cost_per_second: 0
    model_info:
      mode: audio_transcription

the root reasons:
SPECIAL_MODEL_INFO_PARAMS not contains input_cost_per_second , and we also need to add output_cost_per_second for tts model

litellm/litellm/litellm_core_utils/litellm_logging.py

Lines 2560 to 2575 in 55139b8

    
           def use_custom_pricing_for_model(litellm_params: Optional[dict]) -> bool: 
        
               if litellm_params is None: 
        
                   return False 
        
               for k, v in litellm_params.items(): 
        
                   if k in SPECIAL_MODEL_INFO_PARAMS and v is not None: 
        
                       return True 
        
               metadata: Optional[dict] = litellm_params.get("metadata", {}) 
        
               if metadata is None: 
        
                   return False 
        
               model_info: Optional[dict] = metadata.get("model_info", {}) 
        
               if model_info is not None: 
        
                   for k, v in model_info.items(): 
        
                       if k in SPECIAL_MODEL_INFO_PARAMS: 
        
                           return True 
        
               return False

litellm/litellm/types/router.py

Lines 373 to 378 in 55139b8

    
           SPECIAL_MODEL_INFO_PARAMS = [ 
        
               "input_cost_per_token", 
        
               "output_cost_per_token", 
        
               "input_cost_per_character", 
        
               "output_cost_per_character", 
        
           ]

2. `model_prices_and_context_window_backup.json` not contains some groq stt models

whisper-large-v3, whisper-large-v3-turbo, distil-whisper-large-v3-en

3. when i fix 1 and 2 on above issues, appear response_cost Exception

curl:

curl --location 'http://litellm.local/v1/audio/transcriptions' \
--header 'Authorization: Bearer sk-xxx' \
--form 'file=@"/test.ogg"' \
--form 'model="groq/whisper"'

some error log for response_cost :
Exception: This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.\n

The root reason:
groq use OpenAIAudioTranscription and it is hard code to set model name and provider in transcription handler

litellm/litellm/llms/openai/transcriptions/handler.py

Line 145 in 55139b8

hidden_params = {"model": "whisper-1", "custom_llm_provider": "openai"}

4. groq stt miss some unit test

Relevant log output

In S3 file:
{
  "id": "51b6e5bd-021e-40b9-9ea5-030a7e887883",
  "trace_id": "4abd3078-da10-40e9-b107-af12f659f481",
  "call_type": "atranscription",
  "cache_hit": null,
  "stream": null,
  "status": "success",
  "custom_llm_provider": "groq",
  "saved_cache_cost": 0,
  "startTime": 1735818392.247911,
  "endTime": 1735818393.023638,
  "completionStartTime": 1735818393.023638,
  "response_time": 0.7757270336151123,
  "model": "whisper-large-v3",
  "metadata": {
    "user_api_key_hash": "acd92d7246d8cbaa71465db0eceea907e8a56d13dd5458e15e73353fae067b4b",
    "user_api_key_alias": "stt-service",
    "user_api_key_team_id": "d70b5ad8-6e95-4352-ae9c-881291ead2e0",
    "user_api_key_org_id": null,
    "user_api_key_user_id": null,
    "user_api_key_team_alias": "flow",
    "spend_logs_metadata": null,
    "requester_ip_address": "",
    "requester_metadata": {},
    "user_api_key_end_user_id": null
  },
  "cache_key": null,
  "response_cost": 0,
  "total_tokens": 0,
  "prompt_tokens": 0,
  "completion_tokens": 0,
  "request_tags": [],
  "end_user": "",
  "api_base": "https://api.groq.com/openai/v1",
  "model_group": "groq/whisper",
  "model_id": "0309cbac7cf03230d0a72980613e1e26d6df45d4aad37c585560f0d41c7687a5",
  "requester_ip_address": "",
  "messages": [
    {
      "role": "user",
      "content": "02151bfb-3382-4c1e-b77c-3aab54e753c4 (1).ogg"
    }
  ],
  "response": {
    "text": " tagliando auto",
    "language": "Italian",
    "task": "transcribe",
    "duration": 3.2,
    "words": null,
    "segments": [
      {
        "id": 0,
        "avg_logprob": -0.18575256,
        "compression_ratio": 0.8181818,
        "end": 3.2,
        "no_speech_prob": 0.07431701,
        "seek": 0,
        "start": 0,
        "temperature": 0,
        "text": " tagliando auto",
        "tokens": [
          50365,
          6162,
          2081,
          1806,
          8399,
          50525
        ]
      }
    ]
  },
  "model_parameters": {},
  "hidden_params": {
    "model_id": "0309cbac7cf03230d0a72980613e1e26d6df45d4aad37c585560f0d41c7687a5",
    "cache_key": null,
    "api_base": "https://api.groq.com/openai/v1",
    "response_cost": null,
    "additional_headers": {}
  },
  "model_map_information": {
    "model_map_key": "whisper-1",
    "model_map_value": null
  },
  "error_str": null,
  "error_information": {
    "error_code": "",
    "error_class": "",
    "llm_provider": ""
  },
  "response_cost_failure_debug_info": {
    "error_str": "This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.",
    "traceback_str": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4258, in _get_model_info_helper\n    raise ValueError(\nValueError: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/litellm_core_utils/litellm_logging.py\", line 821, in _response_cost_calculator\n    response_cost = litellm.response_cost_calculator(\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 807, in response_cost_calculator\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 795, in response_cost_calculator\n    response_cost = completion_cost(\n                    ^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 745, in completion_cost\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 724, in completion_cost\n    ) = cost_per_token(\n        ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 237, in cost_per_token\n    return openai_cost_per_second(\n           ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/openai/cost_calculation.py\", line 94, in cost_per_second\n    model_info = get_model_info(\n                 ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4442, in get_model_info\n    _model_info = _get_model_info_helper(\n                  ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4364, in _get_model_info_helper\n    raise Exception(\nException: This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.\n",
    "model": "whisper-large-v3",
    "cache_hit": null,
    "custom_llm_provider": "groq",
    "base_model": null,
    "call_type": "atranscription",
    "custom_pricing": false
  },
  "guardrail_information": null
}


Run /health check:
{
  "healthy_endpoints": [],
  "unhealthy_endpoints": [
    {
      "model": "openai/whisper-1",
      "error": "[Errno 2] No such file or directory: '/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/../../../tests/gettysburg.wav'\nHave you set 'mode' - https://docs.litellm.ai/docs/proxy/health#embedding-models\nstack trace: Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 5222, in ahealth_check\n    response = await openai_chat_completions.ahealth_check(\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py\", line 1552, in ahealth_check\n    audio_file = open(file_path, \"rb\")\n                 ^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/../../../tests/gettysburg.wav'\n"
    },
    {
      "input_cost_per_second": 0,
      "output_cost_per_second": 0.0001,
      "model": "groq/whisper-large-v3",
      "cache": {
        "no-cache": true
      },
      "error": "litellm.BadRequestError: GroqException - {\"error\":{\"message\":\"The model `whisper-large-v3` does not support chat completions\",\"type\":\"invalid_request_error\"}}\n\nHave you set 'mode' - https://docs.litellm.ai/docs/proxy/health#embedding-models\nstack trace: Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/openai_like/chat/handler.py\", line 203, in acompletion_function\n    response = await client.post(\n               ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 224, in post\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 181, in post\n    response.raise_for_status()\n  File \"/usr/local/lib/python3.11/site-packages/httpx/_models.py\", line 761, in raise_for_status\n    raise HTTPStatusError(message, request=request, response=self)\nhttpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.groq.com/openai/v1/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site"
    }
  ],
  "healthy_count": 0,
  "unhealthy_count": 2
}

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.56.9

Twitter / LinkedIn details

@Hugo_Liu_X

The text was updated successfully, but these errors were encountered:

liuhu · 2025-03-21T00:58:52Z

@ishaan-jaff @krrishdholakia Any suggestions on these issues? I can create a pull request to try to resolve these issues

krrishdholakia · 2025-03-21T01:01:27Z

Hey @liuhu i missed this

liuhu · 2025-04-01T06:53:08Z

@krrishdholakia according to your suggestions #7610 (comment) I've split the PR, cloud you help me review and approve them. #9649 #9648 #9647

liuhu · 2025-04-02T08:25:30Z

@krrishdholakia according to your suggestions #7610 (comment) I've split the PR, cloud you help me review and approve them. #9649 #9648 #9647

@krrishdholakia @ishaan-jaff I've created new PRs based on @krrishdholakia suggestions 💯, any other comments?

liuhu added the bug Something isn't working label Jan 7, 2025

github-actions bot added the mlops user request label Jan 7, 2025

liuhu changed the title ~~[Bug]: calculate groq audio transcription cost~~ [Bug]: calculate Groq audio transcription cost Jan 7, 2025

liuhu pushed a commit to liuhu/litellm that referenced this issue Jan 7, 2025

fix(asr-groq): calculate Groq audio transcription cost (BerriAI#7601)

60b90f7

This was referenced Jan 7, 2025

fix(asr-groq): calculate Groq audio transcription cost (#7601) #7608

Closed

fix(asr-groq): calculate Groq audio transcription cost (#7601) #7609

Closed

liuhu pushed a commit to liuhu/litellm that referenced this issue Jan 7, 2025

fix(asr-groq): calculate groq audio transcription cost (BerriAI#7601)

52df28b

liuhu mentioned this issue Jan 7, 2025

fix(asr-groq): calculate Groq audio transcription cost #7610

Closed

liuhu changed the title ~~[Bug]: calculate Groq audio transcription cost~~ [Bug]: calculate audio transcription cost Mar 21, 2025

krrishdholakia self-assigned this Mar 21, 2025

This was referenced Mar 30, 2025

fix(asr-groq): fix groq custom pricing #9647

Open

fix(asr-groq): add groq whisper models to model cost map #9648

Merged

fix(asr-groq): fix cost calc on custom pricing for stt #9649

Open

krrishdholakia closed this as completed in #9648 Apr 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: calculate audio transcription cost #7601

[Bug]: calculate audio transcription cost #7601

liuhu commented Jan 7, 2025 •

edited

Loading

liuhu commented Mar 21, 2025

krrishdholakia commented Mar 21, 2025

liuhu commented Apr 1, 2025

liuhu commented Apr 2, 2025

[Bug]: calculate audio transcription cost #7601

[Bug]: calculate audio transcription cost #7601

Comments

liuhu commented Jan 7, 2025 • edited Loading

What happened?

1. custom_pricing not working for stt model.

2. model_prices_and_context_window_backup.json not contains some groq stt models

3. when i fix 1 and 2 on above issues, appear response_cost Exception

4. groq stt miss some unit test

Relevant log output

Are you a ML Ops Team?

What LiteLLM version are you on ?

Twitter / LinkedIn details

liuhu commented Mar 21, 2025

krrishdholakia commented Mar 21, 2025

liuhu commented Apr 1, 2025

liuhu commented Apr 2, 2025

liuhu commented Jan 7, 2025 •

edited

Loading

2. `model_prices_and_context_window_backup.json` not contains some groq stt models