Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Vertex AI Gemini Structured JSON caching not working #9692

Open
adrianlyjak opened this issue Apr 2, 2025 · 0 comments · May be fixed by #9693
Open

[Bug]: Vertex AI Gemini Structured JSON caching not working #9692

adrianlyjak opened this issue Apr 2, 2025 · 0 comments · May be fixed by #9693
Labels
bug Something isn't working

Comments

@adrianlyjak
Copy link
Contributor

What happened?

I noticed that vertex gemini calls are no longer being cached after upgrading.

I believe I tracked down the issue to the removal of a strict: True in response schema, breaking the cache

https://github.com/BerriAI/litellm/blob/655ce2e7451fa81f1da53279bf29a20248fb3d5c/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py#L246C1-L246C50

Logging out the cache key calculation, this is what I see:

cache_key += model: vertex_ai/gemini-2.0-flash
cache_key += messages: [{'role': 'user', 'content': 'Say hello in the language of your choice, structured as {"language": "language", "greeting": "greeting"}'}]
cache_key += temperature: 1
cache_key += response_format: {'type': 'json_schema', 'json_schema': {'schema': {'properties': {'language': {'title': 'Language', 'type': 'string'}, 'greeting': {'title': 'Greeting', 'type': 'string'}}, 'required': ['language', 'greeting'], 'title': 'Salutation', 'type': 'object', 'propertyOrdering': ['language', 'greeting']}}, 'strict': True}
cache_key += stream: False
cache_key (get) 31bb9dda4a7c4a1973cf4cf0471a900a2958ac7309e5d22930ce78b07dbc3495
cache_key += model: vertex_ai/gemini-2.0-flash
cache_key += messages: [{'role': 'user', 'content': 'Say hello in the language of your choice, structured as {"language": "language", "greeting": "greeting"}'}]
cache_key += temperature: 1
cache_key += response_format: {'type': 'json_schema', 'json_schema': {'schema': {'properties': {'language': {'title': 'Language', 'type': 'string'}, 'greeting': {'title': 'Greeting', 'type': 'string'}}, 'required': ['language', 'greeting'], 'title': 'Salutation', 'type': 'object', 'propertyOrdering': ['language', 'greeting']}}}
cache_key += stream: False
cache_key (add) 5e712028c3566ab1457e96a31bbb9e62a9896fe968c391f7bbb766ffdf5f525a

That's from adding this here

print(f"cache_key += {param}: {param_value}")

litellm/litellm/caching/caching.py

Line 264 in 655ce2e

cache_key += f"{str(param)}: {str(param_value)}"

and this here

print("cache_key (get)", cache_key)

litellm/litellm/caching/caching.py

Line 501 in 655ce2e

if cache_key is not None:

and this here:

print("cache_key (add)", cache_key)

litellm/litellm/caching/caching.py

Line 579 in 655ce2e

Relevant log output



Are you a ML Ops Team?


No


What LiteLLM version are you on ?


1.65.0


Twitter / LinkedIn details


https://x.com/rnges_us

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @adrianlyjak
adrianlyjak




          added
  the 

  bug

  Something isn't working
 label


      Apr 2, 2025

    

  



  


  
    
  
  

    
  adrianlyjak 

      added a commit
        to adrianlyjak/litellm
      that referenced
      this issue

    
      Apr 2, 2025
    
      
        
      
    

      


  

    

      


  

      
        @adrianlyjak
  





      

        
          fix BerriAI#9692. Return a copy from strict key removal to not break …
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          5120a83
        
      

    

  

    

      
…cache keys

    




    

  




  

  

    
  


  

      

    @adrianlyjak
adrianlyjak



    linked a pull request
    
      Apr 2, 2025
    

    that will
    close
    this issue





  


  

      
   Open

  





  
  

      
        
      
      

  
    
    
  


    4 tasks
  





  




  


  
    
  
  

    
  adrianlyjak 

      added a commit
        to adrianlyjak/litellm
      that referenced
      this issue

    
      Apr 2, 2025
    
      
        
      
    

      


  

    

      


  

      
        @adrianlyjak
  





      

        
          fix BerriAI#9692. Keep cache key stable during mutation
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          c6eea98
        
      

    

  

    

      
A) Return a copy from strict key removal to not break cache keys
B) Fix issue in existing cache key stabilizer that was not storing a stable
   key across request/response if no litellm_params existed in the request

    




    

  




  


  
    
  
  

    
  adrianlyjak 

      added a commit
        to adrianlyjak/litellm
      that referenced
      this issue

    
      Apr 2, 2025
    
      
        
      
    

      


  

    

      


  

      
        @adrianlyjak
  





      

        
          fix BerriAI#9692. Keep cache key stable during mutation
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          837818d
        
      

    

  

    

      
A) Return a copy from strict key removal to not break cache keys
B) Fix issue in existing cache key stabilizer that was not storing a stable
   key across request/response if no litellm_params existed in the request

    




    

  




  


  
    
  
  

    
  adrianlyjak 

      added a commit
        to adrianlyjak/litellm
      that referenced
      this issue

    
      Apr 2, 2025
    
      
        
      
    

      


  

    

      


  

      
        @adrianlyjak
  





      

        
          fix BerriAI#9692. Keep cache key stable during mutation
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          5e75601
        
      

    

  

    

      
A) Return a copy from strict key removal to not break cache keys
B) Fix issue in existing cache key stabilizer that was not storing a stable
   key across request/response if no litellm_params existed in the request

    




    

  




  


  
    
  
  

    
  adrianlyjak 

      added a commit
        to adrianlyjak/litellm
      that referenced
      this issue

    
      Apr 2, 2025
    
      
        
      
    

      


  

    

      


  

      
        @adrianlyjak
  





      

        
          fix BerriAI#9692. Keep cache key stable during mutation
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          b2e8a82
        
      

    

  

    

      
A) Return a copy from strict key removal to not break cache keys
B) Fix issue in existing cache key stabilizer that was not storing a stable
   key across request/response if no litellm_params existed in the request

    




    

  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    bug

  Something isn't working








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
  
Successfully merging a pull request may close this issue.





    


  
          
      Return a copy from strict key removal to not break cache keys

    
      adrianlyjak/litellm
    





      
    

  




      

    

  
      

  

    

      1 participant
    

    

        
          @adrianlyjak