PubSub: ImportError after deploying Cloud Function with python3.7 and serverless

[woodpav]

Environment details

I am deploying a pubsub function to python3.7 with the serverless framework using "serverless-google-cloudfunctions": "^2.3.2" and "serverless-python-requirements": "^4.3.0".

I am using google-cloud-pubsub==0.45.0 in requirements.txt

Steps to reproduce

1. Run serverless deploy and I get the error.

My Code

I am deploying a Google Cloud function that uses the pubsub API to publish a message. However, I am getting the following error:

ImportError: cannot import name 'cygrpc' from 'grpc._cython'

It seems to be because you can't pip install a C library with requirements.txt. How to I fix this? My code is below.

from google.cloud import pubsub

publisher = pubsub.PublisherClient()
path = publisher.topic_path("my_proj", "my_topic")

publisher.publish(path, "test".encode("utf-8"))

My requirements.txt is the following. I have tried adding grpcio==1.22.0 to no avail.

google-cloud-pubsub==0.45.0

My serverless.yml:

service: my-service

provider:
    name: google
    stage: ${opt:stage, 'dev'}
    runtime: python37
    region: us-central1
    project: ${self:custom.env.PROJECT_NAME}
    credentials: ~/.gcloud/keyfile.json

plugins:
  - serverless-google-cloudfunctions
  - serverless-python-requirements

custom:
    pythonRequirements:
        fileName: private_requirements.txt
        pythonBin: python3
        noDeploy:
            - requirements.txt
    stage:
        ${self:provider.stage}
    env:
        ${file(./.env.${self:provider.stage})}

package:
    include:
        - requirements.txt
    exclude:
        - .git/**
        - .gitignore
        - env*
        - node_modules/**
        - package.json
        - private_requirements.txt
        - yarn.lock

[tseaver]

@JustinBeckwith Can you point us to somebody in the Cloud Functions team who can help?

[plamut]

@woodpav I was able to successfully deploy a function with the following requirements.txt:

google-cloud-pubsub==0.45.0
grpcio==1.22.0

And with the following imports at the top of the main.py file:

import json
import datetime

from google.cloud import pubsub
from grpc._cython import cygrpc

Since grpcio is not a pure Python package (it's partially written in Cython), did you try to use the settings described under the "Cross compiling" section in the docs, if using a non-Linux system? https://serverless.com/plugins/serverless-python-requirements/

When trying to reproduce the reported issue using a Linux machine the deployment went through fine.

[tseaver]

I just answered a similar question in #9142 about AppEngine, where the requirements.txt file on AppEngine is not processed for dependencies during deployment: maybe deployment to Cloud Functions uses a similar approach?

The AppEngine docs suggest using pip freeze in the local virtual environment to specify a complete "known good" set of dependencies in requirements.txt.

[woodpav]

This is a problem with python-serverless-requirements and docker as mentioned in serverless/serverless-python-requirements#272.

Here is a link to my solution on StackOverflow that uses gcloud cli instead of serverless.