Skip to content

Unable to receive events from the local pubsub emulator #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jmirmina opened this issue Feb 22, 2021 · 8 comments · Fixed by #100
Closed

Unable to receive events from the local pubsub emulator #85

jmirmina opened this issue Feb 22, 2021 · 8 comments · Fixed by #100

Comments

@jmirmina
Copy link

I have an extremely basic example that I am trying to get running locally with the pubsub emulator using push subscriptions. Here is my app.rb file:

require 'functions_framework'

FunctionsFramework.cloud_event('send-message') do |event|
	FunctionsFramework.logger.info('We made it!')
	pp event.inspect
end

I start up the server using

 bundle exec functions-framework-ruby --target send-message

I then open up a Rails console with the following to create a topic, subscription, and publish a message:

require "google/cloud/pubsub"
pubsub = Google::Cloud::PubSub.new(project_id: "my-project", emulator_host: 'localhost:8085')
topic = pubsub.create_topic('send-message')
topic.subscribe('test-subscription', endpoint: 'http://localhost:8080')
topic.publish('my-event')

The following error shows up repeatedly in the functions_framework server log:

W, [2021-02-22T09:31:10.829855 #17276]  WARN -- : Unrecognized event format (CloudEvents::HttpContentError)
/.rvm/gems/ruby-2.7.1/gems/functions_framework-0.7.1/lib/functions_framework/server.rb:441:in `decode_event'
/.rvm/gems/ruby-2.7.1/gems/functions_framework-0.7.1/lib/functions_framework/server.rb:421:in `call'
/.rvm/gems/ruby-2.7.1/gems/puma-4.3.7/lib/puma/server.rb:713:in `handle_request'
/.rvm/gems/ruby-2.7.1/gems/puma-4.3.7/lib/puma/server.rb:472:in `process_client'
/.rvm/gems/ruby-2.7.1/gems/puma-4.3.7/lib/puma/server.rb:328:in `block in run'
/.rvm/gems/ruby-2.7.1/gems/puma-4.3.7/lib/puma/thread_pool.rb:134:in `block in spawn_thread'

Thinking that the issue might possibly be related to the pubsub emulator, I created a topic and push subscription using the actual Google PubSub service, and used ngrok to open up port 8080 so it could push messages to my local computer, and I am getting the exact same errors.

However, if I put my basic example in a Google Cloud Function, it works fine. I'm just running into the issue locally.

I found an example on one of the other functions_framework libraries (nodeJS maybe?) where they recommended trying the following:

curl -d "@mockPubsub.json" \
-X POST \
-H "Ce-Type: true" \
-H "Ce-Specversion: 1.0" \
-H "Ce-Source: true" \
-H "Ce-Id: true" \
-H "Content-Type: application/json" \
http://localhost:8080

where mockPubsub.json was a basic message to send, and that works locally.

Any thoughts/suggestions to get this working locally?

@jmirmina
Copy link
Author

Some things I've noticed while testing this a little further. On my local machine, the body of the request looks like this :

{"message"=>{"data"=>"dGVzdA==", "messageId"=>"1989717716705710", "message_id"=>"1989717716705710", "publishTime"=>"2021-02-25T18:08:51.501Z", "publish_time"=>"2021-02-25T18:08:51.501Z"}, "subscription"=>"projects/my-project/subscriptions/gcf-function-1-us-east1-my-topic"}

It appears that the legacy event converter support this format.

In Google Cloud Functions, the body of the request is:

{"context"=>{"eventId"=>"1989668503284241", "timestamp"=>"2021-02-25T18:00:39.653Z", "eventType"=>"google.pubsub.topic.publish", "resource"=>{"service"=>"pubsub.googleapis.com", "name"=>"projects/fs-my-project/topics/my-topic", "type"=>"type.googleapis.com/google.pubsub.v1.PubsubMessage"}}, "data"=>{"@type"=>"type.googleapis.com/google.pubsub.v1.PubsubMessage", "attributes"=>nil, "data"=>"dGVzdA=="}}

which the legacy event converter knows how to convert to a cloud event. I'm just not sure how to get the data in the above format in my dev environment to play around with this locally.

@dazuma
Copy link
Member

dazuma commented Feb 26, 2021

Thanks for the update. I haven't looked at the pubsub emulator, but it's definitely plausible that it's emitting events in a different format than the production service. We'll need to figure out from the pubsub team what's behind the differences you're seeing.

@dazuma dazuma changed the title Unable to get functions_framework running locally Unable to receive events from the local pubsub emulator Feb 26, 2021
@jmirmina
Copy link
Author

The request bodies I posted in my last message were actually from Google PubSub, not the emulator. Sorry for the confusion.

@dazuma
Copy link
Member

dazuma commented Feb 27, 2021

Oh, I see, yes you mentioned in your original description that you were pushing messages from the actual PubSub service to your local machine. You're right: there is special integration between the PubSub service and Cloud Functions, where Cloud Functions converts a raw PubSub message to the format recognized by the Functions Framework's event converter. But this integration doesn't happen if you're running locally (or, I imagine, other non-serverless environments such as a straight GCE VM) and so you see the raw PubSub message that the Functions Framework doesn't know how to convert.

I'm going to need to get some eyes on this from our serverless specialists, so give me a few days. We should be able to enhance the legacy event converter to recognize raw PubSub message format, but I want to make sure this gets handled properly across languages and that we don't clobber other use cases.

Thanks much for the report!

@jmirmina
Copy link
Author

That makes sense. I thought there might be something like that going on behind the scenes, but couldn't find any documentation about it. Now that I know this, I feel a little more comfortable putting in a temporary monkey patch when running locally to work around this for now. Thanks!

@kira4ka
Copy link

kira4ka commented Apr 8, 2021

@dazuma hi, any updates on this issue?

@dazuma
Copy link
Member

dazuma commented May 26, 2021

Sorry for the delay here. It looks like we are adding this logic into the functions frameworks (example from Python), so we'll get it done for Ruby as well.

@dazuma
Copy link
Member

dazuma commented Jun 1, 2021

Released in the gem version 0.10.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants