keypup-io / cloudtasker Goto Github PK

From readme.md, seems the guideline is to create a queue prefix based on App name like my-app-default.

This makes a very big assumption about naming things in the GCP platform. Queues are also project specific, so I don't quite see the need to force clients to specify prefix via CLOUDTASKER_GCP_QUEUE_PREFIX

What ends up happening when a queue is used by the some monolithic app that handles multiple jobs:
my-app-default
my-app-sms-notify-update
my-app-sms-notify-delete
my-app-email-send-campaign

I would prefer to name the queues as:
sms-notify-update
sms-notify-delete
email-send-campaign

So it is less cluttered and I can interpret with an actual meaningful prefix (but not enforced by some way by cloud tasker client)

• sms-<|some-job-action|> -> deals with sms jobs
• email-<|some-job-action|>* -> deals with emails jobs

Not sure if batch would be the way to go though.

Hey guys,
Thanks for your great work on cloudtasker. We started using it and noticed the following issue running the local server. Our server is running on https using a self-signed certificate. It was raising the following exception:

#<Thread:0x0000000115e14270 /Users/dominik/.rvm/gems/ruby-3.1.2/gems/cloudtasker-0.13.2/lib/cloudtasker/local_server.rb:74 run> terminated with exception (report_on_exception is true):
/Users/dominik/.rvm/gems/ruby-3.1.2/gems/net-protocol-0.2.1/lib/net/protocol.rb:237:in `rbuf_fill': end of file reached (EOFError)
	from /Users/dominik/.rvm/gems/ruby-3.1.2/gems/net-protocol-0.2.1/lib/net/protocol.rb:199:in `readuntil'
	from /Users/dominik/.rvm/gems/ruby-3.1.2/gems/net-protocol-0.2.1/lib/net/protocol.rb:209:in `readline'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http/response.rb:42:in `read_status_line'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http/response.rb:31:in `read_new'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1575:in `block in transport_request'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1566:in `catch'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1566:in `transport_request'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1539:in `request'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1532:in `block in request'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:966:in `start'
	from /Users/dominik/.rvm/rubies/ruby-3.1.2/lib/ruby/3.1.0/net/http.rb:1530:in `request'
	from /Users/dominik/.rvm/gems/ruby-3.1.2/gems/cloudtasker-0.13.2/lib/cloudtasker/backend/redis_task.rb:208:in `deliver'
	from /Users/dominik/.rvm/gems/ruby-3.1.2/gems/cloudtasker-0.13.2/lib/cloudtasker/local_server.rb:89:in `process_task'
	from /Users/dominik/.rvm/gems/ruby-3.1.2/gems/cloudtasker-0.13.2/lib/cloudtasker/local_server.rb:74:in `block in process_jobs'

I was able to make it work changing the http_client method in redis_task.rb to:

def http_client
        @http_client ||=
          begin
            uri = URI(http_request[:url])
            http = Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = dispatch_deadline }
            http.use_ssl = true
            http.verify_mode = OpenSSL::SSL::VERIFY_NONE
            http
          end
      end

Do you see any chances to make this configurable, so that this also works locally with https?
Thanks,
Dominik

Rails v. 5.2.2
Cloudtasker gem v. 0.11.rc2

After configuring the project as instructed in README, error is being thrown on launch. Looks like ActiveJob::QueueAdapters::CloudtaskerAdapter is not loaded properly from gem.
config/environments/production.rb|development.rb files, where executing config.active_job.queue_adapter = :cloudtasker are hitting against class that's not initialized.

rails c                                                                                                                                                   ─╯
D, [2020-12-15T15:40:54.183325 #4277] DEBUG -- : [httplog] Connecting: oauth2.googleapis.com:443
D, [2020-12-15T15:40:54.377321 #4277] DEBUG -- : [httplog] Sending: POST http://oauth2.googleapis.com:443/token
D, [2020-12-15T15:40:54.377556 #4277] DEBUG -- : [httplog] Data: grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJ0ZXN0LWdvb2dsZS10YXNrc0BzcGFjZW9zLW1vbml0b3IuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJhdWQiOiJodHRwczovL29hdXRoMi5nb29nbGVhcGlzLmNvbS90b2tlbiIsImV4cCI6MTYwODA0MzMxNCwiaWF0IjoxNjA4MDQzMTk0LCJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvbG9nZ2luZy5hZG1pbiJ9.SIvanU_wFFNp6sKGqzUqTaG9E-LmiL-CMsILp4JYBFIaAtDGbCyLk57BwqTmCpiWVfU6t1oORQA9LQoyZ-NGnxXSwKrZc942NJe8Kr7D7Qr3JUANyTrmUPJLJlIbQNLKqqFe4kbiQ3ik52g1QsYATwslGZYA2HTyde_b11TqfauottpWGvn50p_hqb5WNZYRp6ehFCNZiN91LaSFftMFSFo2-EDfqbgKfN5nI4lV-Mblpq2pO9s4dzVgH3Zgd1h4896lYs-c_pPHUIR3NbitcqOYl3LHQX2HyMvyQstbFnD0StVY_pfyFrKDXuFcpYAziswSkq98LjRMuqE2Xy9XaA
D, [2020-12-15T15:40:54.377690 #4277] DEBUG -- : [httplog] Status: 200
D, [2020-12-15T15:40:54.378064 #4277] DEBUG -- : [httplog] Benchmark: 0.059908 seconds
D, [2020-12-15T15:40:54.378514 #4277] DEBUG -- : [httplog] Response:
{"access_token":"ya29.c.Kp0B6gdcmDLOqbeM6hTKjl9SZTuhbncrIehzMvc7FCmDM8vP4vwzK1masDffsno5iByBCrghZOzW8XKUhbV4NxiIUVD-L7xquThTokP91eOqMJHS1ZN4C-RukYT_u_HGjdfMEftyL1-MgfYsMTM1HyTpoyC3PFBefH3ugXDHPSRCtyjizgSPYPWZs5JFw_P2SrIIRy-KG2C-lP13lSKmLg","expires_in":3599,"token_type":"Bearer"}
/home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/queue_adapters.rb:135:in `const_get': uninitialized constant ActiveJob::QueueAdapters::CloudtaskerAdapter (NameError)
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/queue_adapters.rb:135:in `lookup'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/queue_adapter.rb:35:in `queue_adapter='
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:20:in `block (3 levels) in <class:Railtie>'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:20:in `each'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:20:in `block (2 levels) in <class:Railtie>'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:42:in `each'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/lazy_load_hooks.rb:42:in `on_load'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:19:in `block in <class:Railtie>'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in `instance_exec'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in `run'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/railties-5.2.4.4/lib/rails/initializable.rb:61:in `block in run_initializers'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `each'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `call'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/railties-5.2.4.4/lib/rails/initializable.rb:60:in `run_initializers'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/railties-5.2.4.4/lib/rails/application.rb:361:in `initialize!'
	from /home/janek/rclub/api/config/environment.rb:7:in `<top (required)>'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application.rb:106:in `preload'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application.rb:157:in `serve'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application.rb:145:in `block in run'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application.rb:139:in `loop'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application.rb:139:in `run'
	from /home/janek/.rvm/gems/ruby-2.6.6/gems/spring-2.1.1/lib/spring/application/boot.rb:19:in `<top (required)>'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /home/janek/.rvm/rubies/ruby-2.6.6/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from -e:1:in `<main>'

Trying to create default queue from gcloud CLI and got the following errors.

There is no App Engine app in project [xxx].

Would you like to create one (y/N)? n

ERROR: (gcloud.tasks.queues.create) Could not determine the location for the project. Please try again. It is possible an AppEngine App does not exist for this project.

Wondering if I missed anything during setup? Is App Engine app required to use Cloud Tasks? My web app is a Cloud Run service.

Hello! I've been trying out this library for a few hours and so far it's being inconsistent. I've been testing with the DummyWorker and ExampleJob classes and they work sometimes. What I mean by that is that sometimes the worker works while the job doesn't, and viceversa. I tried starting the server, cloudtasker, and console in various ways: using separate terminals, using a Procfile with foreman; I've tried executing them using bundle exec, calling the worker/job differently (using perform_async and perform_in in the case of the worker, with and without an argument.) Sometimes the worker doesn't work, but I stop and start everything again and it works, but the job doesn't. I can't get them to work properly.

If either of them is working, it keeps working for the rest of the server/console session (or at least for a while.)

The error that I'm getting from the cloudtasker logs is:

#<Thread:0x000055d665c8ebf0 /home/angel/.rvm/gems/ruby-2.7.0/gems/cloudtasker-0.11.0/lib/cloudtasker/local_server.rb:72 run> terminated with exception (report_on_exception is true):
/home/angel/.rvm/gems/ruby-2.7.0/gems/cloudtasker-0.11.0/lib/cloudtasker/local_server.rb:87:in `process_task': undefined method `deliver' for nil:NilClass (NoMethodError)
    from /home/angel/.rvm/gems/ruby-2.7.0/gems/cloudtasker-0.11.0/lib/cloudtasker/local_server.rb:72:in `block in process_jobs'

From what I understand, I guess that process_task callers are not setting the task parameter to the proper value in all cases, such as in process_jobs.

Some info about the environment:

• Ruby 2.7.0
• Rails 6.1.0
• cloudtasker 0.11

I have redis installed (along with the redis gem.) The initializer is the simplest one, with only a config.processor_host line inside the config block.

The docs state that on_batch_complete fires when "all children have successfully completed or died (all retries exhausted)."

It would be helpful to have a callback like on_batch_success that only fires if and when all children in the batch complete successfully.

Hello

I'm trying to give this gem a whirl on ruby 3.0.1 using rails active job and receive an argument error when enqueuing a job.

irb(main):003:0> DummyJob.perform_later("some arg")
E, [2021-04-19T10:10:12.732771 #87494] ERROR -- : Failed enqueuing DummyJob to Cloudtasker(default): ArgumentError (wrong number of arguments (given 1, expected 0))
/XXX/vendor/cache/ruby/3.0.0/gems/cloudtasker-0.11.0/lib/cloudtasker/worker.rb:209:in `schedule_time': wrong number of arguments (given 1, expected 0) (ArgumentError)

I think it is related to the positional arguments changes detailed here.

I checked out the source and can confirm tests are green on 2.7.3 and red on 3.0.1

Is there any plan for 3.x support? I'm happy to pitch in to find a solution i'm just not sure where to start.

Getting this as soon as i install the gem

Post-install message from google-gax:
*******************************************************************************
The google-gax gem is officially end-of-life and will not be updated further.

If your app uses the google-gax gem, it likely is using obsolete versions of
some Google Cloud client library (google-cloud-*) gem that depends on it. We
recommend updating any such libraries that depend on google-gax. Modern Google
Cloud client libraries will depend on the gapic-common gem instead.
*******************************************************************************

what does this mean.. and what effects could this have for this gem in the future..

Thanks for open sourcing a great framework. I didn't see this in the docs or codebase anywhere, but is it possible to specify the OIDC token via service account?

We would need to be able to authenticate target HTTP Tasks, but didn't see an obvious way for us to do that.

Thanks!

Use case:

Have one Cloud Run service for user actions and ActiveJob Tasks, and another Cloud Run Service with more CPU and Memory resources to run heavier schedule tasks.

application_service => 01 vcpu 512M
application_cron_tasks => 04 vcpu 8gb RAM

For ActiveJob tasks, use config.processor_host = "application_service.run.app"

For Cloudtasker::Cron::Schedule ( Workers ), use processor_host = "application_cron_tasks.run.app"

Would it be possible?

Hey @alachaum, any idea why should the app starts to get the wrong project id ?

It's seems google SDK it getting the project ID from the application default user on the environment and not from the cloudtasker initializer configuration.

The documentations says that the configuration and project id on methods has precedency:
https://github.com/googleapis/google-cloud-ruby/blob/main/google-cloud-bigquery/AUTHENTICATION.md#project-and-credential-lookup.

This erro started after migration from long-lived json keys to short-lived workload identity federation. But I guess that any error that I may have made in the service accounts should not take precedence and the hard coded project id that I am passing to cloudtasker and then to cloud SDK and Cloud Task lib.

Example of the error on instance deploy, cloudtasker load, that avoids app to start.

It would be useful to provide an optional UI - at least in development mode - showing the list of jobs pending, running, which ones have failed and the failure reason.

After installing cloudtasker v0.12.2 I got the following warning after bundle install:

The google-gax gem is officially end-of-life and will not be updated further.

If your app uses the google-gax gem, it likely is using obsolete versions of
some Google Cloud client library (google-cloud-*) gem that depends on it. We
recommend updating any such libraries that depend on google-gax. Modern Google
Cloud client libraries will depend on the gapic-common gem instead.

I haven't fully dug in to the code on the usages on google-gax yet but hopefully a dependency update will just work. I can give it a shot if this wasn't already being worked on right now. If it is, is there an existing branch? Thanks in advance!

Having a hard time getting cloud sql and cloud redis to both be available via cloud run/build it seems like if you add a vpc to connect redis you loose cloud sql connection.

Would you be open to a option to store cron config in Postgres?

Callback errors are currently swallowed.
See:

We should instead consider the job to be failed and let retries do their job.

Rails 5.2.2, gem version 0.11.

Cloudtasker::WorkerController uses an ambigious superclass, if Cloudtasker::ApplicationController is not loaded it simply subclasses ::ApplicationController, e.g. i can reproduce in console:

% rails c
Loading development environment (Rails 5.2.2)
[1] pry(main)> Cloudtasker::WorkerController.ancestors.include?(Cloudtasker::ApplicationController)
=> false

% rails c
Loading development environment (Rails 5.2.2)
[1] pry(main)> Cloudtasker::ApplicationController
=> Cloudtasker::ApplicationController
[2] pry(main)> Cloudtasker::WorkerController.ancestors.include?(Cloudtasker::ApplicationController)
=> true

Currently cron jobs are managed using Cloud Tasks and Redis. Investigate a way to also support Cloud Scheduler to eliminate the dependency on Redis.

Hi, we're trying to force the job_id for ActiveJob with CloudTasker (GCP-backed), and it looks like it is not passed down to the worker instance.

This would be useful for deduplication purposes.

Any hints on how to realize this?

Thanks

I sometimes feel that job arguments are not enough or become too expansive when a certain context needs to be passed down to sub jobs (especially in the context of batch jobs)

I feel like it could be nice to have tags or a shared context that can be inherited by jobs so we can know the source/lineage of the job and take actions based on that.

E.g:

• you have generic jobs used to retrieve data from GitHub. These jobs are used for long polling and on user request to refresh data
• GitHub has API credits, so you're limited in the number of calls per hour. You don't want long polling jobs to eat up all your credits because it means that users won't be able to ask for a refresh until credits reset
• When you enqueue these jobs you tag them with "on-request" or "long-polling"
• Based on this tag and the amount of remaining credits you can decide in the job whether to run/reenqueue/abort the job

Tracing the lineage/context of jobs can of course be done with arguments but I find it clunky. Queues can also be used for that purpose but they're heavy to create/use - tags should be lightweight to use.

I'm still evaluating whether this a good idea or an overkill.

Cloud Tasks enforces a limit of 100 KB for the task size. Excluding headers and meta information, this leaves ~80-90KB for free space of the actually job parameters.

Cloudtasker is currently configured to raise a Cloudtasker::MaxTaskSizeExceededError if the job payload is greater than 100 KB.

We could actually improve that behaviour by allowing users to choose Redis for storing payloads (with a configured threshold maybe?).

Cloudtasker.configure do |config|
  # Store all job payloads in Redis
  payload_storage = :redis
  
  # Or configure a threshold - in KB - above which payloads will be stored in Redis
  # payload_storage = { redis: { threshold: 50 } }
end

After taking a look at how the worker processor is implemented (the part that receives the tasks from Google Cloud Tasks to be processed), I found out there's a ready-to-use controller for rails, but for folks using anything else they'll need to re-implement the controller themselves (as shown in the sinatra example)

I think we can have a single implementation of the controller as a rack application (See "How to survive without a framework in Ruby", for an example), and let the gem users mount it using their corresponding routers (or auto-configure it ourselves, maybe?) - notice how all routers use the same method mount with the same options (at, etc):

Rails example - see documentation at guides:

# at `config/routes.rb`
Rails.application.routes.draw do
  mount Cloudtasker::WorkerController, at: '/cloudtasker'
end

Sinatra Example (can't find a good documentation page):

class MySinatraApp
  mount Cloudtasker::WorkerController, at: '/cloudtasker'
end

Hanami example - see documentation:

Hanami::Router.new do
  mount Cloudtasker::WorkerController, at: '/cloudtasker'
end

It would be cool for Cloudtasker to be a backend for Rails ActiveJob

This would require implementing an ActiveJob::QueueAdapter on this library.

I've already done this for another project, so maybe I can help.

The build process using Github Actions is cool, but takes a bit of time.

I think I have a couple of ideas to speed it up. Namely:

• Use caching
• Use docker to run the tests

Hi everyone,

I'm deploying a RoR app on GCP Cloud Run, that uses Cloudtasker to schedule async workloads using GCP Cloud Tasks. While reviewing the gem's docs (specifically https://github.com/keypup-io/cloudtasker#cloud-tasks-authentication--permissions) there is a link to Google Cloud Authentication guide. that mentions that the project id (and other things, like credentials) are loaded auto-magically from the GCP runtime environment, and thus we as devs don't have to worry about loading said values.

Based on that "project id is loaded magically for you", I have an app running on GCP Cloud Run where Cloudtasker.config.gcp_project_id is NOT set from a string/env var/anything. However, when scheduling an async task, I am getting the following on Cloud Run logs:

2022-05-17 17:09:15.255 CEST 15:09:15 web.1 | E, [2022-05-17T15:09:15.252525 #20] ERROR -- : [569d56a4-2004-4661-84ae-c98bd548867b] Missing GCP project ID. Default 2022-05-17 17:09:15.255 CEST 15:09:15 web.1 | Please specify a project ID in the cloudtasker configurator.

When looking into this, I noticed that Cloudtasker's link to Google's docs on authentication actually point to the docs of the google-cloud-bigquery gem, and the equivalent docs for the google-cloud-tasks gem here don't mention anything about the project id being loaded automatically.

So my question is: do I need to explicitly define Cloudtasker.config.gcp_project_id and Cloudtasker.config.gcp_location_id even when running on GCP (and, in which case, it might be good to update the docs here to make this explicit and help noobs like get have a smoother experience), or am I missing something?

In case you think updating the docs will be helpful, I am happy to volunteer for that, as a way of saying "thank you" for this gem.

Thanks

Emails are dropped into the mailer queue

If ActiveJob job fails, then CloudTasker returns 422 error code and that's it. Cloud Tasks will keep re-trying this job for some time, but that's about it. We are using Rollbar to report about exceptions, but in this case exception will never reach Rollbar (or other exception tracking systems I would assume) because every error is rescued here

The X-CloudTasks-TaskExecutionCount header sent by Google Cloud Tasks and providing the number of retries outside of HTTP 503 (instance not reachable) is currently bugged and remains at 0 all the time.

Starting with 0.10.rc3 Cloudtasker uses the X-CloudTasks-TaskRetryCount header to detect the number of retries. This header includes HTTP 503 errors which means that if your application is down at some point, jobs will fail and these failures will be counted toward the maximum number of retries. A bug report has been raised with GCP to address this issue.

Once fixed we will revert to using X-CloudTasks-TaskExecutionCount to avoid counting HTTP 503 as job failures.

After sorting redis to store the cron jobs, now im now seeing an odd error on boot:

ogle::Gax::PermissionDeniedError: GaxError RPC failed, caused by 7:Permission denied on 'locations/eu-west2' (or it may not exist).. debug_error_string:{"created":"@1627941649.186614634","description":"Error received from peer ipv4:142.250.200.10:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Permission denied on 'locations/eu-west2' (or it may not exist).","grpc_status":7}

It used to boot - do you think its just a GCP error? Could permissions be missing after this used to boot?

We need that cloudtasker don't brake in this casa.

CloudTasker::Cron load breaking application starts when worker tasks ETA is more then 720h in the future.

2021-10-01T15:26:04.698271Z/usr/local/bundle/gems/google-gax-1.8.2/lib/google/gax/api_callable.rb:264:in rescue in block in create_api_call': GaxError RPC failed, caused by 3:The Task.scheduleTime, 2021-11-01T03:00:00-07:00, is too far in the future. Schedule time must be no more than 720h in the future.. debug_error_string:{"created":"@1633101964.695316382","description":"Error received from peer ipv4:142.250.148.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"The Task.scheduleTime, 2021-11-01T03:00:00-07:00, is too far in the future. Schedule time must be no more than 720h in the future.","grpc_status":3} (Google::Gax::InvalidArgumentError)
`

Rails 5.2.2, gem version 0.11.

I understand retry_on and company not supported yet (though discard_on simply works), so this is not strictly a bug.

executions (with provider_id and priority though these 2 are not problematic) is filtered from serialization (ActiveJob::QueueAdapters#build_worker and ActiveJob::QueueAdapters::SERIALIZATION_FILTERED_KEYS) and supplied by google cloud tasks (also see #6), however because of retrying/rescheduling (new task id, retry count is 0) this keeps resetting, which in turn leads to never ending retrial.

If retry_on functionality is desirable i was thinking maybe putting this information in job_meta (or even in root worker_payload) it could be retained. If you are not opposing this change i would gladly try to make a PR for it.

When Cloudtasker use with Rails on Docker Compose, Cloudtasker does not work with setting as below.

Cloudtasker.configure do |config|
  config.processor_host = ENV['PROCESSOR_HOST'] # => http://web:3000
end

although, with this setting is working.

Cloudtasker.configure do |config|
  config.processor_host = 'http://web:3000'
end

Can I ask how do I setting processor_host via environment variable?

Rails 5.2.2, gem version 0.11.

It seems the google.cloud.tasks.v2beta3.HttpRequest message has a map :headers, :string, :string, 3 (google-cloud-tasks 1.1.3, lib/google/cloud/tasks/v2beta3/target_pb.rb), however Cloudtasker::Backend::GoogleCloudTask.format_task_payload does a json encode-decode (deep duplication judging by the comment) with symbolize_names: true, which makes payload[::http_request][:headers] have symbol keys, which in turn raises an error in google-gax (in title). (Furthermore the method would overwrite Content-Type but it sets the key as a string, while Cloudtasker::WorkerHandler#task_payload already set it but json encode-decode converted it to a symbol key.)

My quick fix is prepend a module and stringify the keys in the headers (eg in the cloudtasker initializer)

module CloudTaskerFix
  def format_task_payload(payload)
    super.tap { |pld| pld.dig(:http_request, :headers)&.stringify_keys! }
  end
end

require 'cloudtasker/backend/google_cloud_task'
Cloudtasker::Backend::GoogleCloudTask.singleton_class.prepend CloudTaskerFix

First thanks for creating this fantastic gem. It's really well implemented and a joy to migrate to from Resque.

I am experiencing a boot error when loading the cron schedule (rescue in get_task) for a resource project I don't recognize.

{
insertId: "63c7f4110001ae265488ce27"
labels: {1}
logName: "projects/MY_PROJECT_ID/logs/run.googleapis.com%2Fstderr"
receiveTimestamp: "2023-01-18T13:28:49.450574676Z"
resource: {2}
textPayload: "/usr/local/bundle/gems/google-cloud-tasks-v2-0.6.0/lib/google/cloud/tasks/v2/cloud_tasks/client.rb:1537:in `rescue in get_task': 7:Permission denied on resource project 87ec33c5-3856-4826-a81e-483b47bbf038.. debug_error_string:{UNKNOWN:Error received from peer ipv4:142.250.159.95:443 {created_time:"2023-01-18T13:28:49.103259885+00:00", grpc_status:7, grpc_message:"Permission denied on resource project 87ec33c5-3856-4826-a81e-483b47bbf038."}} (Google::Cloud::PermissionDeniedError)"
timestamp: "2023-01-18T13:28:49.110118Z"
}

As far as I understand the Permission denied on resource project 87ec33c5-3856-4826-a81e-483b47bbf038 should contain the my project ID? Instead there is a string which after analyzing everything looks to be the same format as a cloudtaskser job id however I cannot find a job with that ID either so perhaps that's just a coincidence as it also doesn't make sense in the context. I do see 2 tasks appearing in the Cloud Task dashboard however they present the following error message when clicked on Task does not exist. It can be dispatched or deleted. ... Ignore that I just deleted the Redis keys manually and tried redeploy and now there is nothing there so I am assuming it was grabbing the existing task schedule from the previous successful deployment.

I've tried numerous different service accounts and the problem persists. I have also tried rebuilding and deploying with the --no-cache option just in case something strange went on there but that also wasn't the issue.

Any pointers or ideas for further investigation in resolving this would be appreciated.

Hello,

I'm trying to get cloudtasker 0.13 working as an ActiveJob adapter and I'm getting an error when I enqueue a job: ArgumentError (Value 600 must be a Hash or a Google::Protobuf::Duration). I'm not sure if I'm doing something wrong or if I'm running into legitimate bugs. I've read through the docs and don't see any obvious mistakes. I was able to recreate this from a brand new Rails 7.0.5 project, the only additional gem is cloudtasker. I created a simple job:

class EchoJob < ApplicationJob
  queue_as :default

  def perform(message)
    Rails.logger.warn "The message is: #{message}"
  end
end

which is queued like this:

class EchoController < ApplicationController
  def index
    EchoJob.perform_later("Hello, World!")
  end
end

I set the queue adapter in config/development.rb config.active_job.queue_adapter = :cloudtasker. And the cloudtasker initializer is:

Cloudtasker.configure do |config|
  config.gcp_location_id = 'us-central1'
  config.gcp_project_id = 'XXXXX'
  config.gcp_queue_prefix = 'YYYYY'
  config.processor_host = 'ZZZZZ.ngrok-free.app/'
  config.mode = :production
end

The first issue is I'm getting an error that reads: [ActiveJob] Failed enqueuing EchoJob to Cloudtasker(default): ArgumentError (Value 600 must be a Hash or a Google::Protobuf::Duration). The stack trace shows the error coming from lib/cloudtasker/backend/google_cloud_task_v2.rb:146 which passes off the serialized job to the Google Cloud Tasks gem which throws the error. The value 600 is coming from lib/cloudtasker/worker_handler.rb:162 where dispatch_deadline is being set to to 600. I was able to validate that commenting out that line causes that error to go away:

def task_payload
      {
        http_request: {
          http_method: 'POST',
          url: Cloudtasker.config.processor_url,
          headers: {
            Cloudtasker::Config::CONTENT_TYPE_HEADER => 'application/json',
            Cloudtasker::Config::AUTHORIZATION_HEADER => Cloudtasker.config.oidc ? nil : Authenticator.bearer_token
          }.compact,
          oidc_token: Cloudtasker.config.oidc,
          body: worker_payload.to_json
        }.compact,
        # dispatch_deadline: worker.dispatch_deadline.to_i,
        queue: worker.job_queue
      }
    end

That led me to a second protobuf ArgumentError, this one caused by the schedule_time option being set to nil in lib/cloudtasker/backend/google_cloud_task_v2.rb:104. I changed the last line of that method (line 113) to payload.compact which fixed it:

      def self.format_task_payload(payload)
        payload = JSON.parse(payload.to_json, symbolize_names: true) # deep dup

        # Format schedule time to Google Protobuf timestamp
        payload[:schedule_time] = format_schedule_time(payload[:schedule_time])

        # Encode job content to support UTF-8.
        # Google Cloud Task expect content to be ASCII-8BIT compatible (binary)
        payload[:http_request][:headers] ||= {}
        payload[:http_request][:headers][Cloudtasker::Config::CONTENT_TYPE_HEADER] = 'text/json'
        payload[:http_request][:headers][Cloudtasker::Config::ENCODING_HEADER] = 'Base64'
        payload[:http_request][:body] = Base64.encode64(payload[:http_request][:body])

        payload.compact
      end

After making those two changes enqueuing the job was working perfectly. Again, I'm not sure if I have a config error or if these are actually bugs, looking for some expertise to point me in the right direction.

Thanks!

More of a question than an issue: how does CloudTasker handle long running jobs? If I would want to run a jog that takes 30 minutes (some video encoding for example), will it timeout? Because I assume by default HTTP request will timeout pretty fast.

Hi,

We're running into an issue with deliver_later where this error is thrown.

grcp cannot be used before and after forking

Anyone seen this before?

Thanks,
Matt

See: https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-tasks/MIGRATING.md

google-cloud-tasks was locked to v1 in cloudtasker v0.10.rc8