[Proposal] Change Payload to bytes & Add Wrapper around Key-Value pair payload about asynq HOT 11 CLOSED

That's great! I should probably document the message schema in Wiki so that people can write clients in other languages :)

Related: #105

from asynq.

Closing this for now.

from asynq.

Hey @hibiken,
coming back to this, because I am running the same problem here.
Would love to store []byte in the message itself.

What was your reasoning for closing this?

from asynq.

Hey @andygrunwald thanks for your comment!
I'm actually considering changing the signature of NewTask to

func NewTask(typename string, payload []byte) *Task

and add a nice wrapper around the payload to make it easier to do key-value pairs like the current API supports.

something like:

t := NewTask("mytask", asynq.KV(map[string]interface{}{"mykey": 123}))

I'll open this issue again so that it's more visible to people.

In the meantime, is it possible to workaround this by storing the bytes as string in the payload?

from asynq.

Thanks @hibiken.
I thought about something similar.

In the meantime, is it possible to workaround this by storing the bytes as string in the payload?

Didn't check this yet. Need to do another test.
However, I am also running into this issue:

I think the []byte payload is able to solve both.

from asynq.

Related to #245

from asynq.

@andygrunwald Would you mind describing the issue you're seeing?
I want to understand your use case so that I can take that into account when I make API change to support storing payload as []byte.

from asynq.

Hello @hibiken

I have the same issue when consumer tried to execute task with this payload:

WARN: Could not remove task id=f83cca71-620b-45ff-a309-c6917dd2f248 type="app.event" from "asynq:{app.event}:active" err: NOT FOUND; Will retry syncing

Payload:

{"eventType": "Signup", "countryCode": "au"}

The full message is:

{"Type":"app.event","Payload":{"countryCode":"au","eventType":"Signup"},"ID":"5663097f-6cb7-4464-b1d2-b6a8c5a59bb7","Queue":"app.event","Retry":25,"Retried":0,"ErrorMsg":"","Timeout":1800,"Deadline":0,"UniqueKey":""}

If I change key from eventType to abc in the payload everything is ok and no errors occur.

{"abc": "Signup", "countryCode": "au"}

Is there any way to fix it?

from asynq.

^^ UPD:

I found the difference, the keys of object sorted in different way, looks like expected payload sorted alphabetically.

Payload in redis queue asynq:{app.event}:active

"Payload":{"eventType":"Signup","countryCode":"au"}"

Payload in consumer

"Payload":{"countryCode":"au","eventType":"Signup"}"

The problem related to php producer, json_encode method does not work as json.Marshal in golang

from asynq.

@disc Thanks for updating this issue.
Just to clarify, you're enqueueing a task using PHP client? Not using asynq.Client?

FYI: This issue should go away in the next release. I'm currently working on redis key re-design so that we only need to move IDs around and don't need to use JSON data to remove entry from redis sets.

from asynq.

@disc Thanks for updating this issue.
Just to clarify, you're enqueueing a task using PHP client? Not using asynq.Client?

Yes, I wrote PHP client for asynq, it just produces a message to the redis queue as the asynq worker expects

from asynq.