Compatibility with Resque implementations ?

[louisameline]

Hello, a quick question :

I am currently using Resque implementations in PHP (php-resque) and Node (coffee-resque) to queue jobs between different parts of my application and they work fine together. I need to do the same in Python now, is Pyres compatible ? If yes, is it compatible out of the box or are there any configuration pitfalls that I should be aware of ?

It seems to be compatible but it's not clearly stated in the doc and Pyres is not listed in the "Other implementations" section of the Resque project, so I ask beforehand :)

Thanks for the good work.

[bluegod]

Did anyone find out if Pyres is compatible?

[binarymatt]

@louisameline and @bluegod i guess the answer depends on the definition of compatible. Can you enqueue jobs with another implementation and pop them off with pyres? If the json that is push is the same, then yes. For instance, a pyres job looks for a json payload that has the following keys:
class, args, enqueue_timestamp. From the tests you can see what a sample payload looks like
{'class':'tests.Basic','args':['test1'],'enqueue_timestamp':job.enqueue_timestamp}

[louisameline]

Thanks for your answer.

Ok, I see that class and args seem to be the only properties needed to enqueue a job (you add an extra enqueue_timestamp and php-resque adds an extra id but I guess this does not matter).

It gets more complex when you deal with job failures though. Coffee-resque has this payload similar to Resque's one (https://github.com/resque/resque/blob/master/lib/resque/failure/redis.rb) that makes its errors seamlessly monitorable in Resque-web too :

{
    "worker": "node:7236:node_contact_import",
    "queue":"node_contact_import",
    "payload": {
        "class":"import",
        "args":[....],
        "id":"716e8ac85c64e10286df04881c28f70d"
    },
    "exception":"error",
    "error":"my error name",
    "backtrace": [
        " at Connection.parseE",
        " at Connection.parseMessage,
        " at TCP.onread (net.js:526:21)"
    ],
    "failed_at":"Wed Aug 13 2014 11:04:12 GMT+0200 (Paris, Madrid (heure d�été))"
}

Have you done the same for Pyres ?
I could check the source myself but I thought I'd ask you at the same time to include this information in the documentation if you're willing to.
Thank you.

[gkop]

I agree @louisameline that we should unambiguously document the level of compatibility between pyres and resque. I am about to wrap some python code in a worker job of some kind, and would prefer to use pyres, but want to know "Well is it a clone of resque, or not a clone of resque?"

[binarymatt]

@louisameline sorry about the long delay, i've not been seeing github notifications in my gmail. Pyres follows that same failure json. There may be some slight differences because it's not been a priority to track resque in the last few years.

[louisameline]

Hello again !

Thank you, I am eventually implementing Pyres. Unfortunately with difficulty because the documentation falls short of thorough and explained examples for python total newbies like me.

Anyway, I am trying to send a job from Python for Node to pick it up. I have created a parse class that has a queue = 'node' attribute and I call resque.enqueue(parse, arg). The problem is that the class property in the JSON payload is __main__.parse while Node only expects parse. What can I do ?

Thank you.

[louisameline]

Looking at the source, I found a resq.enqueue_from_string(className, queue, *args, **kwargs) method that does just what I need. I think it should be documented.

Next issue : the other way around, when enqueing from Node to Python. From the payload, Pyres seems to expect a class property in the form packageName.className. I guess I could include the package name when I enqueue in my Node app, but that does not really sound right. My Node app should probably not have to care for what package will be used by Python, that's outside of its scope.

If pyres_worker and pyres_manager need the location/package of the class, could we pass it as an IMPORT_PATH option in the command line instead ? I don't know if it would be a very python-like way of doings things but it would :

• solve the issue of the package name being present in the class property
• save me the trouble of having to learn how import works and how I'm supposed to declare my app as a package. I know that sounds lazy :p ^^

Thanks

[louisameline]

I confirm that everything works well when I package my python worker class under the name "workers" and give a "workers.parse" class when I enqueue in Coffee-Resque (Node). However I haven't tested to run ResqueWeb yet.

For newbies like me : to package my parse class (included in a file called "workers.py") so that Pyres will find it, I followed the simple instructions given in this helpful discussion. Basically, you just have to execute this file :

#!/usr/bin/python

from distutils.core import setup

setup(name='workers',
    version='1.0.0',
    py_modules=['workers'],
    )