Working with PHP and Beanstalkd

I have just introduced Beanstalkd into my current PHP project; it was super-easy so I thought I'd share some examples and my thoughts on how a job queue fits in with a PHP web application.

The Scenario

I have an API backend and a web frontend on this project (there may be apps later. It's a startup, there could be anything later). Both front and back ends are PHP Slim Framework applications, and there's a sort of JSON-RPC going on in between the two.

The job queue will handle a few things we don't want to do in real time on the application, such as:

• updating counts of things like comments; when a comment is made, a job gets created and we can return to the user. At some point the job will get processed updating the counts of how many comments are on that thing, how many comments the user made, adding to a news feed of activities ... you get the idea.


• cleaning up; we have had a few cron jobs running to clean up old data but now those cron jobs put jobs into beanstalkd which gives us a bit more visibility and control of them, and also means that those big jobs aren't running on the web servers (we have a separate worker server)


• other periodic things like updating incoming data/content feeds or talking to some of the 3rd party APIs we use like Mailchimp and Bit.ly

Adding Jobs to the Queue

There are two ends to this process, let's start by adding jobs to the queue. Anything you don't want to make a user wait for is a good candidate for a job. As I mentioned, some of our jobs get handled periodically with cron creating jobs, but since they are just beanstalkd jobs I can easily give an admin interface to trigger them manually also. In this case, I'm just making a job to process things we update when a user makes a comment.

A good job is very self-contained; a bit like a stateless web request it should contain anything that is needed to process it and not rely on anything that went before. On a live platform you would typically have many workers all consuming jobs from a single queue so there are no guarantees that one job will be completed before the next one begins to be processed! You can put any data you like into a job; you could send all the data fields to fill in and send an email template for example.

In this example I need to talk to the database anyway so I'm just storing information about which task should be done and including the comment ID with it.

I'm using an excellent library called Pheanstalk which is well-documented and available via Composer. The lines I added to my composer.json:

  "require": {
    "pda/pheanstalk": "2.1.0",
  }

I start by creating an object which connects to the job server and allows me to put jobs on the queue:

    new Pheanstalk_Pheanstalk(
        $config['beanstalkd']['host'] . ":" . $config['beanstalkd']['port']
    )

The config settings there will change between platforms but for my development version of this project, beanstalkd is just running on my laptop so my settings are the defaults:

[beanstalkd]
host=127.0.0.1
port=11300

Once you have the object created, $queue in my example, we can easily add jobs with the put() command - but first you specify which "tube" to use. The tubes would be queues in another tool, just a way of putting jobs into different areas, and it is possible to ask the workers to listen on specific tubes so you can have specialised workers if needed. Beanstalkd also supports adding jobs with different priorities.

Here's adding the simple job to the queue; the data is just a string so I'm using json_encode to wrap up a couple of fields:

  "require": 

Truncated by Planet PHP, read more at the original (another 5142 bytes)

more