goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker is compatible with Resque, so you can push your jobs with Rails and Resque, and consume them with Go in the background.

Install It

Install goworker.

go get github.com/benmanns/goworker

Make a worker.go file.

package main

import (
  "fmt"
  "github.com/benmanns/goworker"
)

func main() {
  if err := goworker.Work(); err != nil {
    fmt.Println("Error:", err)
  }
}

Call It

Create your hello_worker.go

package main

import (
  "fmt"
  "github.com/benmanns/goworker"
)

func init() {
  goworker.Register("Hello", helloWorker)
}

func helloWorker(queue string, args ...interface{}) error {
  fmt.Println("Hello, world!")
  return nil
}

And use Resque from your Ruby application

class Hello
  @queue = :hello
end

Resque.enqueue(Hello)

Run It

Build your worker

go build

And run it!

./worker -queues=hello

How fast is it?

goworker is typically around 70x faster than Resque and 10x faster than Sidekiq.

Hello World

For simple dequeuing, deserialization, execution, and updating stats (as with the Hello worker above), goworker is faster than Resque and faster than Sidekiq.

Additionally, goworker only used 7MB of memory, while Sidekiq used 74MB and Resque 62MB.

puts "Hello World!"

Seconds for N jobs

Computation Heavy

For a 20x20 matrix multiplication operation, goworker is faster than Resque and faster than Sidekiq.

a = Matrix[...]
b = Matrix[...]
c = a * b
puts a * b

Seconds for N jobs

High Wait Time

For high latency jobs, goworker (at concurrency 25) is faster than Resque and faster than Sidekiq (at concurrency 25).

However, because goworker threads are so cheap, it is feasible to run 100,000 concurrent workers on a single 4GB machine, each sharing a single Redis connection. This gives goworker the theoretical 100,000x boost over Resque and 20,000x over Sidekiq.

sleep(1)

Seconds for N jobs

Database Insert

For a worker that inserts records into a Postgres database, goworker is faster than Resque and faster than Sidekiq (at concurrency 25).

Contact.create!({
  name: name,
  email: email,
  phone: phone,
})

Seconds for N jobs

Who's it for?

goworker is for people who are currently running more than two servers or dynos for their background workers. For those who are not running background jobs, yet, starting with goworker may be a case of premature optimization. For those instances, I recommend starting with a Ruby background worker library like Sidekiq or Resque. With these, you will get all the benefits of your existing models and libraries. Use goworker when you're sure that you need the extra speed boost.

Getting Started

To get started with goworker, check out the getting started guides on the wiki.

For more information, updates, tips, and free one-on-one help getting started, sign up for the email list.

Created by

goworker is maintained by Benjamin Manns and community contributors.

Fork me on GitHub