goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.
goworker is compatible with Resque, so you can push your jobs with Rails and Resque, and consume them with Go in the background.
Install goworker.
go get github.com/benmanns/goworker
Make a worker.go
file.
package main
import (
"fmt"
"github.com/benmanns/goworker"
)
func main() {
if err := goworker.Work(); err != nil {
fmt.Println("Error:", err)
}
}
Create your hello_worker.go
package main
import (
"fmt"
"github.com/benmanns/goworker"
)
func init() {
goworker.Register("Hello", helloWorker)
}
func helloWorker(queue string, args ...interface{}) error {
fmt.Println("Hello, world!")
return nil
}
And use Resque from your Ruby application
class Hello
@queue = :hello
end
Resque.enqueue(Hello)
Build your worker
go build
And run it!
./worker -queues=hello
goworker is typically around 70x faster than Resque and 10x faster than Sidekiq.
For simple dequeuing, deserialization, execution, and updating stats (as with the Hello worker above), goworker is faster than Resque and faster than Sidekiq.
Additionally, goworker only used 7MB of memory, while Sidekiq used 74MB and Resque 62MB.
puts "Hello World!"
For a 20x20 matrix multiplication operation, goworker is faster than Resque and faster than Sidekiq.
a = Matrix[...]
b = Matrix[...]
c = a * b
puts a * b
For high latency jobs, goworker (at concurrency 25) is faster than Resque and faster than Sidekiq (at concurrency 25).
However, because goworker threads are so cheap, it is feasible to run 100,000 concurrent workers on a single 4GB machine, each sharing a single Redis connection. This gives goworker the theoretical 100,000x boost over Resque and 20,000x over Sidekiq.
sleep(1)
For a worker that inserts records into a Postgres database, goworker is faster than Resque and faster than Sidekiq (at concurrency 25).
Contact.create!({
name: name,
email: email,
phone: phone,
})
goworker is for people who are currently running more than two servers or dynos for their background workers. For those who are not running background jobs, yet, starting with goworker may be a case of premature optimization. For those instances, I recommend starting with a Ruby background worker library like Sidekiq or Resque. With these, you will get all the benefits of your existing models and libraries. Use goworker when you're sure that you need the extra speed boost.
To get started with goworker, check out the getting started guides on the wiki.
For more information, updates, tips, and free one-on-one help getting started, sign up for the email list.
goworker is maintained by Benjamin Manns and community contributors.