Collective Idea

Collective Idea Logo

Steve Richert

Background Job Monitoring for Atheists

By Steve Richert on November 14, 2013 in dms and god

I’m a Rails developer but I’m not much of an Ops guy. So when it comes to server monitoring, I want a simpler solution than God. God is a “Process Monitoring Framework in Ruby” and while I know it’s awesome, sometimes it’s not quick and dirty enough for my liking.

Fail Loudly

Usually, if one of my application’s dependencies (PostgreSQL, Redis) goes down, it fails loudly. We love Honeybadger for error notifications so we hear about these failures right away and we can snap into action. Perfect.

Background Jobs

Background jobs are similar in that no news is good news. The major difference is that if your background job process were to go down, you still get… no news. So how do we make sure that our background job process is always running?

Dead Man’s Snitch

Dead Man’s Snitch is a process monitoring service for periodic events, like cron jobs. Signing up is easy and your first “snitch” is free. Each snitch has its own URL that you ping every so often to confirm your process is still up and running.

For this case, I created an hourly snitch. Now I can just define a worker and queue up a job!

class SnitchWorker
  include Sidekiq::Worker

  def perform(token)
    `curl https://nosnch.in/#{token}`
    SnitchWorker.perform_in(1.hour, token)
  end
end
 > SnitchWorker.perform_async("c2354d53d2")
=> "d273eb429dc8877aedf6eb0c"

This worker just pings the snitch and then tell itself to do it all again in an hour.

Now if my background job process ever goes down (or even gets really backed up), my snitch will stop receiving pings and Dead Man’s Snitch will let me know.

God not required.

By Steve Richert on November 14, 2013 in dms and god

Post a Comment

Contact Us

Find us on Google Maps
Collective Idea
44 East 8th Street, Suite 410
Holland, Michigan 49423 USA 42.790334-86.105251

Follow us on the Interwebs

We are currently available for medium and long term projects. Please get in touch if we can be of service.