Death by inotify

One of the problems I recently needed to solve at work was moving (and transforming) a lot (TBs) of data from an old system to a new system. Solved it, and things were good. Unfortunately we (co-workers and I) have noticed that the script just stops working. It doesn’t crash it just stops doing anything. Through a bit of luck I discovered this was due to network IO (or lack thereof).

The problem I faced over the last few days is that the script stops transferring data over the network but doesn’t hit a network timeout. The transfer of data grinds to a halt and doesn’t trigger any of our monitoring. I’ve solved this with an inotifywait loop.

While the script is busy copying data it spits information out into a log file. It writes to this file a few times per second with occasional pauses of up to 25ish seconds to collect a new collection of work to transfer. It so turns out that inotifywait has a -t input which is timeout in seconds. If inotifywait gets a notify event it exits with code 0 and if it times out it exits with code 2. With that bit of knowledge in hand, I wrote a wrapper script that launches the aforementioned script in the background and then goes into an infinite loop. It then sets up an inotifywait with -t 60 and -e modify on the log file. If the exit code is 2 then it runs a ps aux | grep to get the pid, kills the script, and relaunches it in the background. In pseudo-code:

./ &
   inotify -t 60 -e modify /path/to/my/logfile
   if exitCode == 2 then
      pid = ps aux | grep myScript | awk '{print $2}'
      kill pid
      ./ &

With that in place the script runs until it stops outputting to the log file for 60 seconds which this wrapper script interprets as it failing. In that event the wrapper script kills the script, restarts it, and we’re back up and running again. Not something I would consider a long term solution but this isn’t a long term problem.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: