[mirrorbrain-commits] r7797 - /trunk/tools/metalink-hasher.py

Author: poeml
Date: Wed Sep  9 04:24:29 2009
New Revision: 7797

URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?rev=7797&view=rev
Log:
metalink-hasher:
- try to prevent a race condition with the directory-wise locking (which I
  didn't expect to happen with the fcntl.lockf wrapper, but well), by:
  - trying harder to not accidentally remove a foreign lockfile, by adding the
    lockfile to the dst_keep set() outside of exception handling trees.
  - directly after the lock was acquired, check if the file still exists -- it
    might have been deleted, because unlinking is not blocked by locks.
  - first unlink the lock file, and close it only then.
  This altogether seems to be effective. The race condition that I could
  reproduce by running two jobs at the same time on a huge tree could be
  reproduced in less then a minute, and now it runs since hours.
- for portability reasons, check for EAGAIN and EACCES errors in addition to
  EWOULDBLOCK when locking

Modified:
    trunk/tools/metalink-hasher.py

Modified: trunk/tools/metalink-hasher.py
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/metalink-hasher.py?rev=7797&r1=7796&r2=7797&view=diff
==============================================================================
--- trunk/tools/metalink-hasher.py (original)
+++ trunk/tools/metalink-hasher.py Wed Sep  9 04:24:29 2009
@@ -259,19 +259,29 @@
                 print 'looking at', src_dir
 
             dst_keep = set()
+            dst_keep.add('LOCK')
 
             lockfile = os.path.join(dst_dir, 'LOCK')
             try:
                 if not opts.dry_run:
                     lock = open(lockfile, 'w')
                     fcntl.lockf(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
-                dst_keep.add('LOCK')
+                    try:
+                        os.stat(lockfile)
+                    except OSError, e: 
+                        if e.errno == errno.ENOENT:
+                            if opts.verbose:
+                                print '====== skipping %s, which we were about to lock' % lockfile
+                            continue
+
                 if opts.verbose:
                     print 'locked %s' % lockfile
             except IOError, e:
-                if e.errno == errno.EWOULDBLOCK:
+                if e.errno in [ errno.EAGAIN, errno.EACCES, errno.EWOULDBLOCK ]:
                     print 'Skipping %r, which is locked' % src_dir
                     continue
+                else:
+                    raise
 
 
             for src_basename in sorted(src_basenames):
@@ -352,8 +362,8 @@
             if opts.verbose:
                 print 'unlocking', lockfile 
             if not opts.dry_run:
+                os.unlink(lockfile)
                 lock.close()
-                os.unlink(lockfile)
 
         if  unlinked_files or unlinked_dirs:
             print 'Unlinked %s files, %d directories.' % (unlinked_files, unlinked_dirs)




_______________________________________________
mirrorbrain-commits mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/

Note: To remove yourself from this list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-commits-request_at_mirrorbrain.org