[mirrorbrain-commits] r8359 - in /trunk: docs/changes.rst mb/mb/crawl.py

Author: poeml
Date: Sun Feb  2 13:33:53 2014
New Revision: 8359

URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?rev=8359&view=rev
Log:
update changelog

Modified:
    trunk/docs/changes.rst
    trunk/mb/mb/crawl.py

Modified: trunk/docs/changes.rst
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/docs/changes.rst?rev=8359&r1=8358&r2=8359&view=diff
==============================================================================
--- trunk/docs/changes.rst	(original)
+++ trunk/docs/changes.rst	Sun Feb  2 13:33:53 2014
_at_@ -113,6 +113,10 @@
 * :program:`tools/geoiplookup_*`: They no longer segfault when opening a GeoIP
   database database fails. Patch courtesy of Dagobert Michelsen. (`issue 138`_)
 
+* :program:`mb scan`: 
+  - When FTP URLs are not correct, and the directory is not found on an
+    FTP server, the scanner bailed out. Fixed with patch from Dago. (`issue 139`_)
+  - when in verbose mode, don't wrongly log symlinks as directories (`issue 141`_) 
 
 * The file ``mb/countries.py`` was never used; remove it so it doesn't confuse anyone.
   Thanks Gokdeniz for the hint.
_at_@ -141,6 +145,8 @@
 .. _`issue 136`: http://mirrorbrain.org/issues/issue136
 .. _`issue 137`: http://mirrorbrain.org/issues/issue137
 .. _`issue 138`: http://mirrorbrain.org/issues/issue138
+.. _`issue 139`: http://mirrorbrain.org/issues/issue139
+.. _`issue 141`: http://mirrorbrain.org/issues/issue141
 
 
 

Modified: trunk/mb/mb/crawl.py
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/mb/mb/crawl.py?rev=8359&r1=8358&r2=8359&view=diff
==============================================================================
--- trunk/mb/mb/crawl.py	(original)
+++ trunk/mb/mb/crawl.py	Sun Feb  2 13:33:53 2014
_at_@ -2,6 +2,7 @@
 
 import sys
 import os
+import re
 
 def get_filelist(url):
     child_stdin, child_stdout, child_stderr = os.popen3(['rsync', '-r', url])
_at_@ -48,10 +49,11 @@
 
 from mechanize import Browser
 
-url = sys.argv[1]
-burl_len = len('http://widehat.opensuse.org/')
-burl_len = len('http://opensuse.unixheads.net/')
-burl_len = len('http://download.opensuse.org/pub/opensuse/')
+burl, url = sys.argv[1], sys.argv[2]
+#burl_len = len('http://widehat.opensuse.org/')
+#burl_len = len('http://opensuse.unixheads.net/')
+#burl_len = len('http://download.opensuse.org/pub/opensuse/')
+burl_len = len(burl)
 
 br = Browser()
 br.open(url)
_at_@ -63,7 +65,7 @@
         .*
         /$
         """, re.X)):
-    #print link
+    #print link.url
     print link.base_url[burl_len:] + link.url
 
 print
_at_@ -76,3 +78,5 @@
     #print link
     print link.base_url[burl_len:] + link.url
 
+for line in get_filelist('rsync.opensuse.org::opensuse-updates'):
+    print line




_______________________________________________
mirrorbrain-commits mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/

Note: To remove yourself from this list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-commits-request_at_mirrorbrain.org