[mirrorbrain-commits] r8176 - in /trunk: mirrorprobe/mirrorprobe.py tools/scanner.pl

Author: poeml
Date: Fri Oct  1 17:04:43 2010
New Revision: 8176

URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?rev=8176&view=rev
Log:
mb scan & mirrorprobe:
- After encountering a web server that ignores requests without Accept header,
  the mirrorprobe and the scanner now send an Accept header with value '*/*'.
  That's a quirk on that particular web server software (or its configuration),
  but this header should not harm.

Modified:
    trunk/mirrorprobe/mirrorprobe.py
    trunk/tools/scanner.pl

Modified: trunk/mirrorprobe/mirrorprobe.py
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/mirrorprobe/mirrorprobe.py?rev=8176&r1=8175&r2=8176&view=diff
==============================================================================
--- trunk/mirrorprobe/mirrorprobe.py (original)
+++ trunk/mirrorprobe/mirrorprobe.py Fri Oct  1 17:04:43 2010
_at_@ -44,11 +44,16 @@
 
         logging.debug("%s probing %s" % (threading.currentThread().getName(), mirror.identifier))
 
+        #def urllib2_debug_init(self, debuglevel=0):
+        #    self._debuglevel = 1
+        #urllib2.AbstractHTTPHandler.__init__ = urllib2_debug_init
+
         #req = urllib2.Request('http://old-cherry.suse.de') # never works
         #req = urllib2.Request('http://doozer.poeml.de/')   # always works
         req = urllib2.Request(mirror.baseurl)
 
         req.add_header('User-Agent', USER_AGENT)
+        req.add_header('Accept', '*/*')
         #req.get_method = lambda: "HEAD"
 
         mirror.status_baseurl_new = False

Modified: trunk/tools/scanner.pl
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/scanner.pl?rev=8176&r1=8175&r2=8176&view=diff
==============================================================================
--- trunk/tools/scanner.pl (original)
+++ trunk/tools/scanner.pl Fri Oct  1 17:04:43 2010
_at_@ -878,6 +878,7 @@
 
   # Create a request
   my $req = HTTP::Request->new(GET => $url);
+  $req->header('Accept' => '*/*');
 
   # Pass request to the user agent and get a response back
   my $res = $ua->request($req);




_______________________________________________
mirrorbrain-commits mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/

Note: To remove yourself from this list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-commits-request_at_mirrorbrain.org