Issue31

Title filenames with spaces not always detected on mirrors
Priority bug Status resolved
Superseder Nosy List poeml, theuni
Assigned To poeml Keywords

Created on 2009-12-05.06:07:20 by theuni, last changed by poeml.

Messages
msg101 (view) Author: theuni Date: 2009-12-09.07:44:41
Whoops, missed the reply.

Indeed this is fixed and working great. Thanks for the quick turnaround as usual.
msg95 (view) Author: poeml Date: 2009-12-05.21:21:25
fixed in 2.11.2. Thanks for the report!
msg94 (view) Author: poeml Date: 2009-12-05.16:10:36
I'm verifying the results that the new scanner brings by running it on all 
OpenOffice.org mirrors that are scanned via FTP; it looks good.
msg93 (view) Author: poeml Date: 2009-12-05.12:39:36
fixed in trunk:
http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/scanner.pl?
r1=7905&r2=7904&pathrev=7905
msg92 (view) Author: poeml Date: 2009-12-05.12:08:27
A verbose scan shows the following:

 % mb scan netcolo -vvvvv --sql-debug
[...]

mirror.netcologne.de: ftp dir: addons/plugins/pictures
mirror.netcologne.de: -rw-r--r--    1 804      804         39209 Nov 09 03:13 Phonebin.tar.gz
-rw-r--r--    1 804      804        436921 Nov 09 03:13 Picasa.tar.gz
-rw-r--r--    1 804      804        255436 Nov 09 03:13 The Big Picture.tar.gz
-rw-r--r--    1 804      804        479633 Nov 09 03:12 flickr.tar.gz
-rw-r--r--    1 804      804       2351538 Nov 09 03:12 iPhoto.tar.gz
-rw-r--r--    1 804      804         21191 Nov 09 03:13 widelec.org.tar.gz
SELECT mirr_add_bypath(?, ?);  <-- 1, addons/plugins/pictures/Phonebin.tar.gz
DELETE FROM temp1 WHERE id = 4
SELECT mirr_add_bypath(?, ?);  <-- 1, addons/plugins/pictures/Picasa.tar.gz
DELETE FROM temp1 WHERE id = 5
SELECT mirr_add_bypath(?, ?);  <-- 1, addons/plugins/pictures/flickr.tar.gz
DELETE FROM temp1 WHERE id = 7
SELECT mirr_add_bypath(?, ?);  <-- 1, addons/plugins/pictures/iPhoto.tar.gz
DELETE FROM temp1 WHERE id = 8
SELECT mirr_add_bypath(?, ?);  <-- 1, addons/plugins/pictures/widelec.org.tar.gz
DELETE FROM temp1 WHERE id = 9
mirror.netcologne.de: committing ftp dir addons/plugins/pictures



The file is seen via FTP (first half of the log), but no action taken when it comes to storing the file into the 
database (second part of log). Staring at the scanner, line 638 now.
msg91 (view) Author: poeml Date: 2009-12-05.11:57:42
I set up a little test instance here. The list of mirrors, and the URLs look fine, when I search in the database. Here's an example with one 
file containing spaces, one not:

mirrorbrain@doozer:~> mb file ls 'addons/plugins/pictures/iPhoto.tar.gz' -u
eu de  100 ok       ok   www.softliste.de                http://www.softliste.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu de  100 ok       ok   mirror.netcologne.de            http://mirror.netcologne.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu es  100 ok       ok   evorq.ugr.es                    http://evorq.ugr.es/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu fr  100 ok       ok   distrib-coffee.ipsl.jussieu.fr  http://distrib-
coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu se  100 ok       ok   ftp.sunet.se                    http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us  100 ok       ok   mirror.its.uidaho.edu           http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us  100 ok       ok   www.gtlib.gatech.edu            http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us  100 ok       ok   ftp.osuosl.org                  http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
mirrorbrain@doozer:~> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -u
eu de  100 ok       ok   www.softliste.de                http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu de  100 ok       ok   mirror.netcologne.de            http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es  100 ok       ok   evorq.ugr.es                    http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr  100 ok       ok   distrib-coffee.ipsl.jussieu.fr  http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big 
Picture.tar.gz
eu se  100 ok       ok   ftp.sunet.se                    http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   mirror.its.uidaho.edu           http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   www.gtlib.gatech.edu            http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   ftp.osuosl.org                  http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz



I also get all 8 mirrors in the mirror list:

http://xbmc.mirrorbrain.org/addons/plugins/pictures/iPhoto.tar.gz?mirrorlist
http://xbmc.mirrorbrain.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist

However, I just added the mirrors with their HTTP URL. Thus, I use HTTP for scanning, which seems to work. Maybe you have FTP and rsync URLs 
configured - which would be preferred by the scanner. 



So, I added the FTP URL for the netcologne mirror, and scan it again:

mirrorbrain@doozer:~> mb scan netcolo
Sat Dec  5 12:55:51 2009 mirror.netcologne.de: starting
Sat Dec  5 12:55:51 2009 mirror.netcologne.de: total files before scan: 96
Sat Dec  5 12:55:51 2009 mirror.netcologne.de: no rsync, trying ftp
Sat Dec  5 12:55:52 2009 mirror.netcologne.de: scanned 69 files (60/s) in 1s
Sat Dec  5 12:55:52 2009 mirror.netcologne.de: files to be purged: 39
Sat Dec  5 12:55:52 2009 mirror.netcologne.de: total files after scan: 69
Sat Dec  5 12:55:52 2009 mirror.netcologne.de: purged old files in 0s.
Sat Dec  5 12:55:52 2009 mirror.netcologne.de: done.
Completed in 1 seconds
mirrorbrain@doozer:~> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -u
eu de  100 ok       ok   www.softliste.de                http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es  100 ok       ok   evorq.ugr.es                    http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr  100 ok       ok   distrib-coffee.ipsl.jussieu.fr  http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big 
Picture.tar.gz
eu se  100 ok       ok   ftp.sunet.se                    http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   ftp.osuosl.org                  http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   www.gtlib.gatech.edu            http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us  100 ok       ok   mirror.its.uidaho.edu           http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz

It's gone from the list. Thus, the problem is in the FTP scanner.
msg90 (view) Author: theuni Date: 2009-12-05.06:09:26
oops, typo in report.

http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz
should be
http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz
msg89 (view) Author: theuni Date: 2009-12-05.06:07:19
See here for reference:
http://mirrors.xbmc.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist

Some mirrors have the file, but aren't added to the list. Using that example the
following mirrors are 2 of the ones missing from that list:
http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz
http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz

I haven't been able to track down why some work and others don't. But it's clear
that it's the whitespace that causes issues.
History
Date User Action Args
2009-12-09 22:16:27poemlsetstatus: chatting -> resolved
2009-12-09 07:44:41theunisetstatus: resolved -> chatting
messages: + msg101
2009-12-05 21:21:25poemlsetstatus: testing -> resolved
messages: + msg95
2009-12-05 16:10:37poemlsetmessages: + msg94
2009-12-05 12:39:36poemlsetstatus: in-progress -> testing
messages: + msg93
2009-12-05 12:08:27poemlsetmessages: + msg92
2009-12-05 11:57:43poemlsetmessages: + msg91
2009-12-05 11:05:02poemlsetstatus: chatting -> in-progress
2009-12-05 06:09:26theunisetstatus: unread -> chatting
messages: + msg90
2009-12-05 06:07:20theunicreate