Author: poeml Date: Fri Jul 31 02:39:13 2009 New Revision: 7737 URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=7737 Log: docs: - set up a directory for developers documentation - move BUGS to docs/devel/bugs.rst (maybe this should be at the top level, on par with the release notes) Added: trunk/docs/devel/ trunk/docs/devel/bugs.rst trunk/docs/devel/developer.rst - copied, changed from r7735, trunk/docs/developer.rst trunk/docs/devel/index.rst Deleted: trunk/docs/developer.rst Modified: trunk/docs/index.rst Added: trunk/docs/devel/bugs.rst ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ trunk/docs/devel/bugs.rst Fri Jul 31 02:39:13 2009 (r7737) @@ -0,0 +1,137 @@ +Bugs +==== + +More important bugs at the top. + + +Metalink timestamps +------------------- + +metalinks will likely use :rfc:`3339` timestamps instead of :rfc:`822` timestamps in +the future (IETF draft w.i.p.). No client makes use of the data yet. Cf. to + +* http://groups.google.com/group/metalink-discussion/browse_thread/thread/44489ba99916aced +* http://groups.google.com/group/metalink-discussion/msg/2b80424f41173cd3 + +``r->request_time``, which is attached to the request object, might be useful +instead of :func:`apr_time_now`, because it's already filled out and can save the +syscall to :func:`time`. + + +Sporadic corruption of ASN and PFX variables in the subprocess environment +-------------------------------------------------------------------------- + +mod_asn lookup data is sometimes garbled (subprocess env table):: + + 87.79.141.235 - - [16/Feb/2009:14:41:38 +0100] "GET /factory/repo/oss/suse/setup/descr/packages.gz HTTP/1.1" 200 2416300 "-" "ZYpp 5.25.0 (curl 7.19.0)" - r:- 145 2416594 -:- ASN:8422 P:87.78.0.0/15 size:- - - "-" + 87.79.141.235 - - [16/Feb/2009:14:41:38 +0100] "GET /factory/repo/oss/suse/setup/descr/patterns HTTP/1.1" 200 164 "-" "ZYpp 5.25.0 (curl 7.19.0)" - r:- 142 431 -:- ASN:{&\x80\x02 P: size:- - - "-" + 87.79.141.235 - - [16/Feb/2009:14:41:39 +0100] "GET /factory/repo/oss/license.tar.gz HTTP/1.1" 200 24492 "-" "ZYpp 5.25.0 (curl 7.19.0)" - r:- 131 24782 -:- ASN:8422 P:87.78.0.0/15 size:- - - "-" + +As a further data point, the following has been seen:: + + 90.182.x.x - - [20/Jul/2009:12:16:19 +0200] "GET /distribution/10.3/repo/oss/content HTTP/1.1" 302 348 "-" "Novell ZYPP Installer" ftp.linux.cz r:country 170 901 EU:CZ ASN:z,ne,ng,re,rw,sc,sd,sh,sl,sn,so,st,td,tf,tg,tn,tz,ug,yt,za,zm,zw a2 ge,kz,ru P:90.180.0.0/14 size:44325 - - "-" + +This further points to a memory corruption, happening while or after +mod_mirrorbrain is running. It seems very hard to trigger though, I'd estimate +20 requests out of 10.000.000. + +Grepping logs for ``xaa`` yields some places for examination:: + + zcat /var/log/apache2/download.opensuse.org/2009/07/download.opensuse.org-20090720-access_log.gz| grep 'xaa' + +Something might be corrupting data in the env table. It doesn't seem to affect +other env variables though, only the two that are set by mod_asn. + +All places where the subprocess env table is manipulated in mod_asn and +mod_mirrorbrain seem safe. + +A "soft" way of debugging would be to add a detailed debugging logging coupled +to a trigger which pulls when non-numeric data is seen in ``ASN`` or ``PFX``. +When actually adding several such triggers at different stages of request +processing, it should be possible to pinpoint the corruption. In addition, a +core dump of the process could be saved to disk while running. + +The bug doesn't seem to have noticeable negative consequences except the messed +up logging of the two values. + + +Inaccurate logging of ASN lookup data for 404s +---------------------------------------------- + +It was noticed that for 404s the lookup is correctly done (can be seen in the +response headers), but ``ASN:- P:-`` is being logged. + +Empty logging also happens for 200s, when e.g. requesting something from ``/icons``, +so that fits the configuration:: + + 87.79.141.235 - - [16/Feb/2009:15:55:43 +0100] "GET /icons/torrent.png HTTP/1.1" 200 3445 "http://download.opensuse.org/distribution/11.1/iso/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Shiira Safari/125" - r:- 405 3744 -:- ASN:- P:- size:- - - "-" + + +But what about redirections exceptions, like size? Example:: + + 78.34.103.82 - - [16/Feb/2009:16:05:54 +0100] "GET /distribution/11.1/repo/oss/suse/setup/descr/patterns HTTP/1.1" 200 170 "-" "ZYpp 5.24.5 (curl 7.19.0)" - r:- 152 448 -:- ASN:8422 P:78.34.0.0/15 size:- - - "-" + + +Inaccurate logging of ASN lookup data for overridden AS +------------------------------------------------------- + +When the AS is overridden with a query parameter, the one being logged/put into +the headers is not the overriding one. + + +Wrong reporting by the scanner about symlinks +--------------------------------------------- + +When scanning via rsync, the scanner reports symlinks (which are mode 0777) as +world-writable directories:: + + lrwxrwxrwx 1 root root 13 2009-03-01 05:29 /mounts/dist/unpacked/head-i586/usr/bin/pnmnoraw -> pnmtoplainpnm* + + rsync dir: 777 13 Sun Mar 1 05:29:31 2009 unpacked/head-i586/usr/bin/pnmnoraw + +That doesn't harm, because the symlinks are (intentionally) ignored anyway, but +it clutters the output and of course is confusing. + + +Pondering a hard dependency on mod_geoip +---------------------------------------- + +One could argue that mod_geoip should be a hard requirement, as mod_form is - +and Apache should check at startup. On the other hand, it could be optional, +because mod_mirrorbrain also works without mod_geoip (distributing requests +world-wide, according to mirror priorities). So it could be a valid use case to +run without mod_geoip. On the other hand, at least a warning should be issued, +so admins get a hint when mod_geoip was simply forgotten. On the other hand, we +could have a check that prevents starting, unless geoip usage is explicitely +disabled (e.g. MirrorBrainRequireGeoIP off) + +Update: explicitely disabling GeoIP would somehow conflict with the `"no GeoIP" +use case`_. + +.. _`"no GeoIP" use case`: http://mirrorbrain.org/docs/configuration/#using-mod-mirrorbrain-without-geoip + +Mirror list shows wrong region when overridden with query parameter +------------------------------------------------------------------- + +Since "override countries" (overridden via query paramter) are not resolved +into region, a wrong region is given in generated mirror lists:: + + http://download.opensuse.org/distribution/11.1/repo/oss/suse/noarch/rubygem-rails-2.1.1-1.14.noarch.rpm?mirrorlist&country=ZA + Found 2 mirrors which handle this country (ZA): <- ok + Found 61 mirrors in other countries, but same continent (EU): <- wrong + + +Mirror list gives inaccurate "number of mirrors", if mirrors were excluded +-------------------------------------------------------------------------- + +The mirrorlist gives inaccurate readings for "number of mirrors", if some +mirrors where not considered, because they are configured ``country-only`` or +``region-only`` (``same_region=1`` or ``same_country=1``) + +As further effect of this bug, it was noticed that a mirror is missing from the +?mirrorlist mirror lists if it is configured as fallback mirror for a country:: + + http://download.opensuse.org/repositories/KDE:/KDE4:/STABLE:/Desktop/openSUSE_11.1/KDE4-DEVEL.ymp?mirrorlist&country=tw + +ftp5 disappears from the list, when configured as fallback for Taiwan. It is +correctly used though and appears on the list *when* actually used as fallback. Copied and modified: trunk/docs/devel/developer.rst (from r7735, trunk/docs/developer.rst) ============================================================================== --- trunk/docs/developer.rst Fri Jul 31 01:53:34 2009 (r7735, copy source) +++ trunk/docs/devel/developer.rst Fri Jul 31 02:39:13 2009 (r7737) @@ -1,7 +1,3 @@ -############################################################################### - Documentation for Developers -############################################################################### - ******************************************************************************** @@ -28,7 +24,8 @@ +******************************************************************************** Debugging notes -================================================================================ +******************************************************************************** To be written. Added: trunk/docs/devel/index.rst ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ trunk/docs/devel/index.rst Fri Jul 31 02:39:13 2009 (r7737) @@ -0,0 +1,9 @@ +Documentation for Developers +============================ + +.. toctree:: + :glob: + + developer + bugs + Modified: trunk/docs/index.rst ============================================================================== --- trunk/docs/index.rst Fri Jul 31 02:14:01 2009 (r7736) +++ trunk/docs/index.rst Fri Jul 31 02:39:13 2009 (r7737) @@ -17,7 +17,7 @@ configuration_reference mirrors known_problems - developer + devel/index changes Indices and tables _______________________________________________ mirrorbrain-commits mailing list Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/ Note: To remove yourself from this list, send a mail with the content unsubscribe to the address mirrorbrain-commits-request_at_mirrorbrain.orgReceived on Fri Jul 31 2009 - 00:39:15 GMT
This archive was generated by hypermail 2.2.0 : Fri Jul 31 2009 - 00:45:25 GMT