Author: poeml Date: Tue Jul 28 17:02:10 2009 New Revision: 7709 URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=7709 Log: create docs/changes.rst, and start maintaining the Release Notes there, obsoleting the NEWS file. Added: trunk/docs/changes.rst Modified: trunk/NEWS Modified: trunk/NEWS ============================================================================== --- trunk/NEWS Tue Jul 28 16:49:42 2009 (r7708) +++ trunk/NEWS Tue Jul 28 17:02:10 2009 (r7709) _at_@ -1,217 +1,7 @@ -Fri Jun 5 14:09:40 CEST 2009 -- MirrorBrain 2.8.1 - - Python 2.6 compatibility fixes: - - mb file ls --md5: use hashlib module, if available (this fixes a - DepracationWarning given by Python 2.6 when importing the md5 module) - - mb list: the --as option had to be renamed to --asn, because "as" is a - reserved keyword in Python, and Python 2.6 is more strict about noticing - this also in cases where just used as an attribute. - - remove the b64_md5 function, which is no longer used since a while - - mb file ls: - - make the --md5 imply the --probe option - - mb export: - - when exporting metadata for import into a VCS, handle additions and deletions - - update docs to point to new RPM packages in the openSUSE build service (in - a repository named Apache:MirrorBrain). The formerly monolithic package has - been split up into subpackages. - - perl-Config-IniFiles added to the list of perl packages required by the - scanner +This document was moved. +View it online at http://mirrorbrain.org/docs/changes/ +(Source file: http://svn.mirrorbrain.org/svn/mirrorbrain/trunk/docs/changes.rst) -Tue Mar 31 02:10:38 CEST 2009 -- MirrorBrain 2.8 -- Improvements in the scanner, mainly with regard to the definition of - patterns for files (and directories) that are to be included from scanning. - Old, hardcoded stuff from the scanner has been removed. Now, excludes can be - defined in /etc/mirrorbrain.conf by scan_exclude and scan_exclude_rsync - directive. - The former takes regular expressions and is effective for FTP and HTTP scans, - while the latter takes rsync patterns, which are passed directly to the - remote rsync daemon. - See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details. - This can decrease the size of the database (>20% for openSUSE), and for many - mirrors it considerably shortens the scan time. -- Fixed a bug where the scanner aborted when encountering filenames in (valid - or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009 -- Improved the implementation of exclusions as well as the top-level-inclusion - pattern, which were not correctly implemented to work in subdir scans. -- The documentation was enhanced in some places. -- mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11. -- mb dirs: new subcommand for showing directories that the database contains, - useful to tune scan exclude patterns. -- mb export: implement a new output format, named "vcs". Can be used to commit - changes to a subversion repository and get change notifications from it. See - http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html -- Partial deletions (for subdir scans) have been implemented. -- mb list accept --country --region --prefix --as --prio options to influence - which details are output by it. -- mb file: support for probing files, with optional md5 hash check of the - downloaded content. -- The latter three changes have already been described in more detail at - http://mirrorbrain.org/news_items/2.7_mb_toolchain_work +poeml, July 2009 - -Wed Mar 4 16:40:06 CET 2009 -- MirrorBrain 2.7 -- Completely reworked the file database. It is 5x faster and one third the - size. Instead of a potentially huge relational table including timestamps (48 - bytes per row), files and associations are now in a single table, using - smallint arrays for the mirror ids. This makes the table 5x faster and 1/3 - the size. In addition, we need only a single index on the path, which is a - small and very fast b-tree. This also gives us a good search, and the chance - to do partial deletions (e.g. for a subtree). -- With this change, MySQL is no longer supported. The core, mod_mirrorbrain, - would still work fine, but the toolchain around is quite a bit specific to - the PostgreSQL database scheme now. If there's interest, MySQL support in the - toolchain can be maintained as well. -- many little improvements in the toolchain were made. -- Notably, the scanner has been improved to be more efficient and give better - output. -- mirror choice can be influenced for testing with a query parameter (as=), - specifying the autonomous system number. - - -Fri Feb 13 08:18:42 CET 2009 -- MirrorBrain 2.6 -- supports additional, finer mirror selection, based on network - topological criteria, network prefix and autonomous system number, using - mod_asn and global routing data. - - -Fri Feb 13 02:39:45 CET 2009 -- updated database schemes and toolchain -- PostgreSQL support is solid now -- work on installation documentation for both MySQL and PostgreSQL - (the latter is recommended now, because it allows for nifty features in the - future. The mb tool has an export subcommand now, perfect to migrate the - database.) -- toolchain work - - -Tue Feb 3 10:38:02 CET 2009 -- version 2.5 -- working on PostgreSQL support -- working on the INSTALL documentation -- scanner: 0.22 - - more efficient SQL statement handling - - output much improved - - added SQL logging option for debugging -- mb (mirrorbrain tool): - - bugfix in the file command: make patterns work which have a wildcard as - first character. - - extend "mb scan" to accept -v and --sql-debug and pass it to the scanner - - -Fri Jan 23 17:20:00 CET 2009 -- version 2.4 -- rename mod_zrkadlo to mod_mirrorbrain -- use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use - the GeoIP city database for instance -- handle satellite "country" called A2 -- auto-reenable dead mirrors -- geoiplookup_city added, new tool to show details from GeoIP city databases -- geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It - also downloads the city database now. -- deprecate "clientip" query parameter, which can no longer work - once we use mod_geoip. Implement 'country" parameter that can be used instead. -- make memcache support optional at compile time - - -Sat Dec 13 20:04:53 CET 2008 -- add commandline tool to edit marker files. (Marker files are used to generate - mirror lists. Each marker file is used to determine whether a mirror mirrors - a certain subtree.) - -Sat Dec 13 13:51:33 CET 2008 -- improvements and few features in the toolchain: - - the mirrorprobe now does GET requests instead of HEAD requests. - - mb, the mirrorbrain tool, has a powerful "probefile" command now that can - check for existance of a file on all mirrors, probing all URLs. This is - especially useful for checking whether the permission setup for staged - content is correct on all mirrors. -- new database fields: public_notes, operator_name, operator_url -- new database tables: country & region - -Sat Dec 6 12:15:13 CET 2008 -- generate mirror lists - -Sat Nov 22 15:58:46 CET 2008 -- release MirrorBrain 2.2: - - simplified database layout, with additional space save. - -Sun Nov 9 15:20:03 CET 2008 -- release MirrorBrain 2.1 - - simplified the Apache configuration: It is no longer needed to configure a - database query. At the same time it's less error-prone and avoids trouble - if one forgets to update the query, when the database schema changes. - - specific mirrors can be now configured to get only requests for files < n bytes - -Mon Nov 3 11:04:04 CET 2008 -- release MirrorBrain 2.0 - - implement better fallback mirror selection - - add tool to list/add/rm files in the mirror database - -Sun Oct 26 18:07:52 CET 2008 -- mod_zrkadlo 1.9: - - add bittorrent links (to found .torrent files) into metalinks - - embed PGP signatures (.asc files) into metalinks - - add configurable CSS stylesheet to mirror lists - -Fri Sep 19 21:08:42 CET 2008 - - implement the redirection exceptions (file too small, mime type not allowed - to be redirected etc) for transparently negotiated metalinks. - - add Vary header on all transparently negotiated resources. -- new, better implementation of rsyncusers tool -- improved mirrordoctor tool -- bugfixes in the scanner, mainly for scanning via HTML -- installation instructions updated - -Fri Aug 22 17:22:14 CEST 2008 -- allow to use the apache module and all tools with multiple instances of - the mirrorbrain. Now, one machine / one Apache can host multiple separate - instances, each in a vhost. -- a number of small bugs in the tools were fixed and several improvements - added. - -Wed Jun 11 01:20:12 CEST 2008 -- added "mirrordoctor", a commandline tool to maintain mirror entries in the - database. Finally! - -Mon Jun 2 17:30:45 CEST 2008 -- mod_zrkadlo 1.8: - use mod_memcache for the configuration and initialization of memcache -- metalink-hasher script added, to prepare hashes for injection into metalink files -- rsyncusers analysis tool added -- scanner bugfix regarding following redirects for large file checks - -Fri May 2 18:54:09 CEST 2008 -- failover testbed for text mirrorlists implemented -- rsyncinfo tool added - -Wed Apr 30 19:34:31 CEST 2008 -- metalinks: switch back to RFC822 format -- new ZrkadloMetalinkPublisher directive -- fix issue with <size> element - -Sun Apr 27 23:02:42 CEST 2008 -- now there is another (more natural) way to request a metalink: by appending - ".metalink" to the filename. - -Sun Apr 27 12:06:47 CEST 2008 -- change metalink negotiation to look for application/metalink+xml in the - Accept header (keep Accept-Features for now, but it is going to be removed - probably) - -Mon Apr 21 17:35:27 CEST 2008 -- new terse text-based mirrorlist -- allow clients to use RFC2295 Accept-Features header to select variants - (metalink or mirrorlist-txt) - - -Mon Apr 21 01:35:31 CEST 2008 -metalink hash includes can now be out-of-tree - -Sat Apr 19 18:21:52 CEST 2008 -mod_autoindex_mb - -Sat Apr 19 18:21:52 CEST 2008 -content-disposition header Added: trunk/docs/changes.rst ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ trunk/docs/changes.rst Tue Jul 28 17:02:10 2009 (r7709) _at_@ -0,0 +1,323 @@ + +Release Notes/Change History +============================ + +Release 2.9.0 (Jul 27, 2009) +---------------------------- + +* the Subversion repository was moved to + http://svn.mirrorbrain.org/svn/mirrorbrain/trunk/ + +* rsync authentication was fixed. Credentials given in rsync URLs in the form of + `rsync://<username>:<password>_at_<host>/<module>` now work as expected. Patch + by Lars Vogdt. + +* the documentation has been moved into a docs subdirectory, and is rewritten in + reStructured Text format, from which HTML is be generated via Sphinx + (http://sphinx.pocoo.org/). Whenever the documentation is changed in subversion, + the changes automatically get online on http://mirrorbrain.org/docs/ + + +* Parallelized mirror probing. Note: for this new feature, the Python module + `processing` or `multiprocessing` needs to be installed. If none of them is + found, the fallback behaviour is to probe serially, like it was done before. + This new feature doesn't affeect the mirrorprobe, which runs threaded, but + the mb probefile and file commands. It also affects the scanner to speed up + the checks done when only a subdirectory is scanned. + +* :program:`mb` tool: + + * :program:`mb probefile`: + + - Implemented downloading (and displaying) of content. + - An `--urls` switch was added, to select the kind of URLs to be probed. + + - `--urls=scan` probes the URLs that would be used in scanning. + - `--urls=http` probes the (HTTP) base URLs used in redirection. + - `--urls=all` probes all registered URLs. + + - Proxy envvars are unset before probing. + - report the mirror identifier for FTP socket timeouts + + * :program:`mb scan`: + + - Considerably improved logging output, avoiding lots of ugly + messages which look like real errors (and tend to cover real ones) + - Now showing how long scanning took. + + * :program:`mb new`: + - try different geoip database locations (GeoIP database was + moved around...). + - prefer the larger city lite database, if available, and prefer updated + copies that were fetched with the geoip-lite-update tool. + + * :program:`mb list`: add `--other-countries` option to allow displaying the + countries that a mirror is configured to handle in addition to its own + country + + +* :program:`mod_mirrorbrain`: in the "generator" tag of metalinks, include + mod_mirrorbrain's version string + +* :program:`metalink-hasher`: + + - automatic removal of old hashes, which don't have a pendant in + the file tree anymore, was implemented. + - A number of things were optimized to run more efficiently on + huge trees, mainly by eliminating all redundant `stat()` calls. + - sha256 was added to the list of digests to generated. + - the need to specify the `-b` (`--base-dir`) option was eliminated, + which makes the command easier to use. + - the order in which the tool works through the todo list of directories + was changed to be alphabetical. + - using a "set" builtin type instead of a list can speed up finding + obsolete files in the destination directory by 10 times, for huge + directories. + - improvements in output and program help + - print a summary of deletions + - catch (and ignore) various errors, as vanishing directories and + exceptions encountered when recursively removing ignored directories. + - correct the indentation of verification containers + - version bumped (1.2) + + +* :program:`geoip-lite-update`: This tool to fetch GeoIP databases has been + updated to use the path that's used in the openSUSE package since recently + (:file:`/var/lib/GeoIP`), and which complies better to the Linux Filesystem + Hierarchy Standard. It still tries the old location (:file:`/usr/share/GeoIP`) as + well, so to continue to work in a previous setup. + + +* mirrorprobe: + - a logrotate snippet was added + - the mirrorprobe logfile was moved to the :file:`/var/log/mirrorbrain/` directory + +* the openSUSE RPM package now creates a user and group named `mirrorbrain` + upon installation. Also, it packages a runtime directory + :file:`/var/run/mirrorbrain` (which is cleaned up upon booting) and a log directory + :file:`/var/log/mirrorbrain`. Some additional Requires have been added, on the + perl-TimeDate, metalink and libapr-util1-dbd-pgsql packages. + + + + + + + +Fri Jun 5 14:09:40 CEST 2009 +- MirrorBrain 2.8.1 + - Python 2.6 compatibility fixes: + - mb file ls --md5: use hashlib module, if available (this fixes a + DepracationWarning given by Python 2.6 when importing the md5 module) + - mb list: the --as option had to be renamed to --asn, because "as" is a + reserved keyword in Python, and Python 2.6 is more strict about noticing + this also in cases where just used as an attribute. + - remove the b64_md5 function, which is no longer used since a while + - mb file ls: + - make the --md5 imply the --probe option + - mb export: + - when exporting metadata for import into a VCS, handle additions and deletions + - update docs to point to new RPM packages in the openSUSE build service (in + a repository named Apache:MirrorBrain). The formerly monolithic package has + been split up into subpackages. + - perl-Config-IniFiles added to the list of perl packages required by the + scanner + + +Tue Mar 31 02:10:38 CEST 2009 +- MirrorBrain 2.8 +- Improvements in the scanner, mainly with regard to the definition of + patterns for files (and directories) that are to be included from scanning. + Old, hardcoded stuff from the scanner has been removed. Now, excludes can be + defined in /etc/mirrorbrain.conf by scan_exclude and scan_exclude_rsync + directive. + The former takes regular expressions and is effective for FTP and HTTP scans, + while the latter takes rsync patterns, which are passed directly to the + remote rsync daemon. + See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details. + This can decrease the size of the database (>20% for openSUSE), and for many + mirrors it considerably shortens the scan time. +- Fixed a bug where the scanner aborted when encountering filenames in (valid + or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009 +- Improved the implementation of exclusions as well as the top-level-inclusion + pattern, which were not correctly implemented to work in subdir scans. +- The documentation was enhanced in some places. +- mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11. +- mb dirs: new subcommand for showing directories that the database contains, + useful to tune scan exclude patterns. +- mb export: implement a new output format, named "vcs". Can be used to commit + changes to a subversion repository and get change notifications from it. See + http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html +- Partial deletions (for subdir scans) have been implemented. +- mb list accept --country --region --prefix --as --prio options to influence + which details are output by it. +- mb file: support for probing files, with optional md5 hash check of the + downloaded content. +- The latter three changes have already been described in more detail at + http://mirrorbrain.org/news_items/2.7_mb_toolchain_work + + +Wed Mar 4 16:40:06 CET 2009 +- MirrorBrain 2.7 +- Completely reworked the file database. It is 5x faster and one third the + size. Instead of a potentially huge relational table including timestamps (48 + bytes per row), files and associations are now in a single table, using + smallint arrays for the mirror ids. This makes the table 5x faster and 1/3 + the size. In addition, we need only a single index on the path, which is a + small and very fast b-tree. This also gives us a good search, and the chance + to do partial deletions (e.g. for a subtree). +- With this change, MySQL is no longer supported. The core, mod_mirrorbrain, + would still work fine, but the toolchain around is quite a bit specific to + the PostgreSQL database scheme now. If there's interest, MySQL support in the + toolchain can be maintained as well. +- many little improvements in the toolchain were made. +- Notably, the scanner has been improved to be more efficient and give better + output. +- mirror choice can be influenced for testing with a query parameter (as=), + specifying the autonomous system number. + + +Fri Feb 13 08:18:42 CET 2009 +- MirrorBrain 2.6 +- supports additional, finer mirror selection, based on network + topological criteria, network prefix and autonomous system number, using + mod_asn and global routing data. + + +Fri Feb 13 02:39:45 CET 2009 +- updated database schemes and toolchain -- PostgreSQL support is solid now +- work on installation documentation for both MySQL and PostgreSQL + (the latter is recommended now, because it allows for nifty features in the + future. The mb tool has an export subcommand now, perfect to migrate the + database.) +- toolchain work + + +Tue Feb 3 10:38:02 CET 2009 +- version 2.5 +- working on PostgreSQL support +- working on the INSTALL documentation +- scanner: 0.22 + - more efficient SQL statement handling + - output much improved + - added SQL logging option for debugging +- mb (mirrorbrain tool): + - bugfix in the file command: make patterns work which have a wildcard as + first character. + - extend "mb scan" to accept -v and --sql-debug and pass it to the scanner + + +Fri Jan 23 17:20:00 CET 2009 +- version 2.4 +- rename mod_zrkadlo to mod_mirrorbrain +- use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use + the GeoIP city database for instance +- handle satellite "country" called A2 +- auto-reenable dead mirrors +- geoiplookup_city added, new tool to show details from GeoIP city databases +- geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It + also downloads the city database now. +- deprecate "clientip" query parameter, which can no longer work + once we use mod_geoip. Implement 'country" parameter that can be used instead. +- make memcache support optional at compile time + + +Sat Dec 13 20:04:53 CET 2008 +- add commandline tool to edit marker files. (Marker files are used to generate + mirror lists. Each marker file is used to determine whether a mirror mirrors + a certain subtree.) + +Sat Dec 13 13:51:33 CET 2008 +- improvements and few features in the toolchain: + - the mirrorprobe now does GET requests instead of HEAD requests. + - mb, the mirrorbrain tool, has a powerful "probefile" command now that can + check for existance of a file on all mirrors, probing all URLs. This is + especially useful for checking whether the permission setup for staged + content is correct on all mirrors. +- new database fields: public_notes, operator_name, operator_url +- new database tables: country & region + +Sat Dec 6 12:15:13 CET 2008 +- generate mirror lists + +Sat Nov 22 15:58:46 CET 2008 +- release MirrorBrain 2.2: + - simplified database layout, with additional space save. + +Sun Nov 9 15:20:03 CET 2008 +- release MirrorBrain 2.1 + - simplified the Apache configuration: It is no longer needed to configure a + database query. At the same time it's less error-prone and avoids trouble + if one forgets to update the query, when the database schema changes. + - specific mirrors can be now configured to get only requests for files < n bytes + +Mon Nov 3 11:04:04 CET 2008 +- release MirrorBrain 2.0 + - implement better fallback mirror selection + - add tool to list/add/rm files in the mirror database + +Sun Oct 26 18:07:52 CET 2008 +- mod_zrkadlo 1.9: + - add bittorrent links (to found .torrent files) into metalinks + - embed PGP signatures (.asc files) into metalinks + - add configurable CSS stylesheet to mirror lists + +Fri Sep 19 21:08:42 CET 2008 + - implement the redirection exceptions (file too small, mime type not allowed + to be redirected etc) for transparently negotiated metalinks. + - add Vary header on all transparently negotiated resources. +- new, better implementation of rsyncusers tool +- improved mirrordoctor tool +- bugfixes in the scanner, mainly for scanning via HTML +- installation instructions updated + +Fri Aug 22 17:22:14 CEST 2008 +- allow to use the apache module and all tools with multiple instances of + the mirrorbrain. Now, one machine / one Apache can host multiple separate + instances, each in a vhost. +- a number of small bugs in the tools were fixed and several improvements + added. + +Wed Jun 11 01:20:12 CEST 2008 +- added "mirrordoctor", a commandline tool to maintain mirror entries in the + database. Finally! + +Mon Jun 2 17:30:45 CEST 2008 +- mod_zrkadlo 1.8: + use mod_memcache for the configuration and initialization of memcache +- metalink-hasher script added, to prepare hashes for injection into metalink files +- rsyncusers analysis tool added +- scanner bugfix regarding following redirects for large file checks + +Fri May 2 18:54:09 CEST 2008 +- failover testbed for text mirrorlists implemented +- rsyncinfo tool added + +Wed Apr 30 19:34:31 CEST 2008 +- metalinks: switch back to RFC822 format +- new ZrkadloMetalinkPublisher directive +- fix issue with <size> element + +Sun Apr 27 23:02:42 CEST 2008 +- now there is another (more natural) way to request a metalink: by appending + ".metalink" to the filename. + +Sun Apr 27 12:06:47 CEST 2008 +- change metalink negotiation to look for application/metalink+xml in the + Accept header (keep Accept-Features for now, but it is going to be removed + probably) + +Mon Apr 21 17:35:27 CEST 2008 +- new terse text-based mirrorlist +- allow clients to use RFC2295 Accept-Features header to select variants + (metalink or mirrorlist-txt) + + +Mon Apr 21 01:35:31 CEST 2008 +metalink hash includes can now be out-of-tree + +Sat Apr 19 18:21:52 CEST 2008 +mod_autoindex_mb + +Sat Apr 19 18:21:52 CEST 2008 +content-disposition header _______________________________________________ mirrorbrain-commits mailing list Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/ Note: To remove yourself from this list, send a mail with the content unsubscribe to the address mirrorbrain-commits-request_at_mirrorbrain.orgReceived on Tue Jul 28 2009 - 15:02:12 GMT
This archive was generated by hypermail 2.3.0 : Mon Feb 20 2012 - 23:47:04 GMT