[mirrorbrain-commits] r7709 - in trunk: . docs

From: <poeml_at_mirrorbrain.org>
Date: Tue, 28 Jul 2009 17:02:11 +0200
Author: poeml
Date: Tue Jul 28 17:02:10 2009
New Revision: 7709
URL: http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=7709

Log:
create docs/changes.rst, and start maintaining the Release Notes there,
obsoleting the NEWS file.

Added:
   trunk/docs/changes.rst
Modified:
   trunk/NEWS

Modified: trunk/NEWS
==============================================================================
--- trunk/NEWS	Tue Jul 28 16:49:42 2009	(r7708)
+++ trunk/NEWS	Tue Jul 28 17:02:10 2009	(r7709)
@@ -1,217 +1,7 @@
-Fri Jun  5 14:09:40 CEST 2009
-- MirrorBrain 2.8.1
-  - Python 2.6 compatibility fixes:
-    - mb file ls --md5: use hashlib module, if available (this fixes a
-      DepracationWarning given by Python 2.6 when importing the md5 module)
-    - mb list: the --as option had to be renamed to --asn, because "as" is a
-      reserved keyword in Python, and Python 2.6 is more strict about noticing
-      this also in cases where just used as an attribute.
-    - remove the b64_md5 function, which is no longer used since a while
-  - mb file ls:
-    - make the --md5 imply the --probe option
-  - mb export: 
-    - when exporting metadata for import into a VCS, handle additions and deletions
-  - update docs to point to new RPM packages in the openSUSE build service (in
-    a repository named Apache:MirrorBrain). The formerly monolithic package has
-    been split up into subpackages.
-  - perl-Config-IniFiles added to the list of perl packages required by the
-    scanner
+This document was moved.
+View it online at http://mirrorbrain.org/docs/changes/
 
+(Source file: http://svn.mirrorbrain.org/svn/mirrorbrain/trunk/docs/changes.rst)
 
-Tue Mar 31 02:10:38 CEST 2009
-- MirrorBrain 2.8
-- Improvements in the scanner, mainly with regard to the definition of
-  patterns for files (and directories) that are to be included from scanning.
-  Old, hardcoded stuff from the scanner has been removed. Now, excludes can be
-  defined in /etc/mirrorbrain.conf by scan_exclude and scan_exclude_rsync
-  directive. 
-  The former takes regular expressions and is effective for FTP and HTTP scans,
-  while the latter takes rsync patterns, which are passed directly to the
-  remote rsync daemon.
-  See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details.
-  This can decrease the size of the database (>20% for openSUSE), and for many
-  mirrors it considerably shortens the scan time.
-- Fixed a bug where the scanner aborted when encountering filenames in (valid
-  or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009
-- Improved the implementation of exclusions as well as the top-level-inclusion
-  pattern, which were not correctly implemented to work in subdir scans. 
-- The documentation was enhanced in some places.
-- mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11.
-- mb dirs: new subcommand for showing directories that the database contains,
-  useful to tune scan exclude patterns.
-- mb export: implement a new output format, named "vcs". Can be used to commit
-  changes to a subversion repository and get change notifications from it. See 
-  http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html
-- Partial deletions (for subdir scans) have been implemented.
-- mb list accept --country --region --prefix --as --prio options to influence
-  which details are output by it.
-- mb file: support for probing files, with optional md5 hash check of the
-  downloaded content.
-- The latter three changes have already been described in more detail at
-  http://mirrorbrain.org/news_items/2.7_mb_toolchain_work
+poeml, July 2009
 
-
-Wed Mar  4 16:40:06 CET 2009
-- MirrorBrain 2.7
-- Completely reworked the file database. It is 5x faster and one third the
-  size. Instead of a potentially huge relational table including timestamps (48
-  bytes per row), files and associations are now in a single table, using
-  smallint arrays for the mirror ids. This makes the table 5x faster and 1/3
-  the size. In addition, we need only a single index on the path, which is a
-  small and very fast b-tree.  This also gives us a good search, and the chance
-  to do partial deletions (e.g. for a subtree).
-- With this change, MySQL is no longer supported. The core, mod_mirrorbrain,
-  would still work fine, but the toolchain around is quite a bit specific to
-  the PostgreSQL database scheme now. If there's interest, MySQL support in the
-  toolchain can be maintained as well.
-- many little improvements in the toolchain were made.
-- Notably, the scanner has been improved to be more efficient and give better
-  output.
-- mirror choice can be influenced for testing with a query parameter (as=),
-  specifying the autonomous system number.
-
-
-Fri Feb 13 08:18:42 CET 2009
-- MirrorBrain 2.6 
-- supports additional, finer mirror selection, based on network
-  topological criteria, network prefix and autonomous system number, using
-  mod_asn and global routing data.
-  
-
-Fri Feb 13 02:39:45 CET 2009
-- updated database schemes and toolchain -- PostgreSQL support is solid now
-- work on installation documentation for both MySQL and PostgreSQL
-  (the latter is recommended now, because it allows for nifty features in the
-  future. The mb tool has an export subcommand now, perfect to migrate the
-  database.)
-- toolchain work
-
-
-Tue Feb  3 10:38:02 CET 2009
-- version 2.5
-- working on PostgreSQL support
-- working on the INSTALL documentation
-- scanner: 0.22
-   - more efficient SQL statement handling
-   - output much improved
-   - added SQL logging option for debugging
-- mb (mirrorbrain tool): 
-   - bugfix in the file command: make patterns work which have a wildcard as
-     first character.
-   - extend "mb scan" to accept -v and --sql-debug and pass it to the scanner
-
-
-Fri Jan 23 17:20:00 CET 2009
-- version 2.4
-- rename mod_zrkadlo to mod_mirrorbrain
-- use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use
-  the GeoIP city database for instance 
-- handle satellite "country" called A2
-- auto-reenable dead mirrors
-- geoiplookup_city added, new tool to show details from GeoIP city databases
-- geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It
-  also downloads the city database now.
-- deprecate "clientip" query parameter, which can no longer work
-  once we use mod_geoip. Implement 'country" parameter that can be used instead.
-- make memcache support optional at compile time
-
-
-Sat Dec 13 20:04:53 CET 2008
-- add commandline tool to edit marker files. (Marker files are used to generate
-  mirror lists. Each marker file is used to determine whether a mirror mirrors
-  a certain subtree.)
-
-Sat Dec 13 13:51:33 CET 2008
-- improvements and few features in the toolchain:
-   - the mirrorprobe now does GET requests instead of HEAD requests.
-   - mb, the mirrorbrain tool, has a powerful "probefile" command now that can
-     check for existance of a file on all mirrors, probing all URLs. This is
-     especially useful for checking whether the permission setup for staged 
-     content is correct on all mirrors.
-- new database fields: public_notes, operator_name, operator_url
-- new database tables: country & region
-
-Sat Dec  6 12:15:13 CET 2008
-- generate mirror lists
-
-Sat Nov 22 15:58:46 CET 2008
-- release MirrorBrain 2.2:
-  - simplified database layout, with additional space save.
-
-Sun Nov  9 15:20:03 CET 2008
-- release MirrorBrain 2.1
-  - simplified the Apache configuration: It is no longer needed to configure a
-    database query. At the same time it's less error-prone and avoids trouble
-    if one forgets to update the query, when the database schema changes. 
-  - specific mirrors can be now configured to get only requests for files < n bytes
-
-Mon Nov  3 11:04:04 CET 2008
-- release MirrorBrain 2.0
-  - implement better fallback mirror selection
-  - add tool to list/add/rm files in the mirror database
-
-Sun Oct 26 18:07:52 CET 2008
-- mod_zrkadlo 1.9:
-  - add bittorrent links (to found .torrent files) into metalinks
-  - embed PGP signatures (.asc files) into metalinks
-  - add configurable CSS stylesheet to mirror lists
-
-Fri Sep 19 21:08:42 CET 2008
- - implement the redirection exceptions (file too small, mime type not allowed
-   to be redirected etc) for transparently negotiated metalinks.
- - add Vary header on all transparently negotiated resources.
-- new, better implementation of rsyncusers tool
-- improved mirrordoctor tool
-- bugfixes in the scanner, mainly for scanning via HTML
-- installation instructions updated
-
-Fri Aug 22 17:22:14 CEST 2008
-- allow to use the apache module and all tools with multiple instances of
-  the mirrorbrain. Now, one machine / one Apache can host multiple separate
-  instances, each in a vhost.
-- a number of small bugs in the tools were fixed and several improvements
-  added.
-
-Wed Jun 11 01:20:12 CEST 2008
-- added "mirrordoctor", a commandline tool to maintain mirror entries in the
-  database. Finally!
-
-Mon Jun  2 17:30:45 CEST 2008
-- mod_zrkadlo 1.8:
-  use mod_memcache for the configuration and initialization of memcache
-- metalink-hasher script added, to prepare hashes for injection into metalink files
-- rsyncusers analysis tool added
-- scanner bugfix regarding following redirects for large file checks
-
-Fri May  2 18:54:09 CEST 2008
-- failover testbed for text mirrorlists implemented
-- rsyncinfo tool added
-
-Wed Apr 30 19:34:31 CEST 2008
-- metalinks: switch back to RFC822 format
-- new ZrkadloMetalinkPublisher directive 
-- fix issue with <size> element
-
-Sun Apr 27 23:02:42 CEST 2008
-- now there is another (more natural) way to request a metalink: by appending
-  ".metalink" to the filename.
-
-Sun Apr 27 12:06:47 CEST 2008
-- change metalink negotiation to look for application/metalink+xml in the
-  Accept header (keep Accept-Features for now, but it is going to be removed
-  probably)
-
-Mon Apr 21 17:35:27 CEST 2008
-- new terse text-based mirrorlist
-- allow clients to use RFC2295 Accept-Features header to select variants
-  (metalink or mirrorlist-txt)
-
-
-Mon Apr 21 01:35:31 CEST 2008
-metalink hash includes can now be out-of-tree
-
-Sat Apr 19 18:21:52 CEST 2008
-mod_autoindex_mb
-
-Sat Apr 19 18:21:52 CEST 2008
-content-disposition header

Added: trunk/docs/changes.rst
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ trunk/docs/changes.rst	Tue Jul 28 17:02:10 2009	(r7709)
@@ -0,0 +1,323 @@
+
+Release Notes/Change History
+============================
+
+Release 2.9.0 (Jul 27, 2009)
+----------------------------
+
+* the Subversion repository was moved to 
+  http://svn.mirrorbrain.org/svn/mirrorbrain/trunk/
+
+* rsync authentication was fixed. Credentials given in rsync URLs in the form of
+  `rsync://<username>:<password>@<host>/<module>` now work as expected. Patch
+  by Lars Vogdt.
+
+* the documentation has been moved into a docs subdirectory, and is rewritten in
+  reStructured Text format, from which HTML is be generated via Sphinx
+  (http://sphinx.pocoo.org/). Whenever the documentation is changed in subversion,
+  the changes automatically get online on http://mirrorbrain.org/docs/
+
+
+* Parallelized mirror probing.  Note: for this new feature, the Python module
+  `processing` or `multiprocessing` needs to be installed.  If none of them is
+  found, the fallback behaviour is to probe serially, like it was done before.
+  This new feature doesn't affeect the mirrorprobe, which runs threaded, but
+  the mb probefile and file commands. It also affects the scanner to speed up
+  the checks done when only a subdirectory is scanned.
+
+* :program:`mb` tool:
+
+  * :program:`mb probefile`:
+  
+    - Implemented downloading (and displaying) of content.
+    - An `--urls` switch was added, to select the kind of URLs to be probed.
+  
+      - `--urls=scan` probes the URLs that would be used in scanning.
+      - `--urls=http` probes the (HTTP) base URLs used in redirection.
+      - `--urls=all` probes all registered URLs.
+  
+    - Proxy envvars are unset before probing.
+    - report the mirror identifier for FTP socket timeouts
+  
+  * :program:`mb scan`:
+  
+      - Considerably improved logging output, avoiding lots of ugly
+        messages which look like real errors (and tend to cover real ones)
+      - Now showing how long scanning took.
+  
+  * :program:`mb new`: 
+    - try different geoip database locations (GeoIP database was
+      moved around...).  
+    - prefer the larger city lite database, if available, and prefer updated
+      copies that were fetched with the geoip-lite-update tool.
+
+  * :program:`mb list`: add `--other-countries` option to allow displaying the
+    countries that a mirror is configured to handle in addition to its own
+    country
+
+
+* :program:`mod_mirrorbrain`: in the "generator" tag of metalinks, include
+  mod_mirrorbrain's version string
+
+* :program:`metalink-hasher`:
+
+  - automatic removal of old hashes, which don't have a pendant in
+    the file tree anymore, was implemented.
+  - A number of things were optimized to run more efficiently on
+    huge trees, mainly by eliminating all redundant `stat()` calls.
+  - sha256 was added to the list of digests to generated.
+  - the need to specify the `-b` (`--base-dir`) option was eliminated,
+    which makes the command easier to use.
+  - the order in which the tool works through the todo list of directories
+    was changed to be alphabetical.
+  - using a "set" builtin type instead of a list can speed up finding
+    obsolete files in the destination directory by 10 times, for huge
+    directories.
+  - improvements in output and program help
+  - print a summary of deletions
+  - catch (and ignore) various errors, as vanishing directories and
+    exceptions encountered when recursively removing ignored directories.
+  - correct the indentation of verification containers
+  - version bumped (1.2)
+
+
+* :program:`geoip-lite-update`: This tool to fetch GeoIP databases has been
+  updated to use the path that's used in the openSUSE package since recently
+  (:file:`/var/lib/GeoIP`), and which complies better to the Linux Filesystem
+  Hierarchy Standard. It still tries the old location (:file:`/usr/share/GeoIP`) as
+  well, so to continue to work in a previous setup.
+
+
+* mirrorprobe:
+  - a logrotate snippet was added
+  - the mirrorprobe logfile was moved to the :file:`/var/log/mirrorbrain/` directory
+
+* the openSUSE RPM package now creates a user and group named `mirrorbrain`
+  upon installation. Also, it packages a runtime directory
+  :file:`/var/run/mirrorbrain` (which is cleaned up upon booting) and a log directory
+  :file:`/var/log/mirrorbrain`. Some additional Requires have been added, on the
+  perl-TimeDate, metalink and libapr-util1-dbd-pgsql packages.
+
+
+
+
+
+
+
+Fri Jun  5 14:09:40 CEST 2009
+- MirrorBrain 2.8.1
+  - Python 2.6 compatibility fixes:
+    - mb file ls --md5: use hashlib module, if available (this fixes a
+      DepracationWarning given by Python 2.6 when importing the md5 module)
+    - mb list: the --as option had to be renamed to --asn, because "as" is a
+      reserved keyword in Python, and Python 2.6 is more strict about noticing
+      this also in cases where just used as an attribute.
+    - remove the b64_md5 function, which is no longer used since a while
+  - mb file ls:
+    - make the --md5 imply the --probe option
+  - mb export: 
+    - when exporting metadata for import into a VCS, handle additions and deletions
+  - update docs to point to new RPM packages in the openSUSE build service (in
+    a repository named Apache:MirrorBrain). The formerly monolithic package has
+    been split up into subpackages.
+  - perl-Config-IniFiles added to the list of perl packages required by the
+    scanner
+
+
+Tue Mar 31 02:10:38 CEST 2009
+- MirrorBrain 2.8
+- Improvements in the scanner, mainly with regard to the definition of
+  patterns for files (and directories) that are to be included from scanning.
+  Old, hardcoded stuff from the scanner has been removed. Now, excludes can be
+  defined in /etc/mirrorbrain.conf by scan_exclude and scan_exclude_rsync
+  directive. 
+  The former takes regular expressions and is effective for FTP and HTTP scans,
+  while the latter takes rsync patterns, which are passed directly to the
+  remote rsync daemon.
+  See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details.
+  This can decrease the size of the database (>20% for openSUSE), and for many
+  mirrors it considerably shortens the scan time.
+- Fixed a bug where the scanner aborted when encountering filenames in (valid
+  or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009
+- Improved the implementation of exclusions as well as the top-level-inclusion
+  pattern, which were not correctly implemented to work in subdir scans. 
+- The documentation was enhanced in some places.
+- mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11.
+- mb dirs: new subcommand for showing directories that the database contains,
+  useful to tune scan exclude patterns.
+- mb export: implement a new output format, named "vcs". Can be used to commit
+  changes to a subversion repository and get change notifications from it. See 
+  http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html
+- Partial deletions (for subdir scans) have been implemented.
+- mb list accept --country --region --prefix --as --prio options to influence
+  which details are output by it.
+- mb file: support for probing files, with optional md5 hash check of the
+  downloaded content.
+- The latter three changes have already been described in more detail at
+  http://mirrorbrain.org/news_items/2.7_mb_toolchain_work
+
+
+Wed Mar  4 16:40:06 CET 2009
+- MirrorBrain 2.7
+- Completely reworked the file database. It is 5x faster and one third the
+  size. Instead of a potentially huge relational table including timestamps (48
+  bytes per row), files and associations are now in a single table, using
+  smallint arrays for the mirror ids. This makes the table 5x faster and 1/3
+  the size. In addition, we need only a single index on the path, which is a
+  small and very fast b-tree.  This also gives us a good search, and the chance
+  to do partial deletions (e.g. for a subtree).
+- With this change, MySQL is no longer supported. The core, mod_mirrorbrain,
+  would still work fine, but the toolchain around is quite a bit specific to
+  the PostgreSQL database scheme now. If there's interest, MySQL support in the
+  toolchain can be maintained as well.
+- many little improvements in the toolchain were made.
+- Notably, the scanner has been improved to be more efficient and give better
+  output.
+- mirror choice can be influenced for testing with a query parameter (as=),
+  specifying the autonomous system number.
+
+
+Fri Feb 13 08:18:42 CET 2009
+- MirrorBrain 2.6 
+- supports additional, finer mirror selection, based on network
+  topological criteria, network prefix and autonomous system number, using
+  mod_asn and global routing data.
+  
+
+Fri Feb 13 02:39:45 CET 2009
+- updated database schemes and toolchain -- PostgreSQL support is solid now
+- work on installation documentation for both MySQL and PostgreSQL
+  (the latter is recommended now, because it allows for nifty features in the
+  future. The mb tool has an export subcommand now, perfect to migrate the
+  database.)
+- toolchain work
+
+
+Tue Feb  3 10:38:02 CET 2009
+- version 2.5
+- working on PostgreSQL support
+- working on the INSTALL documentation
+- scanner: 0.22
+   - more efficient SQL statement handling
+   - output much improved
+   - added SQL logging option for debugging
+- mb (mirrorbrain tool): 
+   - bugfix in the file command: make patterns work which have a wildcard as
+     first character.
+   - extend "mb scan" to accept -v and --sql-debug and pass it to the scanner
+
+
+Fri Jan 23 17:20:00 CET 2009
+- version 2.4
+- rename mod_zrkadlo to mod_mirrorbrain
+- use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use
+  the GeoIP city database for instance 
+- handle satellite "country" called A2
+- auto-reenable dead mirrors
+- geoiplookup_city added, new tool to show details from GeoIP city databases
+- geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It
+  also downloads the city database now.
+- deprecate "clientip" query parameter, which can no longer work
+  once we use mod_geoip. Implement 'country" parameter that can be used instead.
+- make memcache support optional at compile time
+
+
+Sat Dec 13 20:04:53 CET 2008
+- add commandline tool to edit marker files. (Marker files are used to generate
+  mirror lists. Each marker file is used to determine whether a mirror mirrors
+  a certain subtree.)
+
+Sat Dec 13 13:51:33 CET 2008
+- improvements and few features in the toolchain:
+   - the mirrorprobe now does GET requests instead of HEAD requests.
+   - mb, the mirrorbrain tool, has a powerful "probefile" command now that can
+     check for existance of a file on all mirrors, probing all URLs. This is
+     especially useful for checking whether the permission setup for staged 
+     content is correct on all mirrors.
+- new database fields: public_notes, operator_name, operator_url
+- new database tables: country & region
+
+Sat Dec  6 12:15:13 CET 2008
+- generate mirror lists
+
+Sat Nov 22 15:58:46 CET 2008
+- release MirrorBrain 2.2:
+  - simplified database layout, with additional space save.
+
+Sun Nov  9 15:20:03 CET 2008
+- release MirrorBrain 2.1
+  - simplified the Apache configuration: It is no longer needed to configure a
+    database query. At the same time it's less error-prone and avoids trouble
+    if one forgets to update the query, when the database schema changes. 
+  - specific mirrors can be now configured to get only requests for files < n bytes
+
+Mon Nov  3 11:04:04 CET 2008
+- release MirrorBrain 2.0
+  - implement better fallback mirror selection
+  - add tool to list/add/rm files in the mirror database
+
+Sun Oct 26 18:07:52 CET 2008
+- mod_zrkadlo 1.9:
+  - add bittorrent links (to found .torrent files) into metalinks
+  - embed PGP signatures (.asc files) into metalinks
+  - add configurable CSS stylesheet to mirror lists
+
+Fri Sep 19 21:08:42 CET 2008
+ - implement the redirection exceptions (file too small, mime type not allowed
+   to be redirected etc) for transparently negotiated metalinks.
+ - add Vary header on all transparently negotiated resources.
+- new, better implementation of rsyncusers tool
+- improved mirrordoctor tool
+- bugfixes in the scanner, mainly for scanning via HTML
+- installation instructions updated
+
+Fri Aug 22 17:22:14 CEST 2008
+- allow to use the apache module and all tools with multiple instances of
+  the mirrorbrain. Now, one machine / one Apache can host multiple separate
+  instances, each in a vhost.
+- a number of small bugs in the tools were fixed and several improvements
+  added.
+
+Wed Jun 11 01:20:12 CEST 2008
+- added "mirrordoctor", a commandline tool to maintain mirror entries in the
+  database. Finally!
+
+Mon Jun  2 17:30:45 CEST 2008
+- mod_zrkadlo 1.8:
+  use mod_memcache for the configuration and initialization of memcache
+- metalink-hasher script added, to prepare hashes for injection into metalink files
+- rsyncusers analysis tool added
+- scanner bugfix regarding following redirects for large file checks
+
+Fri May  2 18:54:09 CEST 2008
+- failover testbed for text mirrorlists implemented
+- rsyncinfo tool added
+
+Wed Apr 30 19:34:31 CEST 2008
+- metalinks: switch back to RFC822 format
+- new ZrkadloMetalinkPublisher directive 
+- fix issue with <size> element
+
+Sun Apr 27 23:02:42 CEST 2008
+- now there is another (more natural) way to request a metalink: by appending
+  ".metalink" to the filename.
+
+Sun Apr 27 12:06:47 CEST 2008
+- change metalink negotiation to look for application/metalink+xml in the
+  Accept header (keep Accept-Features for now, but it is going to be removed
+  probably)
+
+Mon Apr 21 17:35:27 CEST 2008
+- new terse text-based mirrorlist
+- allow clients to use RFC2295 Accept-Features header to select variants
+  (metalink or mirrorlist-txt)
+
+
+Mon Apr 21 01:35:31 CEST 2008
+metalink hash includes can now be out-of-tree
+
+Sat Apr 19 18:21:52 CEST 2008
+mod_autoindex_mb
+
+Sat Apr 19 18:21:52 CEST 2008
+content-disposition header


_______________________________________________
mirrorbrain-commits mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain-commits/

Note: To remove yourself from this list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-commits-request_at_mirrorbrain.org
Received on Tue Jul 28 2009 - 15:02:12 GMT

This archive was generated by hypermail 2.2.0 : Tue Jul 28 2009 - 15:45:08 GMT