2.8 Release: New Scanning Features, Change Notifications

MirrorBrain 2.8 has been released, which

• improves the scanner
• adds features for database inspection
• adds a database export function that can be used to send out notifications about changes

In addition, this release incorporates other changes described recently.

The 2.8 tarball is up in the "download section":/download.

Details:

The mirror scanner program underwent a cleanup and now offers a better way to include or exclude files on mirrors. Old, hardcoded excludes have been removed from the program, and has been made configurable where one would expect it: in /etc/mirrorbrain.conf. There are two ways to configure excludes:

scan_exclude = REGEXP [...]
scan_exclude_rsync = RSYNC_PATTERN [...]

The former directive takes regular expressions and is effective for FTP and HTTP scans, while the latter takes rsync patterns, which are passed directly to the remote rsync daemon. Therefore, rsync patterns are used in that case. (This constitutes a duplication for the admin, and it would be nice if it would be possible to automatically convert rsync patterns into regexps and vice versa, to be able to specify the excludes only once.)

A mirrorbrain.conf directive with similar effect is scan_top_include. It lists directories at the top level of the tree that are scanned; all others are ignored:

scan_top_include = DIR [...]

With the new configurability, and much better excludes, the size of the openSUSE database could be decreased by 20%. For many mirrors, scan time is considerably shorter with good exclusions. (The reason is that some mirrors that have foreign stuff in-tree, or keep old files.)

A bug was fixed where the scanner could abort when encountering filenames in (valid or invalid) UTF-8 encoding.

There is mb dirs, a new subcommand for showing directories that the database contains, useful to tune scan exclude patterns. See output of "mb help dirs".

There is mb export --format=vcs, which implements a new output format named "vcs". It is suitable to commit changes to a subversion repository and get change notifications from it. The command generates a file tree which can be imported/committed into a version control system (VCS). This mechanism can be used to periodically dump the database into a working copy of such a repository and commit the changes, making use of the standard commit mail mechanism of the VCS to send change notifications.

See NEWS for a complete list of changes.