Re: [mirrorbrain] ToDo for upcoming 2.13.0 release

Hi,

On Wed, May 05, 2010 at 07:03:17PM +0200, Peter Poeml wrote:
> 2.13.0 release is around the corner - and here's a little the list of
> things that I'd like to see fixed beforehand:

"around the corner" proved to be substantially more difficult to make
reality than I thought. With many other things to do, I don't find much
time to work on this.

Anyway, I largely managed to get the stuff done:

> - implement database cleanup for "mb makehashes". So far, it only
>   creates hashes, but never deletes them. It should delete hashes for
>   non-existing files (per directory), as it was done (and is still done)
>   for hashes stored on disk.

Works now.

> - what's the strategy for Torrent files and hashes (.md5 & friends) that
>   already exist on-disk? Should generated torrents/hashes replace them?
>   Or should existing files take precedence?

The strategy is that I simply assume that whoever already have .md5,
.sha1, .sha256, or .torrent files in their tree will exclude them from
redirection. The exclusion takes care of the conflict, by delivering the
real file from disk.

For the small .md5/.sha1/.sha256, that's how things *should* be
configured.

For those larger .torrent files, I only know of one user which serves
those, but there I know that they *do* deliver them directly (because of
wrong mime types on the mirrors).

That's good enough for now.

> - make creation of Zsync hashes configurable (in order to save
>   resources). Eventually, it should be configurable if zsync hashes are
>   created only for certain directories/files. Zsync hashes can get quite
>   large (roughly 10 times larger than torrent hashes). For the upcoming
>   release, it is sufficient to have these hashes switched off in the
>   default configuration; the rest can happen later.

Works now.

Default: off.

(zsync hashes are very consuming in database space and network
bandwidth, and unfortunately, in Apache RAM usage).

> - make creation of Torrent hashes configurable (same as zsync)

Works now.

> - http://mirrorbrain.org/issues/issue54 - the hasher used to ignore
>   unreadable files, but doesn't anymore

Fixed.

> - check for correct handling of empty files by mb hashes (just to make
>   sure it doesn't do something utterly stupid)

Checked & worked.

> - ponder if we can already switch off creation of on-disk hashes in "mb
>   makehashes" (with hashes being saved in the database now)

For now, the files are still written. They should be gotten rid of
later. They help in tracking disappeared files for cleanup, at the
moment.

> - critically eye DBD memory management once again (probably fine, since
>   2.13.0 code runs stable in production since weeks now, and the initial
>   memory consumption has been tracked down to be caused by large zsync
>   hashes)

2.13.0 runs in production since *months* now, and memory consumption is
a dream.

> - quick & easy: include an sha1 top hash into torrents

Works now.

> - torrents: nodes key (http://mirrorbrain.org/issues/issue49)

Works now.


...and there was more.

What remains: 

- I am pondering https://bugzilla.novell.com/show_bug.cgi?id=602434
  which openSUSE ran into. I'll just add their workaround for now, I
  think. It doesn't affect anyone else.

- I need to figure out how to let a post-install script run in the
  Debian/Ubuntu debs that I build. Needed for the database upgrade on a
  live system.

That's what I can think of, at the moment. Then, documentation and the
update can go out. 

I believe that 2.13.0 will be a great release!

Peter



_______________________________________________
mirrorbrain mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain/

Note: To remove yourself from this mailing list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-request_at_mirrorbrain.org