[mirrorbrain] Re: [distro-dev] getting torrents to the trackers (was: Wrong BitTorrent info hash in .meta4)

[removing Tatsuhiro from Cc, to save him from noise not related to the
original thread, and copying the mirrorbrain list instead, in light of
the discussion about mechanisms to notify bittorrent trackers of info
hashes]


Hi,

On Mon, Aug 30, 2010 at 04:31:35PM -0300, Harold Feit wrote:
> The way things currently work, mike's system rsyncs the core tree of
> downloads from the primary mirror grid, then generates the .torrent
> files from the individual downloads themselves once the checksum
> validation completes. After that, ooodev and depshtrike.com pull the
> .torrent files from mike's setup via rsync.

I wonder if this might be a common pattern: generate the torrents
somewhere (where the files are), sync the tree of torrents to another
place (where the tracker runs, or even several places with trackers).
If so, it might make sense to simply take provision in MirrorBrain to
create such a syncable tree of torrents. 

In fact, this would be fairly easy to implement the current MirrorBrain
codebase, because it already creates a tree of files storing metadata -
which was the previous format to store those data before I moved them
into a database for better access. I kept the files for backwards
compatilibity for now, and it would easy to change the mechanism to
write torrent files instead. 

That tree of torrent files could be published by rsync, which is easy to
do in general.

> The largest of the .torrent files I have for openoffice is the 3.1 all
> platforms DVD at 344kbyte.
> 
> The largest .torrent I have for openoffice that isn't a cd/dvd iso is
> OOo_3.2.0_Solaris_Sparc_install-wJRE_de.tar.gz.torrent at 33kbyte.
> 
> Considering we're representing files that exceed 100mbyte in size
> without even trying here, I don't really see the need to attempt to
> conserve space on the scale that we would actually see conserved.
> 
> The entire collection of .torrent files for 3.2.1 with the current
> system is 5.16 MB (5,415,847 bytes).

I'm sorry: with my suggestion about a "space-saving tree" I refered to a
copy of the tree of original files, not of torrents created from them.
(As sparse files, and therefore very lightweight.)

That's what I refered to; sorry for not being clearer.

But since you need the info hashes, that wouldn't have helped anyway.

> > Thanks for the insight. Is there another way of excluding
> > illegitimate access? Like, allowing only torrents with certain names
> > (instead of bt info hashes)? I assume the load spikes would be from
> > illegitimate users, who try to use the trackers for some other
> > stuff? Or how does it happen?
> infohashes only. The announce protocols of the bittorrent protocol
> prevent any other methods of excluding illegitimate access from
> working.

Okay. Makes sense.

> > Maybe I mentioned it before: there are no .torrent files that might
> > be mirrored; the content served is generated in realtime from a
> > database. The content does not primarily exist in torrent form on
> > disk; torrents are just one thing that can be made of them. And I
> > would like to avoid the overhead of writing torrent files for all
> > files to disk, although that would certainly be possible.
> The .torrents only need to be committed to disk at generation time,
> and only need to contain the info dictionary of the torrent, and no
> other part of it for authentication purposes (I actually strip all
> other information out of the .torrent file in the uploader that works
> on my site).

I see. Good to know.

> > [...]
> I have a php based upload script that could work with this that I use
> with my regular users.
> If you upload with this properly, the necessary portion of the .torrent
> can be stored on disk in my system and not actually commit anything to
> disk on the mirrorbrain side. Hard to believe I forgot about this
> considering I wrote the damn thing.
> Nice and oldschool HTTP POST too.

Sounds like an interesting option as well. I'll keep it in mind for
later. 

On Mon, Aug 30, 2010 at 09:37:21PM +0200, Mike wrote:
> If there is any way I could get a hold of the torrents, I could add
> them to the tracker....
> 
> 
> Mike

It sounds as if the local generation of torrent tree, and offering it
via rsync, might be the best option.

I would also fancy a more forward approach, though, where a newly
created torrent is immediately entered into the tracker, without having
to wait for periodic syncs, and cronjobs.

Peter



_______________________________________________
mirrorbrain mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain/

Note: To remove yourself from this mailing list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-request_at_mirrorbrain.org