Re: [mirrorbrain] How to make Squid work with mirrorbrain

Anthony Bryan wrote:

> On Sat, Jun 2, 2012 at 4:12 AM, Jack Bates <grx28t_at_nottheoilrig.com>
> wrote:
>>
>>> When you say "we're using Metalink as the mirror list", what do you
>>> mean?  One annoying item in my setup is the parsing of the HTML
>>> mirror page - you wouldn't happen to know of a way of retrieving the
>>> mirror list in XML format?
>>
>>
>> You can retrieve a Metalink/XML resource that includes information
>> about where a file is mirrored, in XML format. I think the correct
>> way to *discover* this resource is through a 'Link: <...>;
>> rel=describedby; type="application/metalink4+xml"' header. Can anyone
>> (Anthony?) confirm that this is the correct way?
> 
> yes, Jack.
> 
> and that is what I meant, Per, that you could examine the metalink to
> construct a mirror list.

Hi Anthony

I've looked at the metalink xml file from e.g. 

http://download.opensuse.org/distribution/12.1/repo/oss/boot/x86_64/common.meta4

There's no problem working with that XML, but it looks like the
mirror-list will vary slightly depending on something, maybe which file
is being retrieved:

http://download.opensuse.org/distribution/12.1/repo/oss/boot/x86_64/common.meta4

Found 129 mirrors: 0 in the same network prefix, 0 in the same
autonomous system, 1 handling this country, 59 in the same region, 44
elsewhere

http://download.opensuse.org/distribution/12.1/repo/oss/boot/x86_64/yast2-trans-zh_CN.rpm.meta4

Found 127 mirrors: 0 in the same network prefix, 0 in the same
autonomous system, 1 handling this country, 59 in the same region, 42
elsewhere

http://download.opensuse.org/distribution.meta4

Found 35 mirrors: 0 in the same network prefix, 0 in the same autonomous
system, 1 handling this country, 22 in the same region, 2 elsewhere

http://download.opensuse.org/distribution/12.1.meta4

Found 29 mirrors: 0 in the same network prefix, 0 in the same autonomous
system, 1 handling this country, 18 in the same region, 1 elsewhere

I'm wondering if there is one single, easily identified file, that will
give me the complete list?  I quite like the idea of using e.g.
http://download.opensuse.org/distribution.meta4, but the list wasn't
exactly complete.  (should have been complete I think?)

I could no doubt write something to use the mirror-list from multiple
files to cobble up a complete list, but that's even worse than my
primitive parsing of the mirrorlists.py output. 

I would like to just add another output format to mirrorlists.py (in the
mirrorbrain package), unfortunately (for my own purposes) it would take
a while before it would make it to e.g. the opensuse download site. 
Still, my script could test on the availability and default to parsing
the xhtml output. Hmm, that could work. 

> in response to Per, curl metalink support just landed. I think zypper
> supported it on top of libcurl? not sure.

Ah yes, that could be.  I've also seen "multi-curl" mentioned somewhere? 

> what I said about segmented downloads, never mind, didn't fully
> understand...I like how you're doing things transparently. much nicer!
> 

Thanks!


-- 
Per Jessen, Zürich (13.4°C)


_______________________________________________
mirrorbrain mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain/

Note: To remove yourself from this mailing list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-request_at_mirrorbrain.org