Re: [mirrorbrain] [PATCH] IPv6 support for mod_asn - Apparently a very low hanging fruit.

On Wed, Jun 10, 2015 at 01:51:06PM +0200, Peter Pöml wrote:
> On Fri, May 08, 2015 at 08:43:13 +0200, Michael Meier wrote:
> > > bzcat rib.20150508.0000.bz2 | ./zebra-dump-parser.pl | ./filteruniqueprefixes.pl > /tmp/v6routedump.dat
> > > asn_get_routeviews > /tmp/v4routedump.dat
> > > cat /tmp/v6routedump.dat /tmp/v4routedump.dat | asn_import.py
[...]
> That's all great. I'm only struggling to package it in "digestible"
> form, so the existing users don't run into problems. Unfortunately, the
> old mod_asn SQL query doesn't work with the new database schema.
> I think I might add the new column alongside the old one first and
> duplicate the data, so that mod_asn can continue running until it's updated
> as well. 

Trying to simplify importing the new IPv6 data source, by using the
existing scripts, I now provide 
http://mirrorbrain.org/routeviews/ipv6-rib-snapshot-latest.dat.bz2
(nightly updated snapshot from routeviews /route-views6/bgpdata/%Y.%m/RIBS/) 
and also 
http://mirrorbrain.org/routeviews/ipv6-rib-snapshot-latest.txt.bz2
                                                           ^^^
which is the same, as textfile, parsed by zebra-dump-parser.pl by Marco
d'Itri.

The zebra dump parser can be switched to output the data in a format
including neighbouring AS numbers; I chose that format because IPv4 data
comes that way, and the neighbour AS could be imported into the database
one day, which is easy to do and I think could be quite interesting.

I extended asn_get_routeviews to download the new data source as well,
and to filter it similar to filteruniqueprefixes.pl. (The result
verified to be identical.)

I merged your asn_import changes regarding the AS number validity check.
I didn't merge the whole pull request though, since I solved few things
differently in the end. Hope you don't mind.

The result is, for the system administrator, that the scripts can be
used as before, and no changes are necessary. 

Hence, after you guys did the clever work, I have hopefully completed
the dump job as well now; or at least one of them ;-)

> The helper tools used in this and the slightly modified version of
> asn_import.py that I used can be found in
> https://github.com/poempelfox/mod_asn/tree/ipv6support
> While this works, it could need some more streamlining, e.g.:
> - providing a download-tool like for the IPv4 data

check

> - providing a mirror of the data used on mirrorbrain.org/routeviews/

check 

> - install the new helper tool(s) in packaging

check

> - perhaps merging it all together into asn_get_routeviews.py

check

> It would really be nice if this could be integrated into mirrorbrain.

Missing: logic for upgrading the database, possibly without disruption.

Peter


_______________________________________________
mirrorbrain mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain/

Note: To remove yourself from this mailing list, send a mail with the content
 	unsubscribe
to the address mirrorbrain-request_at_mirrorbrain.org