Installation

Prerequirements

A recent enough version of the Apache HTTP server is required. 2.2.6 or later should be used. In addition, the apr-util library needs to be 1.3.0 or newer. This is because the DBD database pool functionality was developed mainly between 2006 and 2007, and reached production quality at the time.

Installing the ip4r data type into PostgreSQL

You need to install the contributed ip4r data type into PostgreSQL. This project is found at http://ip4r.projects.postgresql.org/. To install it, a shared library needs to be built to be loaded into PostgreSQL, and an SQL script needs to be run to make the data type known to PostgreSQL and install functions that use it.

It would be preferable to use a binary package if one exists for your operating system:

openSUSE/SLE rpm package:
http://download.opensuse.org/repositories/server:/database:/postgresql/

The Debian or Ubuntu package is called postgresql-VERSION-ip4r. For Ubuntu 14.04 it’s postgresql-9.3-ip4r, for Debian Jessie it’s postgresql-9.4-ip4r. The install command would look like this:

apt-get install postgresql-9.4-ip4r libapache2-mod-asn
Gentoo portage overlay:
http://github.com/ramereth/ramereth-overlay/tree

If a manual install is required, you need the PostgreSQL devel package of your operating system and compile a shared library, following the procedure described in the installation instructions provided with the software.

After installing the shared object by package or manual install, you will need to register that extension in your database(s). In old PostgreSQL-versions (up to 9.1) you run a SQL script provided with the ip4r sources, in later versions you instead register an extension:

su - postgres
psql -c "CREATE EXTENSION ip4r" template1                     # on Debian/Ubuntu >= postgre 9.1
psql -f /usr/share/postgresql/8.4/contrib/ip4r.sql template1  # on Debian/Ubuntu < postgre 9.1
psql -f /usr/share/postgresql-ip4r/ip4r.sql template1         # on openSUSE

template1 means that all databases that are created later will have the datatype available. To install it onto an existing database, use your database name instead of “template1”.

For instance, if you have an existing mirrorbrain database, you would install the data type on it like this:

su - postgres
psql -c "CREATE EXTENSION ip4r" mirrorbrain                     # on Debian/Ubuntu >= postgre 9.1
psql -f /usr/share/postgresql/8.4/contrib/ip4r.sql mirrorbrain  # on Debian/Ubuntu < postgre 9.1
psql -f /usr/share/postgresql-ip4r/ip4r.sql mirrorbrain         # on openSUSE

It is normal to see a a good screenful of output printed out by the above psql command.

Creating the database table

Assuming that a database exists already, execute the following command to install the pfx2asn table into it. The asn.sql file ships with mod_asn:

psql -U <dbuser> -f asn.sql <dbname>

Note

The command creates a table named pfx2asn in the database named <dbname>. Since the table name is used in some other places, so you should not change its name.

Example: assuming the database already exists (when installing MirrorBrain) and you are on Debian:

su - mirrorbrain
psql -f /usr/share/doc/libapache2-mod-asn/asn.sql

If you see some NOTICE printed out by the command, that’s normal; it’s due to the default logging setup of PostgreSQL which is verbose.

Config file for the import script

If you happen to have a MirrorBrain setup, you’ll have a configuration file named /etc/mirrorbrain.conf, which is automatically used by the asn_import script. No further configuration is needed then. If you have several MirrorBrain instances, the instance into which to import the data can be selected with the -b commandline option.

Alternatively, you need to create config file with the database connection info, named /etc/asn_import.conf, looking like this:

[general]
user = database_user
password = database_password
host = database_server
dbname = name_of_database

Load the database with routing data

The data is downloaded and imported into the database with the following command:

asn_get_routeviews | asn_import

It is recommendable to run the command as unprivileged user, for safety reasons (as any network client).

It might take a few minutes to download and process the data - about 30MB are downloaded, and the data is about 1GB uncompressed (as of 2009) (2010: 13MB compressed, 0.5G uncompressed). The script has to process over 5 million entries, and it is optimized for that job.

In the postgresql database, the data set will be small again.

The command shown above can be used to update the database with fresh routeviews data, by just running it again. This is explained in the next section.

Keep the data up to date

The data changes almost constantly, but most of the changes will be microscopic and won’t directly matter to you. However, you should regularly update from time to time. A weekly (or even monthly) schedule could be entirely sufficient, depending on what you use the data for.

Warning

You should be aware of the fact that routeviews.org kindly provides this data to the public, and you should use their bandwidth with consideration.

Therefore, the MirrorBrain project provides a daily mirror at http://mirrorbrain.org/routeviews/ containing the latest snapshot. This location is used by the provided scripts.

The same command as you ran initially can be used to update the database with fresh routeviews data, by just running it again. This works in production while the database is in active use; it is done in a way that doesn’t block any ongoing connections.

Note

The tarball with the data snapshot will be downloaded only if it doesn’t exist already in the current working directory. To redownload it, remove the file first.

A cron snippet for running the script daily to download and import the data could look as shown below:

35 2 * * *   mirrorbrain  sleep $(($RANDOM/16)); asn_get_routeviews | asn_import

If you have a MirrorBrain setup, and possibly several MirrorBrain instances, you could update each database like this:

# update ASN data in all MB instances
35 2 * * *   mirrorbrain  sleep $(($RANDOM/16)); \
                            for i in $(mb instances); do \
                              asn_get_routeviews | asn_import -b $i; done

The sleep command serves to randomize the job time a bit, and allows the example to be used verbatim. Also note that in the example the scripts are called without the .py extension.

The data is downloaded to the user’s home directory in this case. Make sure the script runs in a directory where other users don’t have write permissions.

Install the Apache module

There are binary packages of mod_asn at the following locations:

openSUSE/SLE:
http://download.opensuse.org/repositories/Apache:/MirrorBrain/
Debian/Ubuntu:
http://download.opensuse.org/repositories/Apache:/MirrorBrain/
Gentoo portage overlay:
http://github.com/ramereth/ramereth-overlay/tree

To manually build mod_asn, all you need to do normally is to use apxs2 with -c to compile and -i to install the module:

apxs2 -ci mod_asn.c

To enable the module to be loaded into Apache, you typically will have to run a command like the following - depending on your platform:

a2enmod asn

Configure Apache / mod_dbd

mod_dbd provides the database connection pool that is used by mod_asn. The module needs to be loaded into Apache:

a2enmod dbd

The DBD module needs a database adapter which connects to the database.

Put the following configuration into server-wide context:

# configure the dbd connection pool.
# for the prefork MPM, this configuration is inactive. Prefork simply uses 1
# connection per child.
<IfModule !prefork.c>
        DBDMin  0
        DBDMax  32
        DBDKeep 4
        DBDExptime 10
</IfModule>

As you might note, the cited configuration is relevant for threaded MPMs only. If you plan to use the prefork MPM, you don’t need it. You should however consider using a threaded MPM if you intend to serve high volumes of requests, because it will scale better, which is partly due to the fact that the threads within one process can share a common database pool, which results in fewer connections that are better utilized, and persistance of connections.

The database driver needs to be configured as well, by putting the following configuration into server-wide or vhost context. Make the file chmod 0640 and owned by root:root, because it will contain the database password:

DBDriver pgsql
DBDParams "host=localhost user=mb password=12345 dbname=mb connect_timeout=15"

Troubleshooting

If Apache doesn’t start, or anything else seems wrong, make sure to check Apache’s error_log. It usually points into the right direction.

A general note about Apache configuration which might be in order. With most config directives, it is important to pay attention where to put them - the order does not matter, but the context does. There is the concept of directory contexts and vhost contexts, which must not be overlooked. Things can be “global”, or inside a <VirtualHost> container, or within a <Directory> container.

This matters because Apache applies the config recursively onto subdirectories, and for each request it does a “merge” of possibly overlapping directives. Settings in vhost context are merged only when the server forks, while settings in directory context are merged for each request. This is also the reason why some of mod_asn’s config directives are programmed to be used in one or the other context, for performance reasons.

The install docs you are reading attempt to always point out in which context the directives belong.

Configure mod_asn

ASLookup

Simply set ASLookup On in the directory context where you want it to be active. The shipped config (mod_asn.conf) shows an example.

ASSetHeaders

Set ASSetHeaders Off if you don’t want the data to be added to the HTTP response headers. In that case, the lookup result is only available through the env table for perusal of other Apache modules.

ASIPHeader

The client IP address looked up is the one that the requests originates from. If mod_asn is running behind a frontend server and can’t see the original client IP address, the frontend may pass the IP via a header and mod_asn can look at the header instead. You can configure this like below:

ASIPHeader X-Forwarded-For
ASIPEnvvar

Alternatively, if you need to use mod_rewrite, you can also make mod_asn look at any variable in Apache’s subprocess environment for the IP, for instance:

ASIPEnvvar CLIENT_IP
ASLookupDebug

ASLookupDebug can be set to On to switch on debug logging. This can be done per directory.

ASLookupQuery

You may use the ASLookupQuery directive (server-wide context) to define a custom SQL query. The compiled in default is:

SELECT pfx, asn FROM pfx2asn WHERE pfx >>= ip4r(%s) ORDER BY ip4r_size(pfx) LIMIT 1

Testing

Once mod_asn is configured, you should be able to verify that it works by doing some arbitrary request and looking at the response:

 % curl -sI 'http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-Addon-Lang-i586.iso'
HTTP/1.1 302 Found
Date: Fri, 26 Jun 2009 22:35:50 GMT
Server: Apache/2.2.11 (Linux/SUSE)
X-Prefix: 87.78.0.0/15
X-AS: 8422
X-MirrorBrain-Mirror: ftp.uni-kl.de
X-MirrorBrain-Realm: country
Location: http://ftp.uni-kl.de/pub/linux/opensuse/distribution/11.1/iso/openSUSE-11.1-Addon-Lang-i586.iso
Content-Type: text/html; charset=iso-8859-1

(The X-Prefix and X-AS headers are not present in the response if mod_asn is configured with ASSetHeaders Off.

When testing with local IP addresses like 192.168.x.x, there’s not much to look up. These addresses are reserved for local use (see RFC 1918). You could however play with sending X-Forwarded-For headers, provided that you configured “ASIPHeader X-Forwarded-For”, and can lookup arbitrary IPs thereby. You can use curl with the following option, causing it to add an X-Forwarded-For header with arbitrary value to the request headers:

% curl -sv -H "X-Forwarded-For: 128.176.216.184" <url>

It can be helpful to set ASLookupDebug On for some directory - you’ll see every step which the module does being logged to the error_log.

Logging

Since the data being looked up is stored in the subprocess environment, it is trivial to log it, by adding the following placeholder to the LogFormat:

ASN:%{ASN}e P:%{PFX}e

That’s it!

Questions, bug reports, patches are welcome at mirrorbrain@mirrorbrain.org.