''Handle mirrors with Peter Poeml''

Jan 27th, 2009

Peter, let's start with common principles of software distribution. Could you please describe briefly how it is possible to distribute content among thousands of users? How does the system cope with peak load hours? Are there any bottlenecks within this period? Are there some difficulties that developers have to deal with?

That is a lot of questions at once. The general task is to provide software distribution to thousands of users world-wide, with the added challenge that many of the files to be distributed are very large. The number of users can be very high for a popular open source project which distributes software, which, for many organizations, clearly exceeds their capacities to handle the load as a simple organization. Luckily, quite some universities, ISPs and private persons are willing to contribute resources to help out. It is 'only' a matter of a suitable infrastructure to make use of the contributed resources, and to do this in a mostly transparent way, in order to hide the inherent complexity from the users.

In principle, the content providers and the helping organizations may form a so-called content delivery network (CDN), that automates this process as much as possible. A manual selection of mirrors, as it was common in the past and is still common in many places, is not practical for users anymore, when a certain, large scale has been reached. A system that deals with this must obviously be highly scalable. To the developers, the process should be just as transparent as to its users. They should not be required to know what's going on, in the complex distribution process.

What problems can be solved with the help of MirrorBrain?

MirrorBrain, as a content delivery network, solves the problem by doing geographical and topological load balancing. This means that it spreads the load, taking into account the location of users, location of mirrors, individual contents of the mirrors, capabilities of each mirror (how much load they can take), and the online status of mirrors (whether they are alive and kicking).

All this is done in a scalable way so that it never becomes 'slow', regardless of the number of users and popularity of software releases, designed to 'always work'. In addition, MirrorBrain offers fine tuning for mirror selection according to certain regional demands, and direct delivery of certain key files for security reasons.

MirrorBrain, as a server infrastructure, attempts to solve all these 'server-side' challenges. In addition, its goal is to support the other end -- the clients -- as well as possible. However, there are client-side problems that cannot be solved on the server-side: clients can face network problems for example, that even the most reliable service cannot prevent because it is simply outside of its control. Clients are often 'simple minds' and not designed to deal well with error conditions. Many download programs, web browser and similar software that users use to download files can not deal with typical failures. Thus, it is very important to monitor outages of mirrors closely, and this is exactly what MirrorBrain does; however, network failures that affect only the client can hardly be monitored and clients need to somehow deal with them. What MirrorBrain can do, though, is offer alternative locations to download from, and there is a new type of download clients (Metalink clients) that makes extensive use of this particular feature.

To support this, MirrorBrain is a full-featured metalink generator. In fact, it is the most advanced metalink generator in existance.

How did you decide to start your own project from scratch? Why not extending the existing ones?

Existing solutions were either not generic, or not scalable enough (far from that!). openSUSE's own home-grown was pushed to its very limits and was not up to the task anymore. It became very clear that a framework was needed which needed to be at least a 100 times as scalable as everything that existed.

In particular, the introduction of the openSUSE Build Service generated a higher demand than what was conceivable before. That Build Service generates a huge amount of Linux RPM packages and constantly updates them -- the changes happen faster than any mirror could be updated. The size of the file tree is the other factor contributing to this, because it encompasses a very large amount of software. So it was highly desirable to have a smart system that deals with this high fluctuation in realtime.

How could you characterize MirrorBrain in comparison with other projects that distribute content? Could you describe cons and pros of these projects?

Here's a short summary of the systems that are most comparable, or solve the same task:

• Fedora's MirrorManager: a pretty similar solution. In fact it and MirrorBrain are converging in some regards. It has the big plus to offer a nice, integrated web frontend for management, something that MirrorBrain is lacking so far, and which I solved with suite of commandline tools. It does not work on file level, but on directory level rather, which wasn't enough granularity for us. It is implemented in Python, while the core of MirrorBrain is written in C for maximum scalability.
• Bouncer: essentially similar approach. There are two versions, the core being implemented in PHP. One used by Mozilla.org, and one [was] used by OpenOffice.org, which doesn't do any geographical load balancing and has scalability issues. Specialized for the layout of Mozilla/OpenOffice.org software packages.
• Mandriva download redirector: it is specific to Mandriva's download client 'urmpi'.
• Sourceforge.net: web-based redirector, requiring Cookies and user interaction. I haven't seen code; I'm not sure if it is available.
• Commercial CDNs like Akamai or Limelight: would do the job fine, but cost money, and are affordable only for commercial software vendors. For open source vendors, they are too expensive.
• Academical CDN approaches: CoralCDN, for instance, is not transparent to client applications. There are several other experimental approaches; CoDeeN seems to be the only one that remotely is in productive use, though.
• GeoDNS (geographically-aware DNS resolution): used by kernel.org, for example. It is a rigid scheme and not flexible enough in itself, as it expects all servers to be fully synced at all times. The granularity is zero, so to speak, and it can't deal with partial mirrors. But it is fully transparent and would actually be a good supplement to MirrorBrain, as it would allow to have a MirrorBrain instance on each continent.
• Debian-like: simple, DNS-round-robin based scheme, which is a solid approach for its purpose, but requires a specialized client, and is not flexible enough, and not generic.

MirrorBrain's key diferentiators are twofold: 1) file-level granularity; 2) all requests pass through a central place. These two characteristics are intertwined, and they result in two key advantages. The first, and more obvious one, is that it is possible to collect detailed download statistics. The second advantage is rather secret, and you don't usually become aware of it, but it is crucial: It gives the power to circumvent problems with intermediate web caches. For short-lived content, this is very important. When clients are sent away to a mirror, and are for further requests talk directly to the mirrors, they typically get varying results, depending on the presence of cached copies of the requested files in intermediate web caches. Mirrored content is served from the cache because no mirror configures appropriate cache control directives. The origin server however (an Apache running MirrorBrain) can control cacheability exactly in a way that is appropriate for the content. Direct mirror access has constantly caused big issues for openSUSE users in the past, before MirrorBrain. Clients would see inconsistencies, and Linux package managers are extremely picky about these issues. This is a crucial thing that MirrorBrain gets right.

Do you have a project team or do you do everything on your own?

The initial design phase was done by a group of 5 or 6 people. My initial part was the implementation of the core, while some other components were contributed by the other folks. After a few months, the first prototype was ready and deployed by me on download.opensuse.org. Since then, I did all further work, which involved working out (ane implementing) the more interesting features, strategic and conceptual work, documentation and presentations, and handling of all mirror issues.

So there isn't really a team working on this project, and I'm pretty absorbed by it. I also spend quite some of my private time on it in an effort to advance it as much as possible, and to explore (and implement) ideas for future improvements. I see a lot of potential for good things, that are worth it!

Could you please illustrate the work of MirrorBrain with some examples?

MirrorBrain can work in several ways. It can either provide a list of mirrors, which can as precise as on file level. Or it can answer normal HTTP requests and send the clients on to mirrors. Or it can be configured to not send certain requests to mirrors, and rather deliver them itself, for security reasons, for files that are too quickly changing, or for files that are so small that they are not worth a redirect.

Alternatively, it can reply with a 'metalink', a mirror list in XML format, that can be used by the new type of client software I mentioned above. This results in a kind of 'knowledge transfer' to the client, which enables the client to work fully automatically and deal with all those error conditions and failures which generally plague people when they try to download files, especially large files like CD or DVD images.

And what about flexibility and reliability, characteristics that are very important for content distribution. How does MirrorBrain solve these tasks? (what hardware is it running on? What is the average load on download.o.o)

MirrorBrain runs without problems since it was installed two years ago. The hardware needs are very moderate - it doesn't need a farm of servers to handle many millions of requests each day. openSUSE's needs are fully served by an old low-end box.

Load has never been an issue (load average being less than 1).

Is it difficult to manage mirrors?

70% of the actual time I spend in communication. The main technical difficulty is mirror monitoring. Mirrors are far away, are a very heterogenous zoo of 200 different operating systems maintained by 200 different people. Working with them requires a rather individual approach, and whatever is implemented needs to stick to a common denominator.

Understanding the many different use cases (and error scenarious) is crucial, and surprisingly diverse - much more than most people would think. And it often proves difficult to explain this to people and generate the right awareness. Most people have a pretty simple picture of these issues, and often a very outdated one.

We discussed some server side aspects and what about the clients? How is the work of MirrorBrain made transparent for the users?

Speaking of the clients. There are different ones: human users, and machine clients running unattended. And they can be simple, standard HTTP-compliant clients like normal web browsers or download programs -- or special clients using optional extensions for failover and error handling. MirrorBrain supports both types and this in a fully transparent way. It can give clients all the means to successfully download --and verify-- files.

Having said that, the weakest link in the system in general is the client, and therefore I try to push most there. Better clients can make better use of the possibilities, that the server side offers.

What about data availability? Can MirrorBrain work in cluster configuration?

Sure. It can be clustered on different levels. Several webservers or several database servers, for load sharing or fallback, are straightforward to deploy. openSUSE's deployment is a bit hindered by lack of resources and doesn't go very far in this regard.

Most interesting is the possibility to place MirrorBrain instances on different continents and use GeoDNS (geographically-aware domain name service) to achieve quicker response times, and use them as fallback servers at the same time. Simple and elegant.

And finally, What about the future? Are you planning any major features?

A web frontend with access to mirror admins is one of the next things that I have in mind. Ah, and I'm working on a feature for automatic awareness of network locality - so MirrorBrain will send clients to a local mirror, if one exists. Simplifying a bit here, but it'll basically mean that when an ISP or a university operates a mirror, all clients from their network will automatically sent to just their mirror. This is something which is possible today with Fedora's MirrorManager, but it requires manual configuration; and this I intend to implement in a way so it's handled automatically, which will allow to put this feature to to much better effect.

I try to evangelize Metalink clients. I would like to push native metalink support in web browsers. That would be the greatest advance in download landscape of this decade!

I am also entertaining the idea to launch a discussion forum for mirror admins and software vendors. Amazingly, there is no such forum. The only forums that exist are vendor-specific mirror mailing lists. I realized that the mirror admins that you meet here and there are always the same folks -- it doesn't matter whether you look on the mailing lists of openSUSE, Fedora, OpenOffice and all those other projects. Thus, it seems logical to gather them in one place, and invite the content providers as well. I want to create a shared placed where to discuss further-reaching efforts and future ideas in the plenum. There is a lot of potential for more collaboration in the future. What's a very loosely organized croud today, could be a group that collaborates much more tomorrow, in the shared interest in distributing open source software.