[mirror discuss] Fwd: [mirrors] tamper-proof mirror network

From: Peter Pml <poeml_at_cmdline.net>
Date: Fri, 4 Dec 2009 20:36:00 +0100
Hi list,

for the archive; from the OpenOffice.org mailing list.

The issues raised in this mail apply to other projects as well.

This is the vision that I would like to elaborate on. Maybe it can  
become a shared vision.


Anfang der weitergeleiteten E-Mail:

> Von: Peter Pml <poeml_at_cmdline.net>
> Datum: 4. Dezember 2009 12:26:47 MEZ
> An: mirrors_at_distribution.openoffice.org
> Kopie: Anthony Bryan <anthonybryan_at_gmail.com>, "Neil M." <nabber00_at_gmail.com 
> >, Hampus Wessman <hampus.wessman_at_gmail.com>, hayden_at_halogenware.com
> Betreff: Re: [mirrors] tamper-proof mirror network
> Antwort an: mirrors_at_distribution.openoffice.org
> Hi Tora,
> thanks for bringing this up!
> Warning: long mail.
> Am 03.12.2009 um 08:01 schrieb tora - Takamichi Akiyama:
>> To prevent attackers from tampering files in the mirror network,
>> what can we do?
> I have a clear vision:
> Sign content, provide these signatures from a trusted source, and  
> enable users to use them as easily as possible.
> Signing content and providing the signatures securely is  
> straightforward.
> The third step, enabling users to easily make use of them, is not so  
> straightforward. Educating users to perform manual steps works only  
> so far. So, making it as easy as possible is the way to go.
> An important part is updating. A major portion of downloads are  
> updates. When the initial OpenOffice.org install was obtained in  
> intact form, it can be the vehicle for all later downloads. In fact,  
> the program already does that - it downloads updates and extensions.  
> In this situation, it would be easy to automate the required  
> verification procedure for downloading, on behalf at the user, and  
> don't bother them. This would be pretty straightforward: the  
> required server-side technology not only exists, but is already in  
> place[1]; the required client-side support can be implemented pretty  
> easily by using an existing Metalink client, or re-using existing  
> code. Thus, it seems very feasible to implement automatic  
> verification for updates, by making use of the provided metadata.
> [1] the new redirector at http://download.services.openoffice.org/  
> provides reliable download descriptions, including MD5, SHA-1 and  
> SHA-256 hashes, for all files. (It would also provide PGP signatures  
> if they existed.) These download descriptions (Metalinks) are  
> provided only by the origin server, and not through the mirror  
> system - which mainly leaves vulnerabilities through man-in-the- 
> middle attacks, and even those could be avoided by adding an HTTPS  
> virtual host to the server.
> So much concerning updating. For initial downloads (or, more  
> generally, downloads initiated by using a web browser, going to  
> download.openoffice.org etc.), the situation is different.  
> Unfortunately, the means that web browsers provide to detect  
> tampering are minimal^Wzero. This is a shortcoming that needs to be  
> fixed over time separately. See also Anthony's post. OpenOffice.org  
> is, of course, not the only content provider who cares about the  
> security of their users (and their good name). Others are working on  
> this as well. IMO, it is very good to join forces on this difficult  
> subject. The mailing list discuss_at_mirrorbrain.org is open for this  
> purpose, and I would like to invite everyone to take the discussion  
> of this subject onto a higher level - above the level of individual  
> projects.
> One idea would be to provide a download client via an applet in the  
> web browser. It is feasible with today's technology to launch a  
> Metalink client from a web browser. It requires a digital  
> certificate for signing the applet/application. And depending on  
> browser security settings a warning dialog may still pop up.  
> However, this does work; Hayden Legendre from Halogenware (author of  
> Retreiver, a download program) has created a proof-of-concept applet  
> of his Retreiver using applet and JNLP technology. I proposed this  
> as GSoC project in 2009 but didn't find the time to search a  
> student...
> Well, guess what the big guys do? (Akamai, the largest operator of a  
> Content Delivery Network.) They have made "Download Manager". Read  
> about it here:
> http://www.akamai.com/html/technology/products/http_downloads.html
> http://www.akamai.com/dl/feature_sheets/Akamai_Download_Manager_FS.pdf
> I quote: "When a user requests a large file download, Download  
> Manager is automatically sent to their computers. It launches  
> immediately, asking the user where the file should be saved." Sounds  
> familiar? You've heard that before? Yes, they do the same that we  
> _want_ to do. And believe me, Akamai know their business ;-)
> A variant of this idea, which might be simpler, would be to provide  
> a downloadable small binary, and the user is asked to execute it  
> after downloading it (which is a procedure that most people are used  
> to). This binary could be a minimal Metalink client that implements  
> the essential features (downloading, verification, and spiced with a  
> little bit of failover). This client would be specialized to  
> download OpenOffice.org, and could start with a little chooser that  
> offers the current OOo variants to choose from, plus optional  
> downloads. Of course it would need to be nice and real easy and user- 
> friendly. But it would do nothing else.
> Since the client would be small, it could be served from trusted  
> servers via HTTPS, and since it would verify all further downloaded  
> content, we would be _fully_ covered.
> Mini Downloader (see http://groups.google.com/group/metalink-discussion/browse_thread/thread/a6e4d2a6863b061f 
>  ) pushes into a similar direction and might be worth a look at.
> And there is AppUpdater (see http://www.nabber.org/projects/appupdater/ 
>  and paper presented at http://www.cs.cmu.edu/~tdumitra/hotswup09/papers/hotswup09_submission_3.pdf 
>  -- http://www.hotswup.org/2009/ -- "Hot Topics in Software Upgrades  
> '09" - what the odds :-)).
> Furthermore, we would be free to use such a downloader as an  
> instrument for feedback at the same time: if anything bad happens,  
> it can submit the log upstreams. Just like a crash reporter that  
> sends backtraces to a server collecting them. This would give us  
> several levers that are important for controlling quality of the  
> distribution:
> - identifying network problems and bottlenecks
> - identifying servers delivering broken content (forged or not)
> - optimizing mirror selection
> - statistics
> And, the best of all, the downloader wouldn't need to be OOo-only.  
> It could be fully reusable -- and would be picked up by other  
> projects, I'm sure.
> It just needs to be small, and platform-independent, user-friendly.
> A download applet (in the browser) would also be a good candidate  
> for reuse by others.
>> In other words, how can we make the OpenOffice.org mirror network
>> unattractive for attackers?
> With the above proposal, it would actually become a _challenge_ ;-)
> Joking aside: currently, I'd find it very attractive: I saw _very_  
> few signatures in the file tree.
> I know that MD5 sums are published somewhere on the web, but afaik  
> they are not immediately visible to the downloading user, and  
> furthermore it'd be a manual process to check them.
> The http://download.openoffice.org/contribute.html page that the  
> user is redirected to, after clicking on the download, doesn't even  
> mention the possibility of verifying the download, as far as I can  
> see. It should!
> Although the new redirector can provide hashes for all files on its  
> own, it would be valuable if the most important ones would already  
> exist in the filesystem: by publishing them together with the files  
> themselves. Adding a .md5 and .sha1 file could go hand in hand with  
> PGP-signing the files (which can only happen in the build system  
> anyway, and not on the download server).
> I do find a few MD5 hashes, but they are not available consistently  
> - I see extended/3.1.1rc2/3.1.1rc2_md5sums.txt and extended/ 
> developer/OOO320_m6/OOO320_m6_md5sums.txt. The former seems outdated  
> (because it is about a release candidate), while the latter is very  
> recent. Both are 90 kilobyte files though, which neither practical  
> to download, nor easily visible in the file system. Having the  
> checksums combined in a large liste makes the list less suited to  
> distribute via HTTPS, and less easy to use (or even download) in an  
> automated way. The common convention to use foobar.md5, foobar.sha1  
> and foobar.asc would be _very_ good to follow here.
>> (a) Geographically adaptable bouncer might help organized attackers.
> Nah, not really, if you meant redirection by client location. Do  
> you? It shouldn't matter if a mirror network is "geographically  
> adaptive" (uses network locality) to distribute traffic. It just  
> means that other users are affected. Exploiting network locality  
> limits the effect of a compromised mirror server to a certain region  
> (a geographical limitation only of course).
>> Assume that an attacker targets at country X. They would deploy
>> a mirror server in the country and ask us to register it as one
>> of the mirror servers. It would be acting as a normal mirror at
>> ordinary times. As a new version of OpenOffice.org is released,
>> it will start to deliver tampered software instead of genuine one.
>> Download requests from users in the country would be transfered
>> to one of the registered mirror servers within the country.
>> Consequently, the users in the targeted country would have tampered
>> ones in the high rate of possibility. Meanwhile, users in other
>> countries would not be affected.
>> (b) How to identify an owner and location of the mirror server.
>> If something wrong happens, we might be involved in the incident
>> and might be required to help authorities in the country.
>> An IP address of the server would identify its Internet service
>> provider. The ISP should know about the subscriber and location,
>> but the ISP, in general, would not disclose such information.
>> A domain name of the server would identify its registrant through
>> WHOIS service. Registrant is obligated to enter correct,
>> up-to-date information on the registrant in the WHOIS database,
>> but in reality, some leave obsolete information, some intentionally
>> enter inaccurate information to protect their privacy, some hide
>> their information behind secret service, etc.
>> In contrast, domain name registrar should know about the registrant
>> such as name, credit card information, billing address, but the
>> registrar, in general, would not disclose such information.
>> (c) How to quickly detect tampered files.
>> Our primary users are casual users. They do not care about MD5  
>> checksum.
>> Digital signature has not yet implemented in OpenOffice.org  
>> installer.
>> Even though it is implemented, it might not help because users would
>> just click on a button Next whatever the installer warns. Moreover,
> "You have been warned."
>> OpenOffice.org can be built by anyone. That implies that anybody
>> should be able to do the digital signature if it is implemented.
> Well. While anyone can build OpenOffice.org, not anyone can access  
> the download infrastructure and distribute builds through the  
> trusted network. I don't thing that anyone should be able to use it.  
> Neither that anyone should be able to sign the builds (with the same  
> keys). On the contrary, we leave this to a comparably small group  
> that we lend our trust. That's okay, isn't it?
>> The longer duration leaving the tampered files available in the
>> malicious mirror server, the more victims.
>> Therefore, quickly detecting tampered files and immediately stopping
>> all bouncers or eliminating the questioned server entries from
>> all bouncers might be crucial under the current circumstances.
>> But how?
> As a start, it would be very useful to start spreading digital  
> signatures for all releases. There are always some people who _do_  
> check them. It's only a small percentage of the mass of downloading  
> people, but nonetheless those are the people who initially find out  
> about possible problems -- some of them will track them down and  
> notify us. At least these people need to be enabled to easily check  
> signatures. This encompasses providing
> - .md5 file with the MD5 sum (which is the hash which is verifiable  
> on most platforms),
> - .sha file with the SHA-1 sum (which is better, but not available  
> everywhere), and a
> - .asc file with the detached PGP signature (which is most secure,  
> but the least people are able to use it)
> A good example is http://www.apache.org/dist/httpd/
> I also like that instructions and keys are available as well: http://www.apache.org/dist/httpd/KEYS
> (Note that the detached hash and signature files are not be  
> redirected to mirrors.)
> Now, while this can work as an alarm system, it is only a start and  
> better than nothing; in the end, verification of downloads must  
> happen on all end-user systems. See above :-)
>> =====
>> I believe 99.999% of the mirror servers currently registered in our
>> mirror network are trusted. However, we would need to consider
>> potential vulnerability of our system and process and to prepare
>> for possible attacks.
> Indeed, the trustfulness that we built up (developers, mirror  
> operators, users) is what connects us all and what keeps the whole  
> thing together. Which is nothing but wonderful. This can't stay  
> unsaid in a discussion like this.
> I definitely share this feeling of trust [*], but:
> It doesn't matter how trustful everybody may be -- because it is  
> trivial to launch a man-in-the-middle attack by IP address spoofing,  
> DNS cache poisening and similar techniques. Yes, it _is_ indeed  
> trivial.
> Such attacks can't be prevented, but their purpose defeated by  
> content verification.
> And it should be kept in mind, that in security there are no 100%.  
> However, the current situation leaves some things to desire. With  
> all the above, we'll achieve a _good_ level of security.
> Thanks for reading!
> Peter
> [*] Although I have been "part" of an experiment of undermining a  
> major mirror network in the past. And I can only second Anthony's  
> suggestion to read http://www.cs.arizona.edu/stork/packagemanagersecurity/ 
>  [**], which is a highly informative reading. It may initially  
> appear to focus on Linux-specific issues; however, that's only  
> because those already go quite far regarding security, and the work  
> really explores the remaining corner issues. But it also covers the  
> basics.
> [**] also published later in a USENIX magazine, in a very readable  
> form - at http://www.usenix.org/publications/login/2009-02/openpdfs/samuel.pdf

discuss mailing list
Archive: http://mirrorbrain.org/archive/discuss/

Note: To remove yourself from this mailing list, send a mail with the content
to the address discuss-request_at_mirrorbrain.org

Received on Fri Dec 04 2009 - 19:36:12 GMT

This archive was generated by hypermail 2.2.0 : Fri Dec 11 2009 - 22:12:59 GMT