June 21, 2012

Introducing http.debian.net, Debian's mirrors redirector

You've been there: you are about to install a package, upgrade, or get the source package, and the mirror fails. It is offline, it is out of date, etc. Whatever the reason, you couldn't do it.

That's only one of the issues that http.debian.net attempts to address. In a nutshell: it works as an http-only content distribution network. It uses the network and geographic location to select the best online and up-to-date mirrors.

APT's sources.list one liner:
deb http://http.debian.net/debian stable main

Use /debian-backports for backports, and /debian-archive for archive.debian.org.

Originally introduced to the Debian mirrors community back in January, the code on the server and client sides has improved since. All is needed is squeeze's APT (or aptitude), but wheezy's and sid's will perform better.

Advantages and more details are discussed at the Debian mirrors redirector's website.

(Thanks to the early testers, everyone who has provided feedback, my friends at PuffinHost, and all the mirrors-related people, obviously including those who sponsor them!)

10 comments:

  1. The parallel downloading sounds fantastic. Is there any plan to migrate existing urls such as http://ftp.us.debian.org to use the new system?

    ReplyDelete
  2. Looks good, it chose perfectly for me. Any particular reason you didn't use mirrorbrain ?

    Also, I hope this'll become http.debian.org very quickly.

    ReplyDelete
    Replies
    1. There's a small list of differences between http.d.n and mirrorbrain in the README file. One major issue is that there are quite a number of http-only mirrors, and some others that are slow due to international links. Scanning them every ten minutes would simply be infeasible.

      Delete
  3. Looks nice. Right now http.debian.net seems to point to single IP address (174.142.22.90). Doesn't this make SPOF? If so, do you plan to solve this (and how)? Can another mirrors for this system help?

    Is there a way to use this with apt-cacher-ng? I tried to use
    deb http://192.168.6.100:3142/http.debian.net/debian squeeze main contrib non-free

    in sources list, but I receive:

    W: Failed to fetch http://192.168.6.100:3142/http.debian.net/debian/dists/squeeze/main/binary-armel/Packages.gz 307 Temporary Redirect

    W: Failed to fetch http://192.168.6.100:3142/http.debian.net/debian/dists/squeeze/contrib/binary-armel/Packages.gz 307 Temporary Redirect

    W: Failed to fetch http://192.168.6.100:3142/http.debian.net/debian/dists/squeeze/non-free/binary-armel/Packages.gz 307 Temporary Redirect

    ReplyDelete
    Replies
    1. More instances will be added for redundancy very soon.

      As for apt-cacher-ng, it is a bug that has already been fixed in sid (and wheezy): http://bugs.debian.org/661971
      As of a few days ago the redirector uses a 302 status code to redirect, which is correctly supported by older versions of apt-cacher-ng. It shouldn't be an issue anymore.

      Delete
    2. I confirm it works with apt-cacher-ng 0.5.1-3 and armel architecture. Thanks for solving this issue and for whole service!

      Delete
  4. Thats a great Service :)
    I'm mirroring currently debian and debian-backports.
    I use the german ftp2.de.debian.org Server.
    But in the last 6 Months he was 2 Times down for more then 3 Days.
    So i need to sync more then 100 GB at once.

    With this Service this Problem gets fixed :)
    Thanks for it.

    Martin

    ReplyDelete
  5. What about security repo?
    Is security repo it mirrored?

    ReplyDelete
    Replies
    1. It is but it might be disabled at some point (by redirecting all requests to security.d.o).
      APT is not very fault tolerant and robust enough. There are concerns regarding the use of third party security mirrors, and it is officially discouraged.
      As soon as those concerns are addressed, the service could be re-enabled.

      Delete