So that they don't know what's inside
Showing posts with label mirrors. Show all posts
Showing posts with label mirrors. Show all posts
May 14, 2015
October 08, 2013
Faster, more stable and new opportunities
A new version of the code behind http.debian.net was deployed a few days ago. It has proved to be far more stable, faster, and scalable compared to the previous mod_perl-based deployment.
There were a couple of glitches during the earlier roll-out for IPv6 users, fixed thanks to the reports by Cyril Brulebois, Michael Stapelberg and Robert Drake.
What's behind?
The redirector is now a plack application (with no middlewere) running under the Starman server with an apache fronted. Requests are processed faster than before and there mod_perl-induced system overload is finally gone.
The redirector is now easier to test and develop. Deploying the live instance is not yet fully streamlined, but it has seen a lot of improvement. Some important changes to the way the redirector works are already on their way to see the light and I am going to be announcing them when they do. Fork the repository and hack a few changes, contributions are welcome :)
It's probably time to move it under debian.org, and finish making it the default everywhere. It's even made its way into the installation manual.
There were a couple of glitches during the earlier roll-out for IPv6 users, fixed thanks to the reports by Cyril Brulebois, Michael Stapelberg and Robert Drake.
What's behind?
The redirector is now a plack application (with no middlewere) running under the Starman server with an apache fronted. Requests are processed faster than before and there mod_perl-induced system overload is finally gone.
The redirector is now easier to test and develop. Deploying the live instance is not yet fully streamlined, but it has seen a lot of improvement. Some important changes to the way the redirector works are already on their way to see the light and I am going to be announcing them when they do. Fork the repository and hack a few changes, contributions are welcome :)
It's probably time to move it under debian.org, and finish making it the default everywhere. It's even made its way into the installation manual.
June 05, 2013
The "let the tool do the work" update
Over the last few weeks I've been making several changes to http.debian.net to detect mirrors that don't follow Debian's mirroring guidelines and end up causing problems to the end users. The changes will mean less hash mismatches and similar errors.
As I wrote back in December, the redirector is becoming nicer but also stricter. Some of the changes I recently made caused over 30 mirrors to be completely disabled from the redirector. This is not ideal and I don't like having to disable mirrors. They are contributions afterall.
The only thing I can do, and can't stress enough, is encourage people to use an up to date ftpsync script (available at project/ftpsync/ on every mirror) to mirror Debian.
It takes care for you of all the little but important things needed. Really. A mirror that uses ftpsync is easier for the administrator to properly configure, and provides a consistent mirror for the benefit of the users.
Speaking of ftpsync, there is a new version! If you use ftpsync please upgrade it as soon as possible.
Other improvements are on their way. Contributions are welcome (if you like refactoring, there's quite a bit of explicitly-redundant code in check.pl that should now give a better idea of the way it needs to be refactored.)
As I wrote back in December, the redirector is becoming nicer but also stricter. Some of the changes I recently made caused over 30 mirrors to be completely disabled from the redirector. This is not ideal and I don't like having to disable mirrors. They are contributions afterall.
The only thing I can do, and can't stress enough, is encourage people to use an up to date ftpsync script (available at project/ftpsync/ on every mirror) to mirror Debian.
It takes care for you of all the little but important things needed. Really. A mirror that uses ftpsync is easier for the administrator to properly configure, and provides a consistent mirror for the benefit of the users.
Speaking of ftpsync, there is a new version! If you use ftpsync please upgrade it as soon as possible.
Other improvements are on their way. Contributions are welcome (if you like refactoring, there's quite a bit of explicitly-redundant code in check.pl that should now give a better idea of the way it needs to be refactored.)
May 08, 2013
Almost one million requests per day
In the first 48 hours after its log files were rotated last Sunday, http.debian.net handled almost 2 million requests, for an average of 11 requests per second.
In the last weeks before the release of Debian wheezy the number of requests had dropped slightly below 2 million per week.
Debian is alive.
In the last weeks before the release of Debian wheezy the number of requests had dropped slightly below 2 million per week.
Debian is alive.
May 02, 2013
An ever-growing mirrors network, a year later
A year ago I wrote about Debian's ever-growing mirrors network, so it is time to review the numbers.
Compared to the numbers from last year, today Debian is being served via http by about 370 mirrors world-wide, and is also available via ftp from 330 mirrors. So that's an increase of 40 mirrors in one year!
The number of countries with Debian mirrors also increased to 76, 3 more since last year.
This has only been possible thanks to the sponsors hosting the mirrors.
During this year some sponsors have had to retire their mirrors, sometimes ceasing years of contributions to the project and its community.
A big thanks is deserved to past and current sponsors.
Compared to the numbers from last year, today Debian is being served via http by about 370 mirrors world-wide, and is also available via ftp from 330 mirrors. So that's an increase of 40 mirrors in one year!
The number of countries with Debian mirrors also increased to 76, 3 more since last year.
This has only been possible thanks to the sponsors hosting the mirrors.
During this year some sponsors have had to retire their mirrors, sometimes ceasing years of contributions to the project and its community.
A big thanks is deserved to past and current sponsors.
April 08, 2013
How the world ended up in Costa Rica
Even though I haven't had much time to dedicate to http.debian.net lately, it has been up and running, or should I say serving?
Part of its job is to detect mirrors that have temporary issues or are entirely gone, down, unavailable. It does so, and many other things, by monitoring the so-called "trace files". A very important one being the "master" (or "origin") trace file.
With the recent integration of backports into the main archive, the master trace file of the backports mirrors also changed. Long story short, this change caused backports mirrors to no longer be considered by the mirror redirector as candidates. As long as they were up to date.
After the usual mirror synchronisation delay, more and more mirrors were disabled and subsets of "up to date" candidates re-calculated. This reached a critical point when only one mirror was left in the database. The mirror had not been synchronised for a couple of weeks.
This mirror is located in Costa Rica, and as the only candidate left in the database it was the only one used to serve requests for the backports archive. No matter where the client was located in the world.
The issue was later noticed and the necessary updates to the mirrors master list made. Mirrors started to be re-considered as they were re-checked (with some delay due to the rate limiter) and the subsets re-calculated. In a few hours everything was back to normality.
Correctness and fault-tolerance don't always get together very well...
Part of its job is to detect mirrors that have temporary issues or are entirely gone, down, unavailable. It does so, and many other things, by monitoring the so-called "trace files". A very important one being the "master" (or "origin") trace file.
With the recent integration of backports into the main archive, the master trace file of the backports mirrors also changed. Long story short, this change caused backports mirrors to no longer be considered by the mirror redirector as candidates. As long as they were up to date.
After the usual mirror synchronisation delay, more and more mirrors were disabled and subsets of "up to date" candidates re-calculated. This reached a critical point when only one mirror was left in the database. The mirror had not been synchronised for a couple of weeks.
This mirror is located in Costa Rica, and as the only candidate left in the database it was the only one used to serve requests for the backports archive. No matter where the client was located in the world.
The issue was later noticed and the necessary updates to the mirrors master list made. Mirrors started to be re-considered as they were re-checked (with some delay due to the rate limiter) and the subsets re-calculated. In a few hours everything was back to normality.
Correctness and fault-tolerance don't always get together very well...
January 21, 2013
January's Debian mirrors update
It's been slightly over a month since December's update to http.debian.net. Since then, Debian's mirrors network has grown by 6 more archive mirrors. Many thanks to the Debian sponsors running them!
There are now about 370 archive mirrors serving it over http, an increase of 40 (12%) since April last year. The number of backports mirrors is now at 82, and 25 for archive.debian.org.
On the http.debian.net front there haven't been many changes since last month. Some major changes are in the works, but they didn't make it into January's code update. There were, however, a few issues with one of the hosts during the first couple of days of January. Apologies for the inconveniences it may have caused.
A new version of ftpsync addressing some issues should hopefully be released some time next month. Stay tuned to the debian-mirrors mailing list for a call for testers and probably a survey for mirror administrators.
There are now about 370 archive mirrors serving it over http, an increase of 40 (12%) since April last year. The number of backports mirrors is now at 82, and 25 for archive.debian.org.
On the http.debian.net front there haven't been many changes since last month. Some major changes are in the works, but they didn't make it into January's code update. There were, however, a few issues with one of the hosts during the first couple of days of January. Apologies for the inconveniences it may have caused.
A new version of ftpsync addressing some issues should hopefully be released some time next month. Stay tuned to the debian-mirrors mailing list for a call for testers and probably a survey for mirror administrators.
December 18, 2012
Nicer, but stricter
Lately I've been working on making the redirector nicer to the mirrors and to some potential users. More specifically, those behind a caching proxy.
The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.
Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.
Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.
Projection of a rate-limited mirror over six weeks. The mirror would have to fail in every attempt for that to happen.
N.b. there's a bump in the scale.
The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:
Comparison of the two behaviours over an 8 weeks period using a logarithmic scale.
Nice chart colours by Libreoffice.
The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.
While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.
Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.
That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.
The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.
Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.
Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.
Projection of a rate-limited mirror over six weeks. The mirror would have to fail in every attempt for that to happen.
N.b. there's a bump in the scale.
The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:
Comparison of the two behaviours over an 8 weeks period using a logarithmic scale.
Nice chart colours by Libreoffice.
The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.
While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.
Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.
That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.
December 03, 2012
Some things you wanted to know about http.debian.net
After quite a bit of, very welcome, feedback I've put together a FAQ page in an attempt to respond to the most common questions about http.debian.net.
Emails have been accumulating for a few weeks now, but I will get to them. So please be patient if you send me an email, or if you have sent me one.
Emails have been accumulating for a few weeks now, but I will get to them. So please be patient if you send me an email, or if you have sent me one.
November 17, 2012
Better routing, less bad apples
Another month, another update to http.debian.net. This time around most of the work was done outside the redirector's code base, as strange as it may sound.
The redirector heavily relies on the mirrors doing at least a couple of things right, for the rest it can and does compensate. When it needs to compensate, certain requests are redirected to automatically-detected good mirrors, thus avoiding mirrors that might work fine for some parts of the day but cause headaches during the rest.
So, part of the work done since the last update was to prod more mirror administrators to upgrade to the latest version of ftpsync. This reduces the number of mirrors for which compensation is needed in order to avoid errors during installations and upgrades. Hopefully, no additional work is needed for the redirector to notice the upgrades. This results in immediate improvements.
However, not all mirrors comply with the bare minimum requirements. As stated in my previous blog post, running rsync once is not enough. When mirrors break these assumptions they lead to the "bad apple" effect. The effects in this case are temporary errors, as experienced by some people. The interesting part of those issues is that the affected population may quickly change given the redirector's use of geo location and the way it creates mirror subsets.
As interesting as the distribution of the effects may be, they are not really welcome. So I put together some code to attempt to detect the bad apples. This resulted in a list of mirrors that have now been disabled in the redirector and whose administrators are going to be contacted so that they comply with the minimum requirements. Given that detection is time-sensitive, there's no 100% guarantee that all of them have been identified so far. The code to detect them will have to be adapted and integrated into the redirector's code base to be proactive on avoiding this kind of issues.
Last but not least, the redirector is now using a database of AS peers for better (re-)routing. This is the next move towards a decision making based more on network location/topology than in geographic location. This first use of a peers database is limited to IPv4 and is based on a recent routing table dump and on feedback provided by interested people. If you are a mirror or network administrator, or you are familiar with the topology of your network please drop me an email so that the redirector can make a better use of your peering agreements.
N.b. in the case of the database, the term peer may also include transit providers. It is used to refer to and establish a relationship between two AS(N)s.
Feedback is, as always, welcome. I read each and every email but it may take me some time to get to it, or reply.
The redirector heavily relies on the mirrors doing at least a couple of things right, for the rest it can and does compensate. When it needs to compensate, certain requests are redirected to automatically-detected good mirrors, thus avoiding mirrors that might work fine for some parts of the day but cause headaches during the rest.
So, part of the work done since the last update was to prod more mirror administrators to upgrade to the latest version of ftpsync. This reduces the number of mirrors for which compensation is needed in order to avoid errors during installations and upgrades. Hopefully, no additional work is needed for the redirector to notice the upgrades. This results in immediate improvements.
However, not all mirrors comply with the bare minimum requirements. As stated in my previous blog post, running rsync once is not enough. When mirrors break these assumptions they lead to the "bad apple" effect. The effects in this case are temporary errors, as experienced by some people. The interesting part of those issues is that the affected population may quickly change given the redirector's use of geo location and the way it creates mirror subsets.
As interesting as the distribution of the effects may be, they are not really welcome. So I put together some code to attempt to detect the bad apples. This resulted in a list of mirrors that have now been disabled in the redirector and whose administrators are going to be contacted so that they comply with the minimum requirements. Given that detection is time-sensitive, there's no 100% guarantee that all of them have been identified so far. The code to detect them will have to be adapted and integrated into the redirector's code base to be proactive on avoiding this kind of issues.
Last but not least, the redirector is now using a database of AS peers for better (re-)routing. This is the next move towards a decision making based more on network location/topology than in geographic location. This first use of a peers database is limited to IPv4 and is based on a recent routing table dump and on feedback provided by interested people. If you are a mirror or network administrator, or you are familiar with the topology of your network please drop me an email so that the redirector can make a better use of your peering agreements.
N.b. in the case of the database, the term peer may also include transit providers. It is used to refer to and establish a relationship between two AS(N)s.
Feedback is, as always, welcome. I read each and every email but it may take me some time to get to it, or reply.
October 31, 2012
rsync is not enough
Ask how can one create a Debian mirror and you will get a dozen different responses. Those who are used to mirroring content, whether it is distribution packages, software in general, documents, etc., will usually come up with one answer: rsync.
Truth is: for Debian's archive, rsync is not enough.
Due to the design of the archive, a single call to rsync leaves the mirror in an inconsistent state during most of the sync process.
Explanation is simple: index files are stored in dists/, package files are stored in pool/. Sort those directories by name (just like rsync does) and you get that the indexes will be updated before the actual packages are downloaded.
There are multiple scripts out there that do exactly that, one of them in Ubuntu's wiki. Plenty more if you search the web.
Now, addressing that issue shouldn't be so difficult, right? after all, all the index files are in dists/, so syncing in two stages should be enough. It's not that simple.
With the dists/ directory containing over 8.5GBs worth of indexes and, erm, installer files, even a two stages sync will usually leave the mirror in an inconsistent state for a while.
How about only deferring to the second stage the bare minimum?, I hear you ask.
That is the current approach, but it leads to some errors when new index files are added and used. The fact that people insist in writing their own scripts doesn't help.
Hopefully, some ideas like moving the installer stuff out of dists/ and
overhauling the repository layout are being considered. An alternative is to make the users of the mirrors more robust and fault-tolerant, but we would be talking about tenths if not hundreds of tools that would need to be improved.
In all cases, the one script that is actively maintained, is rather portable, and improved from time to time is the ftpsync script. Please, do yourself and your users a favour: don't attempt to reinvent the wheel (and forget about calling rsync just once).
Truth is: for Debian's archive, rsync is not enough.
Due to the design of the archive, a single call to rsync leaves the mirror in an inconsistent state during most of the sync process.
Explanation is simple: index files are stored in dists/, package files are stored in pool/. Sort those directories by name (just like rsync does) and you get that the indexes will be updated before the actual packages are downloaded.
There are multiple scripts out there that do exactly that, one of them in Ubuntu's wiki. Plenty more if you search the web.
Now, addressing that issue shouldn't be so difficult, right? after all, all the index files are in dists/, so syncing in two stages should be enough. It's not that simple.
With the dists/ directory containing over 8.5GBs worth of indexes and, erm, installer files, even a two stages sync will usually leave the mirror in an inconsistent state for a while.
How about only deferring to the second stage the bare minimum?, I hear you ask.
That is the current approach, but it leads to some errors when new index files are added and used. The fact that people insist in writing their own scripts doesn't help.
Hopefully, some ideas like moving the installer stuff out of dists/ and
overhauling the repository layout are being considered. An alternative is to make the users of the mirrors more robust and fault-tolerant, but we would be talking about tenths if not hundreds of tools that would need to be improved.
In all cases, the one script that is actively maintained, is rather portable, and improved from time to time is the ftpsync script. Please, do yourself and your users a favour: don't attempt to reinvent the wheel (and forget about calling rsync just once).
October 13, 2012
Debian mirrors map
Ever wondered how the Earth would look like if you added markers for every mirror that is part of Debian's mirrors network?
(the bigger the shadow of the marker, the larger the number of mirrors in that zone)
Mirrors map generated with leaflet, using Openstreetmap.org tiles, and mirrors geolocation using GeoLite data created by MaxMind, available from maxmind.com.
(the bigger the shadow of the marker, the larger the number of mirrors in that zone)
Mirrors map generated with leaflet, using Openstreetmap.org tiles, and mirrors geolocation using GeoLite data created by MaxMind, available from maxmind.com.
September 18, 2012
More mirrors, slightly better IPv6 support and more
As announced in the Debian Project News of September more mirrors have recently joined Debian's mirrors network. Two more have been added since that issue of DPN was released, and about four more might be added any day now. Many thanks to those sponsoring them!
The mirrors redirector has also received some improvements, and around 2.5 million requests a week.
Among the improvements, users of teredo and 6to4 tunnels are now handled as IPv4 users. Not only it gives them access to a wider range of mirrors, but it also avoids the tunnel overhead.
Another improvement is the prompt detection of mirrors that drop architectures without prior notification. This is thanks to a new version of ftpsync, the standard and recommended tool to create Debian mirrors suitable for Debian's mirrors network.
As more mirror administrators upgrade to the newer version, or switch to it, http.debian.net and other tools become more reliable: they no longer need to perform some checks and hope the results mean the mirrors actually include the architectures they say they include.
In this regard, http.debian.net is an improvement over debian-installer's mirrors selection menu: not only it chooses a mirror that is close to the user and up to date, it also ensures the mirror provides the files for an architecture even after the installer image has been built. debian-installer's mirrors list is static and hard-coded into the installation media.
There's more to come, stay tuned.
The mirrors redirector has also received some improvements, and around 2.5 million requests a week.
Among the improvements, users of teredo and 6to4 tunnels are now handled as IPv4 users. Not only it gives them access to a wider range of mirrors, but it also avoids the tunnel overhead.
Another improvement is the prompt detection of mirrors that drop architectures without prior notification. This is thanks to a new version of ftpsync, the standard and recommended tool to create Debian mirrors suitable for Debian's mirrors network.
As more mirror administrators upgrade to the newer version, or switch to it, http.debian.net and other tools become more reliable: they no longer need to perform some checks and hope the results mean the mirrors actually include the architectures they say they include.
In this regard, http.debian.net is an improvement over debian-installer's mirrors selection menu: not only it chooses a mirror that is close to the user and up to date, it also ensures the mirror provides the files for an architecture even after the installer image has been built. debian-installer's mirrors list is static and hard-coded into the installation media.
There's more to come, stay tuned.
July 08, 2012
Updates to http.debian.net
It's been a week with quite some changes to http.debian.net, the Debian mirrors redirector, and it keeps coping very well with the continuously increasing traffic: over 1.5 million requests from APT clients in the last seven days! half-million more from the week before.
The good news is, those rare Hash Sum mismatch errors should be mostly gone. Ditto for some other sort of errors. There is now a second server that takes care of monitoring the mirrors and is ready to handle some of the traffic if there's any need. With this new server, http.debian.net will soon be available over IPv6 too.
Those who are at Debconf12 and followed my advice to use http.debian.net will have noticed that it is redirecting users to the local mirror. So, once again, forget about switching mirrors!
Thanks are also due to Jörg Jaspert and Philipp Kern, for the new server and for the work needed to allow http.debian.net access to Debconf12's local mirror, respectively.
Many thanks again to those who keep providing feedback and have helped the project along the way.
What's next? even more improvements and fixing some issues, some of which involve further collaboration and cooperation with the mirror administrators.
Some parts of the Debian mirrors network are fragile and may bite every once in a while, but those are being worked on. If you administer a mirror, please use ftpsync and submit your Debian mirror.
You might be happy to know that I'm working on restricting mirrors to specific Autonomous System, or countries. More on this later.
The good news is, those rare Hash Sum mismatch errors should be mostly gone. Ditto for some other sort of errors. There is now a second server that takes care of monitoring the mirrors and is ready to handle some of the traffic if there's any need. With this new server, http.debian.net will soon be available over IPv6 too.
Those who are at Debconf12 and followed my advice to use http.debian.net will have noticed that it is redirecting users to the local mirror. So, once again, forget about switching mirrors!
Thanks are also due to Jörg Jaspert and Philipp Kern, for the new server and for the work needed to allow http.debian.net access to Debconf12's local mirror, respectively.
Many thanks again to those who keep providing feedback and have helped the project along the way.
What's next? even more improvements and fixing some issues, some of which involve further collaboration and cooperation with the mirror administrators.
Some parts of the Debian mirrors network are fragile and may bite every once in a while, but those are being worked on. If you administer a mirror, please use ftpsync and submit your Debian mirror.
You might be happy to know that I'm working on restricting mirrors to specific Autonomous System, or countries. More on this later.
June 21, 2012
Introducing http.debian.net, Debian's mirrors redirector
You've been there: you are about to install a package, upgrade, or get the source package, and the mirror fails. It is offline, it is out of date, etc. Whatever the reason, you couldn't do it.
That's only one of the issues that http.debian.net attempts to address. In a nutshell: it works as an http-only content distribution network. It uses the network and geographic location to select the best online and up-to-date mirrors.
APT's sources.list one liner:
Use /debian-backports for backports, and /debian-archive for archive.debian.org.
Originally introduced to the Debian mirrors community back in January, the code on the server and client sides has improved since. All is needed is squeeze's APT (or aptitude), but wheezy's and sid's will perform better.
Advantages and more details are discussed at the Debian mirrors redirector's website.
(Thanks to the early testers, everyone who has provided feedback, my friends at PuffinHost, and all the mirrors-related people, obviously including those who sponsor them!)
That's only one of the issues that http.debian.net attempts to address. In a nutshell: it works as an http-only content distribution network. It uses the network and geographic location to select the best online and up-to-date mirrors.
APT's sources.list one liner:
deb http://http.debian.net/debian stable main
Use /debian-backports for backports, and /debian-archive for archive.debian.org.
Originally introduced to the Debian mirrors community back in January, the code on the server and client sides has improved since. All is needed is squeeze's APT (or aptitude), but wheezy's and sid's will perform better.
Advantages and more details are discussed at the Debian mirrors redirector's website.
(Thanks to the early testers, everyone who has provided feedback, my friends at PuffinHost, and all the mirrors-related people, obviously including those who sponsor them!)
May 22, 2012
On mirrors and why there are so few
Yes, I said few.
Even if in my blog post about Debian's ever-growing mirrors network it seemed like we have a lot of mirrors, truth be told: they are too few. There are many partial or even complete mirrors out there that are not listed.
If you are the administrator of one of those mirrors, please consider submitting your Debian mirror. It helps keep track of them and expose them to more users.
Granted, there are some that due to policies (on remote connections, bandwidth use, etc.) they are kept private. Hidden behind a LAN, hoping that people inside actually use them. As of this time, there's not much that can be done about them. If they were to be listed somewhere, there's no way for somebody to check them from the outside.
Some other mirror networks require that an IP (or a range) be whitelisted, allowing them to access the mirrors from the outside to perform routine checks. For some reason, I thought Fedora's was one of them but, apparently, they use a different approach: they ask private mirrors to run a tool.
I think that each approach has its pros and cons.
Even if in my blog post about Debian's ever-growing mirrors network it seemed like we have a lot of mirrors, truth be told: they are too few. There are many partial or even complete mirrors out there that are not listed.
If you are the administrator of one of those mirrors, please consider submitting your Debian mirror. It helps keep track of them and expose them to more users.
Granted, there are some that due to policies (on remote connections, bandwidth use, etc.) they are kept private. Hidden behind a LAN, hoping that people inside actually use them. As of this time, there's not much that can be done about them. If they were to be listed somewhere, there's no way for somebody to check them from the outside.
Some other mirror networks require that an IP (or a range) be whitelisted, allowing them to access the mirrors from the outside to perform routine checks. For some reason, I thought Fedora's was one of them but, apparently, they use a different approach: they ask private mirrors to run a tool.
I think that each approach has its pros and cons.
April 30, 2012
An ever-growing mirrors network
As of the time of writing, Debian's mirrors network has around 330 mirrors serving the archive over http, and around 300 serving it over ftp.
This month alone, 6 more mirrors were added. Other 6 more were added last month.
There are mirrors in 73 countries. All this couldn't be possible without the help of the sponsors.
However, some interesting questions arise: do people actually use those new mirrors? is having so many mirrors worth the extra load put on the primary mirrors?
I would personally answer: no, and maybe.
Blogosphere: When was the last time you took a look at the mirrors list and/or tried to find a mirror that served you better?
This month alone, 6 more mirrors were added. Other 6 more were added last month.
There are mirrors in 73 countries. All this couldn't be possible without the help of the sponsors.
However, some interesting questions arise: do people actually use those new mirrors? is having so many mirrors worth the extra load put on the primary mirrors?
I would personally answer: no, and maybe.
Blogosphere: When was the last time you took a look at the mirrors list and/or tried to find a mirror that served you better?
Subscribe to:
Posts (Atom)