tag:blogger.com,1999:blog-3939164541028317159.post7990329496791327377..comments2023-05-25T09:20:09.110-06:00Comments on Raphael's blog: rsync is not enoughUnknownnoreply@blogger.comBlogger7125tag:blogger.com,1999:blog-3939164541028317159.post-17514988467779769592012-11-17T15:01:50.865-06:002012-11-17T15:01:50.865-06:00True, but the key point here is feasibility. If an...True, but the key point here is feasibility. If anybody is willing to spend time implementing such tool and then having some mirrors buy the idea and deploy it: good.<br /><br />I'm sceptic of people actually using it. Like I mentioned, there's still people who only run rsync once...<br />Raphael Geisserthttps://www.blogger.com/profile/13007419250201863744noreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-89179455993876214942012-11-17T14:52:03.029-06:002012-11-17T14:52:03.029-06:00The part of making an atomic switch is one of the ...The part of making an atomic switch is one of the biggest issues. Even if that property could easily be satisfied at the file system level, then there is the problem of a user updating when the mirror is in state A and, while the user is still updating its index files, the mirror switches to state B. <br />In that case, and in spite of the atomic property, the mirror is in an inconsistent state from the user's point of view.<br /><br />The current aim is to not have the need of an atomic update.<br />Raphael Geisserthttps://www.blogger.com/profile/13007419250201863744noreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-11555121912076482062012-11-05T17:57:51.431-06:002012-11-05T17:57:51.431-06:00"Jim: due to the diversity and the kind of sy..."Jim: due to the diversity and the kind of systems used on the mirrors it would be unlikely that any mirror would use such approach."<br /><br />That's weird. Even if some mirrors have inferior software and can't play along, it still seems prudent to suggest a non-fragile solution like snapshots for those with modern capabilities.Jimhttps://www.blogger.com/profile/12055216650402612702noreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-83334047285026202432012-11-05T14:26:16.372-06:002012-11-05T14:26:16.372-06:00Well, maybe a quite foolish idea:
The main reposi...Well, maybe a quite foolish idea:<br /><br />The main repository that is replicated should be intact all the time, right? So what if you rsync it to a new place that is not published with rsync --link-dest $published_place and switch it atomically once all syncs completed successfully?Andre Klärnerhttps://www.blogger.com/profile/08282276946915585999noreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-34001015518465018452012-11-04T01:12:57.195-06:002012-11-04T01:12:57.195-06:00Jim: due to the diversity and the kind of systems ...Jim: due to the diversity and the kind of systems used on the mirrors it would be unlikely that any mirror would use such approach.<br /><br />Marcos: it doesn't appear to know about changes to the archive format in the last few years. Moreover, it uses http which is suboptimal for mirroring, and doesn't mirror other files which are important. Perhaps it could work for a local, partial, mirror; but not at all for a mirror that ought to be part of the mirrors network.<br />For those reading at home: <a href="http://packages.debian.org/debmirror" rel="nofollow">debmirror</a> also allows partial mirrors to be created, can use rsync, and is maintained and kept up to date with changes to the Debian archive format.<br /><br />Marcos, please don't take my comment wrong. I believe it is good to have different implementations that explore different approaches. However, I also believe they should eventually converge or allow others to take their place.Raphael Geisserthttps://www.blogger.com/profile/13007419250201863744noreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-88249842992653754792012-11-03T17:53:39.798-06:002012-11-03T17:53:39.798-06:00well, my approach is to:
* download dists databas...well, my approach is to:<br /><br />* download dists databases, read them<br />* download packages listed in those that are not already present in the mirror<br />* if all the files were successfully downloaded<br /> * update the dists databases<br /> * remove files in the mirror that are not anymore in the databases<br />* else<br /> * leave old databases<br /> * also the new packages, so next time I don't have to download the updates again.<br /><br />You can check the script, which not only supports Debian repos, but also yum, urpmi, slackware and cpan, here:<br /><br />https://github.com/StyXman/psync<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3939164541028317159.post-89104166741073143792012-10-31T10:45:33.592-06:002012-10-31T10:45:33.592-06:00What about just keeping your mirror on LVM, making...What about just keeping your mirror on LVM, making a snapshot, and having the public server point to the snapshot while you rsync the master copy?Jimhttps://www.blogger.com/profile/12055216650402612702noreply@blogger.com