Difference between revisions of "Local Mirror (简体中文)"

From ArchWiki
Jump to: navigation, search
(安装准备)
m (替代方法)
(37 intermediate revisions by 14 users not shown)
Line 1: Line 1:
[[Category:Package management (English)]]
+
[[Category:Package management (简体中文)]]
[[Category:HOWTOs (English)]]
+
[[Category:简体中文]]
 +
[[en:Local Mirror]]
 +
[[ru:Local Mirror]]
 +
{{translateme (简体中文)|Article has been updated. To be translated.}}
  
{{i18n_links_start}}
+
{{警告|如果你想创建一个官方镜像,参考 [https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors 这个页面]。}}
{{i18n_entry|English|Local Mirror}}
+
{{i18n_entry|中文(简体)|本地镜像}}
+
{{i18n_links_end}}
+
  
 +
==STOP==
  
{{Translateme}}
+
{{警告|通常不赞成创建一个本地镜像,因为需要很大的带宽。其中一个替代的方法也许会满足你的要求。请查看下面的替代方法。}}
  
'''NOTE:''' 95% of users will NEVER need this. Rsyncing every package from core and extra will give you a lot of stuff you will never need or use. Only follow these instructions if you are running a very large site of Arch machines or want to create an official mirror. Perhaps using a [[Network Shared Pacman Cache]] would serve you better.
+
====替代方法====
 +
*[[Pacman_Tips_(简体中文) #.E5.9C.A8.E7.BD.91.E7.BB.9C.E4.B8.8A.E5.85.B1.E4.BA.ABpacman.E7.BC.93.E5.AD.98|在网络上共享pacman缓存]]
  
'''NOTE:''' Due to traffic issues rsyncing from rsync.archlinux.org is allowed only for official mirrors. If you want to create an official mirror - write to the mailing list about this and entries in /etc/pacman.d/* will be added and your IP address will be allowed.
+
==本地镜像==
  
If you want to get a full mirror for personal use only, you may rsync from '''rsync://distro.ibiblio.org/distros/archlinux/'''
+
===需要牢记的事实===
 +
* 镜像的带宽不是免费的,镜像必须为提供给你的数据付费
 +
** 尽管你已经为你的ISP付费,这也是成立的。
 +
* 有很多你可能下载了却永远用不到的包
 +
* 镜像的管理员会更喜欢让你仅下载需要的包
 +
* 再次请求仔细查看上面的替代方案
  
This document describes how to create a mirror on your local machine of all the packages and iso files on the Arch mirrors, how to update it using cron, how to serve the mirror with vsftpd, and how to set up pacman to use the local mirror.
+
'''如果你完全确定本地镜像是唯一合适的解决方案,那么按照下面的指引操作。'''
  
== 安装准备 ==
+
===服务器设置===
 +
====Building Rsync Command====
 +
* Use the rsync arguments from [https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors DeveloperWiki:NewMirrors]
 +
* Select a server from the above article
 +
* Exclude folder/files you do not want by including {{Ic|1=--exclude-from="/path/to/exclude.txt"}} in the rsync arguments. Example contents might include:
 +
<pre>
 +
iso
  
首先,更新或是安装安装必要的工具:
+
#Exclude i686 Packages
pacman -Sy rsync vsftpd
+
*/os/i686
 +
pool/*/*-i686.pkg.tar.xz
 +
pool/*/*-i686.pkg.tar.gz
  
现在,我们要创建一个用于同步镜像的新用户('''没有登录特权'''),使用FTP获取文件。用户名“mirror”只是一个例子,你可以使用其他任何名称。请'''不要'''使用root或是其他能登录系统的帐号。安全起见,我们使用的帐号只能拥有尽可能少的权限。
+
#Exclude x86_64 Packages
 +
*/os/x86_64
 +
pool/*/*-x86_64.pkg.tar.xz
 +
pool/*/*-x86_64.pkg.tar.gz
 +
</pre>
  
useradd -m -s /bin/false mirror
+
* All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
 +
** As of 9/21/2010 this migration is not yet complete.
 +
*** There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
 +
* Exclude any top-level directories that you do not need
  
现在,我们将继续创建镜像。
+
Example: {{Ic|1=rsync ''$rsync_arguments'' --exclude="/path/to/exclude.txt" ''rsync://example.com/'' /path/to/destination}}
  
== Creating the local mirror directory ==
+
====Example Script====
We will be using <tt>/home/mirror</tt>, the home directory of our unprivileged user, for storage of the scripts, logs, and packages.
+
  
The first thing we need to do is create several directories in /home/mirror:
+
{{Warning|DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ WARNINGS AT THE START OF THIS ARTICLE}}
 +
{{Warning|Only use this script to sync Core/Extra/Community! If you need Testing, gnome-unstable or any other repo, use rsync --exclude instead!}}
  
cd /home/mirror
+
Yes, this script is partially broken '''ON PURPOSE''' to avoid people doing copy-and-paste to create their own mirror. It should be easy to fix if you REALLY want a mirror.
sudo -u mirror mkdir {scripts,files,logs}
+
 
+
== The synchronization script ==
+
 
+
Now lets create the actual rsync script, <tt>scripts/mirrorsync.sh</tt>, using your favorite editor.
+
  
 
  #!/bin/bash
 
  #!/bin/bash
#
 
# The script to sync a local mirror of the Arch Linux repositories and ISOs
 
#
 
# Copyright (C) 2007 Woody Gilk <woody@archlinux.org>
 
# Modifications by Dale Blount <dale@archlinux.org>
 
# and Roman Kyrylych <roman@archlinux.org>
 
# Licensed under the GNU GPL (version 2)
 
 
   
 
   
  # Filesystem locations for the sync operations
+
  #################################################################################################
  SYNC_HOME="/home/mirror"
+
### It is generally frowned upon to create a local mirror due the bandwidth that is required.
  SYNC_LOGS="$SYNC_HOME/logs"
+
### One of the alternatives will likely fulfill your needs.
  SYNC_FILES="$SYNC_HOME/files"
+
### REMEMBER:
  SYNC_LOCK="$SYNC_HOME/mirrorsync.lck"
+
###  * Bandwidth is not free for the mirrors. They must pay for all the data they serve you
 +
  ###      => This still applies although you pay your ISP
 +
  ###      => There are many packages that will be downloaded that you will likely never use
 +
  ###      => Mirror operators will much prefer you to download only the packages you need
 +
  ###  * Really please look at the alternatives on this page:
 +
###      https://wiki.archlinux.org/index.php?title=Local_Mirror
 +
### If you are ABSOLUTELY CERTAIN that a local mirror is the only sensible solution, then this
 +
### script will get you on your way to creating it.
 +
#################################################################################################
 
   
 
   
  # Select which repositories to sync
+
  # Configuration
  # Valid options are: core, extra, unstable, testing, community, iso
+
  SOURCE='rsync://mirror.example.com/archlinux'
  # Leave empty to sync a complete mirror
+
  DEST='/srv/mirrors/archlinux'
  # SYNC_REPO=(core extra unstable testing community iso)
+
  BW_LIMIT='500'
  SYNC_REPO=()
+
REPOS='core extra'
 +
  RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.*"
 +
LCK_FLE='/var/run/repo-sync.lck'
 
   
 
   
  # Set the rsync server to use
+
  # Make sure only 1 instance runs
# Only official public mirrors are allowed to use rsync.archlinux.org
+
  if [ -e "$LCK_FLE" ] ; then
# SYNC_SERVER=rsync.archlinux.org::ftp
+
OTHER_PID=`/bin/cat $LCK_FLE`
SYNC_SERVER=distro.ibiblio.org::distros/archlinux
+
echo "Another instance already running: $OTHER_PID"
+
exit 1
# Set the format of the log file name
+
# This example will output something like this: sync_20070201-8.log
+
LOG_FILE="pkgsync_$(date +%Y%m%d-%H).log"
+
+
# Do not edit the following lines, they protect the sync from running more than
+
# one instance at a time
+
  if [ ! -d $SYNC_HOME ]; then
+
  echo "$SYNC_HOME does not exist, please create it, then run this script again."
+
  exit 1
+
 
  fi
 
  fi
 +
echo $$ > "$LCK_FLE"
 
   
 
   
  [ -f $SYNC_LOCK ] && exit 1
+
  for REPO in $REPOS ; do
touch "$SYNC_LOCK"
+
echo "Syncing $REPO"
  # End of non-editable lines
+
echo /usr/bin/rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
 +
  done
 
   
 
   
  # Create the log file and insert a timestamp
+
  # Cleanup
  touch "$SYNC_LOGS/$LOG_FILE"
+
  /bin/rm -f "$LCK_FLE"
echo "=============================================" >> "$SYNC_LOGS/$LOG_FILE"
+
echo ">> Starting sync on $(date --rfc-3339=seconds)" >> "$SYNC_LOGS/$LOG_FILE"
+
echo ">> ---" >> "$SYNC_LOGS/$LOG_FILE"
+
 
   
 
   
if [ -z $SYNC_REPO ]; then
 
  # Sync a complete mirror
 
  rsync -rptLv --delete-after --delay-updates $SYNC_SERVER "$SYNC_FILES" >> "$SYNC_LOGS/$LOG_FILE"
 
else
 
  # Sync each of the repositories set in $SYNC_REPO
 
  for repo in ${SYNC_REPO[@]}; do
 
    repo=$(echo $repo | tr [:upper:] [:lower:])
 
    echo ">> Syncing $repo to $SYNC_FILES/$repo" >> "$SYNC_LOGS/$LOG_FILE"
 
 
    # If you only want to mirror i686 packages, you can add
 
    # " --exclude=os/x86_64" after "--delete-after"
 
    #
 
    # If you only want to mirror x86_64 packages, use "--exclude=os/i686"
 
    # If you want both i686 and x86_64, leave the following line as it is
 
    #
 
    rsync -rptlv --safe-links --delete-after --delay-updates $SYNC_SERVER/$repo "$SYNC_FILES" >> "$SYNC_LOGS/$LOG_FILE"
 
 
    # Create $repo.lastsync file with timestamp like "2007-05-02 03:41:08+03:00"
 
    # which may be useful for users to know when the repository was last updated
 
    # date --rfc-3339=seconds > "$SYNC_FILES/$repo.lastsync"
 
 
    # Sleep 5 seconds after each repository to avoid too many concurrent connections
 
    # to rsync server if the TCP connection does not close in a timely manner
 
    sleep 5
 
  done
 
fi
 
 
# Insert another timestamp and close the log file
 
echo ">> ---" >> "$SYNC_LOGS/$LOG_FILE"
 
echo ">> Finished sync on $(date --rfc-3339=seconds)" >> "$SYNC_LOGS/$LOG_FILE"
 
echo "=============================================" >> "$SYNC_LOGS/$LOG_FILE"
 
echo "" >> "$SYNC_LOGS/$LOG_FILE"
 
 
# Remove the lock file and exit
 
rm -f "$SYNC_LOCK"
 
 
  exit 0
 
  exit 0
  
Nothing terribly fancy here, just a slightly advanced bash script to do what we need. Let's make it executable.
 
 
chmod +x scripts/mirrorsync.sh
 
 
That's it, you now have an easily modifiable script. You probably don't want to have to run this manually though, so let's set up a cron job to run this for us.
 
 
One note before we move on to the next step: your logs directory is going to keep growing in size. Make sure that you check it regularly so that it doesn't start overtaking the server with garbage. It is highly recommended that you set up [[LogRotate]] to manage this, or write some kind of cleanup script.
 
 
== Running a cron job ==
 
 
Let's make sure we have the necessary cron tools (most Arch installations will):
 
 
pacman -S dcron
 
 
We will be running our cron job with <tt>crontab</tt>. For more infomation, see <tt>man crontab</tt>. The benefit of running the sync with a crontab is a higher level of security, and not cluttering up <tt>/etc/cron.*</tt> with files. It also allows for a higher level of control for when the script is run.
 
 
Create <tt>scripts/mirror.cron</tt> with the following contents:
 
 
0 3 * * * /home/mirror/scripts/mirrorsync.sh
 
 
Now we need to activate our crontab:
 
 
sudo -u mirror crontab scripts/mirror.cron
 
 
Let's make sure that crontab picked up our job:
 
 
sudo -u mirror crontab -l
 
 
You should see the contents of <tt>scripts/mirror.cron</tt> printed out. If not, rerun the previous command and check again.
 
 
This cron setup will run our sync.sh script every night at 3AM. You can adjust this however you want, see http://www.adminschoice.com/docs/crontab.htm for more information on crontab syntax.
 
 
=== Editing the cron job ===
 
If you ever need to edit <tt>mirror.cron</tt>, use the following command:
 
 
sudo -u mirror crontab -e
 
 
If you edit the file by hand, use the following command to update crontab:
 
 
sudo -u mirror crontab scripts/mirror.cron
 
 
Now let's set up pacman to use our local mirror.
 
 
== Setting up pacman to use the local mirror ==
 
 
If you only want to access your mirror on one computer, you can use the following steps.
 
 
=== Single machine ===
 
 
'''NOTE:''' If you are following the above for a single machine, you are using a lot of bandwidth for no reason at all. Save it for the people that need it. This section only applies for those that will follow through with the below section as well.
 
 
You will not need vsftpd for this type of setup, because we are accessing the files via a file:// url, as opposed to a ftp:// url.
 
 
Add the following line to the top of <tt>/etc/pacman.d/core</tt>, at the top of the Servers list:
 
 
Server = file:///home/mirror/files/core/os/i686
 
 
You should also add this line, substituting <tt>/core/</tt> for the name of the repository you are mirroring (e.g.: for community, the line should look like this):
 
 
Server = file:///home/mirror/files/community/os/i686
 
 
Also be sure to change <tt>i686</tt> to <tt>x86_64</tt> if you are using a 64bit version of Arch.
 
 
=== Multiple machines ===
 
 
Syncing this way will allow you to use FTP to access your local mirror from other machines. You can also use this method to sync to your local machine (more details on this later).
 
 
==== FTP server configuration ====
 
 
The first thing we need to do is configure vsftpd. Edit <tt>/etc/vsftpd.conf</tt> to look like this:
 
 
# vsftpd config file /etc/vsftpd.conf
 
#
 
# Setup for a secure anonymous FTP server
 
#
 
# Listen (non-xinetd) mode
 
listen=YES
 
# Use tcp_wrappers to control connections
 
tcp_wrappers=YES
 
# Use localtimes instead of GMT for files
 
use_localtime=YES
 
# Hide the true user/group ID of files
 
hide_ids=YES
 
#
 
# Enable anonymous access (pacman requires this)
 
anonymous_enable=YES
 
# Use this user for anonymous logins
 
ftp_username=mirror
 
# Chroot directory for anonymous user
 
anon_root=/home/mirror/files
 
# Don't require a password for anonymous access (pacman requires this)
 
no_anon_password=YES
 
#
 
# User to run vsftpd as (same as ftp_username)
 
nopriv_user=mirror
 
# Enable recursive "ls" listing
 
ls_recurse_enable=YES
 
#
 
# Forcefully destroy sessions after X seconds of inactivity
 
# (It is highly recommended to not set this above 300)
 
idle_session_timeout=120
 
# Forcefully stop sending data after X seconds of inactivity during a transfer
 
# (It is highly recommended to not set this higher than idle_session_timeout)
 
data_connection_timeout=30
 
 
This setup will offer a very secure FTP server, tailored specifically for our needs. Note that this setup does '''not''' require a password, and should not be used in a publicly accessible network (unless that's what you want). ''Password protecting the FTP and still allowing it to work with pacman is beyond the scope of this document.''
 
 
If you are going to connect to this machine from the outside, you will need to add the following line to <tt>/etc/hosts.allow</tt>:
 
 
vsftpd : ALL : ALL
 
 
Note that this will allow '''anyone''' to download from the mirror. If you want to control downloads more tightly, and don't know how to do so, see [http://linux.about.com/od/commands/l/blcmdl5_hostsal.htm linux.about.com] on the subject.
 
 
Let's make sure vsftpd starts:
 
 
sudo /etc/rc.d/vsftpd start
 
 
If vsftpd does not start, check that the options are set correctly in your <tt>/etc/vsftpd.conf</tt> file.
 
 
==== Enabling the mirror for pacman ====
 
 
Now let's edit <tt>/etc/pacman.d/*</tt> files to use our shiny new mirror. Add the following line to the top of <tt>/etc/pacman.d/core</tt>, at the top of the Servers list:
 
 
Server = ftp://192.168.1.21/core/os/i686
 
 
Note that <tt>192.168.1.21</tt> is the IP address of my test machine. Your address will most likely be different. (Remember that you can get the current IP of an Arch box with <tt>ifconfig -a</tt> or <tt>ifconfig eth0</tt>.)
 
 
If you want to use this same mirror on the local machine, use the following Server line:
 
 
Server = ftp://localhost/core/os/i686
 
  
Repeat this same process for the other repositories you are mirroring. Non-local machines will need to use an IP address to access the repository. Also make sure that the machine serving the mirror has a static IP address.
+
====Another mirror script using lftp====
 +
lftp can mirror via several different protocols: ftp, http, etc. It also restarts on error, and can run in the background. Put this into your $PATH for an easy way to mirror that continues if you log out.
  
== Synchronizing for the first time ==
+
#!/usr/bin/lftp -f
 +
lcd /local/path/to/your/mirror
 +
open ftp.archlinux.org (or whatever your favorite mirror is)
 +
# Use 'cd' to change into the proper directory on the mirror, if necessary.
 +
mirror -cve -X *i686* core &
 +
mirror -cve -X *i686* extra &
 +
mirror -cve -X *i686* community &
 +
mirror -cve -X *i686* multilib &
 +
lcd pool
 +
cd pool
 +
mirror -cve -X *i686* community &
 +
mirror -cve -X *i686* packages &
  
Here comes the pain! Run the following command to start the sync:
+
if you want to see the current status of the mirror. open lftp on terminal and type 'attach <PID>'
  
sudo -u mirror ./scripts/mirrorsync.sh
+
====Partial mirroring====
  
This won't give you any kind of output, but you probably want some. You can use something like this (correct for the name of log file) to monitor the sync progress:
+
Mirroring only some repositories is definitely not easy, due to the centralization of most packages in `pool/`.
 +
See [http://blog.invokk.net/2012/01/mirroring-only-some-repositories-of-archlinux/ this blog post] for an attempt at writing a script for this task.
  
tail -f logs/pkgsync_20070203-9.log
+
====Serving====
 +
*HTTP (LAN)
 +
**[[LAMP]]
 +
**[[Lighttpd]]
 +
*FTP (LAN)
 +
**[[vsftpd]]
 +
*Physical Media
 +
**Flash Drive
 +
**External HD
  
This process will usually take a few hours, depending on the speed of your internet connection and how many repositories you are mirroring. After the first sync, only new packages will be sync'ed, so it will be m
+
===Client Configuration===
 +
*Add the proper Server= variable in /etc/pacman.d/mirrorlist
 +
*For physical media (such as flash drive) the following can be used: Server = file:///mnt/media/repo/$repo/os/$arch (''where /mnt/media/repo is directory where local mirror located'')

Revision as of 14:10, 14 March 2013

Tango-preferences-desktop-locale.png本页面需要更新翻译,内容可能已经与英文脱节。要贡献翻译,请访问简体中文翻译组Tango-preferences-desktop-locale.png

附注: Article has been updated. To be translated.
警告: 如果你想创建一个官方镜像,参考 这个页面

STOP

警告: 通常不赞成创建一个本地镜像,因为需要很大的带宽。其中一个替代的方法也许会满足你的要求。请查看下面的替代方法。

替代方法

本地镜像

需要牢记的事实

  • 镜像的带宽不是免费的,镜像必须为提供给你的数据付费
    • 尽管你已经为你的ISP付费,这也是成立的。
  • 有很多你可能下载了却永远用不到的包
  • 镜像的管理员会更喜欢让你仅下载需要的包
  • 再次请求仔细查看上面的替代方案

如果你完全确定本地镜像是唯一合适的解决方案,那么按照下面的指引操作。

服务器设置

Building Rsync Command

  • Use the rsync arguments from DeveloperWiki:NewMirrors
  • Select a server from the above article
  • Exclude folder/files you do not want by including --exclude-from="/path/to/exclude.txt" in the rsync arguments. Example contents might include:
iso

#Exclude i686 Packages
*/os/i686
pool/*/*-i686.pkg.tar.xz
pool/*/*-i686.pkg.tar.gz

#Exclude x86_64 Packages
*/os/x86_64
pool/*/*-x86_64.pkg.tar.xz
pool/*/*-x86_64.pkg.tar.gz
  • All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
    • As of 9/21/2010 this migration is not yet complete.
      • There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
  • Exclude any top-level directories that you do not need

Example: rsync $rsync_arguments --exclude="/path/to/exclude.txt" rsync://example.com/ /path/to/destination

Example Script

Warning: DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ WARNINGS AT THE START OF THIS ARTICLE
Warning: Only use this script to sync Core/Extra/Community! If you need Testing, gnome-unstable or any other repo, use rsync --exclude instead!

Yes, this script is partially broken ON PURPOSE to avoid people doing copy-and-paste to create their own mirror. It should be easy to fix if you REALLY want a mirror.

#!/bin/bash

#################################################################################################
### It is generally frowned upon to create a local mirror due the bandwidth that is required.
### One of the alternatives will likely fulfill your needs.
### REMEMBER:
###   * Bandwidth is not free for the mirrors. They must pay for all the data they serve you
###       => This still applies although you pay your ISP 
###       => There are many packages that will be downloaded that you will likely never use
###       => Mirror operators will much prefer you to download only the packages you need
###   * Really please look at the alternatives on this page:
###       https://wiki.archlinux.org/index.php?title=Local_Mirror
### If you are ABSOLUTELY CERTAIN that a local mirror is the only sensible solution, then this
### script will get you on your way to creating it. 
#################################################################################################

# Configuration
SOURCE='rsync://mirror.example.com/archlinux'
DEST='/srv/mirrors/archlinux'
BW_LIMIT='500'
REPOS='core extra'
RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.*"
LCK_FLE='/var/run/repo-sync.lck'

# Make sure only 1 instance runs
if [ -e "$LCK_FLE" ] ; then
	OTHER_PID=`/bin/cat $LCK_FLE`
	echo "Another instance already running: $OTHER_PID"
	exit 1
fi
echo $$ > "$LCK_FLE"

for REPO in $REPOS ; do
	echo "Syncing $REPO"
	echo /usr/bin/rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
done

# Cleanup
/bin/rm -f "$LCK_FLE"

exit 0


Another mirror script using lftp

lftp can mirror via several different protocols: ftp, http, etc. It also restarts on error, and can run in the background. Put this into your $PATH for an easy way to mirror that continues if you log out.

#!/usr/bin/lftp -f
lcd /local/path/to/your/mirror
open ftp.archlinux.org (or whatever your favorite mirror is)
# Use 'cd' to change into the proper directory on the mirror, if necessary.
mirror -cve -X *i686* core &
mirror -cve -X *i686* extra &
mirror -cve -X *i686* community &
mirror -cve -X *i686* multilib &
lcd pool
cd pool
mirror -cve -X *i686* community &
mirror -cve -X *i686* packages &

if you want to see the current status of the mirror. open lftp on terminal and type 'attach <PID>'

Partial mirroring

Mirroring only some repositories is definitely not easy, due to the centralization of most packages in `pool/`. See this blog post for an attempt at writing a script for this task.

Serving

Client Configuration

  • Add the proper Server= variable in /etc/pacman.d/mirrorlist
  • For physical media (such as flash drive) the following can be used: Server = file:///mnt/media/repo/$repo/os/$arch (where /mnt/media/repo is directory where local mirror located)