Difference between revisions of "Local Mirror"

From ArchWiki
Jump to: navigation, search
(Server Configuration: Updated information to reflect all packages going into pool)
(Building Rsync Command: Added example script. It is slightly broken ON PURPOSE to avoid copy-pasters.)
Line 30: Line 30:
 
* Use the rsync arguments from [http://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors DeveloperWiki:NewMirrors]
 
* Use the rsync arguments from [http://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors DeveloperWiki:NewMirrors]
 
* Select a server from the above article
 
* Select a server from the above article
* Exclude folder/files you don't want by including --exclude-from="/path/to/exclude.txt" in the rsync arguments
+
* Exclude folder/files you don't want by including <tt>--exclude-from="/path/to/exclude.txt"</tt> in the rsync arguments. Example contents might include:
/path/to/exclude.txt
+
 
<pre>
 
<pre>
 
iso
 
iso
Line 45: Line 44:
 
pool/*/*-x86_64.pkg.tar.gz
 
pool/*/*-x86_64.pkg.tar.gz
 
</pre>
 
</pre>
 +
 
* All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
 
* All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
**As of 9/21/2010 this migration is not yet complete.
+
** As of 9/21/2010 this migration is not yet complete.
***There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
+
*** There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
*Exclude any top-level directories that you do not need
+
* Exclude any top-level directories that you do not need
**Example: iso - no one should download all the ISO's
+
 
 +
Example: <tt>rsync ''$rsync_arguments'' --exclude="/path/to/exclude.txt" ''rsync://example.com/'' /path/to/destination</tt>
 +
 
 +
====Example Script====
 +
 
 +
'''DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ WARNINGS AT THE START OF THIS ARTICLE'''
 +
 
 +
Yes, this script is partially broken '''ON PURPOSE''' to avoid people doing copy-and-paste to create their own mirror. It should be easy to fix if you REALLY want a mirror.
  
Example: rsync ''$rsync_arguments'' --exclude="/path/to/exclude.txt" ''rsync://example.com/'' /path/to/destination
+
#!/bin/bash
 +
 +
#################################################################################################
 +
### It is generally frowned upon to create a local mirror due the bandwidth that is required.
 +
### One of the alternatives will likely fulfill your needs.
 +
### REMEMBER:
 +
###  * Bandwidth is not free for the mirrors. They must pay for all the data they serve you
 +
###      => This still applies although you pay your ISP
 +
###      => There are many packages that will be downloaded that you will likely never use
 +
###      => Mirror operators will much prefer you to download only the packages you need
 +
###  * Really please look at the alternatives on this page:
 +
###      http://wiki.archlinux.org/index.php?title=Local_Mirror
 +
### If you are ABSOLUTELY CERTAIN that a local mirror is the only sensible solution, then this
 +
### script will get you on your way to creating it.
 +
#################################################################################################
 +
 +
# Configuration
 +
SOURCE='rsync://mirror.example.com/archlinux'
 +
DEST='/srv/mirrors/archlinux'
 +
BW_LIMIT='500'
 +
REPOS='core extra'
 +
RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.* --exclude=iso/"
 +
LCK_FLE='/var/run/repo-sync.lck'
 +
 +
# Make sure only 1 instance runs
 +
if [ -e "$LCK_FLE" ] ; then
 +
OTHER_PID=`/bin/cat $LCK_FLE`
 +
echo "Another instance already running: $OTHER_PID"
 +
exit 1
 +
fi
 +
echo $$ > "$LCK_FLE"
 +
 +
for REPO in $REPOS ; do
 +
echo "Syncing $REPO"
 +
echo /usr/bin/rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
 +
done
 +
 +
# Cleanup
 +
/bin/rm -f "$LCK_FLE"
 +
 +
exit 0
  
 
====Serving====
 
====Serving====

Revision as of 01:17, 22 September 2010

This template has only maintenance purposes. For linking to local translations please use interlanguage links, see Help:i18n#Interlanguage links.


Local languages: Català – Dansk – English – Español – Esperanto – Hrvatski – Indonesia – Italiano – Lietuviškai – Magyar – Nederlands – Norsk Bokmål – Polski – Português – Slovenský – Česky – Ελληνικά – Български – Русский – Српски – Українська – עברית – العربية – ไทย – 日本語 – 正體中文 – 简体中文 – 한국어


External languages (all articles in these languages should be moved to the external wiki): Deutsch – Français – Română – Suomi – Svenska – Tiếng Việt – Türkçe – فارسی

Note: If you want to create an official mirror see this page.

STOP

It is generally frowned upon to create a local mirror due the bandwidth that is required. One of the alternatives will likely fulfill your needs. Please look at the alternatives below.


Alternatives:


Local Mirror

Things to keep in mind:

  • Bandwidth is not free for the mirrors. They must pay for all the data they serve you
    • This still applies although you pay your ISP
  • There are many packages that will be downloaded that you will likely never use
  • Mirror operators will much prefer you to download only the packages you need
  • Really please look at the alternatives above

If you are absolutely certain that a local mirror is the only sensible solution, then follow the pointers below.

Server Configuration

Building Rsync Command

  • Use the rsync arguments from DeveloperWiki:NewMirrors
  • Select a server from the above article
  • Exclude folder/files you don't want by including --exclude-from="/path/to/exclude.txt" in the rsync arguments. Example contents might include:
iso

#Exclude i686 Packages
*/os/i686
pool/*/*-i686.pkg.tar.xz
pool/*/*-i686.pkg.tar.gz

#Exclude x86_64 Packages
*/os/x86_64
pool/*/*-x86_64.pkg.tar.xz
pool/*/*-x86_64.pkg.tar.gz
  • All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
    • As of 9/21/2010 this migration is not yet complete.
      • There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
  • Exclude any top-level directories that you do not need

Example: rsync $rsync_arguments --exclude="/path/to/exclude.txt" rsync://example.com/ /path/to/destination

Example Script

DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ WARNINGS AT THE START OF THIS ARTICLE

Yes, this script is partially broken ON PURPOSE to avoid people doing copy-and-paste to create their own mirror. It should be easy to fix if you REALLY want a mirror.

#!/bin/bash

#################################################################################################
### It is generally frowned upon to create a local mirror due the bandwidth that is required.
### One of the alternatives will likely fulfill your needs.
### REMEMBER:
###   * Bandwidth is not free for the mirrors. They must pay for all the data they serve you
###       => This still applies although you pay your ISP 
###       => There are many packages that will be downloaded that you will likely never use
###       => Mirror operators will much prefer you to download only the packages you need
###   * Really please look at the alternatives on this page:
###       http://wiki.archlinux.org/index.php?title=Local_Mirror
### If you are ABSOLUTELY CERTAIN that a local mirror is the only sensible solution, then this
### script will get you on your way to creating it. 
#################################################################################################

# Configuration
SOURCE='rsync://mirror.example.com/archlinux'
DEST='/srv/mirrors/archlinux'
BW_LIMIT='500'
REPOS='core extra'
RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.* --exclude=iso/"
LCK_FLE='/var/run/repo-sync.lck'

# Make sure only 1 instance runs
if [ -e "$LCK_FLE" ] ; then
	OTHER_PID=`/bin/cat $LCK_FLE`
	echo "Another instance already running: $OTHER_PID"
	exit 1
fi
echo $$ > "$LCK_FLE"

for REPO in $REPOS ; do
	echo "Syncing $REPO"
	echo /usr/bin/rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
done

# Cleanup
/bin/rm -f "$LCK_FLE"

exit 0

Serving

Client Configuration

  • Add the proper Server= variable in /etc/pacman.d/mirrorlist
  • For physical media (such as flash drive) the following can be used: Server = file:///mnt/media/repo