Difference between revisions of "Local Mirror"

From ArchWiki
Jump to: navigation, search
(lftp excluding with -X not work,changing to -x)
(simplification of wikilinks, fixing whitespace and section fragments (https://github.com/lahwaacz/wiki-scripts/blob/master/link-checker.py))
 
(11 intermediate revisions by 9 users not shown)
Line 3: Line 3:
 
[[ru:Local Mirror]]
 
[[ru:Local Mirror]]
 
[[zh-CN:Local Mirror]]
 
[[zh-CN:Local Mirror]]
{{Warning| If you want to create an official mirror see [https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors this page].}}
+
{{Warning| If you want to create an official mirror see [[DeveloperWiki:NewMirrors|this page]].}}
  
 
==STOP==
 
==STOP==
Line 10: Line 10:
  
 
====Alternatives:====
 
====Alternatives:====
*[[Network Shared Pacman Cache]]
+
 
 +
* [[Pacman/Tips and tricks#Network shared pacman cache]]
  
 
==Local Mirror==
 
==Local Mirror==
Line 17: Line 18:
 
* Bandwidth is not free for the mirrors. They must pay for all the data they serve you
 
* Bandwidth is not free for the mirrors. They must pay for all the data they serve you
 
** This still applies although you pay your ISP
 
** This still applies although you pay your ISP
 +
** A full mirror (32+64 bit) is over 41GB in size (as of 2015-01-21)
 
* There are many packages that will be downloaded that you will likely never use
 
* There are many packages that will be downloaded that you will likely never use
 
* Mirror operators will much prefer you to download only the packages you need
 
* Mirror operators will much prefer you to download only the packages you need
Line 25: Line 27:
 
===Server Configuration===
 
===Server Configuration===
 
====Building Rsync Command====
 
====Building Rsync Command====
* Use the rsync arguments from [https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors DeveloperWiki:NewMirrors]
+
* Use the rsync arguments from [[DeveloperWiki:NewMirrors]]
 
* Select a server from the above article
 
* Select a server from the above article
 
* Exclude folder/files you do not want by including {{Ic|1=--exclude-from="/path/to/exclude.txt"}} in the rsync arguments. Example contents might include:
 
* Exclude folder/files you do not want by including {{Ic|1=--exclude-from="/path/to/exclude.txt"}} in the rsync arguments. Example contents might include:
Line 47: Line 49:
 
* Exclude any top-level directories that you do not need
 
* Exclude any top-level directories that you do not need
  
Example: {{Ic|1=rsync ''$rsync_arguments'' --exclude="/path/to/exclude.txt" ''rsync://example.com/'' /path/to/destination}}
+
Example: {{Ic|1=rsync ''$rsync_arguments'' --exclude-from="/path/to/exclude.txt" ''rsync://example.com/'' /path/to/destination}}
  
 
====Example Script====
 
====Example Script====
Line 79: Line 81:
 
  RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.*"
 
  RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.*"
 
  LCK_FLE='/var/run/repo-sync.lck'
 
  LCK_FLE='/var/run/repo-sync.lck'
 +
 +
PATH='/usr/bin'
 
   
 
   
 
  # Make sure only 1 instance runs
 
  # Make sure only 1 instance runs
 
  if [ -e "$LCK_FLE" ] ; then
 
  if [ -e "$LCK_FLE" ] ; then
  OTHER_PID=`/bin/cat $LCK_FLE`
+
  OTHER_PID=$(cat $LCK_FLE)
 
  echo "Another instance already running: $OTHER_PID"
 
  echo "Another instance already running: $OTHER_PID"
 
  exit 1
 
  exit 1
Line 90: Line 94:
 
  for REPO in $REPOS ; do
 
  for REPO in $REPOS ; do
 
  echo "Syncing $REPO"
 
  echo "Syncing $REPO"
  /usr/bin/rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
+
  rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
 
  done
 
  done
 
   
 
   
 
  # Cleanup
 
  # Cleanup
  /bin/rm -f "$LCK_FLE"
+
  rm -f "$LCK_FLE"
 
   
 
   
 
  exit 0
 
  exit 0
Line 120: Line 124:
 
Mirroring only some repositories is definitely not easy, due to the centralization of most packages in `pool/`.
 
Mirroring only some repositories is definitely not easy, due to the centralization of most packages in `pool/`.
 
See [http://blog.invokk.net/2012/01/mirroring-only-some-repositories-of-archlinux/ this blog post] for an attempt at writing a script for this task.
 
See [http://blog.invokk.net/2012/01/mirroring-only-some-repositories-of-archlinux/ this blog post] for an attempt at writing a script for this task.
 +
 +
If you can install all of the packages you wish to mirror, you can install them and set up an http server to share your package cache. An example lighttpd.conf would be:
 +
    # This is a minimal example config
 +
    # See /usr/share/doc/lighttpd
 +
    # and http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ConfigurationOptions
 +
   
 +
    server.port            = 80
 +
    server.username        = "http"
 +
    server.groupname        = "http"
 +
    server.document-root    = "/var/cache/pacman/pkg"
 +
    server.errorlog        = "/var/log/lighttpd/error.log"
 +
    dir-listing.activate    = "enable"
 +
    index-file.names        = ( "index.html" )
 +
    mimetype.assign        = ( ".html" => "text/html", ".txt" => "text/plain", ".jpg" => "image/jpeg", ".png" => "image/png", "" => "application/octet-stream" )
 +
 +
Then to update your partial mirror, simply execute pacman -Syu. Note this doesn't work for multiple architectures.
  
 
====Serving====
 
====Serving====

Latest revision as of 11:16, 27 February 2016

Warning: If you want to create an official mirror see this page.

STOP

Warning: It is generally frowned upon to create a local mirror due the bandwidth that is required. One of the alternatives will likely fulfill your needs. Please look at the alternatives below.

Alternatives:

Local Mirror

Things to keep in mind:

  • Bandwidth is not free for the mirrors. They must pay for all the data they serve you
    • This still applies although you pay your ISP
    • A full mirror (32+64 bit) is over 41GB in size (as of 2015-01-21)
  • There are many packages that will be downloaded that you will likely never use
  • Mirror operators will much prefer you to download only the packages you need
  • Really please look at the alternatives above

If you are absolutely certain that a local mirror is the only sensible solution, then follow the pointers below.

Server Configuration

Building Rsync Command

  • Use the rsync arguments from DeveloperWiki:NewMirrors
  • Select a server from the above article
  • Exclude folder/files you do not want by including --exclude-from="/path/to/exclude.txt" in the rsync arguments. Example contents might include:
iso

#Exclude i686 Packages
*/os/i686
pool/*/*-i686.pkg.tar.xz
pool/*/*-i686.pkg.tar.gz

#Exclude x86_64 Packages
*/os/x86_64
pool/*/*-x86_64.pkg.tar.xz
pool/*/*-x86_64.pkg.tar.gz
  • All packages reside in the pool directory. Symlinks are then created from pool to core/extra/testing/etc..
    • As of 9/21/2010 this migration is not yet complete.
      • There may be actual packages, instead of symlinks, in ${repo}/os/${arch}
  • Exclude any top-level directories that you do not need

Example: rsync $rsync_arguments --exclude-from="/path/to/exclude.txt" rsync://example.com/ /path/to/destination

Example Script

Warning: DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ WARNINGS AT THE START OF THIS ARTICLE
Warning: Only use this script to sync Core/Extra/Community! If you need Testing, gnome-unstable or any other repo, use rsync --exclude instead!

Yes, this script is partially broken ON PURPOSE to avoid people doing copy-and-paste to create their own mirror. It should be easy to fix if you REALLY want a mirror.

#!/bin/bash

#################################################################################################
### It is generally frowned upon to create a local mirror due the bandwidth that is required.
### One of the alternatives will likely fulfill your needs.
### REMEMBER:
###   * Bandwidth is not free for the mirrors. They must pay for all the data they serve you
###       => This still applies although you pay your ISP 
###       => There are many packages that will be downloaded that you will likely never use
###       => Mirror operators will much prefer you to download only the packages you need
###   * Really please look at the alternatives on this page:
###       https://wiki.archlinux.org/index.php?title=Local_Mirror
### If you are ABSOLUTELY CERTAIN that a local mirror is the only sensible solution, then this
### script will get you on your way to creating it. 
#################################################################################################

# Configuration
SOURCE='rsync://mirror.example.com/archlinux'
DEST='/srv/mirrors/archlinux'
BW_LIMIT='500'
REPOS='core extra'
RSYNC_OPTS="-rtlHq --delete-after --delay-updates --copy-links --safe-links --max-delete=1000 --bwlimit=${BW_LIMIT} --delete-excluded --exclude=.*"
LCK_FLE='/var/run/repo-sync.lck'

PATH='/usr/bin'

# Make sure only 1 instance runs
if [ -e "$LCK_FLE" ] ; then
	OTHER_PID=$(cat $LCK_FLE)
	echo "Another instance already running: $OTHER_PID"
	exit 1
fi
echo $$ > "$LCK_FLE"

for REPO in $REPOS ; do
	echo "Syncing $REPO"
	rsync $RSYNC_OPTS ${SOURCE}/${REPO} ${DEST}
done

# Cleanup
rm -f "$LCK_FLE"

exit 0

Another mirror script using lftp

lftp can mirror via several different protocols: ftp, http, etc. It also restarts on error, and can run in the background. Put this into your $PATH for an easy way to mirror that continues if you log out.

#!/usr/bin/lftp -f
lcd /local/path/to/your/mirror
open ftp.archlinux.org (or whatever your favorite mirror is)
# Use 'cd' to change into the proper directory on the mirror, if necessary.
mirror -cve -x '.*i686.*' core &
mirror -cve -x '.*i686.*' extra &
mirror -cve -x '.*i686.*' community &
mirror -cve -x '.*i686.*' multilib &
lcd pool
cd pool
mirror -cve -x '.*i686.*' community &
mirror -cve -x '.*i686.*' packages &

if you want to see the current status of the mirror. open lftp on terminal and type 'attach <PID>'

Partial mirroring

Mirroring only some repositories is definitely not easy, due to the centralization of most packages in `pool/`. See this blog post for an attempt at writing a script for this task.

If you can install all of the packages you wish to mirror, you can install them and set up an http server to share your package cache. An example lighttpd.conf would be:

   # This is a minimal example config
   # See /usr/share/doc/lighttpd
   # and http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ConfigurationOptions
   
   server.port             = 80
   server.username         = "http"
   server.groupname        = "http"
   server.document-root    = "/var/cache/pacman/pkg"
   server.errorlog         = "/var/log/lighttpd/error.log"
   dir-listing.activate    = "enable"
   index-file.names        = ( "index.html" )
   mimetype.assign         = ( ".html" => "text/html", ".txt" => "text/plain", ".jpg" => "image/jpeg", ".png" => "image/png", "" => "application/octet-stream" )

Then to update your partial mirror, simply execute pacman -Syu. Note this doesn't work for multiple architectures.

Serving

Client Configuration

  • Add the proper Server= variable in /etc/pacman.d/mirrorlist
  • For physical media (such as flash drive) the following can be used: Server = file:///mnt/media/repo/$repo/os/$arch (where /mnt/media/repo is directory where local mirror located)