Wget

From ArchWiki
Revision as of 23:29, 29 December 2010 by Fukawi2 (Talk | contribs) (pacman integration: typo fix)

Jump to: navigation, search

From Wikipedia:

GNU Wget is a simple computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get, connotative of its primary function. It currently supports downloading via HTTP, HTTPS, and FTP protocols, the most popular TCP/IP-based protocols used for web browsing.

Installing

Wget is normally installed as part of the base setup. If not present, install the Template:Package Official package using pacman:

# pacman -S wget

Configuring

Configuration is performed in Template:Filename. Not only is the default configuration file well documented; altering it is seldom necessary. See the man page for more intricate options.

FTP automation

Normally, SSH is used to securely transfer files among a network. However, FTP is lighter on resources compared to scp and rsyncing over SSH. FTP is not as secure, but when transfering large amounts of data inside a firewall protected environment on CPU-bound systems, using FTP can prove beneficial.

wget ftp://root:somepassword@10.13.X.Y//ifs/home/test/big/"*.tar"

3,562,035,200 74.4M/s   in 47s

In this case, Wget transfered a 3.3 G file at 74.4MB/second rate.

In short, this procedure is:

  • scriptable
  • faster than ssh
  • easily used by languages than can substitute string variables
  • globbing capable

Proxy

Wget uses the standard proxy environment variables. See: Proxy settings

To use the proxy authentication feature:

$ wget --proxy-user "DOMAIN\USER" --proxy-password "PASSWORD" URL

Proxies that use HTML authentication forms are not covered.

pacman integration

To have pacman automatically use Wget and a proxy with authentication, place the Wget command into Template:Filename, in the Template:Codeline section:

XferCommand = /usr/bin/wget --proxy-user "domain\user" --proxy-password="password" --passive-ftp -c -O %o %u
Warning: be aware that storing passwords in plain text is not safe. Make sure that only the root can read the with Template:Codeline.