Difference between revisions of "Disk cloning"

From ArchWiki
Jump to: navigation, search
(Rar: -si is documented in /usr/share/doc/rar/rar.txt; but it also says it cannot be used with -v! The commands given otherwise work just fine)
m (Correcting grammar use/typo)
 
(45 intermediate revisions by 18 users not shown)
Line 1: Line 1:
 
[[Category:Data compression and archiving]]
 
[[Category:Data compression and archiving]]
 
[[Category:System recovery]]
 
[[Category:System recovery]]
[[it:Disk Cloning]]
+
[[it:Disk cloning]]
 +
[[ja:ディスクのクローン]]
 
[[ru:Disk cloning]]
 
[[ru:Disk cloning]]
 
[[tr:Disk klonlama]]
 
[[tr:Disk klonlama]]
[[zh-cn:Disk cloning]]
+
[[zh-hans:Disk cloning]]
'''Disk cloning''' is the process of making an image of a partition or of an entire hard drive. This can be useful for copying the drive to other computers and for [[backup]] and [[File recovery|recovery]] purposes.
+
{{Related articles start}}
 +
{{Related|Synchronization and backup programs}}
 +
{{Related|System maintenance#Backup}}
 +
{{Related|System backup}}
 +
{{Related articles end}}
 +
 
 +
Disk cloning is the process of making an image of a partition or of an entire hard drive. This can be useful for copying the drive to other computers and for [[backup]] and [[File recovery|recovery]] purposes.
  
 
== Using dd ==
 
== Using dd ==
  
The {{ic|dd}} command is a simple, yet versatile and powerful tool. It can be used to copy from source to destination, block-by-block, regardless of their filesystem types or operating systems. A convenient method is to use {{ic|dd}} from a live environment, as in a Live CD.
+
The ''dd'' command is a simple, yet versatile and powerful tool. It can be used to copy from source to destination, block-by-block, regardless of their filesystem types or operating systems. A convenient method is to use ''dd'' from a live environment, as in a Live CD.
  
 
{{Warning|As with any command of this type, you should be very cautious when using it; it can destroy data. Remember the order of input file ({{ic|1=if=}}) and output file ({{ic|1=of=}}) and do not reverse them! Always ensure that the destination drive or partition ({{ic|1=of=}}) is of equal or greater size than the source ({{ic|1=if=}}).
 
{{Warning|As with any command of this type, you should be very cautious when using it; it can destroy data. Remember the order of input file ({{ic|1=if=}}) and output file ({{ic|1=of=}}) and do not reverse them! Always ensure that the destination drive or partition ({{ic|1=of=}}) is of equal or greater size than the source ({{ic|1=if=}}).
 
}}
 
}}
 +
 +
{{Accuracy|1=Are the conv= options safe to use? See:|section=remove conv sync noerror}}
  
 
=== Cloning a partition ===
 
=== Cloning a partition ===
  
 
From physical disk {{ic|/dev/sda}}, partition 1, to physical disk {{ic|/dev/sdb}}, partition 1.
 
From physical disk {{ic|/dev/sda}}, partition 1, to physical disk {{ic|/dev/sdb}}, partition 1.
  # dd if=/dev/sda1 of=/dev/sdb1 bs=512 conv=noerror,sync
+
  # dd if=/dev/sda1 of=/dev/sdb1 bs=64K conv=noerror,sync status=progress
  
{{Warning|If output file {{ic|1=of=}} ({{ic|sdb1}} in the example) does not exist, {{ic|dd}} will create a file with this name and will start filling up your root file system!}}
+
{{Warning|If output file {{ic|1=of=}} ({{ic|sdb1}} in the example) does not exist, ''dd'' will create a file with this name and will start filling up your root file system!}}
  
 
=== Cloning an entire hard disk ===
 
=== Cloning an entire hard disk ===
  
 
From physical disk {{ic|/dev/sd''X''}} to physical disk {{ic|/dev/sd''Y''}}
 
From physical disk {{ic|/dev/sd''X''}} to physical disk {{ic|/dev/sd''Y''}}
  # dd if=/dev/sd''X'' of=/dev/sd''Y'' bs=512 conv=noerror,sync
+
  # dd if=/dev/sd''X'' of=/dev/sd''Y'' bs=64K conv=noerror,sync status=progress
  
 
This will clone the entire drive, including the MBR (and therefore bootloader), all partitions, UUIDs, and data.
 
This will clone the entire drive, including the MBR (and therefore bootloader), all partitions, UUIDs, and data.
* {{ic|noerror}} instructs {{ic|dd}} to continue operation, ignoring all read errors. Default behavior for {{ic|dd}} is to halt at any error.  
+
* {{ic|1=bs=}} sets the block size. Defaults to 512 bytes, which is the "classic" block size for hard drives since the early 1980s, but is not the most convenient. Use a bigger value, 64K or 128K. Also, please read the warning below, because there is more to this than just "block sizes" -it also influences how read errors propagate. See [http://www.mail-archive.com/eug-lug@efn.org/msg12073.html] and [http://blog.tdg5.com/tuning-dd-block-size/] for details and to figure out the best bs value for your use case.
 +
* {{ic|noerror}} instructs ''dd'' to continue operation, ignoring all read errors. Default behavior for ''dd'' is to halt at any error.  
 
* {{ic|sync}} fills input blocks with zeroes if there were any read errors, so data offsets stay in sync.
 
* {{ic|sync}} fills input blocks with zeroes if there were any read errors, so data offsets stay in sync.
* {{ic|1=bs=512}} sets the block size to 512 bytes, the "classic" block size for hard drives. If and only if your hard drives have a 4 Kib block size, you may use "4096" instead of "512". Also, please read the warning below, because there is more to this than just "block sizes" -it also influences how read errors propagate.
+
* {{ic|1=status=progress}} shows periodic transfer statistics which can be used to estimate when the operation may be complete.
  
{{Warning|The block size you specify influences how read errors are handled. Read below.}}
+
{{Warning|The block size you specify influences how read errors are handled. Read below. For data recovery, use [[#Using ddrescue|ddrescue]].}}
  
The {{ic|dd}} utility technically has an "input block size" (IBS) and an "output block size" (OBS). When you set {{ic|bs}}, you effectively set both IBS and OBS. Normally, if your block size is, say, 1 Mib, {{ic|dd}} will read 1024*1024 bytes and write as many bytes. But if a read error occurs, things will go wrong. Many people seem to think that {{ic|dd}} will "fill up read errors with zeroes" if you use the {{ic|noerror,sync}} options, but this is not what happens. {{ic|dd}} will, according to documentation, fill up the OBS to IBS size ''after completing its read'', which means adding zeroes at the ''end'' of the block. This means, for a disk, that effectively the whole 1 Mib would become messed up because of a single 512 byte read error in the beginning of the read: 12ERROR89 would become 128900000 instead of 120000089.
+
The ''dd'' utility technically has an "input block size" (IBS) and an "output block size" (OBS). When you set {{ic|bs}}, you effectively set both IBS and OBS. Normally, if your block size is, say, 1 MiB, ''dd'' will read 1024*1024 bytes and write as many bytes. But if a read error occurs, things will go wrong. Many people seem to think that ''dd'' will "fill up read errors with zeroes" if you use the {{ic|noerror,sync}} options, but this is not what happens. ''dd'' will, according to documentation, fill up the OBS to IBS size ''after completing its read'', which means adding zeroes at the ''end'' of the block. This means, for a disk, that effectively the whole 1 MiB would become messed up because of a single 512 byte read error in the beginning of the read: 12ERROR89 would become 128900000 instead of 120000089.
  
If you are positive that your disk does not contain any errors, you could proceed using a larger block size, which will increase the speed of your copying several fold. For example, changing bs from 512 to 64 Ki changed copying speed from 35 MB/s to 120 MB/s on a simple Celeron 2.7 GHz system. But keep in mind that read errors on the source disk will end up as ''block errors'' on the destination disk, i.e. a single 512-byte read error will mess up the whole 64 Kib output block.
+
If you are positive that your disk does not contain any errors, you could proceed using a larger block size, which will increase the speed of your copying several fold. For example, changing bs from 512 to 64K changed copying speed from 35 MB/s to 120 MB/s on a simple Celeron 2.7 GHz system. But keep in mind that read errors on the source disk will end up as ''block errors'' on the destination disk, i.e. a single 512-byte read error will mess up the whole 64 KiB output block.
  
{{Tip|If you would like to view {{ic|dd}} progressing, you can send it a {{ic|USR1}} signal. {{ic|pidof dd}} will tell you the PID of {{ic|dd}}, then you can use {{ic|kill -USR1}} on that PID: {{ic|kill -USR1 $(pidof dd)}}.}}
+
{{Tip|If you would like to view ''dd'' progressing, use the {{ic|1=status=progress}} option. See [[dd]] for details.}}
  
 
{{Note|
 
{{Note|
* To regain unique UUIDs of an {{ic|ext2/3/4}} filesystem, use {{ic|tune2fs /dev/sd''XY'' -U random}} on every partition.
+
* To regain unique UUIDs of an ''ext2/3/4'' filesystem, use {{ic|tune2fs /dev/sd''XY'' -U random}} on every partition.
* Partition table changes from {{ic|dd}} are not registered by the kernel. To notify of changes without rebooting, use a utility like {{ic|partprobe}} (part of GNU {{ic|parted}}).
+
* Partition table changes from ''dd'' are not registered by the kernel. To notify of changes without rebooting, use a utility like ''partprobe'' (part of [[GNU Parted]]).
 
}}
 
}}
  
=== Backing up the MBR ===
+
=== Backing up the partition table ===
 
 
The MBR is stored in the the first 512 bytes of the disk. It consist of 3 parts:
 
 
 
# The first 446 bytes contain the boot loader.
 
# The next 64 bytes contain the partition table (4 entries of 16 bytes each, one entry for each primary partition).
 
# The last 2 bytes contain an identifier
 
 
 
To save the MBR as {{ic|mbr.img}}:
 
# dd if=/dev/sdX of=/path/to/mbr_file.img bs=512 count=1
 
 
 
To restore (be careful: this could destroy your existing partition table and with it access to all data on the disk):
 
# dd if=/path/to/mbr_file.img of=/dev/sdX
 
 
 
If you only want to restore the boot loader, but not the primary partition table entries, just restore the first 446 bytes of the MBR:
 
# dd if=/path/to/mbr_file.img of=/dev/sdX bs=446 count=1
 
  
To restore only the partition table, one must use:
+
See [[fdisk#Backup and restore partition table]].
# dd if=/path/to/mbr_file.img of=/dev/sdX bs=1 skip=446 count=64
 
 
 
You can also get the MBR from a full dd disk image:
 
# dd if=/path/to/disk.img of=/path/to/mbr_file.img bs=512 count=1
 
  
 
=== Create disk image ===
 
=== Create disk image ===
Line 78: Line 69:
 
  # dd if=/dev/sd''X'' conv=sync,noerror bs=64K | gzip -c  > ''/path/to/backup.img.gz''
 
  # dd if=/dev/sd''X'' conv=sync,noerror bs=64K | gzip -c  > ''/path/to/backup.img.gz''
  
If necessary (e.g. when the format of the external HD is {{ic|FAT32}}) split the disk image in volumes (see also the {{ic|split}} man pages).
+
If necessary (e.g. when the format of the external HD is FAT32) split the disk image in volumes (see also the ''split'' man pages).
  
 
  # dd if=/dev/sd''X'' conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G - ''/path/to/backup.img.gz''
 
  # dd if=/dev/sd''X'' conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G - ''/path/to/backup.img.gz''
 +
 +
If there is not enough disk space locally, you may send the image through ssh:
 +
 +
# dd if=/dev/sd''X'' conv=sync,noerror bs=64K | gzip -c | ssh user@local dd of=backup.img.gz
  
 
5. Save extra information about the drive geometry necessary in order to interpret the partition table stored within the image. The most important of which is the cylinder size.
 
5. Save extra information about the drive geometry necessary in order to interpret the partition table stored within the image. The most important of which is the cylinder size.
Line 86: Line 81:
 
  # fdisk -l /dev/sd''X'' > ''/path/to/list_fdisk.info''
 
  # fdisk -l /dev/sd''X'' > ''/path/to/list_fdisk.info''
  
{{Note|You may wish to use a block size ({{ic|1=bs=}}) that is equal to the amount of cache on the HD you are backing up. For example, {{ic|1=bs=8192K}} works for an 8 MiB cache. The 64 Kib mentioned in this article is better than the default {{ic|1=bs=512}} bytes, but it will run faster with a larger {{ic|1=bs=}}.}}
+
{{Note|You may wish to use a block size ({{ic|1=bs=}}) that is equal to the amount of cache on the HD you are backing up. For example, {{ic|1=bs=8192K}} works for an 8 MiB cache. The 64 KiB mentioned in this article is better than the default {{ic|1=bs=512}} bytes, but it will run faster with a larger {{ic|1=bs=}}.}}
  
 
=== Restore system ===
 
=== Restore system ===
Line 96: Line 91:
 
  # cat ''/path/to/backup.img.gz*'' | gunzip -c | dd of=/dev/sd''X''
 
  # cat ''/path/to/backup.img.gz*'' | gunzip -c | dd of=/dev/sd''X''
  
=== Examples with compression ===
+
== Using ddrescue ==
 +
''ddrescue'' is a tool designed for cloning and recovering data. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying to rescue the good parts first in case of read errors, to maximize the recovered data.
  
When you need to create the hard drive or a single partition compressed backup image file you must use compression tools which can do backup from {{ic|stdout}} and the {{ic|dd}} command. Those compressed files cannot be mounted by the {{ic|mount}} command but it is useful to know how to create and restore them.
+
To clone a faulty or dying drive, run ddrescue twice. First round, copy every block without read error and log the errors to rescue.log.
  
==== 7zip ====
+
# ddrescue -f -n /dev/sdX /dev/sdY rescue.log
  
Install the {{Pkg|p7zip}} package from the [[official repositories]].
+
Second round, copy only the bad blocks and try 3 times to read from the source before giving up.
This backup example will split the {{ic|dd}} command output in the files by up to 100 mebibyte each:
 
# dd if=/dev/sd''XY'' | 7zr a -v100m -t7z -si image-file.7z
 
  
Restore with 7zip:
+
  # ddrescue -d -f -r3 /dev/sdX /dev/sdY rescue.log
  # 7zr x -so image-file.7z | dd of=/dev/sd''XY''
 
{{Note|7zip can split only the ''7z'' compression type files}}
 
  
''Note'' : On certains systems (e.g. CentOS), {{Pkg|p7zip}} installs as {{ic|7za}} instead of {{ic|7zr}}.
+
Now you can check the file system for corruption and mount the new drive.
  
==== Zip ====
+
# fsck -f /dev/sdY
  
Install the {{Pkg|zip}} package from the [[official repositories]], which contains {{ic|zipsplit}} among other utilities for the management of zip archives. It will create a file named "-" inside the {{ic|image-file.zip}} file which will contain data from the {{ic|dd}} command output. To make a raw output of the file you can use the {{ic|-cp}} option with {{ic|unzip}} in {{ic|stdout}} for the {{ic|dd}} command.
+
== Using e2image ==
  
Backup:
+
''e2image'' is a tool included in {{Pkg|e2fsprogs}} for debugging purposes. It can be used to copy ext2, ext3, and ext4 partitions efficiently by only copying the used blocks. Note that this only works for ext2, ext3, and ext4 filesystems, and the unused blocks are not copied so this may not be a useful tool if one is hoping to recover deleted files.
# dd if=/dev/sd''XY'' | zip --compression-method bzip2 ''image-file.zip'' -
 
  
Restore:
+
To clone a partition from physical disk {{ic|/dev/sda}}, partition 1, to physical disk {{ic|/dev/sdb}}, partition 1 with e2image, run
# unzip -cp image-file.zip | dd of=/dev/sd''XY''
 
  
The {{ic|zip}} tool cannot split files on the fly but you can use the {{ic|zipsplit}} utility on an already-created file.
+
  # e2image -ra -p /dev/sda1 /dev/sdb1
  
See also {{ic|man zip}} for more information.
+
{{Note|gparted uses e2image to efficiently copy ext2/3/4 partitions.}}
  
==== Rar ====
+
== Disk cloning software ==
  
Install the {{Aur|rar}} and {{Aur|unrar}} packages from the [[AUR]].
+
These applications allow easy backup of entire filesystems and recovery in case of failure, usually in the form of a Live CD or USB drive. They contain complete system images from one or more specific points in time and are frequently used to record known good configurations.
  
This backs up on the fly:
+
See also [[Synchronization and backup programs]] for other applications that can take full system snapshots, among other functionality.
  
# dd if=/dev/sd''XY'' | rar a -sisd''XY'' ''image-file.rar''
+
* {{App|[[wikipedia:Acronis_True_Image|Acronis True Image]]|Commercial disk cloner for Windows. It allows you to create a live (from within Windows), so you do not need a working Windows install on the actual machine to use it. After registration of the Acronis software on their website, you will be able to download a Linux-based Live CD and/or plugins for BartPE for creation of the Windows-based live CD. It can also create a WinPE Live CD based on Windows. The created ISO Live CD image by Acronis doesn't have the [http://www.syslinux.org/wiki/index.php/Isohybrid hybrid boot] ability and cannot be written to USB storage as a raw file.|http://www.acronis.com/products/trueimage/|}}
  
The ''image-file.rar'' will contain, if opened using {{ic|file-roller}}, for instance, a single file named {{ic|sd''XY''}}
+
* {{App|Arch Backup|A trivial backup script with simple configuration.
<!--
+
** Configurable compression method.
Splitting the archive using -v (see /usr/share/doc/rar/rar.txt):
+
** Multiple backup targets.
-v1000 creates parts of 1000*1000 bytes
+
|https://github.com/p5n/archlinux-stuff/tree/master/arch-backup/|{{AUR|arch-backup}}}}
-v100k creates parts of 100*1024 bytes
 
-v1000b creates parts of 1000 bytes
 
-v10m creates parts of 10*1024*1024 bytes (undocumented)
 
-v10M creates parts of 10*1000*1000 bytes (undocumented)
 
-v10g creates parts of 10*1024*1024*1024 bytes (undocumented)
 
-v10G creates parts of 10*1000*1000*1000 bytes (undocumented)
 
-v<n>f creates parts of <n> bytes unless n<1000, in which case they are 1000 bytes,
 
      except if <n> is one of 360, 720, 1200, 1440 or 2880; the latter are meant
 
      to be standard floppy disk sizes (respectively 362 496, 730 112, 1 213 952,
 
      1 457 664, and 2 915 328 bytes)
 
Yes, the man pages are wrong and/or out of date. Welcome to Linux purgatory.
 
  
Unfortunately, /usr/share/doc/rar/rar.txt states -v cannot be used with -si
+
* {{App|[[Wikipedia:Clonezilla|Clonezilla]]|A disaster recovery, disk cloning, disk imaging and deployment solution.
-->
+
** Boots from live CD, USB flash drive, or PXE server.
This restores:
+
** Supports ext2, ext3, ext4, reiserfs, reiser4, xfs, jfs, btrfs FAT32, NTFS, HFS+ and others.
 +
** Uses Partclone (default), Partimage (optional), ntfsclone (optional), or dd to image or clone a partition.
 +
** Multicasting server to restore to many machines at once.
 +
** Included on the Arch Linux installation media.
 +
|http://clonezilla.org/|{{Pkg|clonezilla}}}}
  
# unrar p -ierr ''image-file.rar'' | dd of=/dev/sd''XY''
+
* {{App|FSArchiver|A safe and flexible file-system backup and deployment tool
 +
** Support for basic file attributes (permissions, owner, ...).
 +
** Support for multiple file-systems per archive.
 +
** Support for extended attributes (they are used by SELinux).
 +
** Support the basic file-system attributes (label, uuid, block-size) for all linux file-systems.
 +
** Support for [http://www.fsarchiver.org/Cloning-ntfs ntfs filesystems] (ability to create flexible clones of a Windows partitions).
 +
** Checksumming of everything which is written in the archive (headers, data blocks, whole files).
 +
** Ability to restore an archive which is corrupt (it will just skip the current file).
 +
** Multi-threaded lzo, gzip, bzip2, lzma compression.
 +
** Support for splitting large archives into several files with a fixed maximum size.
 +
** Encryption of the archive using a password. Based on blowfish from libcrypto from [[OpenSSL]].
 +
** Support backup of a mounted root filesystem (-A option).
 +
** Can be found on the [http://www.sysresccd.org/Main_Page System Rescue CD].
 +
|http://www.fsarchiver.org/|{{Pkg|fsarchiver}}}}
  
The {{ic|-ierr}} prevents the {{ic|unrar}} {{ic|stdout}} from being prefixed and suffixed with {{ic|unrar}}'s comments. You can also use {{ic|-inul}}, which omits the comments entirely.
+
* {{App|[[Wikipedia:Mondo Rescue|Mondo Rescue]]|A disaster recovery solution to create backup media that can be used to redeploy the damaged system.
 +
** Image-based backups, supporting Linux/Windows.
 +
** Compression rate is adjustable.
 +
** Can backup live systems (without having to halt it).
 +
** Can split image over many files.
 +
** Supports booting to a Live CD to perform a full restore.
 +
** Can backup/restore over NFS, from CDs, tape drives and and other media.
 +
** Can verify backups.
 +
|http://www.mondorescue.org/|{{AUR|mondo}}}}
  
You could also use {{ic|rar}} instead of the {{Pkg|unrar}} utility. The {{ic|unrar}} utility is available in the official repositories and can be installed with {{ic|pacman -S unrar}}.
+
* {{App|[[Partclone]]|A tool that can be used to back up and restore a partition while considering only used blocks.
 +
** Supports ''ext2'', ''ext3'', ''ext4'', ''hfs+'', ''reiserfs'', ''reiser4'', ''btrfs'', ''vmfs3'', ''vmfs5'', ''xfs'', ''jfs'', ''ufs'', ''ntfs'', ''fat(12/16/32)'', ''exfat''.
 +
** Supports compression.
 +
** Optionally, an ''ncurses'' interface can be used.
 +
|http://partclone.org/|{{Pkg|partclone}}}}
  
==== Bzip2 ====
+
* {{App|PartedMagic|Live cd/usb with PartImage and other recovery tools.|3=http://partedmagic.com/doku.php?id=start|4=}}
  
Creation by using {{ic|dd}} is safer:
+
* {{App|[[Wikipedia:Partimage|Partimage]]|An ''ncurses'' disk cloning utility for Linux/UNIX environments.
 +
** Has a Live CD.
 +
** Supports the most popular filesystems on Linux, Windows and Mac OS.
 +
** Compression.
 +
** Saving to multiple CDs or DVDs or across a network using Samba/NFS.
 +
** Development stopped in favor of FSArchiver.
 +
|http://www.partimage.org/Main_Page|{{Pkg|partimage}}}}
  
{{hc|1=# dd if=/dev/sd''XY'' <nowiki>|</nowiki> bzip2 -f5 > ''compressedfile.bzip2''|2=
+
* {{App|Q7Z|P7Zip GUI for Linux, which attempts to simplify data compression and backup.  It can create the following archive types: 7z, BZip2, Zip, GZip, Tar.
937016+0 records in
+
** Updates existing archives quickly.
937016+0 records out
+
** Backup multiple folders to a storage location.
479752192 bytes (480 MB) copied, 94.7002 s, 5.1 MB/s
+
** Create or extract protected archives.
}}
+
** Lessen effort by using archiving profiles and lists.
 +
|http://k7z.sourceforge.net/|{{AUR|q7z}}}}
  
And a safe way of restoring with combination of {{ic|dd}}:
+
* {{App|[[Wikipedia:Redo Backup and Recovery|Redo Backup and Recovery]]|A backup and disaster recovery application that runs from a bootable Linux CD image.
 
+
** Is capable of bare-metal backup and recovery of disk partitions.
# bunzip2 -dc ''compressedfile.bzip2'' | dd of=/dev/sd''XY''
+
** Uses [http://www.xpud.org/ xPUD] and [[Partclone]] for the backend.
 
+
|http://www.redobackup.org/|}}
or
 
 
 
# bzcat ''compressedfile.bzip2'' | dd of=/dev/sd''XY''
 
 
 
{{Warning|Never ever use the {{ic|bzip2  -kdc ''imgage.bzip2'' > /dev/sd''XY''}} and {{ic|bzip2  -kc /dev/sd''XY'' > ''imgage.bzip2''}} methods for serious backup of partitions and disks. The errors might be due to the end of the device or partition and the restore process also gives errors due the truncated end.
 
}}
 
 
 
== Using cp ==
 
 
 
The {{ic|cp}} program can be used to clone a disk, one partition at a time. An advantage to using {{ic|cp}} is that the filesystem type of the destination partition(s) may be the same as or different from the source. For safety, perform the process from a live environment.
 
 
 
{{Note|This method should not be considered in the same category as disk cloning on the level at which {{ic|dd}} operates. Also, it has been reported that even with the {{ic|-a}} switch, some extended attributes may not be copied. For better results, {{ic|rsync}} or {{ic|tar}} should be used.}}
 
 
 
The basic procedure from a live environment will be:
 
* Create the new destination partition(s) using {{ic|fdisk}}, {{ic|cfdisk}} or other tools available in the live environment.
 
* Create a filesystem on each of the newly created partitions. Example:
 
# mkfs -t ext3 /dev/sd''XY''
 
* Mount the source and destination partitions. Example:
 
# mount -t ext3 /dev/sd''XY'' ''/mnt/source''
 
# mount -t ext3 /dev/sd''ZY'' ''/mnt/destination''
 
* Copy the files from the source partition to the destination:
 
# cp -a ''/path/to/source/*'' ''/path/to/destination''
 
{{ic|-a}}: preserve all attributes, never follow symbolic links and copy recursively
 
* Change the mount points of the newly cloned partitions in {{ic|/etc/fstab}} accordingly
 
* Finally, install the GRUB bootloader if necessary. (See [[GRUB]])
 
 
 
== Disk cloning software ==
 
  
=== Disk cloning in Arch ===
+
* {{App|System Tar & Restore|Backup and Restore your system using tar or Transfer it with rsync
 +
** GUI and CLI interfaces
 +
** Creates ''.tar.gz'', ''.tar.bz2'', ''.tar.xz'' or ''.tar'' archives
 +
** Supports openssl / gpg encryption
 +
** Uses rsync to transfer a running system
 +
** Supports Grub2, Syslinux, EFISTUB/efibootmgr and Systemd/bootctl
 +
|https://github.com/tritonas00/system-tar-and-restore|{{AUR|system-tar-and-restore}}}}
  
* {{ic|[[Partclone]]}} provides utilities to save and restore used blocks on a partition and supports {{ic|ext2}}, {{ic|ext3}}, {{ic|ext4}}, {{ic|hfs+}}, {{ic|reiserfs}}, {{ic|reiser4}}, {{ic|btrfs}}, {{ic|vmfs3}}, {{ic|vmfs5}}, {{ic|xfs}}, {{ic|jfs}}, {{ic|ufs}}, {{ic|ntfs}}, {{ic|fat(12/16/32)}} and {{ic|exfat}}. Optionally, an {{ic|ncurses}} interface can be used. Partclone is available in the community repository.
+
=== dd spin-offs ===
* [http://www.partimage.org/ Partimage], an {{ic|ncurses}} program, is available in the community repos. Partimage does not currently support {{ic|ext4}} or {{ic|btrfs}} filesystems. {{ic|NTFS}} is experimental.
 
  
=== Disk cloning outside of Arch ===
+
Other ''dd''-like programs feature periodical status output, e.g. a simple progress bar.
  
If you wish to backup or propagate your Arch install root, you are probably better off booting into something else and clone the partition from there. Some suggestions:
+
; dcfldd : {{Pkg|dcfldd}} is an enhanced version of dd with features useful for forensics and security. It accepts most of dd's parameters and includes status output. The last stable version of dcfldd was released on December 19, 2006.<sup>[http://dcfldd.sourceforge.net/]</sup>
  
* [http://partedmagic.com/doku.php?id=start PartedMagic] has a very nice live cd/usb with PartImage and other recovery tools.
+
; ddrescue : GNU {{Pkg|ddrescue}} is a data recovery tool. It is capable of ignoring read errors, which is a useless feature for disk wiping in almost any case. See the [http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html official manual] for details.
* [http://www.mondorescue.org/ Mindi] is a linux distribution specifically for disk clone backup. It comes with its own cloning program, Mondo Rescue.
 
* [[wikipedia:Acronis_True_Image|Acronis True Image]] is a commercial disk cloner for Windows. It allows you to create a live (from within Windows), so you do not need a working Windows install on the actual machine to use it. After registration of the Acronis software on their website, you will be able to download a Linux-based Live CD and/or plugins for BartPE for creation of the Windows-based live CD. It can also create a WinPE Live CD based on Windows. The created ISO Live CD image by Acronis doesn't have the [http://www.syslinux.org/wiki/index.php/Isohybrid hybrid boot] ability and cannot be written to USB storage as a raw file.
 
* [http://www.fsarchiver.org/Main_Page FSArchiver] allows you to save the contents of a file system to a compressed archive file. Can be found on the [http://www.sysresccd.org/Main_Page System Rescue CD].
 
* [http://clonezilla.org/ Clonezilla] is an enhanced partition imager which can also restore entire disks as well as partitions. Clonezilla is included on the Arch Linux installation media.
 
* [http://redobackup.org/ Redo Backup and Recovery] is a Live CD featuring a graphical front-end to {{ic|partclone}}.
 
  
== External Links ==
+
== See also ==
  
 
* [[Wikipedia:List of disk cloning software]]
 
* [[Wikipedia:List of disk cloning software]]
 
* [https://bbs.archlinux.org/viewtopic.php?id=4329 Arch Linux forum thread]
 
* [https://bbs.archlinux.org/viewtopic.php?id=4329 Arch Linux forum thread]

Latest revision as of 19:12, 28 September 2017

Disk cloning is the process of making an image of a partition or of an entire hard drive. This can be useful for copying the drive to other computers and for backup and recovery purposes.

Using dd

The dd command is a simple, yet versatile and powerful tool. It can be used to copy from source to destination, block-by-block, regardless of their filesystem types or operating systems. A convenient method is to use dd from a live environment, as in a Live CD.

Warning: As with any command of this type, you should be very cautious when using it; it can destroy data. Remember the order of input file (if=) and output file (of=) and do not reverse them! Always ensure that the destination drive or partition (of=) is of equal or greater size than the source (if=).

Tango-inaccurate.pngThe factual accuracy of this article or section is disputed.Tango-inaccurate.png

Reason: Are the conv= options safe to use? See: (Discuss in Talk:Disk cloning#remove conv sync noerror)

Cloning a partition

From physical disk /dev/sda, partition 1, to physical disk /dev/sdb, partition 1.

# dd if=/dev/sda1 of=/dev/sdb1 bs=64K conv=noerror,sync status=progress
Warning: If output file of= (sdb1 in the example) does not exist, dd will create a file with this name and will start filling up your root file system!

Cloning an entire hard disk

From physical disk /dev/sdX to physical disk /dev/sdY

# dd if=/dev/sdX of=/dev/sdY bs=64K conv=noerror,sync status=progress

This will clone the entire drive, including the MBR (and therefore bootloader), all partitions, UUIDs, and data.

  • bs= sets the block size. Defaults to 512 bytes, which is the "classic" block size for hard drives since the early 1980s, but is not the most convenient. Use a bigger value, 64K or 128K. Also, please read the warning below, because there is more to this than just "block sizes" -it also influences how read errors propagate. See [1] and [2] for details and to figure out the best bs value for your use case.
  • noerror instructs dd to continue operation, ignoring all read errors. Default behavior for dd is to halt at any error.
  • sync fills input blocks with zeroes if there were any read errors, so data offsets stay in sync.
  • status=progress shows periodic transfer statistics which can be used to estimate when the operation may be complete.
Warning: The block size you specify influences how read errors are handled. Read below. For data recovery, use ddrescue.

The dd utility technically has an "input block size" (IBS) and an "output block size" (OBS). When you set bs, you effectively set both IBS and OBS. Normally, if your block size is, say, 1 MiB, dd will read 1024*1024 bytes and write as many bytes. But if a read error occurs, things will go wrong. Many people seem to think that dd will "fill up read errors with zeroes" if you use the noerror,sync options, but this is not what happens. dd will, according to documentation, fill up the OBS to IBS size after completing its read, which means adding zeroes at the end of the block. This means, for a disk, that effectively the whole 1 MiB would become messed up because of a single 512 byte read error in the beginning of the read: 12ERROR89 would become 128900000 instead of 120000089.

If you are positive that your disk does not contain any errors, you could proceed using a larger block size, which will increase the speed of your copying several fold. For example, changing bs from 512 to 64K changed copying speed from 35 MB/s to 120 MB/s on a simple Celeron 2.7 GHz system. But keep in mind that read errors on the source disk will end up as block errors on the destination disk, i.e. a single 512-byte read error will mess up the whole 64 KiB output block.

Tip: If you would like to view dd progressing, use the status=progress option. See dd for details.
Note:
  • To regain unique UUIDs of an ext2/3/4 filesystem, use tune2fs /dev/sdXY -U random on every partition.
  • Partition table changes from dd are not registered by the kernel. To notify of changes without rebooting, use a utility like partprobe (part of GNU Parted).

Backing up the partition table

See fdisk#Backup and restore partition table.

Create disk image

1. Boot from a live media.

2. Make sure no partitions are mounted from the source hard drive.

3. Mount the external HD

4. Backup the drive.

# dd if=/dev/sdX conv=sync,noerror bs=64K | gzip -c  > /path/to/backup.img.gz

If necessary (e.g. when the format of the external HD is FAT32) split the disk image in volumes (see also the split man pages).

# dd if=/dev/sdX conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G - /path/to/backup.img.gz

If there is not enough disk space locally, you may send the image through ssh:

# dd if=/dev/sdX conv=sync,noerror bs=64K | gzip -c | ssh user@local dd of=backup.img.gz

5. Save extra information about the drive geometry necessary in order to interpret the partition table stored within the image. The most important of which is the cylinder size.

# fdisk -l /dev/sdX > /path/to/list_fdisk.info
Note: You may wish to use a block size (bs=) that is equal to the amount of cache on the HD you are backing up. For example, bs=8192K works for an 8 MiB cache. The 64 KiB mentioned in this article is better than the default bs=512 bytes, but it will run faster with a larger bs=.

Restore system

To restore your system:

# gunzip -c /path/to/backup.img.gz | dd of=/dev/sdX

When the image has been split, use the following instead:

# cat /path/to/backup.img.gz* | gunzip -c | dd of=/dev/sdX

Using ddrescue

ddrescue is a tool designed for cloning and recovering data. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying to rescue the good parts first in case of read errors, to maximize the recovered data.

To clone a faulty or dying drive, run ddrescue twice. First round, copy every block without read error and log the errors to rescue.log.

# ddrescue -f -n /dev/sdX /dev/sdY rescue.log

Second round, copy only the bad blocks and try 3 times to read from the source before giving up.

# ddrescue -d -f -r3 /dev/sdX /dev/sdY rescue.log

Now you can check the file system for corruption and mount the new drive.

# fsck -f /dev/sdY

Using e2image

e2image is a tool included in e2fsprogs for debugging purposes. It can be used to copy ext2, ext3, and ext4 partitions efficiently by only copying the used blocks. Note that this only works for ext2, ext3, and ext4 filesystems, and the unused blocks are not copied so this may not be a useful tool if one is hoping to recover deleted files.

To clone a partition from physical disk /dev/sda, partition 1, to physical disk /dev/sdb, partition 1 with e2image, run

 # e2image -ra -p /dev/sda1 /dev/sdb1
Note: gparted uses e2image to efficiently copy ext2/3/4 partitions.

Disk cloning software

These applications allow easy backup of entire filesystems and recovery in case of failure, usually in the form of a Live CD or USB drive. They contain complete system images from one or more specific points in time and are frequently used to record known good configurations.

See also Synchronization and backup programs for other applications that can take full system snapshots, among other functionality.

  • Acronis True Image — Commercial disk cloner for Windows. It allows you to create a live (from within Windows), so you do not need a working Windows install on the actual machine to use it. After registration of the Acronis software on their website, you will be able to download a Linux-based Live CD and/or plugins for BartPE for creation of the Windows-based live CD. It can also create a WinPE Live CD based on Windows. The created ISO Live CD image by Acronis doesn't have the hybrid boot ability and cannot be written to USB storage as a raw file.
http://www.acronis.com/products/trueimage/ ||
  • Arch Backup — A trivial backup script with simple configuration.
    • Configurable compression method.
    • Multiple backup targets.
https://github.com/p5n/archlinux-stuff/tree/master/arch-backup/ || arch-backupAUR
  • Clonezilla — A disaster recovery, disk cloning, disk imaging and deployment solution.
    • Boots from live CD, USB flash drive, or PXE server.
    • Supports ext2, ext3, ext4, reiserfs, reiser4, xfs, jfs, btrfs FAT32, NTFS, HFS+ and others.
    • Uses Partclone (default), Partimage (optional), ntfsclone (optional), or dd to image or clone a partition.
    • Multicasting server to restore to many machines at once.
    • Included on the Arch Linux installation media.
http://clonezilla.org/ || clonezilla
  • FSArchiver — A safe and flexible file-system backup and deployment tool
    • Support for basic file attributes (permissions, owner, ...).
    • Support for multiple file-systems per archive.
    • Support for extended attributes (they are used by SELinux).
    • Support the basic file-system attributes (label, uuid, block-size) for all linux file-systems.
    • Support for ntfs filesystems (ability to create flexible clones of a Windows partitions).
    • Checksumming of everything which is written in the archive (headers, data blocks, whole files).
    • Ability to restore an archive which is corrupt (it will just skip the current file).
    • Multi-threaded lzo, gzip, bzip2, lzma compression.
    • Support for splitting large archives into several files with a fixed maximum size.
    • Encryption of the archive using a password. Based on blowfish from libcrypto from OpenSSL.
    • Support backup of a mounted root filesystem (-A option).
    • Can be found on the System Rescue CD.
http://www.fsarchiver.org/ || fsarchiver
  • Mondo Rescue — A disaster recovery solution to create backup media that can be used to redeploy the damaged system.
    • Image-based backups, supporting Linux/Windows.
    • Compression rate is adjustable.
    • Can backup live systems (without having to halt it).
    • Can split image over many files.
    • Supports booting to a Live CD to perform a full restore.
    • Can backup/restore over NFS, from CDs, tape drives and and other media.
    • Can verify backups.
http://www.mondorescue.org/ || mondoAUR
  • Partclone — A tool that can be used to back up and restore a partition while considering only used blocks.
    • Supports ext2, ext3, ext4, hfs+, reiserfs, reiser4, btrfs, vmfs3, vmfs5, xfs, jfs, ufs, ntfs, fat(12/16/32), exfat.
    • Supports compression.
    • Optionally, an ncurses interface can be used.
http://partclone.org/ || partclone
  • PartedMagic — Live cd/usb with PartImage and other recovery tools.
http://partedmagic.com/doku.php?id=start ||
  • Partimage — An ncurses disk cloning utility for Linux/UNIX environments.
    • Has a Live CD.
    • Supports the most popular filesystems on Linux, Windows and Mac OS.
    • Compression.
    • Saving to multiple CDs or DVDs or across a network using Samba/NFS.
    • Development stopped in favor of FSArchiver.
http://www.partimage.org/Main_Page || partimage
  • Q7Z — P7Zip GUI for Linux, which attempts to simplify data compression and backup. It can create the following archive types: 7z, BZip2, Zip, GZip, Tar.
    • Updates existing archives quickly.
    • Backup multiple folders to a storage location.
    • Create or extract protected archives.
    • Lessen effort by using archiving profiles and lists.
http://k7z.sourceforge.net/ || q7zAUR
  • Redo Backup and Recovery — A backup and disaster recovery application that runs from a bootable Linux CD image.
    • Is capable of bare-metal backup and recovery of disk partitions.
    • Uses xPUD and Partclone for the backend.
http://www.redobackup.org/ ||
  • System Tar & Restore — Backup and Restore your system using tar or Transfer it with rsync
    • GUI and CLI interfaces
    • Creates .tar.gz, .tar.bz2, .tar.xz or .tar archives
    • Supports openssl / gpg encryption
    • Uses rsync to transfer a running system
    • Supports Grub2, Syslinux, EFISTUB/efibootmgr and Systemd/bootctl
https://github.com/tritonas00/system-tar-and-restore || system-tar-and-restoreAUR

dd spin-offs

Other dd-like programs feature periodical status output, e.g. a simple progress bar.

dcfldd 
dcfldd is an enhanced version of dd with features useful for forensics and security. It accepts most of dd's parameters and includes status output. The last stable version of dcfldd was released on December 19, 2006.[3]
ddrescue 
GNU ddrescue is a data recovery tool. It is capable of ignoring read errors, which is a useless feature for disk wiping in almost any case. See the official manual for details.

See also