Difference between revisions of "Securely wipe disk"

From ArchWiki
Jump to: navigation, search
m (Select a target: gdisk not a package name)
(Select a block size: the physical block size seems to be routinely understated by all but newest AF drives)
 
(231 intermediate revisions by 26 users not shown)
Line 1: Line 1:
 
[[Category:Security]]
 
[[Category:Security]]
 
[[Category:File systems]]
 
[[Category:File systems]]
{{Article summary start}}
+
[[ja:ディスクの完全消去]]
{{Article summary text|Wipe all traces left from (un-)encrypted data and/or prepare for block device encryption}}
+
{{Related articles start}}
{{Article summary heading|Related}}
+
{{Related|Securely wipe disk/Tips and tricks}}
{{Article summary wiki|File Recovery}}
+
{{Related|File recovery}}
{{Article summary wiki|Benchmarking disk wipes}}
+
{{Related|Benchmarking/Data storage devices}}
{{Article summary wiki|Frandom}}
+
{{Related|Frandom}}
{{Article summary wiki|Disk Encryption#Preparing the disk}}
+
{{Related|Disk encryption#Preparing the disk}}
{{Article summary wiki|dm-crypt with LUKS}}
+
{{Related|dm-crypt}}
{{Article summary end}}
+
{{Related articles end}}
 
+
 
Wiping a disk is done by writing new data over every single bit.
 
Wiping a disk is done by writing new data over every single bit.
  
Line 16: Line 15:
  
 
== Common use cases ==
 
== Common use cases ==
 +
 
=== Wipe all data left on the device ===
 
=== Wipe all data left on the device ===
The most common usecase for completely and irrevocably wiping a device will be when the device it going to be given away or also maybe sold. There may be (unencrypted) data left on the device and you want to protect against simple forensic investigation that is mere child's play with for example [[File Recovery]] software.
 
  
If you want to quickly wipe everything from the disk /dev/zero or simple patterns allow maximum performance while adequate randomness can be advantageous in some cases that should be covered up in [[#Data remanence]].
+
The most common usecase for completely and irrevocably wiping a device will be when the device is going to be given away or sold. There may be (unencrypted) data left on the device and you want to protect against simple forensic investigation that is mere child's play with for example [[File recovery]] software.
 +
 
 +
If you want to quickly wipe everything from the disk, {{ic|/dev/zero}} or simple patterns allow maximum performance while adequate randomness can be advantageous in some cases that should be covered up in [[#Data remanence]].
  
 
Every overwritten bit means to provide a level of data erasure not allowing recovery with normal system functions (like standard ATA/SCSI commands) and hardware interfaces. Any file recovery software mentioned above then would need to be specialized on proprietary storage-hardware features.
 
Every overwritten bit means to provide a level of data erasure not allowing recovery with normal system functions (like standard ATA/SCSI commands) and hardware interfaces. Any file recovery software mentioned above then would need to be specialized on proprietary storage-hardware features.
Line 25: Line 26:
 
In case of a HDD data recreation will not be possible without at least undocumented drive commands or fiddling about the device’s controller or firmware to make them read out for example reallocated sectors (bad blocks that [[S.M.A.R.T.]] retired from use).
 
In case of a HDD data recreation will not be possible without at least undocumented drive commands or fiddling about the device’s controller or firmware to make them read out for example reallocated sectors (bad blocks that [[S.M.A.R.T.]] retired from use).
  
There are different wiping issues with different physical storage technologys, most notably all Flash memory based devices and older magnetic storage (old HDD's, floppy disks, tape).
+
There are different wiping issues with different physical storage technologies, most notably all Flash memory based devices and older magnetic storage (old HDD's, floppy disks, tape).
  
 
=== Preparations for block device encryption ===
 
=== Preparations for block device encryption ===
If you want to prepare your drive to securely set up [[Disk Encryption#Block device encryption]] inside the wiped area afterwards you really should use [[#Random data]] generated by a trusted cryptographically strong random number generator (referred to as RNG in this article from now on).
 
  
{{Wikipedia|Random number generation}}
+
If you want to prepare your drive to securely set up [[Disk encryption#Block device encryption]] inside the wiped area afterwards you really should use [[#Random data]] generated by a trusted cryptographically strong random number generator (referred to as RNG in this article from now on).
  
{{Warning|If Block device encryption is mapped on a partition that contains anything else than random/encrypted data, disclosure of usage patterns on the encrypted drive is possible and weakens the encryption being comparable with filesystem-level-encryption. Do never use /dev/zero, simple patterns (badblocks, eg.) or other unrandom data before setting up Block device encryption if you are serious about it!}}
+
See also [[Wikipedia:Random number generation]].
  
== Select a data source for overwriting ==
+
{{Warning|If Block device encryption is mapped on a partition that contains anything else than random/encrypted data, disclosure of usage patterns on the encrypted drive is possible and weakens the encryption being comparable with filesystem-level-encryption. Never use {{ic|/dev/zero}}, simple patterns (badblocks, eg.) or other unrandom data before setting up Block device encryption if you are serious about it!}}
  
As just said If you want to wipe sensitive data you can use anything matching your needs.
+
== Data remanence ==
  
If you want to setup block device encryption afterwards you should always wipe at least with Pseudorandom data.
+
See also [[Wikipedia:Data remanence]].
  
For data that is not truly random your disk's writing speed should be the only limiting factor. If you need random data, the required system performance to generate it may extremely depend on what you choose as source of entropy.
+
The residual representation of data may remain even after attempts have been made to remove or erase the data.
  
{{Note|Everything regarding [[Benchmarking disk wipes]] should get merged there.}}
+
Residual data may get wiped by writing (random) data to the disk with a single or even more than one iteration. However, more than one iteration may not significantly decrease the possibility to reconstruct the data of hard disk drives. See [[#Residual magnetism]].  
  
=== Unrandom data ===
+
=== Operating system, programs and filesystem ===
Overwriting with {{ic|/dev/zero}} or simple patterns is considered secure in most resources. In the case of current HDD's it should be sufficient for fast disk wipes.
+
  
{{Warning|A drive that is abnormally fast in writing patterns or zeroing could be doing transparent compression. It is obviously presumable not all blocks get wiped this way. Some [[#Flash memory]] devices do "feature" that.}}
+
The operating system, executed programs or [[Wikipedia:Journaling file system|journaling file system]]s may copy your unencrypted data throughout the block device. When writing to plain disks this should only be relevant in conjunction with one of the above.
  
==== Pattern write test ====
+
If the data can get exactly located on the disk and was never copied anywhere else, wiping with random data can be thoroughgoing and impressively quick as long there is enough entropy in the pool.
[[#Badblocks]] can write simple patterns to every block of a device and then read and check them searching for damaged areas (just like memtest86* does with memory).
+
  
As the pattern is written to every accesible block this effectively wipes the device.
+
A good example is cryptsetup using {{ic|/dev/urandom}} for [[Dm-crypt/Device encryption#Key management|wiping the LUKS keyslots]].
  
=== Random data ===
+
=== Hardware-specific issues ===
{{Note|Data that is hard to compress (random data) will get written slower, if the drive logic mentioned in the [[#Unrandom data]] warning tries compressing it. This should not lead to [[#Data remanence]] though. As maximum write-speed is not the performance-bottleneck it can get completely neglected while wiping disks with random data.}}
+
  
==== Kernel built-in RNG ====
+
==== Flash memory ====
The kernel build in RNG's /dev/(u)[[random]] are highly recommended for producing reliable random data providing the same security level that is used for the creation of cryptographic keys.  The random number generator gathers environmental noise from device drivers and other sources into an entropy pool.
+
  
; /dev/random: uses an entropy pool of 4096 bits (512 Bytes) to generate random data and stops when the pool is exhausted until it get's (slowly) refilled. /dev/random is absolutely not designed for wiping entire HDD's, but rather to generate cryptographic keys (e.g. SSL/SSH).  
+
[[Wikipedia:Write amplification|Write amplification]] and other characteristics make Flash memory (explicitly including SSDs) a stubborn target for reliable wiping.
; /dev/urandom: reuses existing entropy pool data while the pool is replenished and although not suited for the most crucial cryptographic purposes, for example the generation of longterm keys, its quality should be sufficient for a paranoid disk wipe, [[#Preparations for block device encryption|preparing for block device encryption]], wiping LUKS keyslots, wiping single files and many other purposes.
+
As there is a lot of transparent abstraction in between data as seen by a device's controller chip and the operating system sight data is never overwritten in place and wiping particular blocks or files is not reliable.
  
For much better performance consider using a true [[random#Pseudorandom_number_generator|pseudorandom number generator]].
+
Other "features" like transparent compression (all SandForce SSD's) can compress your /dev/zero or pattern stream so if wiping is fast beyond belief this might be the case.
  
== Select a program ==
+
Disassembling Flash memory devices, unsoldering the chips and analyzing data content without the controller in between is feasible without difficulty using [http://www.flash-extractor.com/manual/reader_models/ simple hardware]. Data recovery companies do it for cheap money.
{{ic|/dev/<drive>}} is the drive to be encrypted.
+
  
=== Coreutils ===
+
For more information see:
{{Merge|Core_Utilities|Basic file operations are '''not specific to disk wiping!''' Unrelated stuff in this section should get merged and then deleted and replaced with a link to [[Core Utilities]]. Did you ever want to write an article about dd and Co? Then just go ahead.}}
+
  
Official documentation for dd and shred is linked to under [[#See also]].
+
* [[SSD memory cell clearing]]
 +
* [http://www.usenix.org/events/fast11/tech/full_papers/Wei.pdf Reliably Erasing Data From Flash-Based Solid State Drives].
 +
* [[#Select a target]]
 +
==== Marked Bad Sectors ====
  
==== Dd ====
+
If a hard drive marks a sector as bad, it cordons it off, and the section becomes impossible to write to via software. Thus a full overwrite would not reach it. However because of block sizes, these sections would only amount to a few theoretically recoverable KB.
{{Wikipedia|Dd_(Unix)}}
+
{{Note|cp does the same as dd without any operands but is not designed for more versatile disk wiping procedures.}}
+
  
===== Checking progress of dd while running =====
+
==== Residual magnetism ====
By default, there is no output of dd until the task has finished.  With kill and the "USR1"-Signal you can force status output without actually killing the program. Open up a 2nd root terminal and issue the following command:
+
# killall -USR1 dd
+
{{Note|This will affect all other running dd-processes as well.}}
+
Or:
+
# kill -USR1 <PID_OF_dd_COMMAND>
+
 
+
For example:
+
# kill -USR1 $(pidof dd)
+
 
+
This causes the terminal in which dd is running to output the progress at the time the command was run. For example:
+
605+0 records in
+
605+0 records out
+
634388480 bytes (634 MB) copied, 8.17097 s, 77.6 MB/s
+
  
===== Dd spin-offs =====
+
A single, full overwrite with zeros or random data does not lead to any recoverable data on a modern high-density storage device.[http://www.howtogeek.com/115573/htg-explains-why-you-only-have-to-wipe-a-disk-once-to-erase-it/] Indications otherwise refer to single residual bits; reconstruction of byte patterns is generally not feasible.[https://web.archive.org/web/20120102004746/http://www.h-online.com/newsticker/news/item/Secure-deletion-a-single-overwrite-will-do-it-739699.html] See also  [https://www.google.com/search?tbs=bks:1&q=isbn:9783540898610], [http://security.stackexchange.com/questions/26132/is-data-remanence-a-myth/26134#26134] and [http://www.nber.org/sys-admin/overwritten-data-guttman.html].
Other dd alike programs feature periodical status output like i.e. a simple progress bar.
+
  
'''dcfldd'''
+
== Select a target ==
  
{{Pkg|dcfldd}} is an enhanced version of dd with features useful for forensics and security. It accepts most of dd's parameters and includes status output. The last stable version of dcfldd was released on December 19, 2006.<sup>[http://dcfldd.sourceforge.net/]</sup>
+
{{Note|Fdisk will not work on [[GPT]] formatted devices. Use gdisk ({{Pkg|gptfdisk}}) instead.}}
 
+
'''ddrescue'''
+
 
+
GNU {{Pkg|ddrescue}} is a data recovery tool. It's capable of ignoring read errors what is a useless feature for disk wiping in almost any case.
+
[http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html GNU ddrescue Manual]
+
 
+
==== shred ====
+
Shred is a Unix command that can be used to securely delete files and devices so that they can be recovered only with great difficulty with specialised hardware, if at all.[http://en.wikipedia.org/wiki/Shred_(Unix)] Shred uses three passes, writing pseudo-[[Securely_wipe_disk#Random data|random data]] to the device during each pass. This can be reduced or increased.
+
 
+
The following command invokes shred with its default settings and displays the progress.
+
# shred -v /dev/<drive>
+
 
+
Alternatively, shred can be instructed to do only one pass with entropy from, e.g. {{ic|/dev/urandom}}.
+
# shred --verbose --random-source=/dev/urandom -n1 /dev/<drive>
+
 
+
=== Badblocks ===
+
For letting [[Badblocks#read-write_Test|badblocks perform a disk wipe]] a destructive read-write test has to be done.
+
 
+
# badblocks -c 10240 -wsv /dev/<drive>
+
 
+
== Select a target ==
+
{{Note|Fdisk will not work on [[GUID Partition Table|GPT]] formatted devices. Use gdisk ({{Pkg|gptfdisk}}) instead.}}
+
 
Use fdisk to locate all read/write devices the user has read acess to.
 
Use fdisk to locate all read/write devices the user has read acess to.
  
Check the output for lines that start with devices such as {{ic|/dev/sdX}}.
+
Check the output for lines that start with devices such as {{ic|/dev/sd"X"}}.
  
 
This is an example for a HDD formatted to boot a linux system:
 
This is an example for a HDD formatted to boot a linux system:
Line 147: Line 106:
 
/dev/sdb1  *          0      802815      401408  17  Hidden HPFS/NTFS</nowiki>}}
 
/dev/sdb1  *          0      802815      401408  17  Hidden HPFS/NTFS</nowiki>}}
  
=== Block size ===
+
If you are worried about unintentional damage of important data on the primary computer, consider using an isolated environment such as a virtual environment (VirtualBox, VMWare, QEMU, etc...) with direct connected disk drives to it or a single computer only with a storage disk(s) that need to be wiped booted from a [[Archiso|Live Media]](USB, CD, PXE, etc...) or use a script to [[Securely wipe disk/Tips and tricks#Prevent wiping mounted partitions|prevent wiping mounted partitions by typo]].
{{Wikipedia|Dd (Unix)#Block size}}
+
If you have a [[Wikipedia:Advanced Format|Advanced Format]] hard drive it is recommended that you specify a block size larger than the default 512 bytes. To speed up the overwriting process choose a block size matching your drive's physical geometry by appending the block size option to the dd command (i.e. {{ic|<nowiki>bs=4096</nowiki>}} for 4KB).
+
  
Fdisk prints physical and logical sector size for every disk.
+
== Select a block size ==
  
Alternatively sysfs does expose information:
+
See also [[Wikipedia:Dd (Unix)#Block size]], [http://people.redhat.com/msnitzer/docs/io-limits.txt blocksize io-limits].
 +
 
 +
If you have an [[Advanced Format]] hard drive it is recommended that you specify a block size larger than the default 512 bytes. To speed up the overwriting process choose a block size matching your drive's physical geometry by appending the block size option to the ''dd'' command (i.e. {{ic|<nowiki>bs=4096</nowiki>}} for 4KB).
 +
 
 +
''fdisk'' prints physical and logical sector size for every disk.  Alternatively sysfs does expose information:
 +
/sys/block/sdX/size
 
  /sys/block/sdX/queue/physical_block_size
 
  /sys/block/sdX/queue/physical_block_size
 
  /sys/block/sdX/queue/logical_block_size
 
  /sys/block/sdX/queue/logical_block_size
  /sys/block/sdX/alignment_offset
+
  /sys/block/sdX/sdXY/alignment_offset
 +
/sys/block/sdX/sdXY/start
 +
/sys/block/sdX/sdXY/size
 +
 
 +
{{Warning|These methods show the block size the drive reports to the kernel. However, many Advanced Format drives incorrectly understate the physical block size as 512.}}
 +
 
 +
{{Tip|This script helps to calculate parameters to wipe a device/partition with dd {{AUR|genwipe.sh}}, e.g.{{ic| genwipe.sh /dev/sd"XY"}}.}}
 +
 
 +
=== Calculate blocks to wipe manually ===
 +
 
 +
In the following the determination of the data area to wipe is done in an example.
 +
 
 +
A block storage devices contains sectors and a size of a single sector that can be used to calculate the whole size of device in bytes. You can do it by multiplying sectors with size of the sector.
 +
 
 +
As an example we use the parameters with the ''dd'' command to wipe a partition:
 +
 
 +
# dd if=''data_source'' of=/dev/sd"X" bs=''sector_size'' count=''sector_number'' seek=''partitions_start_sector''
 +
 
 +
Here you will see only a part of output of {{ic|fdisk -l /dev/sdX}} with root, showing the example partition information:
 +
 
 +
{{bc|Device    Boot      Start        End        Sectors    Size  Id Type
 +
/dev/sd"XA"            2048        3839711231  3839709184  1,8T  83 Linux
 +
/dev/sd"XB"            3839711232  3907029167  67317936    32,1G  5 Extended
 +
}}
 +
 
 +
The first line of the ''fdisk'' output shows the disk size in bytes and logical sectors:
 +
 
 +
Disk /dev/sd"X": 1,8 TiB, 2000398934016 bytes, 3907029168 sectors
 +
 
 +
To calculate size of a single logical sector use {{ic|echo $((2000398934016 / 3907029168))}} or use data from the second line of ''fdisk'' output:
 +
 +
Units: sectors of 1 * 512 = 512 bytes
 +
 
 +
To calculate physical sectors that will make it work faster we can use the third line:
 +
 
 +
Sector size (logical/physical): 512 bytes / 4096 bytes
 +
 
 +
To get disk size in the physical sectors you will need the known disk size in bytes divided with size of a single physical sector {{ic|echo $((2000398934016 / 4096))}}, you can get size of the storage device or partition on it even with the {{ic|blockdev --getsize64 /dev/sd"XY"}} command.
 +
 
 +
{{Note|
 +
* In the examples below we will use the logical sector size.
 +
* You can even wipe unallocated disk space with a {{ic|dd}} command by calculating the difference between the end of one and start of the next partition.}}
 +
 
 +
To wipe partition {{ic|/dev/sd"XA"}} the example parameters with logical sectors would be used like this:
 +
 
 +
{{bc|1=Start=2048
 +
End=3839711231
 +
BytesInSector=512}}
 +
 
 +
By using the starting address of the partition on the device by defining it in the {{ic|1=seek=}} option
 +
 
 +
# dd if=''data_source'' of=/dev/sd"X" bs=${BytesInSector} count=${End} seek=${Start}
 +
 
 +
By using the partitions name
 +
 
 +
{{bc|1=LogicalSectors=3839709184}}
 +
 
 +
# dd if=''data_source'' of=/dev/sd"XA" bs=${BytesInSector} count=${LogicalSectors}
 +
 
 +
Or, to wipe the whole disk by using physical sectors:
 +
 
 +
{{bc|1=AllDiskPhysicalSectors=488378646
 +
PhysicalSectorSizeBytes=4096}}
 +
 
 +
# dd if=''data_source'' of=/dev/sd"X" bs=${PhysicalSectorSizeBytes} count=${AllDiskPhysicalSectors} seek=0
 +
 
 +
{{Note|The {{ic|1=count=}} option not necessary when wiping the physical limited area e.g. {{ic|sd"XY"}} or {{ic|sd"X"}} from begin to the end but will show an error about out of free space when will try to write outside of limits.}}
 +
 
 +
== Select a data source ==
 +
 
 +
As just said If you want to wipe sensitive data you can use anything matching your needs.
 +
 
 +
If you want to setup block device encryption afterwards, you should always wipe at least with an encryption cipher as source or even pseudorandom data.
 +
 
 +
For data that is not truly random your disk's writing speed should be the only limiting factor. If you need random data, the required system performance to generate it may extremely depend on what you choose as source of entropy.
 +
 
 +
=== Non-random data ===
 +
 
 +
Overwriting with {{ic|/dev/zero}} or simple patterns is considered secure in most resources. In the case of current HDD's it should be sufficient for fast disk wipes.
 +
 
 +
{{Warning|A drive that is abnormally fast in writing patterns or zeroing could be doing transparent compression. It is obviously presumable not all blocks get wiped this way. Some [[#Flash memory]] devices do "feature" that.}}
 +
 
 +
==== Pattern write test ====
 +
 
 +
[[#Badblocks]] can write simple patterns to every block of a device and then read and check them searching for damaged areas (just like memtest86* does with memory).
 +
 
 +
As the pattern is written to every accesible block this effectively wipes the device.
 +
 
 +
=== Random data ===
 +
 
 +
For differences between random and pseudorandom data as source, please see [[Random number generation]].
 +
 
 +
{{Note|Data that is hard to compress (random data) will get written slower, if the drive logic mentioned in the [[#Non-random data]] warning tries compressing it. This should not lead to [[#Data remanence]] though. As maximum write-speed is not the performance-bottleneck it can get completely neglected while wiping disks with random data.}}
 +
 
 +
=== Encrypted data ===
 +
 
 +
When preparing a drive for full-disk encryption, sourcing high quality entropy is usually not necessary. The alternative is to use an encrypted datastream. For example, if you will use AES for your encrypted partition, you would wipe it with an equivalent encryption cipher prior to creating the filesystem to make the empty space not distinguishable from the used space.
 +
 
 +
== Overwrite the target ==
 +
 
 +
The chosen drive can be overwritten with several utilities, make your choice. If you only want to wipe a single file, [[Securely wipe disk/Tips and tricks#Wipe a single file]] has considerations in addition to the utilities mentioned below.
 +
 
 +
=== By redirecting output ===
 +
 
 +
The redirected output can be used both for creation of the files to rewrite free space on the partition, wipe the whole device or a single partition on it.
 +
 
 +
In the following are examples that can be used to rewrite the partition or a block device by redirecting [http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html stdout] from other utilities:
 +
 
 +
{{hc|$ cat /dev/urandom > /dev/sd"XY"|
 +
cat: write error: No space left on device}}
 +
 
 +
{{hc|$ xz -z0 /dev/urandom -c > /dev/sd"XY"|
 +
xz: (stdout): Write error: No space left on device}}
 +
 
 +
{{hc|1=$ dd if=/dev/urandom > /dev/sd"XY"|2=
 +
dd: writing to ‘standard output’: No space left on device
 +
20481+0 records in
 +
20480+0 records out
 +
10485760 bytes (10 MB) copied, 2.29914 s, 4.6 MB/s}}
 +
 
 +
The file copy command {{ic|cp}} can also be used to rewrite the device, because it ignores the type of the destination:
 +
 
 +
{{hc|$ cp /dev/urandom /dev/sd"XY"|
 +
cp: error writing ‘/dev/sd"XY"’: No space left on device
 +
cp: failed to extend ‘/dev/sd"XY"’: No space left on device}}
 +
 
 +
To show speed and time you can use {{Pkg|pv}}:
 +
 
 +
# pv --timer --rate --stop-at-size -s "$(blockdev --getsize64 /dev/sd"XY" )" /dev/zero > /dev/sd"XY"
 +
 
 +
=== dd ===
 +
 
 +
See also [[Core utilities#dd]].
  
== Overwrite the disk ==
+
{{Warning|There is no confirmation regarding the sanity of this command so '''repeatedly check''' that the correct drive or partition has been targeted. Make certain that the {{ic|<nowiki>of=...</nowiki>}} option points to the target drive and not to a system disk.}}
{{warning|There is no confirmation regarding the sanity of this command so '''repeatedly check''' that the correct drive or partition has been targeted. Make certain that the {{ic|<nowiki>of=...</nowiki>}} option points to the target drive and not to a system disk.}}
+
  
 
Zero-fill the disk by writing a zero byte to every addressable location on the disk using the [[Wikipedia:/dev/zero|/dev/zero]] stream.
 
Zero-fill the disk by writing a zero byte to every addressable location on the disk using the [[Wikipedia:/dev/zero|/dev/zero]] stream.
Line 165: Line 258:
 
  # dd if=/dev/zero of=/dev/sdX bs=4096
 
  # dd if=/dev/zero of=/dev/sdX bs=4096
  
or the [[Wikipedia:/dev/random|/dev/random]] stream:
+
Or the [[Wikipedia:/dev/random|/dev/urandom]] stream:
 
  # dd if=/dev/urandom of=/dev/sdX bs=4096
 
  # dd if=/dev/urandom of=/dev/sdX bs=4096
  
Line 174: Line 267:
 
  4075290624 bytes (4.1 GB) copied, 1247.7 s, 3.3 MB/s
 
  4075290624 bytes (4.1 GB) copied, 1247.7 s, 3.3 MB/s
  
== Data remanence ==
+
To speed up wiping a large drive, see also:
{{Expansion|This section is too dependent on links to Wikipedia. Links to diverse and high quality resources should be added.}}
+
* [[Securely wipe disk/Tips and tricks#dd - advanced example]] which uses OpenSSL,
 +
* [[Securely wipe disk/Tips and tricks#Using a template file]] which wipes with non-random preset data(e.g. overwrite a whole disk with a single file) but is very fast
 +
* [[Dm-crypt/Drive preparation#dm-crypt specific methods]] which uses dm-crypt.
  
{{Wikipedia|Data remanence}}
+
=== wipe ===
The residual representation of data may remain even after attempts have been made to remove or erase the data.
+
  
Residual data may get wiped by writing (random) data to the disk with a single or even more than one iteration. However, more than one iteration may not significantly decrease the possibility to reconstruct the data of hard disk drives. For more information see [http://www.h-online.com/news/Secure-deletion-a-single-overwrite-will-do-it--/112432 Secure deletion: a single overwrite will do it - The H Security].
+
Specialized on wiping files and is available as the {{Pkg|wipe}} package. To make a quick wipe of a destination you can use something like:
  
=== Random data ===
+
$ wipe -r -q /path/to/wipe
If the data can get exactly located on the disk and was never copied anywhere else, wiping with random data can be thoroughgoing and impressively quick as long there is enough entropy in the pool.
+
  
A good example is cryptsetup using /dev/urandom for [[Dm-crypt_with_LUKS#Wipe_LUKS_keyslots|wiping the LUKS keyslots]].
+
See also:
 +
[http://linux.die.net/man/1/wipe man page].
  
=== Hardware specific issues ===
+
=== shred ===
==== Flash memory ====
+
[[Wikipedia:Write amplification]] and other characteristics make Flash memory a stubborn target for reliable wiping.
+
As there is a lot of transparent abstraction in between data as seen by a device's controller chip and the operating system sight data is never overwritten in place and wiping particular blocks or files is not reliable.
+
  
Other "features" like transparent compression (all SandForce SSD's) can compress your /dev/zero or pattern stream so if wiping is fast beyond belief this might be the case.
+
[https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html ''shred''] (from the {{Pkg|coreutils}} package) is a Unix command that can be used to securely delete individual files or full devices so that they can be recovered only with great difficulty with specialised hardware, if at all. ''shred'' uses three passes, writing [[Random number generation|pseudo-random data]] to the device during each pass. This can be reduced or increased.
  
Disassembling Flash memory devices, unsoldering the chips and analyzing data content without the controller in between is feasible without difficulty using [http://www.flash-extractor.com/manual/reader_models/ simple hardware]. Data recovery companys do it for cheap money.
+
The following command invokes shred with its default settings and displays the progress.
 +
# shred -v /dev/sd''X''
  
For more information see: [http://www.usenix.org/events/fast11/tech/full_papers/Wei.pdf Reliably Erasing Data From Flash-Based Solid State Drives].
+
Alternatively, shred can be instructed to do only one pass, with entropy from e.g. {{ic|/dev/urandom}}.
 +
# shred --verbose --random-source=/dev/urandom -n1 /dev/sd''X''
  
==== Residual magnetism ====
+
=== Badblocks ===
Wiped hard disk drives and other magnetic storage can get disassembled in a cleanroom and then analyzed with equipment like a [[Wikipedia:Magnetic force microscope|magnetic force microscope]]. This may allow the overwritten data to be reconstructed by analyzing the measured [[Wikipedia:Remanence|residual magnetics]].
+
  
This method of data recovery for current HDD's is largely theoretical and would require substantial financial resources. Nevertheless [[Wikipedia:Degaussing#Degaussing magnetic data storage media|degaussing]] is still a practiced countermeasure.
+
For letting badblocks (from the {{Pkg|e2fsprogs}} package) perform a disk wipe, a destructive [[Badblocks#read-write_Test|read-write test]]{{Broken section link}} has to be done:
  
==== Old magnetic storage ====
+
# badblocks -c <NUMBER_BLOCKS> -wsv /dev/<drive>
Securely wiping old magnetic storage (e.g. floppy disks, magnetic tape) is much harder due to much lower [[Wikipedia:Memory storage density|memory storage density]]. Many iterations with random data might be needed to wipe any sensitive data. To ensure that data has been completely erased most resources advise physical destruction.
+
  
==== Operating system, programs and filesystem ====
+
=== hdparm ===
{{Note|This is not specific to any hardware obviously.}}
+
{{warning|Do not attempt to issue a Secure Erase ATA command on a device connected through USB; see https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase and http://www.tomshardware.co.uk/answers/id-1984547/secure-erase-external-usb-hard-drive.html for details.}}
The operating system, executed programs or [[Wikipedia:Journaling file system|journaling file system]]s may copy your unencrypted data throughout the block device. When writing to plain disks this should only be relevant in conjunction with one of the above.
+
 
 +
[[hdparm]] supports [http://tinyapps.org/docs/wipe_drives_hdparm.html ATA Secure Erase], which is functionally equivalent to zero-filling a disk. It is however handled by the hard-drive firmware itself, and includes "hidden data areas". As such, it can be seen as a modern-day "low-level format" command. [[SSD]] drives reportedly achieve factory performance after issuing this command, but may not be sufficiently wiped (see [[#Flash memory]]).
 +
 
 +
Some drives support '''Enhanced Secure Erase''', which uses distinct patterns defined by the manufacturer. If the output of {{ic|hdparm -I}} for the device indicates a manifold time advantage for the '''Enhanced'''  erasure, the device probably has a hardware encryption feature and the wipe will be performed to the encryption keys only.
 +
 
 +
For detailed instructions on using ATA Secure Erase, see the [https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase Linux ATA wiki].
 +
 
 +
=== secure-delete ===
 +
 
 +
The {{AUR|secure-delete}} package from [[AUR]] provides several utilites for secure erasion, including {{ic|sfill}}, which deletes only free space in a specified mount. For example:
  
== See also ==
+
# sfill -v /
* [http://www.gnu.org/software/coreutils/manual/coreutils.html#Basic-operations GNU Coreutils Manpage on Basic operations]. Official documentation for dd and shred.
+
  
* [http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-command-362506/ Learn the DD command]. - linuxquestions.org
+
See [http://superuser.com/questions/19326/how-to-wipe-free-disk-space-in-linux the tools list] for more info.

Latest revision as of 21:54, 13 November 2016

Wiping a disk is done by writing new data over every single bit.

Note: References to "disks" in this article also apply to loopback devices.

Common use cases

Wipe all data left on the device

The most common usecase for completely and irrevocably wiping a device will be when the device is going to be given away or sold. There may be (unencrypted) data left on the device and you want to protect against simple forensic investigation that is mere child's play with for example File recovery software.

If you want to quickly wipe everything from the disk, /dev/zero or simple patterns allow maximum performance while adequate randomness can be advantageous in some cases that should be covered up in #Data remanence.

Every overwritten bit means to provide a level of data erasure not allowing recovery with normal system functions (like standard ATA/SCSI commands) and hardware interfaces. Any file recovery software mentioned above then would need to be specialized on proprietary storage-hardware features.

In case of a HDD data recreation will not be possible without at least undocumented drive commands or fiddling about the device’s controller or firmware to make them read out for example reallocated sectors (bad blocks that S.M.A.R.T. retired from use).

There are different wiping issues with different physical storage technologies, most notably all Flash memory based devices and older magnetic storage (old HDD's, floppy disks, tape).

Preparations for block device encryption

If you want to prepare your drive to securely set up Disk encryption#Block device encryption inside the wiped area afterwards you really should use #Random data generated by a trusted cryptographically strong random number generator (referred to as RNG in this article from now on).

See also Wikipedia:Random number generation.

Warning: If Block device encryption is mapped on a partition that contains anything else than random/encrypted data, disclosure of usage patterns on the encrypted drive is possible and weakens the encryption being comparable with filesystem-level-encryption. Never use /dev/zero, simple patterns (badblocks, eg.) or other unrandom data before setting up Block device encryption if you are serious about it!

Data remanence

See also Wikipedia:Data remanence.

The residual representation of data may remain even after attempts have been made to remove or erase the data.

Residual data may get wiped by writing (random) data to the disk with a single or even more than one iteration. However, more than one iteration may not significantly decrease the possibility to reconstruct the data of hard disk drives. See #Residual magnetism.

Operating system, programs and filesystem

The operating system, executed programs or journaling file systems may copy your unencrypted data throughout the block device. When writing to plain disks this should only be relevant in conjunction with one of the above.

If the data can get exactly located on the disk and was never copied anywhere else, wiping with random data can be thoroughgoing and impressively quick as long there is enough entropy in the pool.

A good example is cryptsetup using /dev/urandom for wiping the LUKS keyslots.

Hardware-specific issues

Flash memory

Write amplification and other characteristics make Flash memory (explicitly including SSDs) a stubborn target for reliable wiping. As there is a lot of transparent abstraction in between data as seen by a device's controller chip and the operating system sight data is never overwritten in place and wiping particular blocks or files is not reliable.

Other "features" like transparent compression (all SandForce SSD's) can compress your /dev/zero or pattern stream so if wiping is fast beyond belief this might be the case.

Disassembling Flash memory devices, unsoldering the chips and analyzing data content without the controller in between is feasible without difficulty using simple hardware. Data recovery companies do it for cheap money.

For more information see:

Marked Bad Sectors

If a hard drive marks a sector as bad, it cordons it off, and the section becomes impossible to write to via software. Thus a full overwrite would not reach it. However because of block sizes, these sections would only amount to a few theoretically recoverable KB.

Residual magnetism

A single, full overwrite with zeros or random data does not lead to any recoverable data on a modern high-density storage device.[1] Indications otherwise refer to single residual bits; reconstruction of byte patterns is generally not feasible.[2] See also [3], [4] and [5].

Select a target

Note: Fdisk will not work on GPT formatted devices. Use gdisk (gptfdisk) instead.

Use fdisk to locate all read/write devices the user has read acess to.

Check the output for lines that start with devices such as /dev/sd"X".

This is an example for a HDD formatted to boot a linux system:

# fdisk -l
Disk /dev/sda: 250.1 GB, 250059350016 bytes, 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00ff784a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      206847      102400   83  Linux
/dev/sda2          206848   488397167   244095160   83  Linux

Or the Arch Install Medium written to a 4GB USB thumb drive:

# fdisk -l
Disk /dev/sdb: 4075 MB, 4075290624 bytes, 7959552 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x526e236e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           0      802815      401408   17  Hidden HPFS/NTFS

If you are worried about unintentional damage of important data on the primary computer, consider using an isolated environment such as a virtual environment (VirtualBox, VMWare, QEMU, etc...) with direct connected disk drives to it or a single computer only with a storage disk(s) that need to be wiped booted from a Live Media(USB, CD, PXE, etc...) or use a script to prevent wiping mounted partitions by typo.

Select a block size

See also Wikipedia:Dd (Unix)#Block size, blocksize io-limits.

If you have an Advanced Format hard drive it is recommended that you specify a block size larger than the default 512 bytes. To speed up the overwriting process choose a block size matching your drive's physical geometry by appending the block size option to the dd command (i.e. bs=4096 for 4KB).

fdisk prints physical and logical sector size for every disk. Alternatively sysfs does expose information:

/sys/block/sdX/size
/sys/block/sdX/queue/physical_block_size
/sys/block/sdX/queue/logical_block_size
/sys/block/sdX/sdXY/alignment_offset
/sys/block/sdX/sdXY/start
/sys/block/sdX/sdXY/size
Warning: These methods show the block size the drive reports to the kernel. However, many Advanced Format drives incorrectly understate the physical block size as 512.
Tip: This script helps to calculate parameters to wipe a device/partition with dd genwipe.shAUR, e.g. genwipe.sh /dev/sd"XY".

Calculate blocks to wipe manually

In the following the determination of the data area to wipe is done in an example.

A block storage devices contains sectors and a size of a single sector that can be used to calculate the whole size of device in bytes. You can do it by multiplying sectors with size of the sector.

As an example we use the parameters with the dd command to wipe a partition:

# dd if=data_source of=/dev/sd"X" bs=sector_size count=sector_number seek=partitions_start_sector

Here you will see only a part of output of fdisk -l /dev/sdX with root, showing the example partition information:

Device     Boot      Start        End         Sectors     Size  Id Type
/dev/sd"XA"            2048         3839711231  3839709184  1,8T  83 Linux
/dev/sd"XB"            3839711232   3907029167  67317936    32,1G  5 Extended

The first line of the fdisk output shows the disk size in bytes and logical sectors:

Disk /dev/sd"X": 1,8 TiB, 2000398934016 bytes, 3907029168 sectors

To calculate size of a single logical sector use echo $((2000398934016 / 3907029168)) or use data from the second line of fdisk output:

Units: sectors of 1 * 512 = 512 bytes

To calculate physical sectors that will make it work faster we can use the third line:

Sector size (logical/physical): 512 bytes / 4096 bytes

To get disk size in the physical sectors you will need the known disk size in bytes divided with size of a single physical sector echo $((2000398934016 / 4096)), you can get size of the storage device or partition on it even with the blockdev --getsize64 /dev/sd"XY" command.

Note:
  • In the examples below we will use the logical sector size.
  • You can even wipe unallocated disk space with a dd command by calculating the difference between the end of one and start of the next partition.

To wipe partition /dev/sd"XA" the example parameters with logical sectors would be used like this:

Start=2048
End=3839711231
BytesInSector=512

By using the starting address of the partition on the device by defining it in the seek= option

# dd if=data_source of=/dev/sd"X" bs=${BytesInSector} count=${End} seek=${Start}

By using the partitions name

LogicalSectors=3839709184
# dd if=data_source of=/dev/sd"XA" bs=${BytesInSector} count=${LogicalSectors}

Or, to wipe the whole disk by using physical sectors:

AllDiskPhysicalSectors=488378646
PhysicalSectorSizeBytes=4096
# dd if=data_source of=/dev/sd"X" bs=${PhysicalSectorSizeBytes} count=${AllDiskPhysicalSectors} seek=0
Note: The count= option not necessary when wiping the physical limited area e.g. sd"XY" or sd"X" from begin to the end but will show an error about out of free space when will try to write outside of limits.

Select a data source

As just said If you want to wipe sensitive data you can use anything matching your needs.

If you want to setup block device encryption afterwards, you should always wipe at least with an encryption cipher as source or even pseudorandom data.

For data that is not truly random your disk's writing speed should be the only limiting factor. If you need random data, the required system performance to generate it may extremely depend on what you choose as source of entropy.

Non-random data

Overwriting with /dev/zero or simple patterns is considered secure in most resources. In the case of current HDD's it should be sufficient for fast disk wipes.

Warning: A drive that is abnormally fast in writing patterns or zeroing could be doing transparent compression. It is obviously presumable not all blocks get wiped this way. Some #Flash memory devices do "feature" that.

Pattern write test

#Badblocks can write simple patterns to every block of a device and then read and check them searching for damaged areas (just like memtest86* does with memory).

As the pattern is written to every accesible block this effectively wipes the device.

Random data

For differences between random and pseudorandom data as source, please see Random number generation.

Note: Data that is hard to compress (random data) will get written slower, if the drive logic mentioned in the #Non-random data warning tries compressing it. This should not lead to #Data remanence though. As maximum write-speed is not the performance-bottleneck it can get completely neglected while wiping disks with random data.

Encrypted data

When preparing a drive for full-disk encryption, sourcing high quality entropy is usually not necessary. The alternative is to use an encrypted datastream. For example, if you will use AES for your encrypted partition, you would wipe it with an equivalent encryption cipher prior to creating the filesystem to make the empty space not distinguishable from the used space.

Overwrite the target

The chosen drive can be overwritten with several utilities, make your choice. If you only want to wipe a single file, Securely wipe disk/Tips and tricks#Wipe a single file has considerations in addition to the utilities mentioned below.

By redirecting output

The redirected output can be used both for creation of the files to rewrite free space on the partition, wipe the whole device or a single partition on it.

In the following are examples that can be used to rewrite the partition or a block device by redirecting stdout from other utilities:

$ cat /dev/urandom > /dev/sd"XY"
cat: write error: No space left on device
$ xz -z0 /dev/urandom -c > /dev/sd"XY"
xz: (stdout): Write error: No space left on device
$ dd if=/dev/urandom > /dev/sd"XY"
dd: writing to ‘standard output’: No space left on device
20481+0 records in
20480+0 records out
10485760 bytes (10 MB) copied, 2.29914 s, 4.6 MB/s

The file copy command cp can also be used to rewrite the device, because it ignores the type of the destination:

$ cp /dev/urandom /dev/sd"XY"
 cp: error writing ‘/dev/sd"XY"’: No space left on device
 cp: failed to extend ‘/dev/sd"XY"’: No space left on device

To show speed and time you can use pv:

# pv --timer --rate --stop-at-size -s "$(blockdev --getsize64 /dev/sd"XY" )" /dev/zero > /dev/sd"XY"

dd

See also Core utilities#dd.

Warning: There is no confirmation regarding the sanity of this command so repeatedly check that the correct drive or partition has been targeted. Make certain that the of=... option points to the target drive and not to a system disk.

Zero-fill the disk by writing a zero byte to every addressable location on the disk using the /dev/zero stream.

# dd if=/dev/zero of=/dev/sdX bs=4096

Or the /dev/urandom stream:

# dd if=/dev/urandom of=/dev/sdX bs=4096

The process is finished when dd reports, No space left on device:

dd: writing to ‘/dev/sdb’: No space left on device
7959553+0 records in
7959552+0 records out
4075290624 bytes (4.1 GB) copied, 1247.7 s, 3.3 MB/s

To speed up wiping a large drive, see also:

wipe

Specialized on wiping files and is available as the wipe package. To make a quick wipe of a destination you can use something like:

$ wipe -r -q /path/to/wipe

See also: man page.

shred

shred (from the coreutils package) is a Unix command that can be used to securely delete individual files or full devices so that they can be recovered only with great difficulty with specialised hardware, if at all. shred uses three passes, writing pseudo-random data to the device during each pass. This can be reduced or increased.

The following command invokes shred with its default settings and displays the progress.

# shred -v /dev/sdX

Alternatively, shred can be instructed to do only one pass, with entropy from e.g. /dev/urandom.

# shred --verbose --random-source=/dev/urandom -n1 /dev/sdX

Badblocks

For letting badblocks (from the e2fsprogs package) perform a disk wipe, a destructive read-write test[broken link: invalid section] has to be done:

# badblocks -c <NUMBER_BLOCKS> -wsv /dev/<drive>

hdparm

Warning: Do not attempt to issue a Secure Erase ATA command on a device connected through USB; see https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase and http://www.tomshardware.co.uk/answers/id-1984547/secure-erase-external-usb-hard-drive.html for details.

hdparm supports ATA Secure Erase, which is functionally equivalent to zero-filling a disk. It is however handled by the hard-drive firmware itself, and includes "hidden data areas". As such, it can be seen as a modern-day "low-level format" command. SSD drives reportedly achieve factory performance after issuing this command, but may not be sufficiently wiped (see #Flash memory).

Some drives support Enhanced Secure Erase, which uses distinct patterns defined by the manufacturer. If the output of hdparm -I for the device indicates a manifold time advantage for the Enhanced erasure, the device probably has a hardware encryption feature and the wipe will be performed to the encryption keys only.

For detailed instructions on using ATA Secure Erase, see the Linux ATA wiki.

secure-delete

The secure-deleteAUR package from AUR provides several utilites for secure erasion, including sfill, which deletes only free space in a specified mount. For example:

# sfill -v /

See the tools list for more info.