Linux and Unix Disk How To

How to take a disk image over SSH

run from remote computer
dd if=/dev/sda | gzip -1 - | ssh user@local dd of=image.gz
run from local computer
ssh user@remote "dd if=/dev/sda | gzip -1 -" | dd of=image.gz
to monitor you can insert pv into the pipeline after gzip or send -USR1 to the last dd in the pipeline

How to mount sshfs on KDE session start

Put the following command into a script
sshfs user@server:/remote/path /mount/point
Copy the ssh keys to allow password-less mounting, test, then add it as a script command to KDE's startup via System-Settings -> Startup and Shutdown -> Autostart


How to check disk transfer speeds

cat /dev/zero | pv > /media/some-disk/temp-file
then flush the cache and and perform the opposite operation using the newly-created file:

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
cat /media/some-disk/temp-file | pv > dev/null

How to create a bootable linux rescue USB stick

get systemrescue cd
https://iweb.dl.sourceforge.net/project/systemrescuecd/sysresccd-x86/5.0.1/systemrescuecd-x86-5.0.1.iso
dont use isohybrid, it messes up the partition table
sudo mount -o loop,exec /path/to/systemrescuecd-x86-x.y.z.iso /media/temp
Plug in the USB stick

cd /mount/temp
sudo ./usb_inst.ssh
sudo umount /media/temp

http://www.system-rescue-cd.org/Installing-SystemRescueCd-on-a-USB-stick/

How to mount a device as a user

This is a command line way to mount the device like KDE/Dolphin does it in GUI
Plain/unencrypted device
udisksctl mount -b /dev/sdx1
To mount an encrypted device

cryptsetup  luksOpen /dev/sdx1 x

Provide luks password
udisksctl mount -b /dev/mapper/x
Provide the current user password (not luks)

To get the list of devices do
udisksctl dump | grep -E "(PreferredDevice|IdLabel)"
or
udisksctl dump | grep /disk/by-label
To list already mounted do
udisksctl dump | grep Configuration | grep tab

To unmount use
udisksctl unmount -b /dev/mapper/x

How to set up an encrypted backup disk to mount automatically at system boot

The idea is to use the main encrypted root partition to store the key for the backup disk. As far as the disk layout goes, LUKS will be set up on a partion, then a file system will be created on LUKS. I.e. it will be like this: MS-DOS Partition table -> Luks -> ext4. An alternative is to have LUKS on the raw disk and then have partitions under it i.e LUKS -> MS-DOS partition table -> ext4. The first way seems to be more compatible with the existing linux disk tools. Ubuntu does it the third way. It creates the partition table, two unencrypted partitions (/boot and /boot/efi), one encrypted LUKS partition. Then it sets up LVM on LUKS, by creating creates a physical volume, a volume group and two logical volumes, root and swap that contain the appropriate file systems. I.e. Partition table -> LUKS -> LVM -> partition tables. To see the setup run lsblk.

Enough with theory. Drop into root with sudo -s. Create the partition table (you know all your existing data on the disk will be gone, right?)
   fdisk /dev/sdi

Within FDISK, press:

  • n (create a new partition table and a partition)
  • p (to specify it as a PRIMARY partition)
  • 1 (to set it as the 1ST primary partition)
  • w (write the changes)

Now create LUKS container

cryptsetup luksFormat /dev/sdi1
cryptsetup luksOpen /dev/sdi1 b1
mkfs.ext4 -m 0 -L Backups /dev/mapper/b1 (-m 0 removes reserved blocks (i.e. set to 0%), since your system's livelihood does not depend on the empty space on this disk)
mount /dev/mapper/b1 /media/Backups

Now, here is the magic. Beside the normal passphase to open the disk, we'll create a random key and then use it to unlock the disk after /root is mounted with it's own passphrase:

dd if=/dev/urandom of=/root/.keyfiles/luks_backups bs=1024 count=4
chmod 0700 /root/.keyfiles
chmod 0400 /root/.keyfiles/luks_backups 
cryptsetup luksAddKey /dev/sdi1 /root/.keyfiles/luks_backups 

Set up chain-opening of the disks:

echo "b1 UUID=$(sudo blkid -s UUID -o value /dev/sdi1) /root/.keyfiles/luks_backups luks" >> /etc/crypttab
update-initramfs -u -k all

Add to /etc/fstab
/dev/mapper/b1                                  /media/Backups  ext4    rw,suid,dev,exec,auto,user,async,relatime       0       2

How to restore from a backup a set of files what was recently changed

Make sure you are in the same folder on your active disk as the backup you are restoring from. Pick the -mmin value in minutes or -mtime in days, keep minus in front of the number to indicate "less than".

find . -type f -mmin -''600'' -exec cp -av "/media/''restore'' ''mount'' ''point''/{}" "{}"  \;

How to view and adjust disk space reserved for root processes

About 5% is reserved for root, so you can still log as root and solve out of disk space. Usually this is 5%. You can see how much exactly is reserved:
tune2fs -l /dev/sda1 | grep -i reserved
Adjust by running
tune2fs -m 3 /dev/sda1
where 3 is the percent of the overall space you want to limit it to.

How to find processes that hold deleted files aka how to find files that are deleted but not yet have their space released back

sudo find /proc/ -mindepth 3 -maxdepth 3 -regex '/proc/[1-9][0-9]*/fd/[1-9][0-9]*' -type l -lname '*(deleted)' -printf '%p:\t%l (%s)\n' 2>/dev/null

How to use find to copy files with folder structure

Use cp --parents key. Here is an example how to copy all files newer than 301 day ago into a "new folder"

mkdir "new folder"
find . -type f -mtime -300 -exec cp -a --parents "{}" "new folder" \;

How to find files not in a specific subfolder with other conditions

Here is an example of locating all files not owned by john, in all directories excluding Backups
find . -path ./Backups -prune -o $-not -user john$ -ls

How to find recently changed files excluding a directory from search

find . -path "./Backups" -prune -o -mtime -1 -ls
-1 gives it within a last day. 0 has the same effect since mday is modulus 24 hrs. For a more natural ls do
find . -path "./Backups" -prune -o -mtime -1 -exec ls -alhd "{}" \;

How to remove files everywhere using locate, instead of find

locate -e0 Thumbs.db | xargs -0 rm -v 
-e is for "existing so that locate does not keep "finding" even after deletion them until you (or cron) run updatedb. More fancy way could be to use regexps:
locate -e0r "/home/user/[\w\d].*/\.directory" | xargs -0 rm -v
This would find all dolphin .directory files in all folders under /home/user, excluding these starting with fancy characters (e.g. the ".blah" hidden folders), which is important because dolphin keeps default settings under .kde in .directory files

How to check hard drive's power on time, internal logs and run hardware checks

First install smartmontools and gsmartcontrol with apt. Then run
sudo gsmartcontrol
Open the drive and look for

  • Power-on time - time hard drive has been running (in hours)
  • Power cycle count - how many times hard drive was shut down and started

You can run the self-test from the "perform tests" tab. Then check the result in the "self-test" tab. However, according to some Google Inc research in 2007 none of the SMART indicators are really predictive of when the drive will fail.

How to mount a USB drive on boot

This assumes you want your disk available to the system regardless whether someone logs in or not, in a permanent fashion, as if it was an internal harddrive. Let's assume the disk is recognized as sdg1.

  • Get the volume ID
sudo blkid /dev/sdg1
  • Create the mount point
mkdir /media/Backups
  • Add to fstab for automounting
UUID=''{volume'' ''id'' ''here}''       /media/Backups  ext4    defaults        0       2

  • Create a custom UDEV rule to act on USB drives - /etc/udev/rules.d/91_usb-drive.rules
ENV{ID_FS_UUID}=="''{volume'' ''id'' ''here}''",   ACTION=="add",      SUBSYSTEMS=="usb", RUN+="mount /dev/%k"

How to quickly wipe empty space on a disk

This would protect you from the curious eyes, but not from a determined expert:

dd if=/dev/zero of=/media/''mountpoint''/zerozerozero bs=100M
rm /media/''mountpoint''/zerozerozero

You can also do it with the sfill comppand from the secure-disk package, but I found it much much slower for the same operation.

sudo sfill -llzv /media/mountpoint

That's two 'L' for a single pass and a 'Z' for zeroing out instead of using random data. If you have SSD and want to nix the whole drive, do it with the hardware, not software:

sudo hdparm --security-erase NULL /dev/sda

How to recursively find files that are in one folder, but not in another

If you simply want to check file existence, and not bother comparing files with the same name, do the following. CD into the source folder (this is important to simplify messing with path names) Then run

find . -not -exec test -e "/''other'' ''folder''/{}" \; -print

You could also do it with rsync with an added benefit of detecting changed files:

 rsync -vrn --size-only ''"first'' ''folder/"'' ''"second'' ''folder/"''

verbose, recursively, no action (simulate only), and ignore timestamps Comparing folders AND checking files for differences can be done with the "diff -q" command, but it's rather slow for big folders. Try rsync here too, with it's rolling checksum speedup:

 rsync -vrnc  ''"first'' ''folder/"'' ''"second'' ''folder/"''

How to duplicate (copy) one disk to another

Clonezilla has a nice interface but is awefully slow and overly complicated if all you want is to clone a disk. Assuming sda is a source and sdb is a destination, do this:

dd if=/dev/sda of=/dev/sdb bs=4K & pid=$!

then run the following to monitor it's progress

watch kill -USR1 $pid

The kill command is used to send a "reporting interrupt". 4K is a typical block size for an SSD. I found copy performance the best at that size. You might also try 512K, since it's a page write size (atom) for a typical SSD.

How to check and repair truecrypt partition

You have to mount the encrypted partition (file) without mounting the internal filesystem. You can do it in the truecrypt GUI by checking the "do not mount" checkbox in the options on the mount prompt, or run the following command as a user who can mount disks (i.e who can sudo)

truecrypt -t --mount ''"encrypted'' ''store"'' --filesystem=none

This will create a fuse mounted structure under /tmp/.truecrypt_aux_mnt1/. Now you can check /tmp/.truecrypt_aux_mnt1/volume directly, but fsck will not be able to fix it since it is presented as an image file, not a device. To get around that mount this file over a loopback device and then do the checking:

sudo losetup -f /tmp/.truecrypt_aux_mnt1/volume
sudo fsck.ext4 -vf /dev/loop0

Use a different fsck if your filesystem is not ext4. When done, unmount the loopback and disconnect the volume

sudo losetup -d /dev/loop0
truecrypt -t --dismount ''"encrypted'' ''store"''

How to profile disk speed in Linux

You can mess with command line or you can do the GUI. GUI is much better:

gnome-disks

It needs an unmounted partition to benchmark properly. It's part of gnome-disk-utility package in Ubuntu.

From the command line, you can test the read speed will be this:

hdparam -tT

But for the write speed you'd have to script something with the "dd". You could also try phoronix benchmarking suite, but it is rather big and cumbersome if all you want to do is check how much your disk performance improved/degraded with some changes.

How to recover/undelete files from an NTFS partition

nftsundelete is actually quite good. Give it a try:

sudo ntfsundelete -p 50 -s /dev/sda8 > "files recoverable by at least 50%"

Undelete

sudo ntfsundelete -u -m "*.doc" -d ~/RestoredDocs/ /dev/sda8

If you want to automate recovering of the files, do the following. Create a folder and dump the list of all fully recoverable content.

sudo ntfsundelete /dev/sda8 -s -p 100 > disk-all 

Trim the list leaving only file extensions with the following regexp

s/.*\.(.*)/$1/g
s/   100%.*//g

Filter to leave only unique extensions

cat disk-all | tr [:upper:] [:lower:] | sort | uniq > disk-ext 

Run the undelete procedure:

for line in $(cat disk-ext); do echo $line; mkdir $line; sudo ntfsundelete /dev/sdb1 -u -q -T -d $line -m *.$line; done

How to sort FAT file system contents for a media player

You can do it with the fatsort tool from the Ubuntu repository. Unfortunately, it is ancient and has a lot of bugs causing segmentation faults. You could also try YAFS, which is newer, but requires compilation from source.

First, find out the device name for your FAT disk. Unmount it. Run 'mount' command to see which device it is. Then install fatsort:

sudo aptitude install fatsort

View your fat system with

sudo fatsort -l /dev/sdc1

And then sort it with

sudo fatsort /dev/sdc1

For YAFS, download the source and unpack it. Install the xerces C lib:

sudo aptitude libxerces-c2-dev

Now compile:

cp Makefile_unix Makefile
make

Once compiled, start by generate the listing for your file system:

sudo ./yafs -d /dev/sdc1 -r -f layout.xml

Edit it with the file editor and change the order as you wish. Write the order back:

sudo ./yafs -d /dev/sdc1 -r -f layout.xml

YAFS has bugs too though. If an ampersand shows up in the short name in layout.xml yafs will crash. To fix that change the XML file with the following command:

sed -e 's/(<short_name>.*)&/$1&/g' layout.xml

How to Make an ISO Image Using DD

Put the media in the drive. Do not mount it, if automounts then unmount it. For DVD

dd if=/dev/dvd of=image.iso

For CD

dd if=/dev/cdrom of=image.iso

How to check disk for bad sectors

Unmount your FS or reboot into a recovery linux OS like Parted Magic.

e2fsck -cc /dev/''sdf1''

-cc runs a non-destructive read-write test. Results are saved into a bad blocks inode on the file system. You can later read its contents by running

dumpe2fs -b /dev/''sdf1''

Alternatively you could use badblocks that would produce the a text file output from the same non-destructive read-write test and stats visible on stdout.

nohup sudo badblocks -snv -o badblocks.log /dev/''sdf1''

Just know that the size of the blocks in the log, and therefore the naming might be different from what fsck thinks the size should be.

How to check file suspected for containing bad sectors

shred -vn 1 ''filename''

How to change UUID of a cloned partition

  • To change the UUID of a non-boot partition, either generate a UUID:

uuidgen

and apply it:

sudo tune2fs /dev/sdc7 -U some-uid

or use the word 'random' to generate it on the fly

sudo tune2fs /dev/sdc7 -U random

reboot. View it with

blkid

  • To change the UUID on the boot partition, do the steps for the non-boot partition and edit /boot/grub/menu.lst and /etc/fstab
  • To do it on a swap partition run a gparted, delete the swap partition and recreate it.

How to see progress of a dd command

sudo kill -USR1 `pidof dd`

How to check what application/process/thread is accessing the disk

So you see that light blinking all the while you are not doing anything and are curious of what that might mean? You could check it in real time by running

sudo iotop -ao

  • -a is to show accumulated usage since iotop started
  • -o is to only show processes that had accessed the disk since iotop started (press "o" during the run-time to switch to showing all processes, including the ones that have not touched the disk)

Keep it running for awhile to monitor your system.

To show currently opened files with their associated processes run the following command:

lsof / | grep "REG" | grep -v "mem" | awk '{print $9}' | sort | uniq

Substitude $1 for $9 to see all processes that currently have files open. Keep in mind that multiple processes may have the same file open just as one process can have multiple files open.

How to clone a disk

The following command clones a disk fast and byte-by-byte so that even blkid remains the same. Before running make sure you've typed in correct disks (and careful copy/pasting 'enter') as there will not be a 'are you sure' prompt

sudo ddrescue -v /dev/sda /dev/sdb

You can get it from

sudo aptitude install ddrescue

How to spin down an HDD to check its noise level

hdparm -Y /dev/sdx

How to check or change how many times a FS was mounted and when the next fsck on boot will run

sudo dumpe2fs -h /dev/sdb1 | grep -i 'mount count'

To change it use

sudo tune2fs -c 50 /dev/hda1

How to restore names from lost+found

find ./lost+found -exec file -b {} \; > filetypes.txt
cat filetypes.txt | sort | uniq > filetypes_uniq.txt

  • Script to copy files based on filetypes, eg. from lost+found
#!/bin/bash
#
# Usage:
# Edit SOURCE and TARGET folders below, then
# ./find.sh filetype
#

# where to copy from (SOURCE) and copy to (TARGET). TARGET will be created if non existing
SOURCE="/home/jane/lost+found"
TARGET="/home/jane/recover"

# file name prefix "#"
FPREFIX="#"

# minimum filesize
FSIZE="100k"

if [ arg$1 != arg ]; then
  filetype=$1
else
  echo "Error: no filetype specified"
  echo "Usage: $0 [filetype]"
  exit
fi

# file extension
fext=".`echo $filetype |tr "[:upper:]" "[:lower:]"`"

# create target folder
targetfolder=$TARGET/$filetype
mkdir -p $targetfolder

FILES=`find "${SOURCE}" -name "${FPREFIX}*" -size +${FSIZE} -exec ls {} \;`
for file in $FILES
do

  fname=`basename "$file"`
  targetfile=${targetfolder}/${fname}${fext}
  check=`file "${file}" | grep -q "${filetype}"`

  if  $? -eq 0
  then
    echo "${file} is ${filetype}, copying to ${targetfile}"
    cp "$file" "$targetfile"
  fi

done

How to force fsck on next boot

By creating /forcefsck file you will force the Linux system (or rc scripts) to perform a full file system check.

sudo touch /forcefsck

Using the shutdown command:

shutdown -rF now

The -F option force fsck on reboot.

How to mount an ISO

On Linux

sudo mount -o loop /yourisofile.iso /mount/point

On AIX

  1. Get the size of the image.

ls -al
-rw-r--r-- 1 root system 374575104 Apr 29 02:59 fim_console_aix_5.iso

It's roughly about 360MB, but we need a number with an increment of 128, so 384MB (always larger) sounds good.

  1. Make a logical volume of the appropriate size for this image. Ensure there is enough space allocated on physical volume hdisk0 (in this case). Try it and if it fails it will inform you that there is not enough space. At that point, increase the size if necessary.

mklv -y cdlv -s n -L /dev/cdlv rootvg 384M hdisk0

  1. Create the pseudo device with 'dd'. Again, ensure that the partition has enough space for pseudo device, /dev/cdlv in this case. This may take along time and will create 2 'dd' processes.

dd if=/opt/software/ISO/fim_console_aix_5.iso of=/dev/cdlv

  1. Mount the device like a cdrom in AIX. Ensure the mount point exists.

mount -v cdrfs -o ro /dev/cdlv /mnt/iso

  1. cd into the directory /mnt/iso

After installing the product you can remove the volume with rmlv 1

How to move OS to a different disk or partition

Just copy all the os files and then

  1. redo grub if you want to boot of that partition
  2. sudo blkid
  3. change fstab to mount appropriate disks
  4. Check that swap line UUID from /etc/fstab matches swap UUID from step 2, if not change fstab.
  5. Check that the UUID in /etc/initramfs-tools/conf.d/resume matches the swap UUID from step 2, if not change resume file.
  6. sudo update-initramfs -u
  7. Restart

How to remount a file system with rw permissions

mount -o remount,rw,noatime -n /dev/root /
To go back :
mount -o remount,ro -n /dev/root /

How to lock file from writing even for root

To make file read only for everyone use additional filesystem attributes:

chattr +i [file]
to see current attr
lsattr

note that these attrs are different from acls set via setacl

How to test flash memory speed

non destructive. May be off because of SSD/Flash wear leveling

hdparm -t /dev/sdb

write testing
sync; rm testing; sync; time ( dd if=/dev/zero of=testing bs=16k count=10000; sync)
Many smallish files write:
sync; rm -rf testing*; sync; time ( for item in `seq 1 1000`; do dd if=/dev/zero of=testing.$item bs=16k count=10; sync; done; )

or

dd count=100 bs=1M if=/dev/urandom of=/media/disk/test

Destructive

I didn’t want to write on the first 1GB of flash memory because that part is probably most often used, I know there are some wear-leveling chips that should take care of that but you can use skip parameter if you wish to be safe. To prefent mesuring your cache performance but actual device performance use iflag and oflag options.


read speed benchmark:

$ sudo dd if=/dev/sdX1 of=/dev/zero bs=1M count=400 iflag=direct

write speed benchmark :

$ sudo dd if=/dev/zero of=/dev/sdX1 bs=1M count=400 skip=1000 oflag=direct

How to see progress of a long copy command

#!/bin/bash
# File copy with progress indicators
# Example: ./test original_file destination_file
usage()
{
   echo "Usage: $0 original_file destination_file"
   exit 1;
}
test $# == 2 || usage

echo Preparing to copy
orig_size=$(stat -c %s $1)

>$2
dest_size=0
cp -f $1 $2 &

while [ $orig_size -gt $dest_size ] ; do
   dest_size=$(stat -c %s $2)
   pct=$((( 100 * $dest_size ) / $orig_size ))

if [ $pct -lt 10 ] ; then
   echo -en "#  $pct%\b\b\b\b"
else
   echo -en "#  $pct%\b\b\b\b\b"
fi
sleep 1
done
echo

or

#!/bin/bash
# Example: ./test original_file destination_file
usage()
{
   echo "Usage: $0 original_file destination_file"
   exit 1;
}

test $# == 2 || usage
orig_size=$(stat -c %s $1)

>$2
dest_size=0
cp -f $1 $2 &

while [ $orig_size -gt $dest_size ] ; do
   dest_size=$(stat -c %s $2)
   pct=$((( 69 * $dest_size ) / $orig_size ))

    echo -en "\r["
    for j in `seq 1 $pct`; do
        echo -n "="
    done
    echo -n ">"
    for j in `seq $pct 68`; do
        echo -n "."
    done
    echo -n "] "
    echo -n $((( 100 * $pct ) / 69 ))
    echo -n "%"
done
echo


How to clean up disk space in Ubuntu

  • Clean package infromation and obsolete/unused packages

sudo aptitude clean
sudo aptitude autoclean
sudo apt-get autoremove
sudo apt-get install localepurge
sudo localepurge
sudo apt-get install deborphan
sudo deborphan | xargs sudo apt-get -y remove --purge

Start Synaptic Package Manager. Click the Status button and remove all residiual configurations (if present)

  • Clean kernels

Edit /boot/grub/menu.lst and change

# howmany=all

to

# howmany=2

then save it and run

sudo update-grub

this keeps the kernels installed, but limits your grub menu to the latest 2 kernels at all times.

  • Other tools that could help: fslint, gtkorphan



@HowTo @LinuxandUnix