Linux and Unix Scripting How To

[edit] How to perform a mass rename of files

Use rename regex in conjunction with find. For example to strip off a pattern:

find . -name "*pattern*" -exec rename -n -v 's|^(.*/)(.*) pattern (.*)\s*$|$1$2$3|g' {} \;

[edit] How to change file contents to upper or lower case with bash

cat file.csv | tr [:lower:] [:upper:] > file_up.csv

[edit] How to extract data with bash

Just an example of getting data with bash commands

df -h | awk '{print $5}' | grep % | grep -v Use | sort -n | tail -1 | cut -d "%" -f1 -

This will show the topmost utilization of a mounted linux partition.

[edit] How to iterate over lines in a file with BASH

while read i; do touch "$i"; done < ../list

[edit] How to do a mass search and replace

Following is a handy little command to get the job done. Assuming you are in the directory where you want to effect this search/replace --

perl -pi -e 's/lookFor/replaceWith/' *.fileExtension
perl -pi -e 's/lookFor/replaceWith/g' *.fileExtension
perl -pi -e 's/lookFor/replaceWith/gi' *.fileExtension

NOTES: - lookFor is the word you wish to look for. - replaceWith is the word you wish to replace your original word with - g stands for global, it will basically replace all the occurences of lookFor with replaceWith in all your files in a directory - i stands for case insensitive

To load a series of replace expressions from a text file:

perl -pi -e "`</path/to/replace-patterns.txt`" *.fileExtension

where replace-patterns.txt looks like (don't forget the semicolon!):

s/lookForPatternOne/replaceWithOne/;
s/lookForPatTwo/replaceWithTwo/g;
s/lookForPatThree/replaceWithThree/gi;

or

#!/usr/local/bin/perl
#
# Replaces a string within multiple files
# specified on the command line
$mv = '/bin/mv';
$op = shift || die("Usage: $0 perlexpr [filenames]\n");
if (!@ARGV) {
  @ARGV = <STDIN>;
  chop(@ARGV);
}
foreach $file (@ARGV) {
  if (!-f $file) {
       print "Skipping non-regular file: $file\n";
       next;
  }
  if (-B $file) {
       print "Skipping binary file: $file\n";
       next;
  }
$outfile = "/usr/tmp/$file.$$";
open(FILE, $file) ||
     die("Couldn't open $file: $!\n");
undef $/;
$_ = <FILE>;
close(FILE);
if (eval $op) {
  open(OFILE, "> $outfile") ||
  die("Couldn't open $outfile: $!\n");
  print OFILE;
  close(OFILE);
  system($mv, '-f', $file, "$file.bak");
  system($mv, '-f', $outfile, $file);
  print "File changed: $file\n";
}
else {
  print "No change to file: $file\n";
}
}
exit(0);

[edit] How to find all files in subdirectories containing a string

find . -name "*.jsp" | xargs grep "login b"

[edit] How to get a list of files with atypical owners

sudo ls -AlR / | grep -v -E 'root|total|www-data|\.|/|^$'

[edit] How to get directory sizes in the current folder

ls -d \* | xargs -i du -kasm {}

[edit] How to get size of folders in the current folder

ls | xargs du -sh

or

du -sh `ls`

Attention - does not work for folders with spaces in the name, escape with sed 's/\s/\\ /'

[edit] How to make a command survive term or putty or xterm termination

Just running it with ampersand wont work First run it with ampo

./command &

then disown it

disown [process number]

where process number is the process number from the previous command (or you can get it by doing ps -ef | grep [command])

disown is a bash command to detach the child-process from the parent and allow it to run "freely" (actually, it changes the parent to whatever is farther up the process chain).

Another handy way of doing this is the nohup command. This will even further detach the given command to "survive" even a full logout - which is not recommended for UI programs. It's used quite a bit in the system services at startup.

[edit] How to search for relevant man pages

apropos [substring]

[edit] How to see progress of a long copy command

#!/bin/bash
# File copy with progress indicators
# Example: ./test original_file destination_file
usage()
{
   echo "Usage: $0 original_file destination_file"
   exit 1;
}
test $# == 2 || usage
 
echo Preparing to copy
orig_size=$(stat -c %s $1)
 
>$2
dest_size=0
cp -f $1 $2 &
 
while [ $orig_size -gt $dest_size ] ; do
   dest_size=$(stat -c %s $2)
   pct=$((( 100 * $dest_size ) / $orig_size ))
 
if [ $pct -lt 10 ] ; then
   echo -en "#  $pct%\b\b\b\b"
else
   echo -en "#  $pct%\b\b\b\b\b"
fi
sleep 1
done
echo

or

#!/bin/bash
# Example: ./test original_file destination_file
usage()
{
   echo "Usage: $0 original_file destination_file"
   exit 1;
}
 
test $# == 2 || usage
orig_size=$(stat -c %s $1)
 
>$2
dest_size=0
cp -f $1 $2 &
 
while [ $orig_size -gt $dest_size ] ; do
   dest_size=$(stat -c %s $2)
   pct=$((( 69 * $dest_size ) / $orig_size ))
 
    echo -en "\r["
    for j in `seq 1 $pct`; do
        echo -n "="
    done
    echo -n ">"
    for j in `seq $pct 68`; do
        echo -n "."
    done
    echo -n "] "
    echo -n $((( 100 * $pct ) / 69 ))
    echo -n "%"
done
echo

[edit] How to set a prompt to use current folder

export PS1="`whoami`@`hostname`"'($PWD)# '

[edit] How to set display variable automagically in .profile

D1=`who am i| awk '{print substr($6,2,(length($6)-2))}'`
 
if [ -z "$D1" ]
then
        D1=`hostname`
fi
 
D2=`echo $D1 | awk '{print index($1,":")}'`
 
if [ $D2 -gt 0 ]
  then
        export DISPLAY=$D1
  else
        #display=`host $D1 | awk '{print $3}'`
        display=$D1
        export DISPLAY=${display}:0.0
        echo "DISPLAY Set to: ${display}:0.0"
  fi

[edit] How to change permissions on all files that have a certain other set of permissions (or type or name)

file . -perm 600 | sed -n "s/\(.*\)/\"\1\"/gp" | xargs chmod 600

[edit] How to pipe gunzip into tar for on the fly unpacking

If your tar supports it run

tar zxvf file.tar.gz

otherwise do

gunzip -c file.tar.gz | tar -xvf -

[edit] How to set default shell to bash in Makefiles under Ubuntu

Add the following at the top of each file:

SHELL = /bin/bash

[edit] How to get the PID of an X window

Run the following verbatim

xprop _NET_WM_PID

then click on the window you're interested in. To list all processes with windows do this:

wmctrl -pl | awk '{print $3}' | xargs ps -p

To list all windows on the screen

xlsclients -al

To get info on a window id (0x12345)

xprop -id 0x12345