Exclude Sub-Directories while Copying (cp) Files / Directory using tar

I recently had the need to copy a directory to another portion of the file system, but wanted to skip some sub-directories. Normally what one would do is just copy the whole directory and delete the sub-directories that you don’t want after the copy operation has completed (and not make a big deal about it). However that wasn’t feasible this time, because each of these sub-directories had GBs of data and I did not have enough disk space to accomodate those extra files. I went through the cp man page and did a few google searches but did not find anything that would work for me. However I realized that I had just used the “exclude” feature of tar a few days back which lets you skip portions of a directory while tarring and I knew tar could be used to copy over files as well (tar pipe). So thats what I did. Continue reading

Indirectly reference bash variable

If you know the name of a variable which contains a value of interest, you need indirect referencing to retrieve the value, this is how you do it.


j=1 #variable containg a value
i=j #variable containing the name of the variable which holds the value
eval k=\$$i #k now contains the value of j which is 1
echo $k
k=${!i}  #another way of indirectly referencing
echo $k

Bash based Busy Handler for Sqlite

Sqlite is a serverless database and hence doesn’t allow multiple synchronous write requests. Sqlite API for most programming languages have a busy handler functionality which lets you define a busy handler function or a timeout value, thus enabling your program to wait for a certain amount of time and then access the database again.
Continue reading

Bash Tricks: Deal with a Flaky Internet Connection

Its been about 9 months in Vietnam now. One problem that I have to deal with on a day to day basis here is the flaky Internet connection. There are good days and bad days and bad days are incredibly frustrating. The Internet is ON and OFF every five minutes. I wrote this simple script which will wait for the Internet to be active before executing commands given to the script as an argument. I have found this very useful for commands like git pull/push, svn update etc.

PING=$(ping -q -c 1 &>/dev/null; echo $?)
while [ $PING -gt 0 ]; do
  PING=$(ping -q -c 1 &>/dev/null; echo $?)

I call it "wfi", wait-for-internet. Dump this script into your bin directory and give it executable permissions.

To test this script, disconnect your wifi/ethernet connection and then run the following command.

wfi ls -l

Reconnect your wifi/ethernet, and you'll see that the command gets executed as soon as your Internet connection is up.

wfi git pull

Now I can execute these commands through wfi and then go back to my work and be less exasperated.

Bash Tricks: Get the Value of a Variable whose Name is Stored in Another Variable

If want to find out the value of a variable whose name is stored in another variable, you need to dereference the variable. This is how you do it.

Lets say the name of the variable is test123 and it has a value equal to 123. Also the name test123 is stored in another variable varname. To find out the value of test123, you will use the ! dereferencing operator.

@~$ test123=123
@~$ varname=test123
@~$ echo ${!varname}

Took me a while to figure this out.

Bash Tricks: Delete the Last Substring from a Delimited String

I wrote a post about how to print just the last substring of a delimited string. Instead if you’d want to delete just the last substring, and keep rest of the string intact, thats a little trickier.

Here’s how you do it.

@~$ echo "abc:cde:efg" | awk 'BEGIN {FS=ORS=":"} {for(i=1;i<NF;i++) print $i}' | sed 's/$/\n/'

FS is the field separator, while ORS is the output record separator. ORS by default is the new line character \n. What we are doing here is splitting the string using awk, and then printing all but the last records i < NF. The sed statment is just adding a newline at the end.

If you want to skip the trailing delimiter, you could use the code below.

@~$ echo "abc:cde:efg" | awk 'BEGIN {FS=ORS=":"} {for(i=1;i<NF;i++) print $i}' | sed 's/:$/\n/'

Here the sed statement replaces the trailing : with a newline. To skip the newline, you can change the sed statement to sed 's/:$//'.

Bash Tricks: Split / Cut a String with Multi Character Delimiters Using AWK

Some time back I wrote this post showing how to split a string into substrings separated by multi character delimiters. Didn’t realize then that there’s a much easier solution using awk. Using the same example as used in the previous post, here’s the solution.

echo "abcd<>efgh<>ijkl<>mn op<>qr st<>uv wx<>yz" | awk 'BEGIN {FS="<>"} {for(i=1;i<=NF;i++)print $i}'

The delimiter here is "<>".

This will print out all the substrings. If you want an individual substring, you can use something like

echo "abcd<>efgh<>ijkl<>mn op<>qr st<>uv wx<>yz" | awk 'BEGIN {FS="<>"} {print $1}'
echo "abcd<>efgh<>ijkl<>mn op<>qr st<>uv wx<>yz" | awk 'BEGIN {FS="<>"} {print $2}'

Thats how easy it is.