Working with data feeds is sometimes frustrating when you have over 50 columns and are trying to find the field that is too long for your insert into your database system of choice. I have found processing the files with different Linux utilities useful and one that I regularly use it “awk”. I was trying to get the substring of all fields in a product feed the other day and used the following trick.

awk -F 't' '{for(i = 1;i<NF;i++) printf("%s,",substr($i,0,50));print;};

The for loop iterates over all the fields separated by a tab in each line and the prints out the field with a length of 50 characters or less. The extra print is for the newline at the end of each line.

Tags: awk, Linux

When you are using the Linux tool screen, there are times
that I need to send a ctrl-a character. This is a special command key in
screen that drives most of the other commands. But I use the ctrl-a to go
to the beginning of some of my commands and in order for the terminal to
get a ctrl-a you need to use the following key sequence:

ctrl-a  a

So to explain, you need to hold ‘control’ or ‘ctrl’ key and the ‘a’ key together
and then release the ‘ctrl’ key and press ‘a’ key and it will send a ctrl-a to
the terminal.

Tags: Linux, Utilities

When you have to process log files, it can be very useful to highlight the terms you are seeking. One way to highlight a term is to use grep with the –color option. The drawback to using this method is that if you are chaining the grep command after another command then the last grep is all that would be highlighted.
I found that you could use the following to highlist all terms:

grep --color -e 'search|search2' 

So to break down the command line, first we have the grep command followed by the –color option which (if your terminal allows it) will highlight the search term within the screen results. This means that if you put this into a file, then the color highlighting will be lost. The ‘-e’ option states that the next argument is going to be a regular expression based argument.
The full syntax of regular expressions is beyond the scope of this article but basically is a pattern to match strings. The reason I am using a -e option is to support the ‘|’ option that allows me to search for more than one term. The reason that the pipe character is escaped is that otherwise it would be considered as a character to search for instead of a separator. This also will color either term within the output. So I am searching for search or search2 anywhere within the files that I pass in.

I have found myself using Vim as my Linux editor of choice.  It is not dependent on X11 being available and I can edit pretty much anything fairly quickly.  I have been using a Redhat flavored linux(Redhat,Fedora, Centos) for quite sometime and because I couldn’t guarantee that Emacs would be installed I just got used to using Vi.

When I first attempted to use Vi after using Emacs from my college days, I was initially very frustrated with modes and key sequences.  But after a very short time I found I could edit files very rapidly and make changes quite easily.

Quick key list:

1) The thing that I do quite often is to replace all Ctrl-M (Windows Returns) within a file.  (Yes, I could do this with a call to dos2unix, but this is a Vi list).   I found the following key sequence works quite well.

:%s/Ctrl-v<enter>//g

breaking this sequence down

2) Another sequence I use quite often is to reformat source code.

:1 <enter> <shift-v> <shift>-g =

breaking the sequence down

:1<enter>   – states go to the first line of the file

<shift>-v    – Turns on visual line mode which selects the entire current line

<shift>-g    -Makes the cursor go to the last line in the file and with the last command

selects the entire file

=                   – Tells the editor to reformat the selection area

This page will be updated as I figure out more useful commands. This is only a small amount of the power of VI!

I have had to process data files from customers that have come in as delimited files. The files are usually in CSV formats and the delimiters might change depending on the content of the files. Sometimes the delimiters are usually tabs”t”, pipes “|”, comma “,” or semi-colon “;”. A useful tool for processing this type of file is to use awk.

Example command line:

awk -F 't' '{OFS="|"; if (NR != 1) print $3,$2,$1}'

The ‘-F’ argument is followed by the character that you have the file separated on. The example above uses a tab escape sequence ‘t’ but you could have any character. One potential problem is if the data contains the same character that you are attempting to split on. As in a comma separated list using commas within a long text sequence.
The Second sequence of quoted text above starts the processing of the file. The if statement uses the NF keyword which stands for Number of Row. This tells the script that it should not fire for the first row. The print statement allows you to pick and choose the columns(numbered from 1) to use or allows you to reorder them.
The ‘OFS’ part of the command line that helps with the print statement describes the Output Field Separator. In the case above it is using a pipe character ‘|’ to separate the text as it is being printed out. This allows you comma separate the fields that you want instead of having to use a more sophisticated printf statement.

The input file is of the form

Label,name,address

And the output is going to be of the form:

address,name,Label

When I am writing build automation scripts, there are times that I have to insure that the previous command completed
with no errors. I have found that ERRORLEVEL gets filled in with the return value from a command.

dir %1

if (%ERRORLEVEL%) == (0) echo Found File: %1
if (%ERRORLEVEL%) == (1) echo Missing File: %1

So if you save the previous in a file and run the command with one (1) argument, it will display the output of the dir command
but also print either “Found file: ” or “Missing File: “.

Tags: DOS, scripting

So in 3DSMax it is sometimes useful to insert a background image usually when you are modelling
something and using the background image as a guide. There is no easy way to remove it. You
can reassign another image or set it to not display the background image, but this can lead to
problems if the file is moved or removed as it is still linked to your max file.

MaxScript to the rescue! You can use the MaxScript Listener(F11) to type in the following to
clear the background filename:

backgroundimagefilename=""

This will clear the entry for the filename. I have noticed that you do need to exit 3DSMax to
fully make the change complete.

Tags: 3D, 3DSMax

The other day I had a requirement to fill in a date value while creating an entity in ClearQuest. The field that was required was a date and was formatted as “mm/dd/yyyy hh:MM::SS AM|PM”. I used the localtime function to return the date and used sprintf to format the date to the proper output.
The reason I used sprintf was that for hours, months and days less 10 the output was 1 digit instead of two and the input was not allowed. The %02d states that the digits used will be a minumum of 2 in this case.

Here is the sample program I wrote to test the expected output:

 my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;

# formatted just by concatenating output
my $datestring = $mon . "/" . $mday . "/" . $year . "  " . $hour . ":" . $min.
":" . $sec  ;

# formatted using sprintf to ensure proper digit counts are output
my $datestring2 = sprintf("%02d/%02d/%04d %d:%02d:%02d %s",$mon,$mday,1900 +$yea
r,$hour,$min,$sec, $hour >= 12? "PM" : "AM");

 print $datestring;
  print $datestring2;

Tags: perl, scripting

Recently I was moving data between an old server and new web server. I had some 250 GB of data and thought about just using Secure Copy (scp). When I started the process, I walked away
thinking “No Problem”, but after a few files, I got an error due to file permissions and I did not want to figure out what files had not been copied yet or where to try to restart.
So I used rsync to mirror the drive. I was able to set it up and due to my slow network speed, it still took several days, but I was certain that I had gotten all the data.

rsync -azuv -e ssh  user@originserver:public_html/* public_html/

The options I used were for :

This took some time to run, but I did not have to babysit the command and it duplicated the data from my old server to the new one.

I was writing a batch file the other day and needed to change the argumets to a command based on if the user had specified his password on the command line.

SET PASS=somepassword
: or
:SET PASS=
if x%PASS%= X (
  SET PASSARG=--password somepassword
) else
(
  SET PASSARG=
)

commandtorun %PASSARG%

Now if the user comments out his password, the script will make the PASSARG environment variable be empty or if they set a password the passarg variable will be filled in.

Tags: DOS, scripting