Using a Node script to query a database to do corrections was working fine on my local Windows based system until we tried to automate the process. We moved it to a Jenkins server which happened to be on a Linux system and the script stopped working. After some debugging it turned out that the way we were attempting to differentiate data returns from the queries depended on the line ending.

On a Windows system, the line endings are ‘\r\n’ or carriage return, new line. This had worked well while we were running the script locally, but once it went to the Linux system the line endings were just a ‘\n’ or new line. The line ending was not being matched so this led us to figure out how to tell what platform was being used. The process.platform variable contains the platform name.

The process.platform returns ‘win32’ for Windows, ‘darwin’ for MacOS and ‘linux’ for linux systems. There are other platforms supported but these are all that mattered to us. Using this variable we were able to replace the line endings and the data started working as expected on all our platforms.

The following code snippet sets the lineendings for our sql code:

let platform = process.platform;
let lineendings = '\n';
if (platform == 'win32') {
    lineendings = '\r\n';
}

Tags: Linux, node

To install PostgreSQL in ubuntu

Add the repo for postgres to the apt repo list:

 sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
 sudo wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -

To install the postgres database packages:

 sudo apt-get update
 sudo apt-get install postgresql postgresql-contrib

There is a simple Linux utility for counting characters, lines and words in files called wc. This allows you to give a file as an argument or a stream and get the counts of the aforementioned items.

wc <filename>

or

cat <filename> | wc

The options that wc accepts

The ouput with no arguments will be three numbers (words,lines,characters. Otherwise the the number will reflect the argument you have passed.

And if you pass multiple filenames the output displays counts for all files and totals.

 >wc -l HelloWorld.java sample.txt
    8 HelloWorld.java
   25 sample.txt
   33 total

Tags: Shell, Utilities

Working with data feeds is sometimes frustrating when you have over 50 columns and are trying to find the field that is too long for your insert into your database system of choice. I have found processing the files with different Linux utilities useful and one that I regularly use it “awk”. I was trying to get the substring of all fields in a product feed the other day and used the following trick.

awk -F 't' '{for(i = 1;i<NF;i++) printf("%s,",substr($i,0,50));print;};

The for loop iterates over all the fields separated by a tab in each line and the prints out the field with a length of 50 characters or less. The extra print is for the newline at the end of each line.

Tags: awk, Linux

When you are using the Linux tool screen, there are times
that I need to send a ctrl-a character. This is a special command key in
screen that drives most of the other commands. But I use the ctrl-a to go
to the beginning of some of my commands and in order for the terminal to
get a ctrl-a you need to use the following key sequence:

ctrl-a  a

So to explain, you need to hold ‘control’ or ‘ctrl’ key and the ‘a’ key together
and then release the ‘ctrl’ key and press ‘a’ key and it will send a ctrl-a to
the terminal.

Tags: Linux, Utilities

I have had to process data files from customers that have come in as delimited files. The files are usually in CSV formats and the delimiters might change depending on the content of the files. Sometimes the delimiters are usually tabs”t”, pipes “|”, comma “,” or semi-colon “;”. A useful tool for processing this type of file is to use awk.

Example command line:

awk -F 't' '{OFS="|"; if (NR != 1) print $3,$2,$1}'

The ‘-F’ argument is followed by the character that you have the file separated on. The example above uses a tab escape sequence ‘t’ but you could have any character. One potential problem is if the data contains the same character that you are attempting to split on. As in a comma separated list using commas within a long text sequence.
The Second sequence of quoted text above starts the processing of the file. The if statement uses the NF keyword which stands for Number of Row. This tells the script that it should not fire for the first row. The print statement allows you to pick and choose the columns(numbered from 1) to use or allows you to reorder them.
The ‘OFS’ part of the command line that helps with the print statement describes the Output Field Separator. In the case above it is using a pipe character ‘|’ to separate the text as it is being printed out. This allows you comma separate the fields that you want instead of having to use a more sophisticated printf statement.

The input file is of the form

Label,name,address

And the output is going to be of the form:

address,name,Label

Recently I was moving data between an old server and new web server. I had some 250 GB of data and thought about just using Secure Copy (scp). When I started the process, I walked away
thinking “No Problem”, but after a few files, I got an error due to file permissions and I did not want to figure out what files had not been copied yet or where to try to restart.
So I used rsync to mirror the drive. I was able to set it up and due to my slow network speed, it still took several days, but I was certain that I had gotten all the data.

rsync -azuv -e ssh  user@originserver:public_html/* public_html/

The options I used were for :

This took some time to run, but I did not have to babysit the command and it duplicated the data from my old server to the new one.

I use find and grep quite a bit. I found that I was regularly searching for strings in file and was typing commands like:

find . -exec grep "somestring" {} ; --print

This find command will search from the current directory and for each file will run grep on it and if it finds a matching string, it will print the string and then the filename. So I took this and moved it into a script.

#!/bin/sh

if [ $# -lt 1 ]; then
  echo 1>&2 Usage: $0 ""
  exit 127
fi
icase=
search=

while [ $# -ge 1 ]; do
  case $1 in
     -i) icase=$1;;
      *) search=$1 ;;
  esac
  shift
done

find . -type f -print0 | xargs -0 grep $icase $search

This allows me to run a command such as

ffind -i Super

This command will search all file names starting from the current directory and recurse directories and the “-i” will tell it to ignore case.

Tags: scripting

There are times that I get on a Linux box and want to know about some of the resources available to me. There are tools that can accomplish this, but there are some files that contain basic settings about the resources available to the system. One of those file is:

/proc/cpuinfo

This file contains information about the Processors and you can do a simple:

more /proc/cpuinfo

to see all the details or you can grep for certain fields:

processor       -- This will show you how many processor entries there are.
cpu MHz  or MHz -- This will show you the speed of the processors.
cache size      -- This will show you the cache size available to the processor.
model           -- This will show you the type and model of the processor you are using.

There are quite a few more fields, but these are the ones that I check the most.

Tags: Linux, processor, Utilities

I was asked about system utilization on Linux systems. I found thesystatset of tools which consist of: sar, sadf, mpstat, iostat, nfsiostat, cifsiostat, pidstat and sa tools.

I will talk about the iostat tool in this article and follow up about some of the other tools.

On Ubuntu or debian systems you can install this toolset with:

sudo apt-get install sysstat

Then you will have access to the iostat command. The command output from executing with no options is below:

# iostat
Linux 2.6.32-33-generic (sumo)  09/01/2011      _i686_  (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.11    0.00    0.47    0.01    0.00   99.41

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.54         1.85         8.91    1611385    7740024

This displays the avg-cpu utilization and a device usage list with basic utilization.
Some options you can use are

One more way to use the iostat tool is to specify an interval and count to the command

iostat -d -x 10 5

This will display the output 5 more times with 10 seconds between each run.

Tags: Linux, system tools, utilization