Using a Node script to query a database to do corrections was working fine on my local Windows based system until we tried to automate the process. We moved it to a Jenkins server which happened to be on a Linux system and the script stopped working. After some debugging it turned out that the way we were attempting to differentiate data returns from the queries depended on the line ending.

On a Windows system, the line endings are ‘\r\n’ or carriage return, new line. This had worked well while we were running the script locally, but once it went to the Linux system the line endings were just a ‘\n’ or new line. The line ending was not being matched so this led us to figure out how to tell what platform was being used. The process.platform variable contains the platform name.

The process.platform returns ‘win32’ for Windows, ‘darwin’ for MacOS and ‘linux’ for linux systems. There are other platforms supported but these are all that mattered to us. Using this variable we were able to replace the line endings and the data started working as expected on all our platforms.

The following code snippet sets the lineendings for our sql code:

let platform = process.platform;
let lineendings = '\n';
if (platform == 'win32') {
    lineendings = '\r\n';
}

Working with data feeds is sometimes frustrating when you have over 50 columns and are trying to find the field that is too long for your insert into your database system of choice. I have found processing the files with different Linux utilities useful and one that I regularly use it “awk”. I was trying to get the substring of all fields in a product feed the other day and used the following trick.

awk -F 't' '{for(i = 1;i<NF;i++) printf("%s,",substr($i,0,50));print;};

The for loop iterates over all the fields separated by a tab in each line and the prints out the field with a length of 50 characters or less. The extra print is for the newline at the end of each line.

When you are using the Linux tool screen, there are times
that I need to send a ctrl-a character. This is a special command key in
screen that drives most of the other commands. But I use the ctrl-a to go
to the beginning of some of my commands and in order for the terminal to
get a ctrl-a you need to use the following key sequence:

ctrl-a  a

So to explain, you need to hold ‘control’ or ‘ctrl’ key and the ‘a’ key together
and then release the ‘ctrl’ key and press ‘a’ key and it will send a ctrl-a to
the terminal.

There are times that I get on a Linux box and want to know about some of the resources available to me. There are tools that can accomplish this, but there are some files that contain basic settings about the resources available to the system. One of those file is:

/proc/cpuinfo

This file contains information about the Processors and you can do a simple:

more /proc/cpuinfo

to see all the details or you can grep for certain fields:

processor       -- This will show you how many processor entries there are.
cpu MHz  or MHz -- This will show you the speed of the processors.
cache size      -- This will show you the cache size available to the processor.
model           -- This will show you the type and model of the processor you are using.

There are quite a few more fields, but these are the ones that I check the most.

I was asked about system utilization on Linux systems. I found thesystatset of tools which consist of: sar, sadf, mpstat, iostat, nfsiostat, cifsiostat, pidstat and sa tools.

I will talk about the iostat tool in this article and follow up about some of the other tools.

On Ubuntu or debian systems you can install this toolset with:

sudo apt-get install sysstat

Then you will have access to the iostat command. The command output from executing with no options is below:

# iostat
Linux 2.6.32-33-generic (sumo)  09/01/2011      _i686_  (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.11    0.00    0.47    0.01    0.00   99.41

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.54         1.85         8.91    1611385    7740024

This displays the avg-cpu utilization and a device usage list with basic utilization.
Some options you can use are

One more way to use the iostat tool is to specify an interval and count to the command

iostat -d -x 10 5

This will display the output 5 more times with 10 seconds between each run.

I regularly have to find where a file is located and usually use find:

find . | grep 

Where is replaced with the filename or partial name to match. So I finally decided to write a script to just run this and allow me to print the matching names and even potentially pass the -i flag to grep.

#!/bin/sh


if [ $# -lt 1 ]; then
  echo 1>&2 Usage: $0 ""
  exit 127
fi
icase=

while [ $# -ge 1 ]; do
   case $1 in
     -i)  icase=$1;;
      *)  search=$1 ;;
   esac
   shift
done

find .  | grep $icase $search

This is just a simple script which I have named ffind that searches starting from the current directory. This lets me search with the following commands:

ffind bak$

This will search for all files that end with “bak” .

ffind -i edr

This will search for all files with “edr” regardless of case

So, after a Linux crash that caused my previous ddrescue to fail(after 350GB), I started to look into how to speed it up, because 30+ hours did not sound like a good idea to me.
I look into ways to speed it up and using a larger block size did the trick. I experimented with a few different sizes and figured out that using 2M gave me the best throughput. So instead of a couple of megs a second i was getting over 24 MB/s. So about a 4 1/2 hour recovery after that.

ddrescue -b 2M /dev/sda2 /media/newdisk/olddata /media/newdisk/logfile

The ddrescue process will attempt to recover any lost sectors and retrim the lost data. Luckily, I only lost about 3100k of data so the most of my data is safe.

You can attempt to rerun ddrescue with the -r <# of times to retry> and it will continue to attempt to read those failed sectors that were unrecoverable. There is no guarantee that you will get back all the data.

Once this data is recovered, you can (hopefully) mount this file as a disk under linux.

 mount /media/newdisk/olddata /mnt/newdisk 

Now remember to back up your data and you will not go through the pain that I did to recover this data.

Part 1 of article

So I was trying to do some work the other morning in my hotel room and my Windows 7 computer gave me the dreaded blue screen of death and would not boot past loading a certain library file. I was able to boot into an Ubuntu disk that I happened to have and found that there were many S.M.A.R.T. errors on the disk and wished I had thought to monitor this. So it seemed that my drive was cooked and I could not get any of the data from it.

    I was disappointed that Spinrite was unable to save me from the problems that I was having and I paid 90 bucks for it. So I turned to tools in Linux that might do the job. I had used the dd utility to copy the contents of my drive to a new drive in the past, but dd will exit on the first error. So I found that there is an addition to the dd program called ddrescue which is part of the GNU Project.

This utility will let you copy the contents of a disk or partition and allow you to skip the errors or even retry the error spots a number of times.

The command line is something like:

    ddrescue -r 3 /dev/sda1 /media/newvolume/diskimage.iso /media/newvolume/logfilefordiskimage

The really nice thing is if you have to stop the command for some reason, it will restart from the end of the log file. It does take quite a bit of time. I am 6 1/2 hours into recovering a 500GB disk, but it beats losing all my data. I will follow up this post with the results of the rescue and the next steps I took to recover the data.

Part 2 of article

Sometimes you have a drive that your want to mount under Linux. The mount command can make the drive available to you. But you need to also create a directory to hold the mount point.

mkdir /mnt/drivepoint

Then you can mount the drive to that mount point by using the mount command and giving the device identifier, in this case it is /dev/sda1, and the mount point which we created as /mnt/drivepoint. One thing to note, is that the mount will use whatever directory you specify, so if there are files located in the directory you use as a mount point, they will no longer be visible.

mount /dev/sda1 /mnt/drivepoint

This must be run as root or with the sudo command.

If you have every wanted to set certain IP addresses in your network without depending on a nameserver, you can edit the ‘hosts’ file in your operating system.  This can be done to stop access to any inappropriate sites as well.

The hosts file in Windows is:

%systemroot%System32driversetchosts

This is typically  in:

c:windowsSystem32driversetchosts

On linux this file is:

/etc/hosts

The common layout of the file is

<IP address> hostname [hostname]

So if would look something like this:

127.0.0.1  localhost
192.168.2.222 homeserver homeserver.mydomain.net
127.0.0.1 badlocation

Remember that these lookups happen before looking to a nameserver. So this will override any name lookup. So if you happen to do this on a laptop that you regularly use outside of your home network, make sure that you do not override an address that might be needed when you are not at home.

You will need appropriate privileges to access the file. So you may have to be root or use sudo on Linux and you may have to open an editor as administrator before opening the file on Windows.