Working with data feeds is sometimes frustrating when you have over 50 columns and are trying to find the field that is too long for your insert into your database system of choice. I have found processing the files with different Linux utilities useful and one that I regularly use it “awk”. I was trying to get the substring of all fields in a product feed the other day and used the following trick.

awk -F 't' '{for(i = 1;i<NF;i++) printf("%s,",substr($i,0,50));print;};

The for loop iterates over all the fields separated by a tab in each line and the prints out the field with a length of 50 characters or less. The extra print is for the newline at the end of each line.