I have been enjoying learning Hadoop and have had to debug issues within a current process job to enhance it for new data points. It is quite the challenge to debug a distributed system but I have found two ways to get some meaningful input from the system. One way is using Counters and the other is to use MultipleOutputs to capture output from errors(This will be in a later post). This article will show how counters can be used.

The Counter method allows you to specify a set of enum values to specify the counter name:

public enum Counters {
  CHOCOLATE,
  VANILLA,
  MINT_CHOCO_CHIP
}

This will be used as the identifier for the counter that is in use. The same code can be used within the Mapper or the reducer depending on where you are looking for the counts. The difference is where in the final output of the Hadoop process that the counts show up.

if (flavor.contains("CHOCOLATE")) {
    context.getCounter(Counters.CHOCOLATE).increment(1);
}
if (flavor.contains("VANILLA")) {
    context.getCounter(Counters.VANILLA).increment(1);
}
if (flavor.contains("MINT_CHOCO_CHIP")) {
    context.getCounter(Counters.MINT_CHOCO_CHIP).increment(1);
}

This way you will find a line for each counter that was incremented during the Hadoop process.

I have used it to highlight problems with in the code or just to make sure that a segment of code has been run.

Tags: BigData, hadoop

Variables in Python are declared on first use.
Variable names in Python must:

• Start with a letter or an underscore character
• Not start with a number
• Only contain alpha-numeric characters and underscores (A-Z,a-z, 0-9 and _)
• Be case sensitive (ex. name and Name are different variables)

Good variable names

apple
b_funky
_tester

The underscore as the first character usually indicates that the function is for internal use. This convention is mentioned in PEP 8.

Bad variable names

8million
$dog
+happy

The scope of a Python variable depends where it is defined.

If it defined outside of any function or class, then that variable is global. This means that all functions can access that variable.

some_var= "Eric"

If it is defined within a class or function, than that variable is considered to have local scope.

def print_name(name):
    greeting = "Good Morning,"
    print(greeting,name)

The variable some_var is global and can be used throughout the program, but the variables name and greeting are local to the function and are not available outside of the function.

Tags: Python

If you want to get your hands on Crypto Currency but are not willing to put up cash up front, there are a collection of sites out there that are called faucets. These faucets “drip” crypto currencies. This will not make you rich today but it will give you a foothold into the crypto currency domain.

Tags: bitcoin, cryptocurrency, dogecoin, litecoin

When I develop in the web, there are plenty of times that I want to see information about where I am in the program or what is going on in the Javascript when I load the page. I use the Console to watch updates but it can be difficult to read when you have possibly hundreds of log statements.
Usually you can just use the standard console.log function:

> console.log("Some information");
Some information
or 

> console.debug("Some information");
Some information

or 

> console.info("Some information");
Some information

You also get the line number and file that the output comes from in the console.

But did you know that you can use the following:

console.clear()
console.error(object)
console.table(array)
console.warn()

Let’s say you want to clear the current console. You can place in your code, or use it within the console itself.

console.clear();

Or you can get color coded lines with the warn or error functions:

> console.warn("This is a warning");

> console.error("Error: Danger");

And if you need to display an array that and want a better layout use console.table():
> console.table(['a','b','c']);

I was talking to a friend of mine on the phone the other day and we were talking about how debugging in Chrome was sometimes frustrating. We both had some ideas on how that could have been made better for each of us.

While you can load your page, find the line of code, and add a set of breakpoints. He let me know kicking off the debugger in a certain part of your code just by adding the following line of code where you want the code to stop:

debugger;

Once this brings up the debugging window, this behaves like a breakpoint and stops execution. Now you can see the value of your variables, or you can step through the rest of your code line by line.

My next post will talk about the different kind of console logging functions that are availabole.

There are times you might want to dynamically generate elements on a page and add classes to style them. You can create elements with ids and/or classes with plain Javascript with the following commands:

var newElement = document.createElement(“div”);
//to add a class:
newElement.setAttribute(“class”, “card”);
//to add an id
newElement.setAttribute(“id”, “id_name”);
//to add it to the body:
document.body.appendChild(newElement);
// or to add it to some element:
document.getElementById(“card-box”).appendChild(newElement);

You could also create the string for an element and attach it to the content of a div.

var newElement = '<div id="id_name" class="card">Some Content</div>';
 // This will replace the contents with your new element. 
document.body.innerHTML = newElement;

I was editing the ~/.vagrant.d/boxes//metadata_url file in a box I was testing and when I ran “vagrant box update” I received the following:

$ vagrant box update
==> default: Checking for updates to 'edev-developer'
    default: Latest installed version: 2.0.5
    default: Version constraints: 
    default: Provider: virtualbox
There was an error while downloading the metadata for this box.
The error message is shown below:

Illegal characters found in URL

The illegal character happens to be an EOL char added automatically by your editor, mine was vi, and that is causing the error. I used the following commands to remove the eol character from the file.

perl -pe 'chomp if eof' .vagrant.d/boxes//metadata_url > .vagrant.d/boxes//metadata_url2
mv ~/.vagrant.d/boxes//metadata_url2 ~/.vagrant.d/boxes//metadata_url

I regularly have to test internal systems by adding an entry to the hosts file on my computer. As I keep quite a few tabs open this can cause issues when I need to use the original host system to do some work as I do not want to close all of my tabs to restart Chrome. And when I finish my testing and remove the entry from the hosts file, I need to restart the browser to make sure that I do not have the test DNS entry for my host.

IF you place the following in the address bar:

chrome://net-internals/#dns

You will get a page for DNS that has the following button:

This will clear the browser DNS cache and you can use the list below to see the current state of DNS entries below to verify your DNS is as expected.

To install PostgreSQL in ubuntu

Add the repo for postgres to the apt repo list:

 sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
 sudo wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -

To install the postgres database packages:

 sudo apt-get update
 sudo apt-get install postgresql postgresql-contrib

So you have found yourself in vi or vim and are unable to exit. The following options are available.

First you need to get to command mode, so either hit or Ctrl-c will switch to command mode from insert mode.
Then use the following:
Exit if there are no changes to the current buffer without writing

:q

Exit even though there are changes to the buffer without writing

:q!

Write and exit ( those are capital Zs)

ZZ

Exit and do not write

ZQ