SOAP vs. REST

These two are both popular for Web Services.
Here we assume that readers already have basic knowledge for these two technology.
The differences are:

1. SOAP is a protocol over HTTP. REST just directly uses HTTP operations. 
2. SOAP is a protocol. It's a standard, thus all the SOAP services follow this standard/protocol. REST is not a protocol. Instead, REST is an architecture. Therefore, REST services may have large varieties. 
3. REST is a light-weight web service. It does not have service description and discovery mechanism like SOAP does. (SOAP has WSDL for service description and UDDI for service discovery.

Some quick summary for REST web services:

REST is REpresentational Stateful Transfer service. It refers each object with a URL and uses HTTP operations to manipulate the object. "Post" operation is used for creating the object. "Put" is used to update the object. "Get" is for retrieving the object and "Delete" is for deleting the object.  

Hadoop cluster's memory usage

I run the TeraSort application over my Hadoop cluster, but get the exception:

java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory

This is caused by that Map/Reduce task can not get sufficient memory. In my cluster, each node only has 1GB memory. I did research on Hadoop cluster's memory requirement. Here is the summary.

By default, datanode takes 1000MB RAM, tasktracker takes 1000MB and each task(map or reduce) takes 200MB RAM.

By default, maximumly 2 map tasks and 2 reduce tasks can concurrently run on a single node.

Hence, it’s better to have 1000+1000+2*200+2*200=2800MB RAM on a worker node.

We can increase or reduce Hadoop cluster's heap size by modifying conf/hadoop-env.sh The variable is HADOOP_HEAPSIZE. 

But we first need to make sure that we have enough physical RAM on the worker node. The recommend minimum RAM requirement is 2GB. 

Reference: 

The whole exception stack is:

11/04/08 15:33:25 INFO mapred.JobClient:  map 100% reduce 5%

11/04/08 15:33:32 INFO mapred.JobClient:  map 100% reduce 8%

11/04/08 15:33:53 INFO mapred.JobClient:  map 100% reduce 9%

11/04/08 15:33:56 INFO mapred.JobClient:  map 100% reduce 10%

11/04/08 15:34:05 INFO mapred.JobClient: Task Id : attempt_201104081512_0004_r_000000_0, Status : FAILED

java.io.IOException: Task: attempt_201104081512_0004_r_000000_0 - The reduce copier failed

            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)

            at org.apache.hadoop.mapred.Child.main(Child.java:170)

Caused by: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory

            at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)

            at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)

            at org.apache.hadoop.util.Shell.run(Shell.java:134)

            at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)

            at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)

            at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)

            at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160)

            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2537)

            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2501)

Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory

            at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)

            at java.lang.ProcessImpl.start(ProcessImpl.java:81)

            at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)

            ... 8 more

attempt_201104081512_0004_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.ReduceTask).

attempt_201104081512_0004_r_000000_0: log4j:WARN Please initialize the log4j system properly.

attempt_201104081512_0004_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.ReduceTask).

attempt_201104081512_0004_r_000000_0: log4j:WARN Please initialize the log4j system properly.

How to make Tomcat automatically start when system boots

By default, Tomcat server won't automatically start when system boots in most hosting services (actually, by default Tomcat is not installed in most cases.) This means your website is not available if for any reason the machine restarts.

To make Tomcat automatically start when system boots, you can add Tomcat as a service with the following steps (Assume you have Linux machine):

Step 1: add a script using command "sudo vi /etc/init.d/tomcat"

paste the following content into the file:

# Tomcat auto-start
# description: Auto-starts tomcat
# processname: tomcat
# pidfile: /var/run/tomcat.pid

export JAVA_HOME=/usr/lib/jvm/java-6-sun

case $1 in
start)
sh /usr/local/tomcat/bin/startup.sh
;;
stop)
sh /usr/local/tomcat/bin/shutdown.sh
;;
restart)
sh /usr/local/tomcat/bin/shutdown.sh
sh /usr/local/tomcat/bin/startup.sh
;;
esac
exit 0

In this example, we assume the java path is /usr/lib/jvm/java-6-sun and tomcat path is /usr/local/tomcat.

Step 2: make the script executable by running the chmod command:

sudo chmod 755 /etc/init.d/tomcat

Step 3: link this script to the startup folders by running the following commands

sudo ln -s /etc/init.d/tomcat /etc/rc1.d/K99tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc2.d/S99tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc3.d/K04tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc3.d/S98tomcat
chkconfig --level 2345 tomcat on

Now, Tomcat has been set as auto-starting for system restart.

Use shell script to change password for big cluster

I'm building a hadoop cluster with 50 nodes and I need to create a hadoop user and setup password on each node in the cluster.

If Ubuntu is running on the nodes of the cluster, I can do this in the shell script like this:

for i in `nodelist`; do
ssh $i /usr/sbin/groupadd hadoop
ssh $i /usr/sbin/useradd -g hadoop hadoop
ssh $i echo "hadoop" | passwd --stdin hadoop 

done

where nodelist is a text file and all the cluster nodes are listed there.

However, this does not work for Redhat Linux 6. You can not first do "ssh $i" to log on the remote machine and then do "echo 'hadoop' | passwd --stdin hadoop". What you can do is to create two script files. File A just covers adding user and changing password, script file B handles copying file A to each node on the cluster and run file A locally on each node.

For example, my file A looks like:

/usr/sbin/groupadd hadoop

 /usr/sbin/useradd -g hadoop hadoop

echo "hadoop" | passwd --stdin hadoop 

my file B looks like:

for i in `nodelist`; do

scp fileA $i:./

ssh $i ./fileA

done

Thus, I just need to create file A on the master node run fileB from there.