Thursday, December 13, 2012

How to add custom infowindow for Google MAP API V3

InfoWindow is a commonly used feature for Google Map. When users click a marker, a popup window can be displayed to provide more information. Although google map API provides easy-to-use default InfoWindow feature, it is often much cooler to have custom InfoWindow, which will make your application unique. I will introduce two tutorials with simple examples to demonstrate how to develop custom infowindow on Google MAP API V3.

1) Multi-Tab InfoWindow
Multi-Tab InfoWindow is supported in Google MAP API V2, but not in V3. It is useful if you have a lot of information to display. 

The package InfoBubble can be used to implement multi-tab infowindow in V3. Here is a very good and easy-to-understand tutorial.

2) Customized InfoWindow Style
If you want to develop an InfoWindow with very different styles, you can use the Elabel package, which allows developers to flexibly control everything. You can design your own window style and actions. Here is the tutorial.


Friday, May 13, 2011

SOAP vs. REST

These two are both popular for Web Services.
Here we assume that readers already have basic knowledge for these two technology.
The differences are:

  1. SOAP is a protocol over HTTP. REST just directly uses HTTP operations. 
  2. SOAP is a protocol. It's a standard, thus all the SOAP services follow this standard/protocol. REST is not a protocol. Instead, REST is an architecture. Therefore, REST services may have large varieties. 
  3. REST is a light-weight web service. It does not have service description and discovery mechanism like SOAP does. (SOAP has WSDL for service description and UDDI for service discovery.
Some quick summary for REST web services:
REST is REpresentational Stateful Transfer service. It refers each object with a URL and uses HTTP operations to manipulate the object. "Post" operation is used for creating the object. "Put" is used to update the object. "Get" is for retrieving the object and "Delete" is for deleting the object.  

Friday, April 8, 2011

Hadoop cluster's memory usage

I run the TeraSort application over my Hadoop cluster, but get the exception:

java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory

This is caused by that Map/Reduce task can not get sufficient memory. In my cluster, each node only has 1GB memory. I did research on Hadoop cluster's memory requirement. Here is the summary.


By default, datanode takes 1000MB RAM, tasktracker takes 1000MB and each task(map or reduce) takes 200MB RAM.
By default, maximumly 2 map tasks and 2 reduce tasks can concurrently run on a single node.
Hence, it’s better to have 1000+1000+2*200+2*200=2800MB RAM on a worker node.

We can increase or reduce Hadoop cluster's heap size by modifying conf/hadoop-env.sh The variable is HADOOP_HEAPSIZE. 

But we first need to make sure that we have enough physical RAM on the worker node. The recommend minimum RAM requirement is 2GB. 

Reference: 
Hadoop: The Definitive Guide


The whole exception stack is:

11/04/08 15:33:25 INFO mapred.JobClient:  map 100% reduce 5%
11/04/08 15:33:32 INFO mapred.JobClient:  map 100% reduce 8%
11/04/08 15:33:53 INFO mapred.JobClient:  map 100% reduce 9%
11/04/08 15:33:56 INFO mapred.JobClient:  map 100% reduce 10%
11/04/08 15:34:05 INFO mapred.JobClient: Task Id : attempt_201104081512_0004_r_000000_0, Status : FAILED
java.io.IOException: Task: attempt_201104081512_0004_r_000000_0 - The reduce copier failed
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
            at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
            at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
            at org.apache.hadoop.util.Shell.run(Shell.java:134)
            at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
            at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
            at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
            at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2537)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2501)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
            at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
            at java.lang.ProcessImpl.start(ProcessImpl.java:81)
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
            ... 8 more

attempt_201104081512_0004_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.ReduceTask).
attempt_201104081512_0004_r_000000_0: log4j:WARN Please initialize the log4j system properly.
attempt_201104081512_0004_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.ReduceTask).
attempt_201104081512_0004_r_000000_0: log4j:WARN Please initialize the log4j system properly.

Saturday, April 2, 2011

How to make Tomcat automatically start when system boots

By default, Tomcat server won't automatically start when system boots in most hosting services (actually, by default Tomcat is not installed in most cases.) This means your website is not available if for any reason the machine restarts.

To make Tomcat automatically start when system boots, you can add Tomcat as a service with the following steps (Assume you have Linux machine):

Step 1: add a script using command "sudo vi /etc/init.d/tomcat"

paste the following content into the file:

# Tomcat auto-start
# description: Auto-starts tomcat
# processname: tomcat
# pidfile: /var/run/tomcat.pid

export JAVA_HOME=/usr/lib/jvm/java-6-sun

case $1 in
start)
sh /usr/local/tomcat/bin/startup.sh
;;
stop)
sh /usr/local/tomcat/bin/shutdown.sh
;;
restart)
sh /usr/local/tomcat/bin/shutdown.sh
sh /usr/local/tomcat/bin/startup.sh
;;
esac
exit 0

In this example, we assume the java path is /usr/lib/jvm/java-6-sun and tomcat path is /usr/local/tomcat.

Step 2: make the script executable by running the chmod command:

sudo chmod 755 /etc/init.d/tomcat

Step 3: link this script to the startup folders by running the following commands

sudo ln -s /etc/init.d/tomcat /etc/rc1.d/K99tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc2.d/S99tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc3.d/K04tomcat
sudo ln -s /etc/init.d/tomcat /etc/rc3.d/S98tomcat
chkconfig --level 2345 tomcat on

Now, Tomcat has been set as auto-starting for system restart.


Tomcat: The Definitive Guide

Professional Apache Tomcat 6

Friday, April 1, 2011

Use shell script to change password for big cluster

I'm building a hadoop cluster with 50 nodes and I need to create a hadoop user and setup password on each node in the cluster.
If Ubuntu is running on the nodes of the cluster, I can do this in the shell script like this:

for i in `nodelist`; do
ssh $i /usr/sbin/groupadd hadoop
ssh $i /usr/sbin/useradd -g hadoop hadoop
ssh $i echo "hadoop" | passwd --stdin hadoop 
done
where nodelist is a text file and all the cluster nodes are listed there.

However, this does not work for Redhat Linux 6. You can not first do "ssh $i" to log on the remote machine and then do "echo 'hadoop' | passwd --stdin hadoop". What you can do is to create two script files. File A just covers adding user and changing password, script file B handles copying file A to each node on the cluster and run file A locally on each node.

For example, my file A looks like:

/usr/sbin/groupadd hadoop
 /usr/sbin/useradd -g hadoop hadoop
echo "hadoop" | passwd --stdin hadoop 

my file B looks like:
for i in `nodelist`; do
scp fileA $i:./
ssh $i ./fileA
done

Thus, I just need to create file A on the master node run fileB from there.






Hadoop: The Definitive Guide

Linux Command Line and Shell Scripting Bible

Hadoop in Action