1. Install needed packages

Sun Java JDK:

Downloaded form: http://java.sun.com/
For example, it is installed into /usr/java/jdk1.6.0_16 which may different depending on jdk‘s version.

Install JDK on ALL nodes of the Hadoop cluster.

Hadoop:

Downloaded from: http://hadoop.apache.org/

Unpack it to somewhere. We store it in /lhome/zma/hadoop in this example

For our convenience that we needn’t to specify the full path when calling hadoop:

$ export PATH=/lhome/zma/hadoop-0.20.2/bin/:$PATH

or

$ ln -s /lhome/zma/hadoop-0.20.2/bin/* /home/zma/bin/

2. Hosts files

Add lines like this to /etc/hosts on ALL nodes:

We use four machines here.

10.0.0.103  vm103
10.0.0.104  vm104
10.0.0.105  vm105
10.0.0.106  vm106

These host’s name will automatically change to the name described in the hosts file after rebooting.

Configure iptables:

We can configure iptables to allow all connections if these nodes are in a secure local area network by this command on all nodes:

# iptables -F
# service iptables save

3. Password-less SSH login

More details about password-less ssh login can be found here.

Generate private and public key:

On vm103 the master node:

$ ssh-keygen -t rsa
$ chmod 755 .ssh
$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
$ chmod 755 .ssh/authorized_keys

Copy .ssh directory to all nodes’ home directories:

$ for ((i=103; i<=106; i++)); do scp -r .ssh vm$i:~/; done;

Try to ssh to all these nodes:

$ for ((i=103; i<=106; i++)); do ssh vm$i ; done;

4. Hadoop Configurations

hadoop/conf/hadoop-env.h

Add or change these lines depending on your configuration:

export JAVA_HOME=/usr/java/jdk1.6.0_16
export HADOOP_HOME=/lhome/zma/hadoop-0.20.2

hadoop/conf/core-site.xml

Add these lines:
Here the namenode is vm103.

  <property>
    <name>fs.default.name</name>
    <value>hdfs://vm103:9000</value>
  </property>

hadoop/conf/hdfs-site.xml

Add these lines:

  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>

hadoop/conf/mapred-site.xml

Add these lines:
Here jobtracker is vm103.

<property>
    <name>mapred.job.tracker</name>
    <value>vm103:9001</value>
</property>

hadoop/conf/masters

Delete localhost and add:

vm103

hadoop/conf/slaves

Delete localhost and add:

vm104
vm105
vm106

5. Duplicate Hadoop to all nodes

Here we are on vm103:

$ for ((i=104; i<=106; i++)); \
do scp -r /lhome/zma/hadoop vm$i:/lhome/zma/ ; \
done;

6. Start Hadoop

We need to start both the HDFS and MapReduce to start Hadoop.

On vm103 (namenode):

Format a new distributed-filesystem HDFS:

$ hadoop namenode -format

Remember to delete HDFS’s local files on all nodes before reformat it:

$ rm /lhome/zma/hadoop/logs /tmp/hadoop-zma -rf

Start HDFS:

$ start-dfs.sh

Check the HDFS status:

$ hadoop dfsadmin -report

There may be less nodes listed in the report than we actually have. We can try it again.

Start mpared:

On vm103 (jobtracker):

$ start-mapred.sh

Check job status:

$ hadoop job -list

7. Enjoy the fun now

A simple example

Copy the input files into the distributed filesystem:

$ hadoop fs -put hadoop-0.20.2/conf input

Run some of the examples provided:

$ cd hadoop-0.20.2
$ hadoop jar hadoop-0.20.2-examples.jar grep input output 'dfs[a-z.]+'

Examine the output files:

Copy the output files from the distributed filesystem to the local
filesytem and examine them:

$ hadoop dfs -get output output
$ cat output/*

or

View the output files on the distributed filesystem:

$ hadoop dfs -cat output/*

8. Shut down Hadoop cluster

$ stop-dfs.sh
$ stop-mapred.sh

or simply stop bothe DFS and MapReduce in one command:

$ stop-all.sh

References

[1] http://hadoop.apache.org/common/docs/current/quickstart.html

Updated history
Feb. 23, 2010. Format this post.
May. 1, 2010. Change the part that copy .ssh directory to all machines.
Jun. 30, 2010. Format the post.
Jul. 13, 2010. Revise the article. The jobtracker and namenode can be same master machine.
Jul. 14, 2010. Add iptables configuration commands.

• mrcc – A Distributed C Compiler System on MapReduce
• Vim Formatting C/C++ Code
• Generate a mixed source and assembly list using gcc

A C++/Java programmer’s introduction to Objective Caml

• Java与C++在语言方面上的不同
• Vim Formatting C/C++ Code
• The C Programming Style that I Follow

截图一张：

• A C++/Java programmer’s introduction to Objective Caml
• SetProxy 一个设置IE代理的小工具
• DebugTerm – telnet二进制显示与发送调试程序

//驱动程序名
String driverName=”com.mysql.jdbc.Driver”;

//数据库用户名
String userName=”username”;

//密码
String userPasswd=”password”;

//数据库名
String dbName=”dbname”;

//表名
String tableName=”tablename”;

//联结字符串
String url=”jdbc:mysql://localhost/”+dbName+”?user=”+userName+”&password=”+userPasswd;

Class.forName(“com.mysql.jdbc.Driver”).newInstance();
Connection conn=DriverManager.getConnection(url);
Statement statement = conn.createStatement();
String sql=”SELECT * FROM “+tableName;
ResultSet rs = statement.executeQuery(sql);

//获得数据结果集合
ResultSetMetaData rmeta = rs.getMetaData();

//确定数据集的列数，亦字段数
int numColumns=rmeta.getColumnCount();

// 输出每一个数据值
while(rs.next()) {
out.print(rs.getString(1)+” “);
out.print(“|”);
out.print(rs.getString(2));
out.print(“
“);
}

out.print(“
“);
out.print(“数据库操作成功，恭喜你”);
rs.close();

statement.close();
conn.close();

• Java中调用本地代码
• Java中控制符的作用范围
• Java与C++在语言方面上的不同

环境:
Microsoft Windows XP [版本 5.1.2600]
(C) 版权所有 1985-2001 Microsoft Corp.

JDK:
C:\Documents and Settings\mzq>java -version
java version “1.6.0_05″
Java(TM) SE Runtime Environment (build 1.6.0_05-b13)
Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode, sharing)

本地代码编译环境:
MS Visual Studio 2008

1) 编写Java代码
示例代码如下:

public class JNITest {
public static void main (String args[]) {
  if (args.length > 0) {
    int x = new Integer(args[0]).intValue();
    int a = new mynative().cubecal(x);
    System.out.println(a);
  }
}
}

class mynative {
public native int cubecal(int x);
static {
  System.loadLibrary("mynative");
}
}

注意mynative类内容即可.

2)生成为本地方法调用的mynative.h头文件

D:\workspace_java\JNI\src>javac JNITest.java

D:\workspace_java\JNI\src>javah -jni mynative

3)编写本地代码
注意:代码扩展名一定要用.c

根据mynative.h编写

#include "jni.h"

int cube(int x) {
return (x*x*x);
}

JNIEXPORT jint JNICALL Java_mynative_cubecal
(JNIEnv *e, jobject o, jint x) {
return cube(x);
}

4)解决依赖关系
将$JAVA_HOME$/include/jni.h $JAVA_HOME$/include/win32/jni_md.h
及mynative.h复制到vs2008 proj目录下

5)编译生成mynative.dll

6)复制mynative.dll到存放Java项目目录中运行即可

D:\workspace_java\JNI\bin 的目录

2008-06-02 14:52 413 mynative.class
2008-06-02 14:52 719 JNITest.class
2008-06-02 15:39 7,168 mynative.dll
3 个文件 8,300 字节
2 个目录 7,816,396,800 可用字节

D:\workspace_java\JNI\bin>java JNITest 9
729

• A C++/Java programmer’s introduction to Objective Caml
• JSP连接MySQL
• Java中控制符的作用范围

同一类中可见 Y Y Y Y Y 同一包中子类可见 N Y Y Y Y 同一包中非子类可见 N Y Y Y N 不同包中子类可见 N N Y Y Y 不同包中对非子类可见 N N N Y N

Read more:

• A C++/Java programmer’s introduction to Objective Caml
• JSP连接MySQL
• Java中调用本地代码

基本数据类型的长度和范围：

• A C++/Java programmer’s introduction to Objective Caml
• JSP连接MySQL
• Java中调用本地代码

Java抛弃的C++中的这些机制和结构多数都是“危险”的，减少了语言的复杂性，增强了安全性，虽然在一定程度上减少了其灵活性。

• A C++/Java programmer’s introduction to Objective Caml
• Vim Formatting C/C++ Code
• The C Programming Style that I Follow

相应设置方法如下：

在/etc/profile.d/下建立一脚本java.sh用来设置环境变量。
在脚本中export这三个环境变量

下面列出我的java.sh，不同版本设置方法类似。

#/etc/profile.d/java.sh — set java environment

export JAVA_HOME=/usr/java/jdk1.6.0_04
export CLASSPATH=.:/usr/java/jdk1.6.0_04/lib:/usr/java/jdk1.6.0_04/jre/lib
export PATH=/usr/java/jdk1.6.0_04/bin:/usr/java/jdk1.6.0_04/jre/bin:$PATH

之后分配java.sh权限：
chmod 755 /etc/profile.d/java.sh

手动运行一次java.sh或重新启动linux后就可以使用了。
运行 java -version 如果正常显示版本信息就OK了。

类似如下：

[mzq@mzq ~]$ java -version
java version “1.6.0_04″
Java(TM) SE Runtime Environment (build 1.6.0_04-b12)
Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode, sharing)

• A C++/Java programmer’s introduction to Objective Caml
• Setting up eCryptFS in Linux
• Add more loop device on Linux

设置如下：

[root@localhost fonts]# mkdir fallback
[root@localhost fonts]# cd fallback/
[root@localhost fallback]# pwd
/usr/java/jdk1.6.0_05/jre/lib/fonts/fallback
[root@localhost fallback]# ln -s /usr/share/fonts/zh_CN/TrueType/zysong.ttf

• A C++/Java programmer’s introduction to Objective Caml
• Setting up eCryptFS in Linux
• Speeding up Firefox on Linux