Hadoop入门-WordCount示例
WordCount的过程如图，这里记录下入门的过程，虽然有很多地方理解的只是皮毛。
hadoop的安装
安装比较简单，安装完成后进行单机环境的配置。
hadoop-env.sh:指定JAVA_HOME。
# The only required environment variable is JAVA_HOME. All others are# optional. When running a distributed configuration it is best to# set JAVA_HOME in this file, so that it is correctly defined on# remote nodes.# The java implementation to use.export JAVA_HOME="$(/usr/libexec/java_home)"
core-site.xml：设置Hadoop使用的临时目录，NameNode的地址。
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/Cellar/hadoop/hdfs/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property></configuration>
hdfs-site.xml：一个节点，副本个数设为1。
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>
mapred-site.xml:指定JobTracker的地址。
<configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9010</value> </property></configuration>
启动Hadoop相关的所有进程。
➜ sbin git:(master) ./start-all.shThis script is Deprecated. Instead use start-dfs.sh and start-yarn.sh16/12/03 19:32:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableStarting namenodes on [localhost]Password:localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-namenode-vonzhoudeMacBook-Pro.local.outPassword:localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-datanode-vonzhoudeMacBook-Pro.local.outStarting secondary namenodes [0.0.0.0]Password:0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-secondarynamenode-vonzhoudeMacBook-Pro.local.out16/12/03 19:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablestarting yarn daemonsstarting resourcemanager, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/yarn-vonzhou-resourcemanager-vonzhoudeMacBook-Pro.local.outPassword:localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/yarn-vonzhou-nodemanager-vonzhoudeMacBook-Pro.local.out
（可以配置ssh无密码登录方式，否则启动hadoop的时候总是要密码。）
看看启动了哪些组件。
➜ sbin git:(master) jps -l5713 org.apache.hadoop.hdfs.server.namenode.NameNode6145 org.apache.hadoop.yarn.server.nodemanager.NodeManager6044 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager5806 org.apache.hadoop.hdfs.server.datanode.DataNode5918 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
访问 http:// localhost:50070/ 可以看到DFS的一些状态。
WordCount 单词计数
WordCount就是Hadoop学习的hello world，代码如下：
public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setJarByClass(WordCount.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); /** * 设置一个本地combine,可以极大的消除本节点重复单词的计数,减小网络传输的开销 */ job.setCombinerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); }}
构造两个文本文件, 把本地的两个文件拷贝到HDFS中：
➜ hadoop-examples git:(master) ✗ ln /usr/local/Cellar/hadoop/2.7.1/bin/hadoop hadoop➜ hadoop-examples git:(master) ✗ ./hadoop dfs -put wordcount-input/file* inputDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.16/12/03 23:17:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable➜ hadoop-examples git:(master) ✗ ./hadoop dfs -ls input/ DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.16/12/03 23:21:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 2 items-rw-r--r-- 1 vonzhou supergroup 42 2016-12-03 23:17 input/file1-rw-r--r-- 1 vonzhou supergroup 43 2016-12-03 23:17 input/file2
编译程序得到jar:
mvn clean package
运行程序（指定main class的时候需要全包名限定）：
➜ hadoop-examples git:(master) ✗ ./hadoop jar target/hadoop-examples-1.0-SNAPSHOT.jar com.vonzhou.learnhadoop.simple.WordCount input output16/12/03 23:31:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/12/03 23:31:20 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id16/12/03 23:31:20 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=16/12/03 23:33:21 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.16/12/03 23:33:21 INFO input.FileInputFormat: Total input paths to process : 216/12/03 23:33:21 INFO mapreduce.JobSubmitter: number of splits:216/12/03 23:33:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local524341653_000116/12/03 23:33:22 INFO mapreduce.Job: The url to track the job: http://localhost:8080/16/12/03 23:33:22 INFO mapreduce.Job: Running job: job_local524341653_000116/12/03 23:33:22 INFO mapred.LocalJobRunner: OutputCommitter set in config null16/12/03 23:33:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 116/12/03 23:33:22 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter16/12/03 23:33:22 INFO mapred.LocalJobRunner: Waiting for map tasks16/12/03 23:33:22 INFO mapred.LocalJobRunner: Starting task: attempt_local524341653_0001_m_000000_016/12/03 23:33:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 116/12/03 23:33:22 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.16/12/03 23:33:22 INFO mapred.Task: Using ResourceCalculatorProcessTree : null16/12/03 23:33:22 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/vonzhou/input/file2:0+4316/12/03 23:33:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)16/12/03 23:33:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 10016/12/03 23:33:22 INFO mapred.MapTask: soft limit at 8388608016/12/03 23:33:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760016/12/03 23:33:22 INFO mapred.MapTask: kvstart = 26214396; length = 655360016/12/03 23:33:22 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer16/12/03 23:33:22 INFO mapred.LocalJobRunner: 16/12/03 23:33:22 INFO mapred.MapTask: Starting flush of map output16/12/03 23:33:22 INFO mapred.MapTask: Spilling map output16/12/03 23:33:22 INFO mapred.MapTask: bufstart = 0; bufend = 71; bufvoid = 10485760016/12/03 23:33:22 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/655360016/12/03 23:33:22 INFO mapred.MapTask: Finished spill 016/12/03 23:33:22 INFO mapred.Task: Task:attempt_local524341653_0001_m_000000_0 is done. And is in the process of committing16/12/03 23:33:22 INFO mapred.LocalJobRunner: map16/12/03 23:33:22 INFO mapred.Task: Task 'attempt_local524341653_0001_m_000000_0' done.16/12/03 23:33:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local524341653_0001_m_000000_016/12/03 23:33:22 INFO mapred.LocalJobRunner: Starting task: attempt_local524341653_0001_m_000001_016/12/03 23:33:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 116/12/03 23:33:22 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.16/12/03 23:33:22 INFO mapred.Task: Using ResourceCalculatorProcessTree : null16/12/03 23:33:22 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/vonzhou/input/file1:0+4216/12/03 23:33:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)16/12/03 23:33:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 10016/12/03 23:33:22 INFO mapred.MapTask: soft limit at 8388608016/12/03 23:33:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760016/12/03 23:33:22 INFO mapred.MapTask: kvstart = 26214396; length = 655360016/12/03 23:33:22 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer16/12/03 23:33:22 INFO mapred.LocalJobRunner: 16/12/03 23:33:22 INFO mapred.MapTask: Starting flush of map output16/12/03 23:33:22 INFO mapred.MapTask: Spilling map output16/12/03 23:33:22 INFO mapred.MapTask: bufstart = 0; bufend = 70; bufvoid = 10485760016/12/03 23:33:22 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/655360016/12/03 23:33:22 INFO mapred.MapTask: Finished spill 016/12/03 23:33:22 INFO mapred.Task: Task:attempt_local524341653_0001_m_000001_0 is done. And is in the process of committing16/12/03 23:33:22 INFO mapred.LocalJobRunner: map16/12/03 23:33:22 INFO mapred.Task: Task 'attempt_local524341653_0001_m_000001_0' done.16/12/03 23:33:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local524341653_0001_m_000001_016/12/03 23:33:22 INFO mapred.LocalJobRunner: map task executor complete.16/12/03 23:33:22 INFO mapred.LocalJobRunner: Waiting for reduce tasks16/12/03 23:33:22 INFO mapred.LocalJobRunner: Starting task: attempt_local524341653_0001_r_000000_016/12/03 23:33:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 116/12/03 23:33:22 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.16/12/03 23:33:22 INFO mapred.Task: Using ResourceCalculatorProcessTree : null16/12/03 23:33:22 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: [email protected]64accbd916/12/03 23:33:23 INFO mapreduce.Job: Job job_local524341653_0001 running in uber mode : false16/12/03 23:33:23 INFO mapreduce.Job: map 100% reduce 0%16/12/03 23:33:53 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=1016/12/03 23:33:53 INFO reduce.EventFetcher: attempt_local524341653_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events16/12/03 23:33:53 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524341653_0001_m_000001_0 decomp: 86 len: 90 to MEMORY16/12/03 23:33:53 INFO reduce.InMemoryMapOutput: Read 86 bytes from map-output for attempt_local524341653_0001_m_000001_016/12/03 23:33:53 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 86, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->8616/12/03 23:33:53 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524341653_0001_m_000000_0 decomp: 87 len: 91 to MEMORY16/12/03 23:33:53 INFO reduce.InMemoryMapOutput: Read 87 bytes from map-output for attempt_local524341653_0001_m_000000_016/12/03 23:33:53 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 2, commitMemory -> 86, usedMemory ->17316/12/03 23:33:53 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning16/12/03 23:33:53 INFO mapred.LocalJobRunner: 2 / 2 copied.16/12/03 23:33:53 INFO reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs16/12/03 23:33:53 INFO mapred.Merger: Merging 2 sorted segments16/12/03 23:33:53 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 162 bytes16/12/03 23:33:53 INFO reduce.MergeManagerImpl: Merged 2 segments, 173 bytes to disk to satisfy reduce memory limit16/12/03 23:33:53 INFO reduce.MergeManagerImpl: Merging 1 files, 175 bytes from disk16/12/03 23:33:53 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce16/12/03 23:33:53 INFO mapred.Merger: Merging 1 sorted segments16/12/03 23:33:53 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 165 bytes16/12/03 23:33:53 INFO mapred.LocalJobRunner: 2 / 2 copied.16/12/03 23:33:53 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords16/12/03 23:33:53 INFO mapred.Task: Task:attempt_local524341653_0001_r_000000_0 is done. And is in the process of committing16/12/03 23:33:53 INFO mapred.LocalJobRunner: 2 / 2 copied.16/12/03 23:33:53 INFO mapred.Task: Task attempt_local524341653_0001_r_000000_0 is allowed to commit now16/12/03 23:33:53 INFO output.FileOutputCommitter: Saved output of task 'attempt_local524341653_0001_r_000000_0' to hdfs://localhost:9000/user/vonzhou/output/_temporary/0/task_local524341653_0001_r_00000016/12/03 23:33:53 INFO mapred.LocalJobRunner: reduce > reduce16/12/03 23:33:53 INFO mapred.Task: Task 'attempt_local524341653_0001_r_000000_0' done.16/12/03 23:33:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local524341653_0001_r_000000_016/12/03 23:33:53 INFO mapred.LocalJobRunner: reduce task executor complete.16/12/03 23:33:54 INFO mapreduce.Job: map 100% reduce 100%16/12/03 23:33:54 INFO mapreduce.Job: Job job_local524341653_0001 completed successfully16/12/03 23:33:54 INFO mapreduce.Job: Counters: 35 File System Counters FILE: Number of bytes read=54188 FILE: Number of bytes written=917564 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=213 HDFS: Number of bytes written=89 HDFS: Number of read operations=22 HDFS: Number of large read operations=0 HDFS: Number of write operations=5 Map-Reduce Framework Map input records=5 Map output records=14 Map output bytes=141 Map output materialized bytes=181 Input split bytes=222 Combine input records=0 Combine output records=0 Reduce input groups=11 Reduce shuffle bytes=181 Reduce input records=14 Reduce output records=11 Spilled Records=28 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=7 Total committed heap usage (bytes)=946864128 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=85 File Output Format Counters Bytes Written=89➜ hadoop-examples git:(master) ✗
查看执行的结果：
➜ hadoop-examples git:(master) ✗ ./hadoop dfs -ls outputDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.16/12/03 23:36:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 2 items-rw-r--r-- 1 vonzhou supergroup 0 2016-12-03 23:33 output/_SUCCESS-rw-r--r-- 1 vonzhou supergroup 89 2016-12-03 23:33 output/part-r-00000➜ hadoop-examples git:(master) ✗ ./hadoop dfs -cat output/part-r-00000DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.16/12/03 23:37:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablebig 1by 1data 1google 1hadoop 2hello 2learning 1papers 1step 2vonzhou 1world 1
大数据时代，Hadoop培训、大数据培训、培训班，就选光环大数据！
大数据培训、人工智能培训、Python培训、大数据培训机构、大数据培训班、数据分析培训、大数据可视化培训，就选光环大数据！光环大数据，聘请专业的大数据领域知名讲师，确保教学的整体质量与教学水准。讲师团及时掌握时代潮流技术，将前沿技能融入教学中，确保学生所学知识顺应时代所需。通过深入浅出、通俗易懂的教学方式，指导学生更快的掌握技能知识，成就上万个高薪就业学子。 更多问题咨询，欢迎点击------>>>>在线客服！
