[综合]Apache Hadoop 2.2.0集群安装(1)[翻译]

标签: apache hadoop 集群 | 发表时间:2013-11-21 12:27 | 作者:
出处:http://www.iteye.com

用途

此文档描述了如何安装、配置和维护一个重大集群从几个节点到上千节点。

初次接触hadoop建议先从 单节点集群开始。

 

前提

从Apache 上下载了稳定的版本。

 

安装

安装hadoop集群通常需要在所有的节点上解压软件或者prm安装。

通常集群中的某一个节点被当做NameNode,其他节点作为ResourceManager,这些是主控节点。其他节点被当做DataNode和NodeManager,这些是从节点。

 

非安全模式启动Hadoop

接下来的章节将会阐述如何配置hadoop集群。

配置文件

hadoop中的配置文件有两大类型:

只读型默认配置: core-default.xmlhdfs-default.xmlyarn-default.xml and  mapred-default.xml

定制化配置:conf/core-site.xml, conf/hdfs-site.xml, conf/yarn-site.xml and conf/mapred-site.xml.

此外:你可以自己操作hadoop的脚本,在bin目录下可以找到,还有一些配置的环境变量在conf/hadoop-env.sh and yarn-env.sh中。

站点配置:

配置hadoop集群你首先要配置hadoop守护进程执行的环境。

hadoop的守护进程包括NameNode/DataNode and ResourceManager/NodeManager.

hadoop守护进程环境配置

管理员需要使用conf/hadoop-env.sh and conf/yarn-env.sh脚本对hadoop守护进程做环境配置。

首先你要验证JAVA_HOME在所有的节点上是否正确

有时候你需要  HADOOP_PID_DIR and  HADOOP_SECURE_DN_PID_DIR目录只能被启动守护进程的用户执行写操作。否则就会出现软连接攻击。

管理员可以利用配置项单独配置进程,配置项如下:

Daemon Environment Variable
NameNode HADOOP_NAMENODE_OPTS
DataNode HADOOP_DATANODE_OPTS
Secondary NameNode HADOOP_SECONDARYNAMENODE_OPTS
ResourceManager YARN_RESOURCEMANAGER_OPTS
NodeManager YARN_NODEMANAGER_OPTS
WebAppProxy YARN_PROXYSERVER_OPTS
Map Reduce Job History Server HADOOP_JOB_HISTORYSERVER_OPTS

如要配置Namenode 为parallelGC,那么可以添加如下到hadoop-env.sh中:

 

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"

其他有用的可定制化参数包括:

 

HADOOP_LOG_DIR /  YARN_LOG_DIR :进程日志目录,如果不存在会自动创建。

HADOOP_HEAPSIZE /  YARN_HEAPSIZE:内存堆大小默认单位为M,如果变量设置成1000 那么堆内存会设置成1000M,默认为1000,如果你需要配置他那么你可以为每个节点单独配置。

 

Daemon Environment Variable
ResourceManager YARN_RESOURCEMANAGER_HEAPSIZE
NodeManager YARN_NODEMANAGER_HEAPSIZE
WebAppProxy YARN_PROXYSERVER_HEAPSIZE
Map Reduce Job History Server HADOOP_JOB_HISTORYSERVER_HEAPSIZE

hadoop守护进程非安全模式配置:

此章节是比较重要的参数配置,涉及信息如下:

conf/core-site.xml

Parameter Value Notes
fs.defaultFS NameNode URI hdfs://host:port/
io.file.buffer.size 131072 SequenceFiles的读/写缓冲区大小

conf/hdfs-site.xml

NameNode的配置:

Parameter Value Notes
dfs.namenode.name.dir Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.
dfs.namenode.hosts / dfs.namenode.hosts.exclude List of permitted/excluded DataNodes. If necessary, use these files to control the list of allowable datanodes.
dfs.blocksize 268435456 HDFS blocksize of 256MB for large file-systems.
dfs.namenode.handler.count 100 More NameNode server threads to handle RPCs from large number of DataNodes.

DataNode配置:

Parameter Value Notes
dfs.datanode.data.dir Comma separated list of paths on the local filesystem of a  DataNode where it should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices.

 

conf/yarn-site.xml

ResourceManager和NodeManager配置:

Parameter Value Notes
yarn.acl.enable true / false Enable ACLs? Defaults to  false.
yarn.admin.acl Admin ACL ACL to set admins on the cluster. ACLs are of for  comma-separated-users space comma-separated-groups. Defaults to special value of  * which means  anyone. Special value of just  space means no one has access.
yarn.log-aggregation-enable false Configuration to enable or disable log aggregation

ResourceManager配置:

Parameter Value Notes
yarn.resourcemanager.address ResourceManager host:port for clients to submit jobs. host:port
yarn.resourcemanager.scheduler.address ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. host:port
yarn.resourcemanager.resource-tracker.address ResourceManager host:port for NodeManagers. host:port
yarn.resourcemanager.admin.address ResourceManager host:port for administrative commands. host:port
yarn.resourcemanager.webapp.address ResourceManager web-ui host:port. host:port
yarn.resourcemanager.scheduler.class ResourceManager Scheduler class. CapacityScheduler (recommended),  FairScheduler(also recommended), or  FifoScheduler
yarn.scheduler.minimum-allocation-mb Minimum limit of memory to allocate to each container request at the  Resource Manager. In MBs
yarn.scheduler.maximum-allocation-mb Maximum limit of memory to allocate to each container request at the  Resource Manager. In MBs
yarn.resourcemanager.nodes.include-path / yarn.resourcemanager.nodes.exclude-path List of permitted/excluded NodeManagers. If necessary, use these files to control the list of allowable NodeManagers.

NodeManager配置:

 

Parameter Value Notes
yarn.nodemanager.resource.memory-mb Resource i.e. available physical memory, in MB, for given NodeManager Defines total available resources on the  NodeManager to be made available to running containers
yarn.nodemanager.vmem-pmem-ratio Maximum ratio by which virtual memory usage of tasks may exceed physical memory The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio.
yarn.nodemanager.local-dirs Comma-separated list of paths on the local filesystem where intermediate data is written. Multiple paths help spread disk i/o.
yarn.nodemanager.log-dirs Comma-separated list of paths on the local filesystem where logs are written. Multiple paths help spread disk i/o.
yarn.nodemanager.log.retain-seconds 10800 Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.
yarn.nodemanager.remote-app-log-dir /logs HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled.
yarn.nodemanager.remote-app-log-dir-suffix logs Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled.
yarn.nodemanager.aux-services mapreduce_shuffle Shuffle service that needs to be set for Map Reduce applications.

运行历史配置:

Parameter Value Notes
yarn.log-aggregation.retain-seconds -1 How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.
yarn.log-aggregation.retain-check-interval-seconds -1 Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node.

 

conf/mapred-site.xml

MapReduce应用配置:

Parameter Value Notes
mapreduce.framework.name yarn Execution framework set to Hadoop YARN.
mapreduce.map.memory.mb 1536 Larger resource limit for maps.
mapreduce.map.java.opts -Xmx1024M Larger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb 3072 Larger resource limit for reduces.
mapreduce.reduce.java.opts -Xmx2560M Larger heap-size for child jvms of reduces.
mapreduce.task.io.sort.mb 512 Higher memory-limit while sorting data for efficiency.
mapreduce.task.io.sort.factor 100 More streams merged at once while sorting files.
mapreduce.reduce.shuffle.parallelcopies 50 Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.

MapReduce 执行历史服务配置:

Parameter Value Notes
mapreduce.jobhistory.address MapReduce JobHistory Server  host:port Default port is 10020.
mapreduce.jobhistory.webapp.address MapReduce JobHistory Server Web UI host:port Default port is 19888.
mapreduce.jobhistory.intermediate-done-dir /mr-history/tmp Directory where history files are written by MapReduce jobs.
mapreduce.jobhistory.done-dir /mr-history/done Directory where history files are managed by the MR JobHistory Server.

 

Hadoop机架感知

HDFS和YARN服务可机架感知的

NameNode 和ResourceManager通过调用api来获取集群中每个从节点的机架信息。

api以dns名称(或ip)作为一个机架id

这个模块也是可配置的,通过topology.node.switch.mapping.impl来配置,可以通过命令行参数topology.script.file.name来配置,如果topology.script.file.name没有配置那么默认其ip为机架id。



已有 0 人发表留言,猛击->> 这里<<-参与讨论


ITeye推荐



相关 [apache hadoop 集群] 推荐:

[综合]Apache Hadoop 2.2.0集群安装(2)[翻译]

- - ITeye博客
NodeManager节点健康监控. hadoop提供一个检测一个节点健康状态的机制,那就是管理员可以配置NodeManager去周期性执行一个脚本. 管理员可以在这个脚本中做任何的状态监控从而决定此节点是否健康. 如果某节点不健康了,那么他们会有一个标准的错误输出,NodeManager的脚本周期性检测输出,如果节点输出中包含了ERROR字符串,那么此节点会被上报为 unhealthy ,并且此节点会被ResourceManager放入黑名单.

[综合]Apache Hadoop 2.2.0集群安装(1)[翻译]

- - ITeye博客
此文档描述了如何安装、配置和维护一个重大集群从几个节点到上千节点. 初次接触hadoop建议先从 单节点集群开始. 从Apache 上下载了稳定的版本. 安装hadoop集群通常需要在所有的节点上解压软件或者prm安装. 通常集群中的某一个节点被当做NameNode,其他节点作为ResourceManager,这些是主控节点.

Apache Hadoop 2.0.3发布了

- - 董的博客
Dong | 新浪微博: 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及 版权声明. 网址: http://dongxicheng.org/mapreduce-nextgen/apache-hadoop-2-0-3-published/. Apache Hadoop 2.0.3发布了,在这次版本更新中,主要增加了以下几个特性:.

Apache Sqoop 1.4.3 发布,Hadoop 数据迁移

- - 开源中国社区最新新闻
Sqoop是一个用来将 Hadoop和关系型数据库中的数据相互转移的工具,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导入到Hadoop的HDFS中,也可以将HDFS的数据导入到关系型数据库中. Sqoop 1.4.3 完整的改进记录请看 这里. 下载地址: http://www.apache.org/dyn/closer.cgi/sqoop/.

Apache Hadoop 2.2.0 稳定版发布

- - 开源中国社区最新新闻
Apache Hadoop 2.2.0 稳定版发布了,建议用户升级. 该版本更加稳定,同时在 API 和协议上兼容老的版本. 与 Hadoop 1.x 比较,该版本显著的改进包括:. YARN - A general purpose resource management system for Hadoop to allow MapReduce and other other data processing frameworks and services.

Apache整合Tomcat、集群

- - ITeye博客
Apache 整合 Tomcat 、集群. 1.1     使用mod_proxy整合. 1.2     使用mod_jk整合. 1.3.1    Tomcat集群配置. 1.3.2    mod_proxy进行负载均衡. 1.3.3    mod_jk进行负载均衡.        Apache整合Tomcat主要有两种方式,通过mod_proxy整合和通过mod_jk整合.

Hadoop集群与Hadoop性能优化

- - 学着站在巨人的肩膀上
本文讲解一下Hadoop集群、Hadoop性能优化、Hadoop机架感知实现、Hadoop配置等,下面是有关这些命令的具体介绍. Hadoop性能优化:Hadoop机架感知实现及配置:分布式的集群通常包含非常多的机器,由于受到机架槽位和交换机网口的限制,通常大型的分布式集群都会跨好几个机架,由多个机架上的机器共同组成一个分布式集群.

[hadoop] 搭建自己的hadoop集群

- - CSDN博客系统运维推荐文章
       a>  五台centos6.2虚拟机,配置主机名、IP地址、yum源、.        b>  准备所需要的软件包. 2> 配置我自己的hadoop 集群.       a>  修改5台机器的hosts文件.       b>  配置master无密码登录slave,在master01和master02上执行以下命令:   .

Hadoop 集群基准测试

- - IT瘾-dev
生产环境中,如何对 Hadoop 集群进行 Benchmark Test. 本文将通过 Hadoop 自带的 Benchmark 测试程序:TestDFSIO 和 TeraSort,简单介绍如何进行 Hadoop 的读写 & 计算性能的压测. 回顾上篇文章: 认识多队列网卡中断绑定. (本文使用 2.6.0 的 hadoop 版本进行测试,基准测试被打包在测试程序 JAR 文件中,通过无参调用 bin/hadoop jar ./share/hadoop/mapreduce/xxx.jar 可以得到其列表 ).

[hadoop] 基于Hadoop集群的HBase集群的配置

- - CSDN博客系统运维推荐文章
       a> 已经配置完成的Hadoop集群.        b> 所需要的软件包. 2>  单独安装的ZooKeeper集群,不基于HBase集群管理.        a> 在master01上解压zookeeper-3.4.4.tar.gz.        b> 修改Zookeeper的配置文件.