Hadoop 2.9.0核心配置详解：core-default.xml属性解析
需积分: 12 144 浏览量
更新于2024-09-08
收藏 188KB PDF 举报
"Hadoop 2.9.0的core-default.xml配置文件包含了Hadoop核心组件的关键属性及其默认值。此文件对于理解和优化Hadoop集群的性能至关重要。部分重要的属性已经通过PDF注释进行了标注，方便用户关注和调整。"
在Hadoop 2.9.0版本中，`core-default.xml`是核心配置文件，它定义了Hadoop系统运行所需的基础设置。以下是一些关键属性的详细说明：
1. `hadoop.common.configuration.version`: 这个属性表示配置文件的版本，其默认值为0.23.0。这个信息对于追踪配置文件的变化和升级历史很有用。
2. `hadoop.tmp.dir`: 这个属性定义了一个基础路径，用于创建其他临时目录。默认值为`/tmp/hadoop-${user.name}`，其中`${user.name}`会被替换为当前用户的用户名。这个目录是存放Hadoop运行时产生的临时文件的地方。
3. `io.native.lib.available`: 如果这个属性设为`true`，Hadoop会尝试使用本地库（如bz2和zlib压缩库）来提升性能。如果不需要使用本地库，可以将其设为`false`。
4. `hadoop.http.filter.initializers`: 这个属性定义了一组Filter初始化器的类名列表，每个类都必须继承`org.apache.hadoop.http.FilterInitializer`。这些过滤器会被应用到所有的用户界面JSP和Servlet上，列表顺序决定了过滤器的执行顺序。
5. `hadoop.security.authorization`: 如果设为`true`，Hadoop服务级别的授权将被启用。这用于控制对Hadoop服务的访问权限，确保只有授权用户才能进行操作。
6. `hadoop.security.instrumentation.requires.admin`: 如果这个属性设为`true`，则访问监控Servlet（如JMX、METRICS、CONF、STACKS）需要管理员权限。默认情况下，任何人都可以查看这些信息。
7. `hadoop.security.authentication`: 这个属性决定了Hadoop的安全认证方式，默认为`simple`，即无认证。另一种可能的值是`kerberos`，它支持Kerberos认证，提供更强的安全性。
8. `hadoop.security.group.mapping`: 这个属性指定了用户与组映射的实现类，用于确定用户所属的Hadoop群组。通常，在分布式环境中，这个属性会被设置为一个特定的GroupMappingServiceProvider实现，如LDAP或Windows AD。
以上只是一部分关键属性，`core-default.xml`中还有许多其他属性，涵盖了如日志级别、网络设置、DFS操作限制等多个方面。理解并适当地调整这些属性可以帮助优化Hadoop集群的性能和安全性。在实际部署和管理Hadoop集群时，需要根据具体需求来调整配置，确保系统的高效稳定运行。
name value description
hadoop.common.configuration.version 0.23.0 version of this configuration file
hadoop.tmp.dir /tmp/hadoop-${user.name} A base for other temporary directories.
io.native.lib.available true
Controls whether to use native libraries for bz2 and zlib compression codecs
or not. The property does not control any other native libraries.
hadoop.http.filter.initializers org.apache.hadoop.http.lib.StaticUserWebFilter
A comma separated list of class names. Each class in the list must extend
org.apache.hadoop.http.FilterInitializer. The corresponding Filter will be
initialized. Then, the Filter will be applied to all user facing jsp and servlet
web pages. The ordering of the list defines the ordering of the filters.
hadoop.security.authorization false Is service-level authorization enabled?
hadoop.security.instrumentation.requires.admin false
Indicates if administrator ACLs are required to access instrumentation
servlets (JMX, METRICS, CONF, STACKS).
hadoop.security.authentication simple Possible values are simple (no authentication), and kerberos
hadoop.security.group.mapping org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
Class for user to group mapping (get groups for a given user) for ACL. The
default implementation,
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, will
determine if the Java Native Interface (JNI) is available. If JNI is available
the implementation will use the API within hadoop to resolve a list of groups
for a user. If JNI is not available then the shell implementation,
ShellBasedUnixGroupsMapping, is used. This implementation shells out to
the Linux/Unix environment with the bash -c groups command to resolve a
list of groups for a user.
hadoop.security.dns.interface
The name of the Network Interface from which the service should determine
its host name for Kerberos login. e.g. eth2. In a multi-homed environment,
the setting can be used to affect the _HOST substitution in the service
Kerberos principal. If this configuration value is not set, the service will use
its default hostname as returned by InetAddress.getLocalHost
().getCanonicalHostName(). Most clusters will not require this setting.
hadoop.security.dns.nameserver
The host name or IP address of the name server (DNS) which a service Node
should use to determine its own host name for Kerberos Login. Requires
hadoop.security.dns.interface. Most clusters will not require this setting.
hadoop.security.dns.log-slow-lookups.enabled false
Time name lookups (via SecurityUtil) and log them if they exceed the
configured threshold.
hadoop.security.dns.log-slow-lookups.threshold.ms 1000
If slow lookup logging is enabled, this threshold is used to decide if a lookup
is considered slow enough to be logged.
hadoop.security.groups.cache.secs 300
This is the config controlling the validity of the entries in the cache
containing the user->group mapping. When this duration has expired, then
the implementation of the group mapping provider is invoked to get the
groups of the user and then cached back.
hadoop.security.groups.negative-cache.secs 30
Expiration time for entries in the the negative user-to-group mapping
caching, in seconds. This is useful when invalid users are retrying frequently.
It is suggested to set a small value for this expiration, since a transient error
in group lookup could temporarily lock out a legitimate user. Set this to zero
or negative value to disable negative user-to-group caching.
hadoop.security.groups.cache.warn.after.ms 5000
If looking up a single user to group takes longer than this amount of
milliseconds, we will log a warning message.
hadoop.security.groups.cache.background.reload false
Whether to reload expired user->group mappings using a background thread
pool. If set to true, a pool of
hadoop.security.groups.cache.background.reload.threads is created to update
the cache in the background.
hadoop.security.groups.cache.background.reload.threads 3
Only relevant if hadoop.security.groups.cache.background.reload is true.
Controls the number of concurrent background user->group cache entry
refreshes. Pending refresh requests beyond this value are queued and
processed when a thread is free.
hadoop.security.groups.shell.command.timeout 0s
Used by the ShellBasedUnixGroupsMapping class, this property controls
how long to wait for the underlying shell command that is run to fetch
groups. Expressed in seconds (e.g. 10s, 1m, etc.), if the running command
takes longer than the value configured, the command is aborted and the
groups resolver would return a result of no groups found. A value of 0s
(default) would mean an infinite wait (i.e. wait until the command exits on its
own).
hadoop.security.group.mapping.ldap.connection.timeout.ms 60000
This property is the connection timeout (in milliseconds) for LDAP
operations. If the LDAP provider doesn't establish a connection within the
specified period, it will abort the connect attempt. Non-positive value means
no LDAP connection timeout is specified in which case it waits for the
connection to establish until the underlying network times out.
hadoop.security.group.mapping.ldap.read.timeout.ms 60000
This property is the read timeout (in milliseconds) for LDAP operations. If
the LDAP provider doesn't get a LDAP response within the specified period,
it will abort the read attempt. Non-positive value means no read timeout is
specified in which case it waits for the response infinitely.
hadoop.security.group.mapping.ldap.url
The URL of the LDAP server to use for resolving user groups when using
the LdapGroupsMapping user to group mapping.
hadoop.security.group.mapping.ldap.ssl false Whether or not to use SSL when connecting to the LDAP server.
hadoop.security.group.mapping.ldap.ssl.keystore
File path to the SSL keystore that contains the SSL certificate required by the
LDAP server.
hadoop.security.group.mapping.ldap.ssl.keystore.password.file
The path to a file containing the password of the LDAP SSL keystore. If the
password is not configured in credential providers and the property
hadoop.security.group.mapping.ldap.ssl.keystore.password is not set,
LDAPGroupsMapping reads password from the file. IMPORTANT: This file
should be readable only by the Unix user running the daemons and should be
a local file.
hadoop.security.group.mapping.ldap.ssl.keystore.password
The password of the LDAP SSL keystore. this property name is used as an
alias to get the password from credential providers. If the password can not
be found and hadoop.security.credential.clear-text-fallback is true
LDAPGroupsMapping uses the value of this property for password.
hadoop.security.credential.clear-text-fallback true
true or false to indicate whether or not to fall back to storing credential
password as clear text. The default value is true. This property only works
when the password can't not be found from credential providers.
hadoop.security.credential.provider.path
A comma-separated list of URLs that indicates the type and location of a list
of providers that should be consulted.
hadoop.security.credstore.java-keystore-provider.password-file
The path to a file containing the custom password for all keystores that may
be configured in the provider path.
hadoop.security.group.mapping.ldap.bind.user
The distinguished name of the user to bind as when connecting to the LDAP
server. This may be left blank if the LDAP server supports anonymous binds.
hadoop.security.group.mapping.ldap.bind.password.file
The path to a file containing the password of the bind user. If the password is
not configured in credential providers and the property
hadoop.security.group.mapping.ldap.bind.password is not set,
LDAPGroupsMapping reads password from the file. IMPORTANT: This file
should be readable only by the Unix user running the daemons and should be
a local file.
hadoop.security.group.mapping.ldap.bind.password
The password of the bind user. this property name is used as an alias to get
the password from credential providers. If the password can not be found and
hadoop.security.credential.clear-text-fallback is true LDAPGroupsMapping
uses the value of this property for password.
1/
9
2018/3/15file:///F:/Hadoop/hadoop-2.9.0/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
下载后可阅读完整内容，剩余8页未读，立即下载
2015-07-11 上传
2018-06-02 上传
2018-04-04 上传
2023-05-18 上传
介绍hadoop的core-site.xml、hadoop-env.sh、hdfs-site.xml、mapred-site.xml、 yarn-env.sh、yarn-site.xml文件1200字
2023-05-24 上传
2023-05-24 上传
2023-04-11 上传
2023-05-24 上传
2023-06-03 上传
devalone
- 粉丝: 108
- 资源: 37
我的内容管理
展开
最新资源
- 探索AVL树算法：以Faculdade Senac Porto Alegre实践为例
- 小学语文教学新工具：创新黑板设计解析
- Minecraft服务器管理新插件ServerForms发布
- MATLAB基因网络模型代码实现及开源分享
- 全方位技术项目源码合集：***报名系统
- Phalcon框架实战案例分析
- MATLAB与Python结合实现短期电力负荷预测的DAT300项目解析
- 市场营销教学专用查询装置设计方案
- 随身WiFi高通210 MS8909设备的Root引导文件破解攻略
- 实现服务器端级联：modella与leveldb适配器的应用
- Oracle Linux安装必备依赖包清单与步骤
- Shyer项目：寻找喜欢的聊天伙伴
- MEAN堆栈入门项目： postings-app
- 在线WPS办公功能全接触及应用示例
- 新型带储订盒订书机设计文档
- VB多媒体教学演示系统源代码及技术项目资源大全
