|
|
楼主 |
发表于 2022-11-14 13:52:09
|
显示全部楼层
<?xml version="1.0"?>! s" f( @% b3 M) n+ U
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>. X! ?; I0 M* R: v) i
<!--# G0 H5 k' a! U+ D! D! X' t: U' T# `
Licensed to the Apache Software Foundation (ASF) under one or more% h* T) T' x" Z. Z
contributor license agreements. See the NOTICE file distributed with! ~, n. K( R9 u
this work for additional information regarding copyright ownership.
4 F* }7 }# P+ G' h6 P4 M2 I The ASF licenses this file to You under the Apache License, Version 2.0* d$ x L( b2 g* M/ x0 ` g
(the "License"); you may not use this file except in compliance with/ r, D9 j6 U( l" ?
the License. You may obtain a copy of the License at' a5 p- l, S9 R1 z4 O' r! V
http://www.apache.org/licenses/LICENSE-2.01 X8 {5 X& k) \/ n2 a+ S
Unless required by applicable law or agreed to in writing, software4 Y3 {5 {4 i3 O, T3 L! O( D
distributed under the License is distributed on an "AS IS" BASIS,
2 u- j0 ?/ ^- ?( _/ V I WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.0 r! m9 d/ t& ]8 A1 L0 m3 h4 L
See the License for the specific language governing permissions and
2 F9 Z# y* P- P4 F0 ]3 S+ {/ x limitations under the License.& u! v/ e/ K. u# I: O
-->
7 D( C3 `" N! l+ n<!-- Do not modify this file directly. Instead, copy entries that you -->
, v+ q; K9 j9 K z<!-- wish to modify from this file into core-site.xml and change them -->
( C+ p( j& C, K$ c2 G<!-- there. If core-site.xml does not already exist, create it. -->& J! b& q" X6 y4 b3 ~
<configuration>9 a# `/ G0 R! m) O1 V
<!--- global properties -->
' \* G u' I5 H8 O5 n: C+ _6 E<property>
' K! a+ @) s9 {# I& q- e <name>hadoop.common.configuration.version</name>
" `* Q2 U( Y: W, a <value>3.0.0</value>
. M+ v# w6 q( \1 z' v <description>version of this configuration file</description>2 [4 [! T! ~% [, S
</property>
# c$ r6 ^1 }" e- q" `<property>
9 G9 i# Q* A) T- m+ c+ | <name>hadoop.tmp.dir</name>. _( O% t% a- ~
<value>/tmp/hadoop-${user.name}</value>
/ M/ @( Y% C( d) r <description>A base for other temporary directories.</description>8 X; d; l; H% n
</property>/ |& N. p: S* c
<property>
5 i4 j9 S w' d" N. D" ~ <name>hadoop.http.filter.initializers</name>6 G: T0 _( V* J. R4 ]- u7 ?* G
<value>org.apache.hadoop.http.lib.StaticUserWebFilter</value>. @; U' \6 a$ Z9 F% u2 w0 P5 g# T; e
<description>A comma separated list of class names. Each class in the list
. c) N, T$ C$ N1 [ must extend org.apache.hadoop.http.FilterInitializer. The corresponding
3 u9 k7 j2 q2 U% c0 X Filter will be initialized. Then, the Filter will be applied to all user
$ L4 \% E4 J' Q; ^1 c \* m+ q facing jsp and servlet web pages. The ordering of the list defines the' O% ?# R/ P( m6 D1 T# F9 F
ordering of the filters.</description>
' W6 `# Z& t9 p# @- ` Q) ^</property># k* y( D" f! l) u. K. ~
<!--- security properties -->! V2 |7 J2 t( B! C$ V* W C
<property>
/ u5 P8 S" J1 @7 ]7 M <name>hadoop.security.authorization</name>$ J, |3 S% ~3 w$ f* q6 l
<value>false</value>
8 ?. J' w, T3 w$ M3 ~ <description>Is service-level authorization enabled?</description>* I1 w( ^4 U4 F
</property># l0 L0 S7 Q1 x+ }: o4 c2 y
<property>- l C- T8 J/ z a- `
<name>hadoop.security.instrumentation.requires.admin</name>! y" i" S3 d3 d+ ]
<value>false</value>2 B& h. B) ]/ D8 p' O! c+ r6 [
<description>7 Y& S5 ?6 T6 w0 b
Indicates if administrator ACLs are required to access
0 C3 ~3 J8 s6 H. T! M& C instrumentation servlets (JMX, METRICS, CONF, STACKS).
( _2 K0 \7 \9 C' [1 v </description>* Z# M" N. F6 Z& U' \9 V/ r5 X
</property>& r! X7 H% H) _9 V4 y
<property>6 ^, s$ ^+ [6 w7 O
<name>hadoop.security.authentication</name>: F& H/ z- w: Y5 i! e! \1 z; N2 D0 R
<value>simple</value>
, U# U( ~ |# E* i3 I: L/ T* F <description>Possible values are simple (no authentication), and kerberos4 Q, E4 Q a: p( o! f
</description>
6 x; I* f7 [# U</property>8 s E8 L3 P( e0 I ^4 d$ P
<property>
- C- m% @0 T$ ~! U# k <name>hadoop.security.group.mapping</name>! H8 Z7 x: Z- i' C! ~1 d7 \; }
<value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>% ~( b$ x |$ L0 y/ E# U
<description>
9 }! |, X/ |; i8 X5 m: Q3 v4 u! s Class for user to group mapping (get groups for a given user) for ACL.* `7 o: l* G; c1 ?
The default implementation,) @' @- B- {* _7 U6 D8 ~% Y
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback,
: z. U3 k2 }5 `7 N, P6 l l# q' P will determine if the Java Native Interface (JNI) is available. If JNI is
4 N# \! y/ W0 R available the implementation will use the API within hadoop to resolve a
( Q! a$ R) C6 @ list of groups for a user. If JNI is not available then the shell
% B4 ^, r+ [ c' m5 d implementation, ShellBasedUnixGroupsMapping, is used. This implementation1 ~. B* P* T4 U u! o2 s" S
shells out to the Linux/Unix environment with the2 Z- O! x& T# y: o- t
<code>bash -c groups</code> command to resolve a list of groups for a user.9 a; e2 w U0 l" ~% s, v4 h5 B
</description>
8 d- s. d/ F: N: H& [</property>- U# x- v5 K8 j
<property>
7 c( Q& f# k# o( F% ?9 { <name>hadoop.security.dns.interface</name>5 r$ I; t3 @: P0 H4 M
<description>
" U5 g! A- u! Y7 Y1 f+ R5 @6 d The name of the Network Interface from which the service should determine/ D" ^& C3 Q `: _8 D, Q
its host name for Kerberos login. e.g. eth2. In a multi-homed environment,
/ d' Z* v" c7 h( L. e the setting can be used to affect the _HOST substitution in the service
0 a7 J7 l+ q5 W4 }% y) j Kerberos principal. If this configuration value is not set, the service
# P( a2 H Z o- ]" r! E will use its default hostname as returned by
2 R( c- d: Z4 w" ?; C% x0 Z InetAddress.getLocalHost().getCanonicalHostName().6 F0 U6 f! E* ]# K; _% m- j
Most clusters will not require this setting.
! O9 i3 J' ?/ v! e </description>5 H3 h5 w- q. j+ `8 y
</property>. ~0 ]* t D- ^7 R3 z$ u. h
<property>
* b E9 ?8 c# O, S3 k, l7 M <name>hadoop.security.dns.nameserver</name>
) ~4 ]) \/ [: U9 d <description>7 R( _9 w' S/ O( @
The host name or IP address of the name server (DNS) which a service Node
, G, i. H8 h7 y should use to determine its own host name for Kerberos Login. Requires
) y2 u8 O! k8 X q: E% n hadoop.security.dns.interface.+ ^, F+ @% ?5 |9 r
Most clusters will not require this setting.1 W3 g3 D1 j! ?* c8 ?
</description>2 j* v0 D! Z8 \6 b i9 w
</property>
; O! W# V2 j8 A) W+ I) V<property>' J% _# E( ^8 `. k9 }) R2 x3 n
<name>hadoop.security.dns.log-slow-lookups.enabled</name>
' t, s* C- s. C# q$ P <value>false</value>
+ e; ?- U+ m9 v: l1 U <description>7 h. P/ v5 E T# x( c* P
Time name lookups (via SecurityUtil) and log them if they exceed the% _: |' A4 c( B! k3 |- G( F
configured threshold.
! ~. \2 ?1 k2 V( ^# G/ |$ L </description>& g0 s: J/ ~: e4 B) I: o
</property>
2 f( q, D- v. `4 V<property>
" Y& `4 d2 K4 H3 s <name>hadoop.security.dns.log-slow-lookups.threshold.ms</name>
! o& [( o% u8 _! R <value>1000</value>
. y- ?0 [8 A1 I' u. H <description>- Q, L2 t: J' U2 T, W4 Y
If slow lookup logging is enabled, this threshold is used to decide if a
; e3 f+ g! C& Z: b, o lookup is considered slow enough to be logged.) [% m- l% J) d6 g! D9 c8 K
</description>
7 j9 ?# E8 ?) N9 s8 f; ~</property>
' @0 e1 ?; G0 o& ^, ?' K& P1 O<property>9 F2 l9 n4 I( G1 d9 G: Y
<name>hadoop.security.groups.cache.secs</name>: y8 I6 @9 m, q0 d1 B; q; k
<value>300</value>
* l3 ~+ w* L2 N <description>
u& x$ z* ?: Z4 a This is the config controlling the validity of the entries in the cache
& G# H* c8 G* n6 Y2 P: o$ f containing the user->group mapping. When this duration has expired,4 D8 M( c" {# [ s B
then the implementation of the group mapping provider is invoked to get2 s2 ^( u! U5 v8 k, M
the groups of the user and then cached back.
) G6 r4 a |0 J( ?4 D0 m: H </description>- @8 T* H3 c) u* e7 x* ?% }
</property>
2 |' X; t& b2 H9 H& Z8 c; \<property>7 T$ i4 \* r/ ]3 D( P g+ f- N
<name>hadoop.security.groups.negative-cache.secs</name>
, h+ C6 X# W) M3 x <value>30</value>
6 A" p# R* Y: |* H <description>/ P% Y, [) r: T* [' }" r. `; q8 s
Expiration time for entries in the the negative user-to-group mapping
" u, A3 u! _+ |, d& n2 B! Y caching, in seconds. This is useful when invalid users are retrying
, L+ i! }; Y- `7 p frequently. It is suggested to set a small value for this expiration, since
' p$ s) F6 b) T$ O u a transient error in group lookup could temporarily lock out a legitimate
1 j& [ J) l' C* Y- K" x% G user.+ Z' h( e: @2 X: x: S3 o
Set this to zero or negative value to disable negative user-to-group caching.
4 ?" W1 O. @4 Q6 | </description>: J/ G( O" ?; l1 B* [8 t5 Y
</property>6 g: s% {/ k. J$ J3 i
<property>
5 x" X' C% g3 ~" Z G <name>hadoop.security.groups.cache.warn.after.ms</name>
# v% t5 X a# H( a" i& e4 d* F <value>5000</value>& T/ _( V8 C6 A1 c/ K- B/ f; R
<description>+ {% D" Z0 P: O
If looking up a single user to group takes longer than this amount of; p9 j. B( l+ ^9 {2 V0 \. F
milliseconds, we will log a warning message.
. @& o& W$ C4 |+ ^ </description>4 g3 K! h: Y" y" I3 g
</property>
9 M/ a+ y# J# g O$ e<property>; O- }6 l$ `: ?/ o, ^
<name>hadoop.security.groups.cache.background.reload</name>
4 m1 V- r) c8 e/ ^7 |( }+ @ <value>false</value>+ ^" W! b6 j% J s# A
<description>* b% F3 \2 \% t. f$ S5 w
Whether to reload expired user->group mappings using a background thread
# h8 k' h4 {/ X( D pool. If set to true, a pool of
3 n# i- }2 ^8 ?* X6 a hadoop.security.groups.cache.background.reload.threads is created to
7 R) ]' g" D% H( D% n0 o/ Y+ f update the cache in the background.0 X7 [$ [- j6 C
</description>7 U4 W- p3 a! ~ _
</property>
/ G( z, Y3 P S2 K<property>' |1 E1 @ ]# D' _. T$ ]2 j
<name>hadoop.security.groups.cache.background.reload.threads</name>
8 [2 @7 `% r6 x b C <value>3</value>! _$ M$ e4 M. `' U5 {$ M! P$ M/ P
<description>
# l+ C: ^ u: ?: Q1 L0 C Only relevant if hadoop.security.groups.cache.background.reload is true.( u) Y+ _5 `8 G
Controls the number of concurrent background user->group cache entry
6 [/ ^' E. Y/ `9 K* [9 A7 P8 k refreshes. Pending refresh requests beyond this value are queued and
/ B, ?1 Q9 W' y" u processed when a thread is free.
, V( b* @. ^ n% x8 k8 p </description>
6 }& Y8 ` F) q/ M9 A* w</property>
. j& n* V; O ~7 Q4 M3 b6 n: O<property>
1 g8 u; t6 e% e6 \6 x5 [9 M <name>hadoop.security.groups.shell.command.timeout</name>, ^* n( Q* a4 d6 ^7 i. @* R
<value>0s</value>/ ]2 n/ `# p- ^( F6 B. } ?" q
<description>9 [: {; k. ~- v
Used by the ShellBasedUnixGroupsMapping class, this property controls how
& P# v7 s# X2 f l5 \+ w' u long to wait for the underlying shell command that is run to fetch groups.6 p( V4 `+ {8 Q% T2 w( g0 L
Expressed in seconds (e.g. 10s, 1m, etc.), if the running command takes
; v) U4 y% }. u# [( \ longer than the value configured, the command is aborted and the groups H7 z3 H, X+ R0 q
resolver would return a result of no groups found. A value of 0s (default)- i0 X7 O, T( _0 r
would mean an infinite wait (i.e. wait until the command exits on its own).
' [$ }) _0 j, J$ ]$ x </description>
2 A" t1 r4 S+ {6 [% d( S1 }) g</property>
: T9 z z: X4 Y2 W<property>
$ P6 m1 @! T0 K% w t( W- v0 O, B: C <name>hadoop.security.group.mapping.ldap.connection.timeout.ms</name>
0 q5 D4 l I8 C* b2 F' Y. R8 ?# H <value>60000</value>0 \) c' u+ a. Z3 \6 Q/ T
<description>
a+ }% x0 U' B8 I; u This property is the connection timeout (in milliseconds) for LDAP
; n+ A& ]5 k7 O6 X operations. If the LDAP provider doesn't establish a connection within the( U: M2 p& {" J4 o4 {/ R
specified period, it will abort the connect attempt. Non-positive value
3 k7 H$ b: P8 B+ x means no LDAP connection timeout is specified in which case it waits for the
% N# f2 i& }' i, V6 Z# {6 | connection to establish until the underlying network times out.. ?1 ^0 @4 z3 |- i# I) t
</description>
; D- W4 ]+ Z$ `7 o' k</property>. @5 S, Y# X: n& M6 [ R; G4 F
<property>
6 ?" U8 H3 z) z! I8 j& | <name>hadoop.security.group.mapping.ldap.read.timeout.ms</name>5 e( h9 u% n& M/ ^* [
<value>60000</value>+ X+ O+ u# G. Q6 S6 c# [( j9 W1 M
<description>
4 X1 M- ~5 D0 u9 q. I6 U m. Z This property is the read timeout (in milliseconds) for LDAP5 w4 S) t4 p9 \" s T: @% |
operations. If the LDAP provider doesn't get a LDAP response within the
~4 y) O% G+ u4 j! ~# R( X* H specified period, it will abort the read attempt. Non-positive value
9 i2 `4 P2 l2 q+ O6 f" h7 L means no read timeout is specified in which case it waits for the response
, r0 {( T4 I# s$ v! A4 p% B infinitely.
: R# c: A3 L9 {- [1 P! i$ U </description>
7 y% a' M7 ]) |9 B6 _* t7 i</property>
. b: l7 |* w7 F& g) W! b: x<property>* h% K2 |, A9 n
<name>hadoop.security.group.mapping.ldap.url</name>
. u3 n/ T7 a9 I2 L1 `2 J6 n! C <value></value>2 _6 a8 a4 W( s/ {3 n' e( c$ X
<description>7 ^+ b8 Z* E: G4 D
The URL of the LDAP server to use for resolving user groups when using
4 H V" C- X" z9 o4 S& v# z the LdapGroupsMapping user to group mapping.
) w) u2 o( F4 n4 x) X </description>
$ K% @. Y V+ i2 U- a</property>7 p. ]4 A' k: D% h
<property>
- m& t! G- D& p) N <name>hadoop.security.group.mapping.ldap.ssl</name>
1 A9 U! K$ e3 \; M) H. H5 X4 [ <value>false</value>
! o" C$ [* o) l. k8 B7 x& H <description>
' ?+ Z4 @0 B& ?9 C2 Y Whether or not to use SSL when connecting to the LDAP server.5 [* Z Z! j J( K& I
</description>, |1 e3 [3 v0 ^
</property>. q) p ]+ o8 H) E- t! n2 [6 ^
<property>
- X: `3 f2 ]$ R: a1 r <name>hadoop.security.group.mapping.ldap.ssl.keystore</name>
1 V/ E. `7 J# V* H, R <value></value>9 A, V* z* O' ]* z8 T
<description>
+ u1 l' ^6 _ `8 ] File path to the SSL keystore that contains the SSL certificate required
* r2 O1 Y% U: _3 ?* ? by the LDAP server.
# a6 d; p& S8 L, }% c </description>
! N0 O$ J6 i2 O5 g! F</property>
# |6 e5 h8 T& l: L<property>
) k0 |+ D5 ^2 D+ `: ~ <name>hadoop.security.group.mapping.ldap.ssl.keystore.password.file</name>
% V6 _+ h+ G& [# p5 C# A <value></value>( A3 v- F: S. G
<description>
% }( U4 F+ P2 ^/ Y6 p l The path to a file containing the password of the LDAP SSL keystore. If
v2 |- i ^( I7 `' Y" @" H2 z the password is not configured in credential providers and the property- S9 X) C- H! j+ j
hadoop.security.group.mapping.ldap.ssl.keystore.password is not set,
& @3 S( p$ C; K$ {" t LDAPGroupsMapping reads password from the file.
. Y: w- F; F4 T' h+ o/ a IMPORTANT: This file should be readable only by the Unix user running
/ Y- J3 O2 t8 d9 Z' e the daemons and should be a local file.
# V: S3 l: B! K </description>/ a% Z, i; Y- F: K( _, }* f" }
</property>! p# C1 Q" T S% s
<property>
& A9 R2 A' c/ [1 `. V: ~5 F8 A <name>hadoop.security.group.mapping.ldap.ssl.keystore.password</name>8 y9 ~) o" \4 i, P4 p8 {
<value></value>' d* F; m4 T; j% `+ u3 c" r
<description>/ F7 W; k4 x% K( A5 J$ N) B/ O! f
The password of the LDAP SSL keystore. this property name is used as an! a* N! }6 r; h' u) i- i/ b$ j5 i
alias to get the password from credential providers. If the password can+ R( N( c, J% O& c: p
not be found and hadoop.security.credential.clear-text-fallback is true
9 C9 n% J" _) m& o; N9 t, t LDAPGroupsMapping uses the value of this property for password.6 z, |" b6 `" U6 d' I4 a
</description>: f/ a" A* h4 j$ K4 l4 d& }
</property>
% M/ {9 c r% J: [2 z1 H<property>- H9 C0 }2 l/ Z8 [
<name>hadoop.security.group.mapping.ldap.conversion.rule</name>
8 Q2 D: u' T6 O. F' R" S# V <value>none</value>; T8 {/ ~4 O# c+ R
<description>
4 T, U3 ^6 X) E7 U1 n The rule is applied on the group names received from LDAP when% C9 B& x- |3 K% ?
RuleBasedLdapGroupsMapping is configured.- g$ k' O3 m, u8 ? o% O8 ?
Supported rules are "to_upper", "to_lower" and "none".# U" x; d, H7 }) Z- i1 ?6 Q. J U2 X4 m
to_upper: This will convert all the group names to uppercase.
$ i( r( m4 x" h to_lower: This will convert all the group names to lowercase.
m( i9 c) Y# V2 i* X7 e M none: This will retain the source formatting, this is default value." b6 g. t3 U6 G, R; w; c
</description>, v* A4 [3 c5 m/ t* }
</property>5 d$ D q3 H P$ h, I6 t
<property>0 X4 b& O Q4 h/ x
<name>hadoop.security.credential.clear-text-fallback</name># n% y, Y% u# o
<value>true</value>
# D2 A# A- F1 B; l <description>- T4 V* s& p% R4 s3 b. n0 m
true or false to indicate whether or not to fall back to storing credential3 A6 z& Y1 Z) v! R
password as clear text. The default value is true. This property only works
/ }$ l$ k; k1 S4 t. s+ | when the password can't not be found from credential providers.( ?6 t( M# `3 ?+ E
</description># M' ^$ g3 D ?+ I
</property>
3 P9 [# _, ~% X) |! F<property>
3 \# E8 `% N0 y <name>hadoop.security.credential.provider.path</name>
2 a& ~6 V/ d# s$ F: n <value></value>
' R/ q6 [% V6 q$ k% p/ ?% P3 v$ W% g <description>, D$ O2 j5 u* G2 P, b J1 _3 B
A comma-separated list of URLs that indicates the type and |- o. C, l s/ ?! x
location of a list of providers that should be consulted.
1 m& c7 G$ ^* G4 ^3 }% ` </description>
* \: _8 z$ _. w/ z( y4 C</property>& b$ K! {. w, s& @" w& k0 H# Z
<property>
Z# T/ I C1 E+ @5 p <name>hadoop.security.credstore.java-keystore-provider.password-file</name>
' s% E+ v2 k) Q4 x" s. y5 q, {% A; V <value></value>
6 y/ z! R* h; l <description> v' |" }4 F$ c2 S7 G" C
The path to a file containing the custom password for all keystores& o1 B0 x5 N! a; J
that may be configured in the provider path., g( x1 S) v D2 {% i
</description>
$ C. c9 F; o, _9 }</property>
0 g2 c5 a! q' h' Y) A3 O<property>
8 h) e' E: H( x" a. f: l* r <name>hadoop.security.group.mapping.ldap.ssl.truststore</name>
1 g$ N) J0 p% M& U: u5 r$ M- g& c <value></value>
% x& `1 a' |+ c <description>
; o6 ]# \. Q- O& \* D2 R4 n4 I File path to the SSL truststore that contains the root certificate used to
$ B* M$ ]) p$ H% ~ sign the LDAP server's certificate. Specify this if the LDAP server's' x1 M- O2 l' C, I1 i# a5 F+ c3 @4 X
certificate is not signed by a well known certificate authority.: E# a8 a: h' X# U! s9 B
</description>0 ~7 z0 m1 P/ W$ ]9 D
</property>6 q0 \; m4 l, I- O0 |0 @
<property>
9 D2 p( {& f/ Q( Q! C4 A( u <name>hadoop.security.group.mapping.ldap.ssl.truststore.password.file</name>
6 u; K) ]$ j- z7 H, r+ d. |4 p <value></value>; b& P! y8 o- ], k: I
<description>
8 m* K2 W6 g6 t1 {" B7 J The path to a file containing the password of the LDAP SSL truststore.
( x7 \6 k6 s: r, |! q0 v IMPORTANT: This file should be readable only by the Unix user running4 s/ E+ U. Q6 Q- U$ a
the daemons.3 F" ^' U/ p+ x
</description>
6 w2 Z/ X4 ]. L" I6 f/ k" q</property>0 P% m9 ^: h N) R9 {/ k) Y# S0 ?! ~
<property>* x v5 {5 t2 R2 Q3 b1 w' V
<name>hadoop.security.group.mapping.ldap.bind.user</name>7 q3 ]2 ^ W- w
<value></value>1 i) i9 i8 q) C
<description>7 B6 u! P9 W; C: `6 g( }
The distinguished name of the user to bind as when connecting to the LDAP
/ _$ t/ x5 A K6 n7 Q server. This may be left blank if the LDAP server supports anonymous binds.' X5 y1 J3 \) H* a; y9 Q; S l
</description>
6 T6 v9 k; s8 [8 \: p8 ~</property>: w3 }* u& G- N. h( |: Q
<property>7 T9 v. f2 D9 ?8 h. ^% U7 O- _( ?5 t
<name>hadoop.security.group.mapping.ldap.bind.password.file</name>- u4 u' b, s, o7 Q
<value></value>
; |8 O: o5 M( y5 B <description>
1 `2 J+ L6 j8 v The path to a file containing the password of the bind user. If Z/ j& t T. z& f9 t( s
the password is not configured in credential providers and the property* T0 R& `. _9 ~3 {; }; k& ?
hadoop.security.group.mapping.ldap.bind.password is not set,0 d: E, G- L* Y+ x
LDAPGroupsMapping reads password from the file.2 L4 H% ?# `$ Z4 Y* n
IMPORTANT: This file should be readable only by the Unix user running
. u% a9 I' F2 {# Z% Q; Z$ V9 P- X8 j the daemons and should be a local file.
$ g# {' y4 R- D. {$ r& T# d/ S </description>; N h/ _( Y4 t* p
</property>; C, n: O2 L3 a
<property>& \! y8 m* E% E( H
<name>hadoop.security.group.mapping.ldap.bind.password</name>6 R1 x- M8 W3 B
<value></value>
" W$ Q; y/ @, `! @7 `8 B <description>/ R7 e- A7 y% U, F$ ]% ]1 h3 n; S
The password of the bind user. this property name is used as an
% h3 `* i$ G# S# }* s% Z' x alias to get the password from credential providers. If the password can
' A' i& _0 ^. C0 k" ^ not be found and hadoop.security.credential.clear-text-fallback is true
3 }0 _( ~; D5 o1 i9 h5 S. A LDAPGroupsMapping uses the value of this property for password.* h9 v. n6 i! A) \, ]) F5 w
</description>
/ R4 f+ K* z1 z$ \ [% m$ h( D</property>; ?8 @) I' p; {5 J
<property>2 l: ~3 G2 f1 p* n) G5 U5 S
<name>hadoop.security.group.mapping.ldap.base</name>
7 g! ]- |- k1 F2 t) F <value></value>
, j. s! l9 }7 O( h# V9 Q. V; U; k+ J <description>
4 Q; u% e. }$ M$ Z! h+ s( b. I The search base for the LDAP connection. This is a distinguished name,
( I: ^0 I+ J1 d2 L* O" [+ y9 @( F$ Z and will typically be the root of the LDAP directory.
( j2 t% G% m9 J& f </description>. ?3 F1 E1 q9 T5 b" T8 @$ q! I8 n
</property>) S' y6 d2 Y( t" E8 X( ~# m. M
<property>
5 Y7 l" ]( M: g& i; G7 G Q' u <name>hadoop.security.group.mapping.ldap.userbase</name>
4 s$ ~$ M+ r; x' I <value></value>
. ?- w* O$ Y8 B0 M: i <description>( a& ~! C6 P# K# Z+ m1 J
The search base for the LDAP connection for user search query. This is a
5 g$ p* I% @2 m6 _( {4 l distinguished name, and its the root of the LDAP directory for users.
5 N/ z7 V% v( \, Q/ m8 O( z If not set, hadoop.security.group.mapping.ldap.base is used.' }# L ]* [1 M% F
</description>/ U7 G1 G1 d, }" f3 F1 G
</property>+ H2 ?5 X1 L. f7 P# d$ ?) O
<property>! Y0 ~6 h+ p% c w7 E1 A
<name>hadoop.security.group.mapping.ldap.groupbase</name>
+ p! P& h2 O m8 C+ r2 z$ P* Z, q <value></value>& Q% r q4 F. H
<description>
& m6 h/ M4 T9 m2 Z4 U! h The search base for the LDAP connection for group search . This is a7 V4 j2 m" U2 N" T+ l% i
distinguished name, and its the root of the LDAP directory for groups.* p. Z r% I! N9 b! A4 U6 F1 _
If not set, hadoop.security.group.mapping.ldap.base is used.3 c7 h7 c6 ]8 r! R" D! k
</description>: o4 b* q- b% ?3 [0 X
</property>3 b+ N7 Z+ W$ Z
<property>
% p8 q* e4 A- m% ^( @1 _ <name>hadoop.security.group.mapping.ldap.search.filter.user</name>' T' `' \4 ] G4 H* C1 R
<value>(&(objectClass=user)(sAMAccountName={0}))</value>$ k; G- x3 V; V. D1 B4 @
<description>
0 d# r E6 B1 p, y2 `0 p- o2 v An additional filter to use when searching for LDAP users. The default will
& D+ B) p0 M' a: ? usually be appropriate for Active Directory installations. If connecting to! e% T. v+ { i V! n* b
an LDAP server with a non-AD schema, this should be replaced with7 D B9 R: U6 ~0 W$ a$ L4 s1 a$ z
(&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to8 n7 I2 k b4 t# e6 Z( C$ ?! x, c
denote where the username fits into the filter.
) \$ m: f- t; [4 H, h+ J If the LDAP server supports posixGroups, Hadoop can enable the feature by
6 @& K+ Q f2 I5 ~7 X5 S8 t setting the value of this property to "posixAccount" and the value of- I- B+ D: y( c/ G
the hadoop.security.group.mapping.ldap.search.filter.group property to) M9 Q3 [- P, p- i: w$ \9 T8 v
"posixGroup"." v) z5 ?3 C2 [* L# T8 W
</description>
! d/ V7 t% ]2 I3 v7 ~! `- b) O9 V</property>& {9 e! u! D4 }1 F4 u
<property>
' Y3 A1 Z5 K# n9 S+ [: L E <name>hadoop.security.group.mapping.ldap.search.filter.group</name>" a' H. `' k7 v5 V T# g' @
<value>(objectClass=group)</value>
9 l! w& k9 ?2 u <description>
* w) n0 A3 f1 u2 W5 [ An additional filter to use when searching for LDAP groups. This should be2 P& p/ n3 ]& a* y( w8 `/ Y# ]& ~! p
changed when resolving groups against a non-Active Directory installation.
) G4 K, `; E8 |; N. p" ~ See the description of hadoop.security.group.mapping.ldap.search.filter.user
& R- O$ n; R1 W8 k* k" P to enable posixGroups support. X4 h0 n5 ~4 N* Z9 g% w
</description> E y7 k6 f- i
</property> U7 a8 g' X4 C
<property># D* w9 U9 I% ?9 w# f5 X( x
<name>hadoop.security.group.mapping.ldap.search.attr.memberof</name>( f* f) V, n, ?1 _5 T+ U
<value></value>
2 b" a( g+ j6 i4 _ <description>$ ?- m" C/ K0 u5 ?" n( S7 [* i0 l
The attribute of the user object that identifies its group objects. By
- J7 x( z- G6 M$ q% K" S default, Hadoop makes two LDAP queries per user if this value is empty. If+ C* T( w3 {8 }2 S) Y4 h0 y. p# q
set, Hadoop will attempt to resolve group names from this attribute, ?! Z$ k+ z0 t$ K0 S
instead of making the second LDAP query to get group objects. The value, x6 M; ~# v+ e/ o5 J3 P5 l
should be 'memberOf' for an MS AD installation.
- W1 L! a. ^- g2 }- h' J </description>
9 s* M8 k3 @6 ]4 d6 C5 I, Z: ]</property> j p' G, I5 C8 N% Y* n
<property>2 R/ [4 C; o$ c! G/ y+ s8 J
<name>hadoop.security.group.mapping.ldap.search.attr.member</name>! S& H% ]+ t( n. F8 a: T. m- e
<value>member</value>8 o( U# n Q2 g9 @# Y2 n& q- M
<description>- U( T8 y: Y; g
The attribute of the group object that identifies the users that are6 Q! g# }. w4 T" x
members of the group. The default will usually be appropriate for
' h8 P% z- Q) o. l5 y any LDAP installation.! D+ c9 q, ~) k5 e9 k: I' {1 U. @
</description>
+ G( f5 Z! U) D+ N7 v- A</property>4 j+ h1 a' {& Y. A9 S( P
<property>
5 Z% P6 f$ @ a <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>, a4 t- F$ |$ l; P. t/ v
<value>cn</value>
, x* v3 ]4 @% s3 i1 J& d( y3 S <description>) j" O1 W: e# t5 h% Y- |. z
The attribute of the group object that identifies the group name. The/ @& N' U& ?4 U) w/ K6 C
default will usually be appropriate for all LDAP systems.
" I. C9 J5 _- |) Y' y( C </description>
4 E* y8 @4 y% i# s</property>
, u* ]- j4 u' R' O$ d8 Z$ B" I7 D<property>% A- Z' A3 A7 }% j
<name>hadoop.security.group.mapping.ldap.search.group.hierarchy.levels</name>3 g. b6 ]1 V; L4 O4 w0 S9 x6 t
<value>0</value>
4 b) z- F" W- e& D <description>+ _1 ]; y+ y& y8 |
The number of levels to go up the group hierarchy when determining2 G9 f0 r* b! J- {( x
which groups a user is part of. 0 Will represent checking just the$ L" i3 f7 a- _- N
group that the user belongs to. Each additional level will raise the6 j+ D: @3 \ l& J. o Z' T
time it takes to execute a query by at most+ \# Q- v0 J' D
hadoop.security.group.mapping.ldap.directory.search.timeout.
' p j2 C* q0 D' z) M& N0 a The default will usually be appropriate for all LDAP systems.6 j0 Y( l. m" i. f+ b3 g/ G
</description>
. ] R( }0 [+ }/ [' I; ]' c$ ?6 D</property>7 r6 D7 H3 A5 H$ l+ H; Z: N c2 [, @
<property>, j& H& r2 t- T! X3 m
<name>hadoop.security.group.mapping.ldap.posix.attr.uid.name</name>
6 c4 S3 Z# n# _' n/ r2 a <value>uidNumber</value>
. _" p: |( o, O: ] <description>, _: Q. U" X, R# q1 U
The attribute of posixAccount to use when groups for membership.
0 @3 n* X% T' z5 e# x+ W) w j Mostly useful for schemas wherein groups have memberUids that use an
- F( A y' q0 c3 t attribute other than uidNumber.
1 Z! c. q3 p% `" `7 a- u </description>
3 [& {4 m9 ]- j8 U" c7 e</property>( f2 I0 b6 ?% w6 Y
<property>+ Z/ T0 j. F+ |% \7 r3 }+ ]
<name>hadoop.security.group.mapping.ldap.posix.attr.gid.name</name>
' V, L9 [ u% f/ s, J) K <value>gidNumber</value>( X! i' X! U1 V4 J9 q# e$ W
<description>
* |# n. \% X+ L; [ The attribute of posixAccount indicating the group id." t: l# E P+ `- B4 P3 R
</description>
& {9 s" v6 S! u2 x</property>' d% q- x: W# k, O' ]
<property>6 V' t' i/ f+ L0 m E; ~
<name>hadoop.security.group.mapping.ldap.directory.search.timeout</name>, l2 Z1 a8 @# y) e: ~0 s
<value>10000</value>% e6 O. `' p& @/ f* Y0 j# Q/ x
<description>- X( l0 ~7 f9 j- n, I/ b5 M
The attribute applied to the LDAP SearchControl properties to set a. C/ i& o. q0 t3 H+ Q8 a
maximum time limit when searching and awaiting a result.3 S) h; C8 _ X, e' {4 `! t
Set to 0 if infinite wait period is desired.* }6 U/ I9 g1 i
Default is 10 seconds. Units in milliseconds.
! ?0 m. {6 J) } </description>6 x, W0 v% x a3 u
</property>
( }' ^6 B! x! c% W& j0 r<property>
: {9 Y. E. z! p& O N$ c3 u <name>hadoop.security.group.mapping.providers</name>
1 z$ x9 u& ]* W- J; G <value></value>8 A. j' `. L \* z* W4 g& n5 V
<description>) G1 k( v$ F; P" C% p# c* ?
Comma separated of names of other providers to provide user to group
( g, N1 Z) d/ |/ h mapping. Used by CompositeGroupsMapping.' x5 @( d. i% u1 S9 j+ i
</description>8 c: b1 I2 {1 _
</property>! m+ d& k# _9 O$ k. P
<property>$ Y$ x) D3 a: i# C
<name>hadoop.security.group.mapping.providers.combined</name>
. e* F! R# ~0 i& m4 L% c <value>true</value>
7 G9 S8 @: ]' }1 Z/ v3 K+ f$ N( Y <description>
" n* D( c8 j# u' n true or false to indicate whether groups from the providers are combined or
; D5 Q' N" J8 w* K# c not. The default value is true. If true, then all the providers will be6 n7 C0 l `$ F$ @. h* d9 z8 S+ F8 c
tried to get groups and all the groups are combined to return as the final
6 L0 M6 ^% l3 w results. Otherwise, providers are tried one by one in the configured list& i8 ^9 o9 m' \) H8 d
order, and if any groups are retrieved from any provider, then the groups8 E# C, @5 U- Q1 n
will be returned without trying the left ones.7 h2 Q/ ~" m/ `* ] Y4 V
</description>+ [3 _/ h- R3 a) }1 y
</property>
* h5 u- P @4 e; V7 U8 S<property>) A" t: a6 V8 Y4 a1 n
<name>hadoop.security.service.user.name.key</name>
+ B$ G9 u$ M& h <value></value>
# I, r: z/ H) w8 D- R3 O3 u& ` <description>3 Q. K/ ?) {( f: r" k, |
For those cases where the same RPC protocol is implemented by multiple
+ W1 h1 C2 F! ?# n" j3 X* c servers, this configuration is required for specifying the principal
* k% o' S- m# q O" K1 ]7 ]5 ] name to use for the service when the client wishes to make an RPC call.$ _, _/ ] T" y! H
</description>
4 M3 Y$ r5 D! ~7 S</property>, @ v+ ]# ]9 W9 a8 @. [& a: @
<property>
- s s( w/ K Z/ A <name>fs.azure.user.agent.prefix</name>
3 c( V8 `1 v, @ <value>unknown</value>" L% w' O8 v; T( [# y; U
<description>$ x9 I. S5 P6 l, Y& z2 ~
WASB passes User-Agent header to the Azure back-end. The default value
: N6 {- L; P! h j contains WASB version, Java Runtime version, Azure Client library version, s$ S7 k- f; ?
and the value of the configuration option fs.azure.user.agent.prefix.3 @6 T7 G. I3 t/ u
</description>
' A) _! u! t ]! p6 }3 R7 ], H </property>
7 P: x4 r* s. P( [<property>5 Y) p7 H0 l& Z# M+ n. Z; X3 t
<name>hadoop.security.uid.cache.secs</name>
6 j3 X. @ M+ ~! V7 ~9 M$ [ <value>14400</value>
, h; e4 Z( q/ _0 e4 B/ D <description>
6 S" n4 p4 G: O0 q) C$ M- F This is the config controlling the validity of the entries in the cache! s; k9 `7 F9 O- W2 U/ `# v
containing the userId to userName and groupId to groupName used by. p! y1 \" B. B
NativeIO getFstat().
/ G2 O3 {, X; E6 A0 O& n </description>
' S& H0 }) M3 ~; H- G</property>( r; k0 ^$ p0 ^4 ?" k; M" h/ j3 C' F4 y
<property>
- i' P; ?( U/ v/ { <name>hadoop.rpc.protection</name>
# a# Q2 y3 ~- D; v `6 i+ \) d <value>authentication</value>
% M( A) r7 w# n9 z/ l6 o <description>A comma-separated list of protection values for secured sasl" ~& a. Q- q9 z2 S* `4 y
connections. Possible values are authentication, integrity and privacy.
9 z* \2 Q. g5 `+ A authentication means authentication only and no integrity or privacy;" V9 n- K; J/ f1 C3 B
integrity implies authentication and integrity are enabled; and privacy
6 H3 a* `8 D; B implies all of authentication, integrity and privacy are enabled.
H* b% L3 `0 l9 C hadoop.security.saslproperties.resolver.class can be used to override, M) s6 |6 i, p5 `5 l$ o9 W) o
the hadoop.rpc.protection for a connection at the server side.
# }. E; l, ?' l* f0 W, ^ </description>9 g" ]3 c" }8 }' G6 {7 a
</property>+ s0 x( o t3 l( v
<property>
/ z1 J, M7 v( \6 q7 q" z0 i- c6 Y <name>hadoop.security.saslproperties.resolver.class</name>% z7 f7 p* E/ H
<value></value>" [9 F' t6 E+ N+ e! B/ h
<description>SaslPropertiesResolver used to resolve the QOP used for a$ N+ I. `( ~7 n& E+ C; ~' O% A5 Z
connection. If not specified, the full set of values specified in
$ w( M T5 J3 |# E7 i2 q2 m* e$ w hadoop.rpc.protection is used while determining the QOP used for the7 E& P! x* i) q9 ?$ F: E" o
connection. If a class is specified, then the QOP values returned by& [8 d n2 m b! \3 H! U/ h" W
the class will be used while determining the QOP used for the connection.( ~4 w, L m/ ?+ |6 p5 C( V% Y
</description>+ _- F ^# L8 { r
</property>
" q$ m) [# Z% A0 ~# g$ Y2 r: I<property>. K0 C$ z. ~5 |3 R1 f. g% U
<name>hadoop.security.sensitive-config-keys</name>; L' q! Z. P/ @; ~- \3 k
<value>3 F* t( P& V i$ H3 o
secret$
9 |- J8 `- ]* O. m S9 Y password$: y8 x& E I+ a$ c8 m5 L# H
ssl.keystore.pass$/ T0 e" L8 O. v6 P$ B- x
fs.s3.*[Ss]ecret.?[Kk]ey
6 }( F0 a A) L; @7 U( H& n fs.s3a.*.server-side-encryption.key& _ t# P/ F! U: e$ L7 a6 v. q( O
fs.azure.account.key.*
s+ ]6 U) ]) f O4 j% W" I credential$5 v, z0 F5 r* l+ h+ j$ d
oauth.*token$
( I: T! h* x9 F0 Y) j, S% {/ | hadoop.security.sensitive-config-keys0 P$ j$ C2 m- o4 y$ W/ q
</value>
2 W( X% i6 a# o% J <description>A comma-separated or multi-line list of regular expressions to
2 r2 B: E* Q; `& j0 [- l match configuration keys that should be redacted where appropriate, for
# g' f% m1 Q) x& a, V X example, when logging modified properties during a reconfiguration,) |7 m2 u, j. M
private credentials should not be logged.
6 g3 {1 H: m: V# y </description>
6 y3 v/ S$ Z F$ A</property>
9 N7 z4 E! b3 \% E) J' j<property>
! o& E5 s: P% \ <name>hadoop.workaround.non.threadsafe.getpwuid</name>
6 U( J* I' J+ ]' ]# q9 d$ u* i <value>true</value> b: J7 ]2 t: z+ X- M* K6 j1 d
<description>Some operating systems or authentication modules are known to
1 r6 h& n1 ~& y g2 J% J' @# n have broken implementations of getpwuid_r and getpwgid_r, such that these e* @) W0 [2 l+ @% {, O( z" q7 r0 V
calls are not thread-safe. Symptoms of this problem include JVM crashes% f @$ z: H* T" z% D
with a stack trace inside these functions. If your system exhibits this
* f0 Z* C6 c5 T8 e issue, enable this configuration parameter to include a lock around the( z3 ? `0 ]" ` @' f- ^+ H& _
calls as a workaround.
/ b7 F( K; s+ p S An incomplete list of some systems known to have this issue is available
( x Y! ?9 ] W9 j( B ? at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations/ S: u7 s( `4 L
</description>9 W/ W9 `3 ^3 f# _1 |- J
</property>
% ^0 R' p k& S9 P% k" y<property>
5 i/ ` X7 O q <name>hadoop.kerberos.kinit.command</name>
( G- }& q( m2 s6 Y, S' r' S <value>kinit</value>4 K! @) [) H, g1 k6 a
<description>Used to periodically renew Kerberos credentials when provided
$ Q3 L7 @( p \# x% i to Hadoop. The default setting assumes that kinit is in the PATH of users
9 V. Y* ^+ M- k) O running the Hadoop client. Change this to the absolute path to kinit if this' Q9 T4 F0 K- x/ _# Q7 o
is not the case.: E n5 X; o! Y& c) c" h) m
</description>
/ b4 {& U( `; v' q</property>, N6 t/ A1 E3 I+ t1 c
<property>3 X1 y4 G5 h9 J6 h) s; s
<name>hadoop.kerberos.min.seconds.before.relogin</name>2 H' j. J- O/ J: G( H1 ^
<value>60</value>
7 D( N: @0 a! M6 t; X <description>The minimum time between relogin attempts for Kerberos, in% W" N& p/ s4 d4 e' V2 W
seconds.& \% t: a; O9 J" s4 }5 A1 N
</description>1 L- x* Y% Y+ z: C _
</property>2 c3 _ i3 I3 }+ y
<property>
: v0 R% w9 `7 \, z$ X <name>hadoop.security.auth_to_local</name>3 B1 ?. X5 U& g2 C1 K+ k* P
<value></value>
) |' p/ Q1 _/ d8 k- G9 z: Y! T <description>Maps kerberos principals to local user names</description>6 x5 R+ n8 }. K
</property>) ]7 N* O! }2 a
<property>' J; `( `* ~; H$ s4 }* s
<name>hadoop.token.files</name>. k3 S2 q9 k1 y7 u
<value></value>2 q/ D* L7 G1 V7 N
<description>List of token cache files that have delegation tokens for hadoop service</description>4 a' g$ i2 I/ Z; Z0 R) t
</property>% Q5 _8 f$ Q5 L) V1 n1 c; p
<!-- i/o properties -->6 o1 \+ L# O& q s$ M5 [; g# `
<property>
( T4 a, w9 A$ W, a, u <name>io.file.buffer.size</name>
+ t- y" F {. b& \, C; W <value>4096</value>( C+ U$ i1 Y# j1 t4 n0 f, ^5 x
<description>The size of buffer for use in sequence files.
" B! x2 ^+ G5 [! d The size of this buffer should probably be a multiple of hardware6 f4 J- f( [0 \8 T
page size (4096 on Intel x86), and it determines how much data is4 J7 D1 F$ L/ q' I8 F7 I& z
buffered during read and write operations.</description>
8 M' c9 h R: F</property>" v9 Y9 ^ F; N/ J7 K; F, Z0 Z# E% m
<property>& c7 S) ~6 f5 a; R1 ]! K' l
<name>io.bytes.per.checksum</name>
- o; N* ~- E. q. g! A0 _( {! q <value>512</value>
4 U7 i$ r+ f+ H z0 i% [ <description>The number of bytes per checksum. Must not be larger than
/ r4 @" Y6 R: N1 j- f4 J" M io.file.buffer.size.</description>
8 @: g7 J$ }, J- _! {</property>
, C( O6 J% {/ A: H+ |<property> V+ x- Q) K# F6 I3 R) v9 w, ~1 B
<name>io.skip.checksum.errors</name>
" e1 p! ?2 v" q( ]' B# ^2 ? <value>false</value>. u1 j. ?! T0 ^+ Z* u
<description>If true, when a checksum error is encountered while" H$ T2 e. @1 p8 P" _: P
reading a sequence file, entries are skipped, instead of throwing an
8 Y, z1 p0 t, U exception.</description>- e/ P9 c9 y p6 ^
</property>. D& h$ O7 q' H; K' z
<property>1 U2 \3 U' {& S
<name>io.compression.codecs</name>. O% }, s' h7 i( O
<value></value>9 w7 |- }2 A0 W& i; I
<description>A comma-separated list of the compression codec classes that can. A9 N5 L( I0 S& c" F1 i& s P
be used for compression/decompression. In addition to any classes specified4 M" `" \% T6 r" e+ n
with this property (which take precedence), codec classes on the classpath
+ ?* B! |$ \* C' |- [) {- c' e are discovered using a Java ServiceLoader.</description>
* D' U% `7 K) Y7 M" y' N</property>* e5 r6 g- i0 P3 u
<property>
" b' Z0 Z: b" \8 e' ^' J: W: ?" F <name>io.compression.codec.bzip2.library</name>1 n0 ~; Q6 ^/ Q$ c0 _/ p: u% Y
<value>system-native</value>
' c0 _& l+ P- i; p <description>The native-code library to be used for compression and
u- ?% d, e) ]9 ?$ |" t$ O2 g decompression by the bzip2 codec. This library could be specified$ R! y3 |. w3 O& M
either by by name or the full pathname. In the former case, the
# z! z/ s6 p) R" ]! o" b library is located by the dynamic linker, usually searching the; C. V( T' \6 C ?5 @
directories specified in the environment variable LD_LIBRARY_PATH.
4 w) _" \! g# i The value of "system-native" indicates that the default system
$ B9 a& U6 m1 P! O, q) d library should be used. To indicate that the algorithm should
o% s/ _ k4 c8 \. Q* N ]2 Q operate entirely in Java, specify "java-builtin".</description>0 }* \+ n$ g' b/ D3 Y* a5 {
</property># e& w8 i' z, W
<property>6 n1 N4 ~9 W7 d" E" t; k
<name>io.serializations</name>
' T( U, m& y' F" v* N# u3 Z4 v <value>org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization</value>
+ H. H3 ` j: U( b' H/ K) F <description>A list of serialization classes that can be used for5 R1 F/ ~: v4 F
obtaining serializers and deserializers.</description>: @+ Y' I, l3 E+ \+ N8 m* o
</property>
5 ? \6 J/ s& u' _4 i2 _" g<property>* G9 U8 Y8 k# @$ N
<name>io.seqfile.local.dir</name>; G; g( t/ J$ n( B7 ~& Q$ n
<value>${hadoop.tmp.dir}/io/local</value>
% s" Z( n4 h4 h" ` <description>The local directory where sequence file stores intermediate
2 E& H$ L$ a \# `6 T0 c8 h data files during merge. May be a comma-separated list of6 w4 `7 {. p& ] j7 W4 N
directories on different devices in order to spread disk i/o.
) E7 X& \) z0 }0 f( L$ T. U" h Directories that do not exist are ignored.
; p$ {) U& G/ a+ }. ^ </description>6 Z8 C! n3 Y$ U4 U7 E# t
</property>
6 C2 x5 [: [4 g; R' A0 R; {0 q1 a1 @<property>
+ |( @1 v4 C1 \5 f. I <name>io.map.index.skip</name>/ i" n0 M. l" W3 A
<value>0</value>
% G/ r2 {* }: W. d <description>Number of index entries to skip between each entry.
O: M5 S/ X8 C4 {1 `7 w Zero by default. Setting this to values larger than zero can: K8 @9 V- i( ^5 K4 q
facilitate opening large MapFiles using less memory.</description>
8 C% _ u, @* }7 ]- m9 Q N* P</property>$ k$ D' X7 N- I; K
<property>
7 p% Q+ j4 a# e: _ <name>io.map.index.interval</name>1 @, r$ }, ^/ `( m) u) W
<value>128</value>
% I& T: S; v$ D1 H7 X6 S% Z <description>$ O6 k3 m& S# H6 W
MapFile consist of two files - data file (tuples) and index file" }6 N p5 i2 ?$ r# G% |/ n, F) r! c6 |
(keys). For every io.map.index.interval records written in the
$ H6 O& S7 c$ n, X2 T* n& ? data file, an entry (record-key, data-file-position) is written# w/ i; v& u) Q1 D* T
in the index file. This is to allow for doing binary search later
$ E3 ]) |" `7 L. z9 f, g& [ within the index file to look up records by their keys and get their H; T$ V! _- e; A, Y" W3 h+ J7 A
closest positions in the data file.
3 }' K3 w: ]* d0 ]1 T: J </description>
9 x) D1 }/ z. K/ A! w" X2 G. k( v</property>
& C$ ~1 J& s9 S# \<property>: j$ R' Q0 D ~# z ^
<name>io.erasurecode.codec.rs.rawcoders</name>
" U( C/ g' h: d- |8 X. ] <value>rs_native,rs_java</value>9 D% _+ G* F1 |2 o
<description>6 `& c+ b. p: |7 G; h; o! Z" b! n! s' I
Comma separated raw coder implementations for the rs codec. The earlier J6 e8 f& M7 E5 z. \
factory is prior to followings in case of failure of creating raw coders.
, M1 J6 d5 C. P. B$ g </description>
% p$ A! r$ g/ G9 u. P</property>. p5 K$ W- X% J. ^: E0 n
<property>) T9 v6 f0 k0 l: {0 i6 i4 @
<name>io.erasurecode.codec.rs-legacy.rawcoders</name>
2 i3 d- B2 v( Z <value>rs-legacy_java</value>
& ?/ P) [* t% v" L7 I0 u4 h <description>' |1 Y+ \/ v0 S$ Y
Comma separated raw coder implementations for the rs-legacy codec. The earlier# b2 x+ K' U# d8 A K/ N, N. Q, [3 w
factory is prior to followings in case of failure of creating raw coders.9 I* p* B# R2 p" T& Q
</description>. J7 I* G; a7 N+ e9 P9 E/ J
</property>3 b1 [6 Y2 g% y3 _# X* J
<property>
% _+ f# p8 X W3 r c. \3 K <name>io.erasurecode.codec.xor.rawcoders</name>
7 ]+ T5 D. W, ]; M( x2 m' C <value>xor_native,xor_java</value>
q6 `+ ~, {! ~; ?3 z! r: f# c' S <description>6 D/ z* g0 l" d9 ^# ]) S$ C3 l4 N
Comma separated raw coder implementations for the xor codec. The earlier$ C# H$ c" W$ b
factory is prior to followings in case of failure of creating raw coders.
5 w: {. ~; S" j" u- @5 }2 ^/ ]7 _- l, R </description>3 B. Y+ P1 J& t8 W2 y6 _
</property>) }' k- T+ { c! M9 X3 A
<!-- file system properties -->
8 F' u; d7 X6 G<property>0 {8 n- ]+ M @, |' L7 v& ]
<name>fs.defaultFS</name>
, Q! W* J& U( h/ ^7 T7 e <value>file:///</value>
* x) M, t' l3 r; d9 w$ t& S' I5 k, B <description>The name of the default file system. A URI whose
0 E" I. l5 ? `1 ~# I0 v1 [/ w1 c scheme and authority determine the FileSystem implementation. The8 t/ V; N' q, F
uri's scheme determines the config property (fs.SCHEME.impl) naming0 C( {. Z) W( m/ ]
the FileSystem implementation class. The uri's authority is used to
0 ]' q: E. U* u* i% [5 \ determine the host, port, etc. for a filesystem.</description>7 k5 M g1 Y6 y
</property>/ q3 r4 Q! {7 T4 T1 H& ]
<property>
7 H7 {. h w- X' N) E& g; W <name>fs.default.name</name>5 q1 T% `, `. U7 N! a4 s
<value>file:///</value>! W7 H: s' G( E
<description>Deprecated. Use (fs.defaultFS) property
" K( t& O- r5 n1 j; A( q# I, x3 g instead</description>
$ R2 v5 P2 a/ ~" J</property>
8 l4 T; Y T j$ d<property>
4 K' O) Q, @9 D, g2 \+ h6 G <name>fs.trash.interval</name>3 T2 ^( v/ N) Q$ u. e3 O* G7 j! F
<value>0</value>
* S5 P) c' G% M4 v2 G <description>Number of minutes after which the checkpoint) K6 w9 J; u5 o3 X# {
gets deleted. If zero, the trash feature is disabled.
* p X) N* x2 v5 C! N9 L8 e2 g This option may be configured both on the server and the; i8 m4 S5 ]4 c" K
client. If trash is disabled server side then the client
2 [2 _, U6 E+ j5 B Q4 s9 D side configuration is checked. If trash is enabled on the# z2 H, b) s( D# Q( f5 x0 v, @# n
server side then the value configured on the server is
) D9 g( t: H3 }( p- @( t5 q used and the client configuration value is ignored.
( n9 Y/ ]: \1 Y! B1 U* G X3 i5 ]" ~ </description>- |* L1 D% a! x: q( M$ q6 z. @
</property>* N M* L# O; V1 a Y5 p1 {
<property>8 g3 L: y6 z( i) |+ s
<name>fs.trash.checkpoint.interval</name>
& n3 g& y. m- e" t <value>0</value>. E, c* [0 s5 N% I# Z/ u
<description>Number of minutes between trash checkpoints.
" D+ x# }) h2 i+ A G. c, C- h2 U Should be smaller or equal to fs.trash.interval. If zero,
( b, P3 F, X! ]2 i9 E% ^ the value is set to the value of fs.trash.interval." i2 `! X7 v5 h$ v
Every time the checkpointer runs it creates a new checkpoint5 F$ {' i2 }3 M
out of current and removes checkpoints created more than/ Y4 D- t% l/ |/ I& ^3 y$ W
fs.trash.interval minutes ago.5 `; A$ M$ E' X& G+ E
</description>
( i* z: X2 F$ P U5 |</property>4 W; D# D Y# r
<property>
: o( `' |# K: i9 B3 \1 n' j <name>fs.protected.directories</name>3 M5 \; p8 q* p2 E
<value></value>
1 k5 i0 k) P! E; t% N0 I <description>A comma-separated list of directories which cannot
4 i7 @: o% q& ?- r! \9 j3 ~ be deleted even by the superuser unless they are empty. This
$ ~. U, Y. J |1 Z setting can be used to guard important system directories
* D% K- A4 p+ K0 O: _$ W against accidental deletion due to administrator error.
" X$ V3 f# c1 g, ? </description>. E- f; C, q: ] e1 m0 s% w
</property>
& }- y/ O- i- g9 }<property>: i: H% [$ u. o0 ?# Z* O
<name>fs.AbstractFileSystem.file.impl</name>
3 Z0 t: X0 t- @, ~/ n5 ]% K <value>org.apache.hadoop.fs.local.LocalFs</value>
3 {) k* L8 R2 v) a <description>The AbstractFileSystem for file: uris.</description>
, F1 S* [+ I6 Y) P6 S/ r1 q</property>
# ?% U; S: Y8 o h! P' I6 a<property>
' d9 F$ \) x% |: o) T* J+ Y0 A# I <name>fs.AbstractFileSystem.har.impl</name>* B8 p! | s3 X% a& C. T8 ^$ ]# H
<value>org.apache.hadoop.fs.HarFs</value>4 o1 ?& I4 R4 {1 [8 r5 R7 L# ?: O! S5 O
<description>The AbstractFileSystem for har: uris.</description>
% Y: F6 l/ F( }/ A</property>
% F3 y7 K& D/ e<property>4 }( i# F7 ~# K: U* ~5 Q* i
<name>fs.AbstractFileSystem.hdfs.impl</name>
5 L# f# S8 N$ j5 v6 d <value>org.apache.hadoop.fs.Hdfs</value>0 Y/ I6 R4 e {- I8 Z
<description>The FileSystem for hdfs: uris.</description>0 _. o. |& ^( ^) l0 e$ z
</property>; `( r" K# U" ?. v& ]2 E/ V
<property>& b5 ~4 N* W3 }# E) D
<name>fs.AbstractFileSystem.viewfs.impl</name>5 y/ ?9 ^: a5 x0 f! C6 C6 ]" M
<value>org.apache.hadoop.fs.viewfs.ViewFs</value>
" Y& i' ]8 I+ ]. x6 l <description>The AbstractFileSystem for view file system for viewfs: uris" l. k1 P! n# M; l# Z
(ie client side mount table:).</description>& ?, g2 [7 E4 W0 }& c1 L
</property>" F6 U7 k1 M! @) u$ C& T
<property>
' q2 X3 t% g+ h5 K5 S! {" g* `% E <name>fs.viewfs.rename.strategy</name>
: U" h# J5 @+ q( D4 Y$ h <value>SAME_MOUNTPOINT</value>
7 _0 r/ M- \$ }6 x <description>Allowed rename strategy to rename between multiple mountpoints.
" n) T$ G! d8 n+ Q; q Allowed values are SAME_MOUNTPOINT,SAME_TARGET_URI_ACROSS_MOUNTPOINT and6 t. T! E! A8 D) R6 L6 ^* {3 L! ~
SAME_FILESYSTEM_ACROSS_MOUNTPOINT.% d# g! t; B/ K# r- V
</description>( M" y4 X% z+ r# D
</property>" ]" U9 f2 T3 \: V- ?- {
<property>1 v; G" U8 f$ O) q2 ~
<name>fs.AbstractFileSystem.ftp.impl</name>1 Z5 e8 L- N8 ]9 L. X$ R7 X
<value>org.apache.hadoop.fs.ftp.FtpFs</value>% T: d6 B7 N% I$ d7 A
<description>The FileSystem for Ftp: uris.</description>
- n& |4 i/ Q% L4 q9 ]& d</property>' i, t# i: s r9 y# I# _
<property>
8 p" m/ }: E. w" W* {+ ` <name>fs.ftp.impl</name>4 s4 [! ]5 G+ v- x+ Z$ d
<value>org.apache.hadoop.fs.ftp.FTPFileSystem</value>
2 m" B; U6 Q' [3 D <description>The implementation class of the FTP FileSystem</description>
* Z a8 s8 |1 E( @$ z7 ~- g3 n- ^3 i</property> R- E' s' U# X4 x# [9 O K9 g
<property>! _+ ~; L, Z' f; {+ g7 W4 k
<name>fs.AbstractFileSystem.webhdfs.impl</name>
9 k) K: O6 k# {6 X4 A <value>org.apache.hadoop.fs.WebHdfs</value>% P) I5 s$ ` Y, `# b
<description>The FileSystem for webhdfs: uris.</description>' ~/ C6 m) x# N3 q' o' W$ ?
</property>) u* i8 m9 I- P" M+ E
<property>7 o+ X) u9 A- W5 @5 J$ I! A' W V
<name>fs.AbstractFileSystem.swebhdfs.impl</name>
8 Z3 S& ]- ^0 O7 r* @" ]5 ~ <value>org.apache.hadoop.fs.SWebHdfs</value>2 K# P) f' u# I: d
<description>The FileSystem for swebhdfs: uris.</description>
: N1 |( h& O% B1 O0 F$ c) e</property>: @7 A/ J7 P% E5 }
<property>
- r6 a, ?& h K <name>fs.ftp.host</name>
0 s S" m# L/ ]! x7 l <value>0.0.0.0</value>
8 N9 N' S: J3 c <description>FTP filesystem connects to this server</description>
" d- G8 \+ e4 Z5 f. v$ u</property>5 G. R X6 p1 b7 Z: I% h0 o% a
<property> ^1 Q; A% t) U4 ^8 X) E, d9 _
<name>fs.ftp.host.port</name>
5 z& O V5 j- W, v2 ^ <value>21</value>: o2 t2 K" P# a0 \3 P) u3 d5 k
<description>7 T1 }. J6 S! _4 P
FTP filesystem connects to fs.ftp.host on this port0 u/ k7 ?0 `4 O0 a
</description>5 _. V( ^7 ?! @/ d3 \0 [
</property>+ Q# Z3 D) A6 {8 s* R
<property>" W" D. I2 {! L Y6 X% O5 h2 {
<name>fs.ftp.data.connection.mode</name>; t- M& s' n0 [7 H4 V! Q7 n
<value>ACTIVE_LOCAL_DATA_CONNECTION_MODE</value>2 K# r! q" v7 ^$ T% J
<description>Set the FTPClient's data connection mode based on configuration.
0 q- ?- I& t& A% x/ v% \. J Valid values are ACTIVE_LOCAL_DATA_CONNECTION_MODE,
: W9 \! [ }" \8 Z# W% V PASSIVE_LOCAL_DATA_CONNECTION_MODE and PASSIVE_REMOTE_DATA_CONNECTION_MODE.3 W) {2 ]9 q* l) b; K- k
</description>9 c8 a! u9 ~( E2 |
</property>
1 ?& m" u: |' x# w0 q<property>: g& b: \; {% F: v2 N6 v
<name>fs.ftp.transfer.mode</name>
% |$ o) @# C0 F6 t4 @7 T* l8 h6 q <value>BLOCK_TRANSFER_MODE</value>) [3 v- i' f3 P' W! j
<description>" a, L' w* h/ l# Q7 Z
Set FTP's transfer mode based on configuration. Valid values are; Y8 Y6 [" m3 t( \
STREAM_TRANSFER_MODE, BLOCK_TRANSFER_MODE and COMPRESSED_TRANSFER_MODE.
3 a( a6 R. O( l$ u </description>
& i( d( C0 o& x5 o$ w8 c5 Q</property>) e" F8 X) {" D3 Y( H
<property>
/ x, F! n6 j& K; W* H <name>fs.df.interval</name>
& l: \9 W9 i# R+ { <value>60000</value>' e0 Z7 |) c- L4 s4 @7 U# d
<description>Disk usage statistics refresh interval in msec.</description>
& `7 _6 r: w: l( Y* u8 n) m6 l</property>
4 @1 I6 M% @' g<property>
1 D4 f" B" C n0 O2 k <name>fs.du.interval</name>: C" f) K" N+ V
<value>600000</value>* r9 N+ _, O: _
<description>File space usage statistics refresh interval in msec.</description>; W/ p4 S3 W! i0 Y9 u" f
</property>
; k; [* n- \% K" @ e Y<property>' c5 Y7 X. {+ b$ T
<name>fs.swift.impl</name>
1 v- b3 g+ j: }) P) h <value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>
, L- L0 b; s! \ r$ ]2 P* Y1 c <description>The implementation class of the OpenStack Swift Filesystem</description>
3 Y1 M4 F! V% V% c- a</property>5 e/ A. m/ {6 D: S# j @
<property>
. J7 W# x; F& `( @5 `5 N& P <name>fs.automatic.close</name>' `$ u$ _( M& e7 T0 {
<value>true</value>) Z# V( p1 n8 m) G- J
<description>By default, FileSystem instances are automatically closed at program* } N y& r" k. B
exit using a JVM shutdown hook. Setting this property to false disables this/ u( [2 ~- r6 q! Y7 r# l3 N6 Y
behavior. This is an advanced option that should only be used by server applications# O$ y, ]- h5 S Y; @' r
requiring a more carefully orchestrated shutdown sequence.+ v/ x- e/ \7 U% K$ X
</description>
0 p. R. [4 \1 `</property>
+ L: @" D4 v4 e& ^: q6 C3 T3 Z$ C<property>
" o7 W) @: Z' P4 X' N# ? <name>fs.s3a.access.key</name>
t! j# c, i9 X1 k( {/ n" Q <description>AWS access key ID used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
+ ~$ R4 U; f/ ~" I</property>. C; f# E; x1 u& f
<property>
/ K% w3 a3 j% Y; r$ L" i <name>fs.s3a.secret.key</name>9 k# H" t- C6 j
<description>AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
- ]! H e' e( `- Z6 x3 Q& a! o</property>) N7 M/ _: P' |7 G+ y: _
<property>8 a0 T2 F' O2 p/ o( y
<name>fs.s3a.aws.credentials.provider</name>
- M2 Z0 _4 ~# } <description>/ Y/ Y m1 d: d- U/ b) v7 X5 T& Z
Comma-separated class names of credential provider classes which implement
6 V' ~4 M; P3 U. m7 {1 U, ] com.amazonaws.auth.AWSCredentialsProvider.0 u1 V1 ^( R3 f# Z( d) }
These are loaded and queried in sequence for a valid set of credentials.* i! s5 y/ o' L" I0 }
Each listed class must implement one of the following means of! z$ `) f# }8 b h
construction, which are attempted in order:# b7 T+ p& n* N2 I5 p" J
1. a public constructor accepting java.net.URI and! M" q/ K: X; D4 T) ?0 S1 v. w
org.apache.hadoop.conf.Configuration,
3 k: \ |: ~% }* h% h' ~ 2. a public static method named getInstance that accepts no- f2 O4 F6 g+ j9 f. W. Q! ?
arguments and returns an instance of# z% k: E6 P+ [1 v) s6 V5 E ~8 E/ o
com.amazonaws.auth.AWSCredentialsProvider, or
2 \: R q; K3 d% B& ~/ q2 ~ 3. a public default constructor.2 T' P: \7 J+ M) b* e0 I+ \
Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows: q/ m9 M) H% d8 I1 x* E* {
anonymous access to a publicly accessible S3 bucket without any credentials.$ ?! A v! ^. {) w( E( S" s# i
Please note that allowing anonymous access to an S3 bucket compromises& W' G1 L' W) e7 s% P, d
security and therefore is unsuitable for most use cases. It can be useful) A* H+ U0 i; f, F, Y
for accessing public data sets without requiring AWS credentials.
7 a1 z! k! V- z: a: V If unspecified, then the default list of credential provider classes,
3 h& g' t, V- Q o* r; ` queried in sequence, is:
' v+ N2 T: ~% K7 W/ \" a& q 1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static
, p2 q k3 A: `. F configuration of AWS access key ID and secret access key. See also8 F5 |- c4 X: H- E
fs.s3a.access.key and fs.s3a.secret.key.0 Y' W4 O$ e; }8 Y4 `. l
2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports
: j9 u! u, u" s- b* @; d! E configuration of AWS access key ID and secret access key in9 P% i! e1 u; T% h" \
environment variables named AWS_ACCESS_KEY_ID and+ C1 V$ K- N( p, f* P
AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK.( J: p/ j: t1 R- e* e0 H
3. com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use
/ E: R" d3 v9 z; _+ p of instance profile credentials if running in an EC2 VM.$ F4 y# @' ^) S) Z0 g- m
</description>! D8 W. D5 K4 j5 ]
</property>
, p8 P' b5 s" B U<property>
& D+ i$ ]' {# s+ Q. x$ T <name>fs.s3a.session.token</name>- N7 f1 e' l: }1 }3 j1 u/ _8 E
<description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider6 V' F3 T; M9 C7 _6 P2 F& @& f2 ^
as one of the providers.0 l- S* C- K; V. m
</description>
- ` b0 p7 P0 ]. Q9 N% ^" u5 p</property>
, `( {" V: z0 Y. R& Z<property>
3 L0 b" J" t, E3 k <name>fs.s3a.security.credential.provider.path</name>" n- U0 z3 w' A7 l) g
<value />
2 P/ S7 H* [+ N* M/ g @ ]( O <description>
7 X; H) z- V7 K Optional comma separated list of credential providers, a list
/ h' y" Z: \: ]7 z6 y% P" j7 F which is prepended to that set in hadoop.security.credential.provider.path* [% ]+ u s( b# F% c8 B
</description>* l" \, x7 a# s$ E7 y' u
</property>
! N" z T: U- G' s<property>4 R; A" r; ~) e
<name>fs.s3a.assumed.role.arn</name>
. S' r* s- \& P) [. J4 S1 | k% h <value />
2 ~* y$ G) b' T, J <description>) _* I/ v! F( D
AWS ARN for the role to be assumed.
; c2 t* u/ q" q B7 n r2 u& ]! @ Required if the fs.s3a.aws.credentials.provider contains6 p# t0 p" A q
org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider6 n0 t' F5 I7 e
</description>
, ~! h) X, o2 A5 U</property>
! p, M) Y& B) J% v7 E& }2 Y<property>* @+ _4 [. A( h# y, t) J$ ?: f& [
<name>fs.s3a.assumed.role.session.name</name>
0 {9 @/ x; U7 l% r5 r6 Q <value />' U. ]4 g! A! Z0 A
<description>
$ H2 M9 }6 M! ~2 b" V Session name for the assumed role, must be valid characters according to
8 p. |& u& v, v: U) \7 P the AWS APIs.
: U3 ^* u$ f+ G8 `( K Only used if AssumedRoleCredentialProvider is the AWS credential provider.
3 u6 h, F5 N I/ ]% [6 g5 `5 Y If not set, one is generated from the current Hadoop/Kerberos username.+ {$ w1 p7 x0 p& I" J: {" {4 b! R
</description>, E4 t3 ]% P/ T# ~% j) j
</property>
4 s5 `( `! Y) B<property>
$ ^% ]8 o$ A* P7 Q ^5 D( F <name>fs.s3a.assumed.role.policy</name>
' f: L$ O! _: \) J- J; { <value/>
) u# P% h5 @4 H4 m <description>
" W% Y; ?! W" K JSON policy to apply to the role.5 P, p | [7 b2 B+ U0 V
Only used if AssumedRoleCredentialProvider is the AWS credential provider.2 A, {7 k0 K6 p6 R$ u
</description>
6 V8 k1 P+ f* v' x</property>) t3 D X! X' H' e d, K
<property>1 C1 K1 l& D- G, i4 p' ]' Z, W) ` L
<name>fs.s3a.assumed.role.session.duration</name>
5 F& ^1 n" T+ ?' ?3 p! i7 v8 Y( A <value>30m</value>
8 d6 o: I6 S N! J$ B <description>4 v( N# K% }, W* K; [ b' R3 ^
Duration of assumed roles before a refresh is attempted.8 S A& d- k& f( u. y# O6 w
Only used if AssumedRoleCredentialProvider is the AWS credential provider.
$ Q1 J' e( g1 `( [ Range: 15m to 1h2 }3 v% v* k8 K* j
</description>% E# Y0 c2 t- O2 ? Z8 A- ^
</property>
. ?9 V% n+ ]( P8 X" a; z8 u$ U* C<property>
- H! u: J. t0 ^5 Q <name>fs.s3a.assumed.role.sts.endpoint</name>
! Y6 k( G6 A3 b3 V" G% {- s <value/>
2 E7 u1 z% x. P ? <description>
) j6 G4 \5 U) X& R' y1 y5 Y% u' V/ B AWS Simple Token Service Endpoint. If unset, uses the default endpoint.
; ^3 A+ e/ l$ M8 m3 F Only used if AssumedRoleCredentialProvider is the AWS credential provider.2 }; [) T3 `- O- G: r
</description>
9 j1 E/ J$ |2 h8 j4 T$ ?5 O</property>3 h6 b# L+ m. \+ f1 `
<property>* A& t) g0 q1 |# |, e
<name>fs.s3a.assumed.role.credentials.provider</name> F0 J: B9 [, q$ @
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
! v$ ?2 P# q+ m" s. f% E <description>
' v+ F! J: J; B) i. i B List of credential providers to authenticate with the STS endpoint and$ c- a9 O0 }, S4 c
retrieve short-lived role credentials.) Q- J4 c7 k0 t' x' c* D% x
Only used if AssumedRoleCredentialProvider is the AWS credential provider.
( J4 p! L' z; k, a7 |7 P: j If unset, uses "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider".& f; T: }( ~/ e/ S
</description>9 c' Z& T- ]! p1 Q5 |0 r
</property>% J0 s/ o [- [9 p: A
<property>
: l6 q/ n/ O' w <name>fs.s3a.connection.maximum</name>
/ t2 |: t% p2 I4 U <value>15</value>; G& V5 v* d: J6 H3 F# ]! A
<description>Controls the maximum number of simultaneous connections to S3.</description>. w% z G7 a# n5 l" O
</property>; F' S$ _9 X! a& R& K3 e
<property>9 e h! _; O8 M- l- p1 Z
<name>fs.s3a.connection.ssl.enabled</name>: t# c& b, W( e7 \4 w, e# X
<value>true</value>
) p& B' k4 W I( P0 q <description>Enables or disables SSL connections to S3.</description>
0 r6 c: U9 p7 m* g/ g; l- H</property>
: b5 {. X; l, A0 O1 M<property>% M+ q% i$ a" b
<name>fs.s3a.endpoint</name>& [/ [5 C6 N# p. ?
<description>AWS S3 endpoint to connect to. An up-to-date list is
+ N& r% H0 F+ b( r7 Q9 x provided in the AWS Documentation: regions and endpoints. Without this7 W# p9 \9 K; M
property, the standard region (s3.amazonaws.com) is assumed./ _' z" ], S2 d
</description>" R! p" r, G, d4 l: [
</property>5 i( k1 ~' M- n+ t
<property>+ |: r4 X: j' f9 a/ J9 |% g: y9 u
<name>fs.s3a.path.style.access</name>5 i; w$ x* H" Y& y" c3 y
<value>false</value>
2 Z/ `5 U2 c. N4 o8 u/ r <description>Enable S3 path style access ie disabling the default virtual hosting behaviour./ }: C2 k+ L% |1 t2 f1 ^$ x' J
Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting.
7 v4 B$ y3 O. K% P; d. p, R4 T </description>, s; L+ B' \* K H- D0 J' H
</property>* }) |% e9 _. }" t
<property>
2 t( K% R9 t* s, f$ A1 B <name>fs.s3a.proxy.host</name>$ i1 P) [+ A7 K2 @6 z* L
<description>Hostname of the (optional) proxy server for S3 connections.</description>) e8 A& R5 L, ?% D" @
</property>
' P4 o; \' `' K<property>
7 D% e, y) a% {) \; N* N. O: y <name>fs.s3a.proxy.port</name>
0 T. n( I/ d3 N2 I! ^5 p <description>Proxy server port. If this property is not set* X: v1 U2 [. a( U; k# g9 Q! ^6 U
but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with1 n/ }( {4 w; r" A; v
the value of fs.s3a.connection.ssl.enabled).</description>7 @: Q* t/ O9 K+ G
</property>0 F! {4 q( f9 n/ j" I) b8 ?9 E
<property>
, W! u: {( Y1 z' s: m4 N <name>fs.s3a.proxy.username</name>& ?( r0 G7 d/ Z: Y
<description>Username for authenticating with proxy server.</description>
8 f: H7 M' J# J0 }/ ]</property>
9 K( m. v0 z3 l<property>2 u' L$ Y* s$ D/ J! |3 W" E
<name>fs.s3a.proxy.password</name>/ M1 G$ w5 m$ ]' _' H& s7 W* T+ F/ m
<description>Password for authenticating with proxy server.</description>
; E9 `4 E6 R3 v3 o2 k) h4 j1 |</property>
8 c: x1 s5 y; P$ h3 ~4 I' m<property>" \7 X; e9 P' J. B4 _
<name>fs.s3a.proxy.domain</name>
7 |9 ` x, f* V <description>Domain for authenticating with proxy server.</description>
, x$ K/ a/ Y5 l# t2 h& l</property>: r# e( ^0 J% G# q9 F. f- A- I
<property>
~# K y3 O' Y1 Q6 w' D <name>fs.s3a.proxy.workstation</name>% @2 G. h4 s2 D9 h. x
<description>Workstation for authenticating with proxy server.</description> M( }( x) P ]8 C5 O! `1 Z" B9 I& t6 a
</property>
) z3 e# t* |* U( a& i& e, X- O<property>. L8 U, K$ Q+ R+ k' O0 d6 e
<name>fs.s3a.attempts.maximum</name>
7 \8 I9 F, \7 i$ _! T& Y ?8 M& ]$ R% f <value>20</value>
" ?& L' K% c0 m' T! _% f <description>How many times we should retry commands on transient errors.</description>
4 I& y7 A( S$ Y7 }$ l</property>
7 g/ x K' ~) s( Q: C% c1 l<property>' A! ?+ Z. g* ]! L' L- f
<name>fs.s3a.connection.establish.timeout</name>
# K7 ^6 |/ u( |& A: }9 `0 w( D <value>5000</value>5 D: t# j2 u) `/ q5 s# \+ t
<description>Socket connection setup timeout in milliseconds.</description>; g; }3 O1 C2 M6 a+ m
</property>
' z( d; D: V4 l$ x<property>
3 D7 j* o, p g T3 g) z5 O( L <name>fs.s3a.connection.timeout</name>
' `. Q$ l( o5 U# O <value>200000</value>7 b. N [' C5 `* D
<description>Socket connection timeout in milliseconds.</description>
- L. E- a3 B( n t2 M8 [' D</property># X4 O1 b+ s5 l1 H1 C% L- o# D
<property>, V# F! }/ K5 k+ H
<name>fs.s3a.socket.send.buffer</name> Y4 S: H7 o$ J, D
<value>8192</value>
( | D1 f* a( w2 N5 H+ R <description>Socket send buffer hint to amazon connector. Represented in bytes.</description>6 D$ W7 r O. \1 x% `2 a, z" P- y
</property>
1 H% f9 E( p3 P+ q* l! b6 t7 |+ j8 B<property>
3 J+ k& J6 r2 Q# y5 p. P* q) @- u <name>fs.s3a.socket.recv.buffer</name>
+ u! o7 [1 u5 d# Q( ~: d <value>8192</value>
) m( [: h2 I$ o <description>Socket receive buffer hint to amazon connector. Represented in bytes.</description>
% z$ s2 w, _4 T3 l' ~</property>4 p5 ~3 W( k2 v! s' J
<property>
) R/ _% z/ f" P" r/ r" q( c; s <name>fs.s3a.paging.maximum</name>
7 L: t! F, ]+ Z4 S <value>5000</value>; Y x+ d/ ~/ ~" B# T
<description>How many keys to request from S3 when doing0 |3 d6 r$ n0 k. s. [5 I( @2 V1 {
directory listings at a time.</description>; a% J. I6 X% w4 f7 k+ S+ K
</property>
/ ?; I/ n: G+ v- P<property>
7 P! H0 `0 S0 ^0 H: t5 H V7 ] <name>fs.s3a.threads.max</name>
8 M: H4 s! o$ d9 E8 T0 C <value>10</value>' _' h, j U9 i0 H) c) g5 d) ?' r
<description>The total number of threads available in the filesystem for data
" T1 Z" }0 H5 @: B9 b8 X0 x uploads *or any other queued filesystem operation*.</description>3 [2 t, g! T# m8 f
</property>1 O; p. n0 a6 J
<property>
5 @( |, W1 }# i3 p, }- f <name>fs.s3a.threads.keepalivetime</name>8 w* Z9 v/ L2 o5 z/ x" ?
<value>60</value>
& m9 \ D4 R; u <description>Number of seconds a thread can be idle before being) _, x1 H3 b. v
terminated.</description>
% |/ V+ g) Z& E( \! H' a, z</property>
; _' `4 d2 M7 g J) m; h) ]<property>( x' i* z, ^+ I5 H F
<name>fs.s3a.max.total.tasks</name>
0 J; Y* M k& U <value>5</value>5 y* G4 Y! F& k$ S4 o
<description>The number of operations which can be queued for execution</description>
* W$ t/ G( }* ?9 |</property>1 o# B$ o: a" A+ H$ o* B1 R$ S
<property>
5 `, w ?$ U6 A1 P, b' ~ <name>fs.s3a.multipart.size</name>
/ @/ j% I5 p3 E" W; m <value>100M</value>
: L" g$ S/ ] [, z! @ <description>How big (in bytes) to split upload or copy operations up into.0 L: O& S: Z5 e, Z
A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
. K0 t7 |% n) Z# k8 L </description>
( ]1 Y; A! M: C</property>
6 B; C2 m6 Y+ i g1 [3 X: {6 V, U<property>* A O9 P% F7 O' s* y$ u7 w
<name>fs.s3a.multipart.threshold</name>
, L5 A W1 w- N6 G: c9 b <value>2147483647</value>
. \/ a1 Q. }, ]0 T7 X1 i <description>How big (in bytes) to split upload or copy operations up into.
8 k- y, n' D9 u# V This also controls the partition size in renamed files, as rename() involves
; T$ D+ |% c1 e4 r) n! e copying the source file(s).# ?7 S0 N+ t. L
A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.& R0 ^) }% f4 b6 Z7 r
</description>) Q1 `8 r! g* K# j3 x+ K( Y( O
</property>; H3 g k# i0 k' v, B; Z$ z
<property>
6 F! J8 v+ F E1 }+ O <name>fs.s3a.multiobjectdelete.enable</name>; S3 S' ?( K! q0 ^
<value>true</value>
4 y. O6 L" @- {6 l5 y3 ?2 N <description>When enabled, multiple single-object delete requests are replaced by
) D$ B; _0 F$ q1 r/ V a single 'delete multiple objects'-request, reducing the number of requests.
" @. }5 R: G6 K0 p5 }. d! `2 f Beware: legacy S3-compatible object stores might not support this request.5 i! t9 a6 v) ?1 u2 T# X; M
</description>9 O# \7 @0 s# Q" Y* C
</property>7 G/ z( w" U& C( o4 s
<property>% g3 Z1 [1 b5 C! J9 _- E
<name>fs.s3a.acl.default</name>
) v5 @0 U. P9 L <description>Set a canned ACL for newly created and copied objects. Value may be Private,$ A8 i) X( o5 D' M; s
PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead,) ?' ^( w l& c4 e
or BucketOwnerFullControl.</description>0 }2 k: }" f, J3 n
</property>
# t$ Y' C$ F7 a& t/ V' N2 B8 A9 G0 x<property>
4 e ?" j$ n) A1 s G6 L5 ` <name>fs.s3a.multipart.purge</name>% K8 P6 g, D; Y, C
<value>false</value>
9 X* `3 f6 M8 ~2 } <description>True if you want to purge existing multipart uploads that may not have been8 I1 i6 q! u! l4 b4 G7 w. F
completed/aborted correctly. The corresponding purge age is defined in0 _ }) m G9 V* T# R- ?# ]
fs.s3a.multipart.purge.age.+ G0 O1 ^8 z/ n' b
If set, when the filesystem is instantiated then all outstanding uploads6 ^' n6 V4 X# a m. p& I+ @; J
older than the purge age will be terminated -across the entire bucket.+ Q! R, t8 a/ ~7 }0 ?1 r
This will impact multipart uploads by other applications and users. so should t/ k! g: `# t1 ]8 V
be used sparingly, with an age value chosen to stop failed uploads, without- a& }% Q! G. E# s) P3 s
breaking ongoing operations.# n: L0 S" }" O8 n& v* L
</description>
# g7 \$ L6 g3 k. T c A7 e$ [* V</property>6 i3 E0 L/ J1 z g. U$ G
<property>
4 M, x, M' j+ M# a <name>fs.s3a.multipart.purge.age</name>& ]. _& {( D3 J( J- A& o& d1 m5 r
<value>86400</value>0 @! Q% O/ k1 ^" O. b8 S
<description>Minimum age in seconds of multipart uploads to purge
" l7 l. ~: k5 q on startup if "fs.s3a.multipart.purge" is true
6 _: Y& v. e0 `/ m </description>& T, Q/ c* Y4 y- F4 \, } X
</property>1 J6 x8 c7 B' x: `* ^. q2 P
<property>
$ e/ S- d% u$ A5 ~ <name>fs.s3a.server-side-encryption-algorithm</name>% T1 t" G+ z+ m% I6 Y
<description>Specify a server-side encryption algorithm for s3a: file system.
( Y9 V% c9 R: v! I* m6 Q Unset by default. It supports the following values: 'AES256' (for SSE-S3)," N1 e7 w( l6 u4 r0 A. @" u1 h
'SSE-KMS' and 'SSE-C'./ j+ {9 v8 I4 ~# a7 q/ z, u
</description>; ^" t$ W7 g9 i" _# v- Q5 ~& u
</property>
5 g1 I a+ b0 N, ?# a3 p/ f) v<property>% }4 X& a2 V3 `! {) M U
<name>fs.s3a.server-side-encryption.key</name>2 p( a! q+ ?: {0 G/ S2 f6 e
<description>Specific encryption key to use if fs.s3a.server-side-encryption-algorithm
+ C0 |, G' ~0 P, c$ d- c& X1 d6 r has been set to 'SSE-KMS' or 'SSE-C'. In the case of SSE-C, the value of this property0 T1 |9 t- M6 J
should be the Base64 encoded key. If you are using SSE-KMS and leave this property empty,
6 L: {$ P9 ^5 j" x& ]/ } you'll be using your default's S3 KMS key, otherwise you should set this property to0 {% h% Z0 i7 F3 d$ w: ~" d
the specific KMS key id. V; K+ W1 I' y" q
</description>" U% k+ [6 ^- v2 G: }0 F
</property>- q+ D: V1 F a# T, H( e! F
<property>
. x7 `5 G& T! J <name>fs.s3a.signing-algorithm</name>
% c# m6 y2 v$ y# ~! d <description>Override the default signing algorithm so legacy
( V/ G& ^7 ~; r" c0 k implementations can still be used</description>
* a9 K s5 ]5 q* d* O</property> H: P3 z( w5 G3 \7 X5 K: A- `6 @! h+ c
<property>
' r7 B6 g7 ]# s4 ?2 b <name>fs.s3a.block.size</name>
. C7 J1 B5 f0 p" S <value>32M</value>
: j. P! Q% U' B/ ] <description>Block size to use when reading files using s3a: file system.
! o( f1 h u( r* k8 }8 a* m A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.% Z \3 e$ n2 l% ] C1 I
</description>! H5 u6 a. O# N7 J1 n$ @8 a
</property>
9 S7 }9 \5 N: i/ F3 b0 X$ L<property>5 Y6 s8 w, g: Q3 }$ Y; P* G
<name>fs.s3a.buffer.dir</name>
. T/ g& R* f% s r# f8 O <value>${hadoop.tmp.dir}/s3a</value> }' j$ C( q0 v0 y1 n
<description>Comma separated list of directories that will be used to buffer file9 ^+ j, Y# \' _( k
uploads to.</description>
+ t3 R) v% D7 }- J. _( i</property>4 T. }/ w! C& C8 V- C9 g/ m% q
<property>
# ]! U2 n% C8 ]$ X) a <name>fs.s3a.fast.upload.buffer</name># r; P5 N1 o0 k8 S" d. M. B$ m
<value>disk</value>
5 e0 J) f4 C I( U X <description>, j2 O% {' O0 f! Z+ R5 K& }
The buffering mechanism to for data being written.! q) {7 `5 E% h/ y3 a4 D6 n7 j
Values: disk, array, bytebuffer.: G: x' ?- m4 h4 r
"disk" will use the directories listed in fs.s3a.buffer.dir as
# k( ?( _& n1 e# }* p the location(s) to save data prior to being uploaded.
1 Z) U$ w5 ^% C% n' B "array" uses arrays in the JVM heap# U- R; H: I1 W Y% e
"bytebuffer" uses off-heap memory within the JVM.' W# O8 v1 R& Y) Y
Both "array" and "bytebuffer" will consume memory in a single stream up to the number% ^( a6 R1 T7 V0 R' [8 t
of blocks set by:
D) b9 ^, O/ {, [: W! ?+ ` fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks.( Y$ H4 w6 b* B" j
If using either of these mechanisms, keep this value low2 e% B/ g+ N! R# D- L$ s
The total number of threads performing work across all threads is set by
. T- p0 ]6 L: X, s) w5 s fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued, w' _2 G- G9 h
work items.
8 G2 `. j# N& e H* p+ O: ]- a </description> b/ Z0 {! K' |$ f2 X. E
</property>
, v x# h }, F2 N5 y0 ?% ]3 d4 m<property>8 A9 s$ f- P1 j0 q
<name>fs.s3a.fast.upload.active.blocks</name>
0 V9 b8 h9 r4 O <value>4</value>
9 [0 p6 a3 k( a0 L' P8 M: I <description>
: _2 {0 H, b& E2 p- X Maximum Number of blocks a single output stream can have
% F" w6 L8 n& u active (uploading, or queued to the central FileSystem( w- d# q5 n1 s/ b. z
instance's pool of queued operations.' |. y- L" E! ]$ J( S1 w2 _
This stops a single stream overloading the shared thread pool.
3 J1 y' d# I, f# n" N: r3 u </description>4 y: y6 m J) d s% |
</property>2 w" ~# Q) p3 l2 q
<property>
8 t" ~& x2 [$ w <name>fs.s3a.readahead.range</name>/ [# _' P/ n% m
<value>64K</value>
& t- v2 U" F8 w5 S9 s0 |) c <description>Bytes to read ahead during a seek() before closing and
7 z8 o y. H0 n ~0 y# R re-opening the S3 HTTP connection. This option will be overridden if+ Z. Z# r! @: c5 L& I) L
any call to setReadahead() is made to an open stream.
" M+ |: N [4 e' K7 H A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.* b, B$ R# F& |& j
</description>+ ]8 J! {+ h; F' M8 k! U+ O
</property>
5 W- i2 Z* x x# t$ X* g<property>: l* h: {2 h* P7 v0 W* P
<name>fs.s3a.user.agent.prefix</name>' K( d' M6 d; {2 @0 M
<value></value>9 J! q/ ^8 ]6 D4 {/ h' H: m
<description>& }& p4 ^9 z# W% b7 ^$ j
Sets a custom value that will be prepended to the User-Agent header sent in
+ X/ o4 c S& s/ L9 g# q3 {! d HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header4 E% |! ]2 v h! t1 J) I# {# t
always includes the Hadoop version number followed by a string generated by) l0 o7 b6 `; j0 D& y
the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6".
" p4 A# b# I. _* d0 a If this optional property is set, then its value is prepended to create a
/ A0 @5 Q8 y) z. s customized User-Agent. For example, if this configuration property was set
6 h; t8 O1 r9 u5 Q to "MyApp", then an example of the resulting User-Agent would be" h- f9 |! Z7 x- L6 G, C
"User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6".
8 V& t' ^4 D% P5 d" x1 F </description>: N5 m/ J. M$ A. I% b6 u/ ~& L* M
</property>
$ P# ~4 L0 a5 f$ k: V# e! E) |<property>. Z: L/ O2 G. f+ S8 ] H. U& ^
<name>fs.s3a.metadatastore.authoritative</name>6 J- z2 A5 V& E! Y
<value>false</value>1 E A# x6 \; j# t% g
<description>& Y) `" U7 E5 `
When true, allow MetadataStore implementations to act as source of
' A) r3 [# s3 i# P. M% E truth for getting file status and directory listings. Even if this
8 J& a- h7 {* n; c) ]8 N L: d! M is set to true, MetadataStore implementations may choose not to
* s R" V+ y) x& Q# l" X return authoritative results. If the configured MetadataStore does
3 e; O: p% N( j0 g$ f j not support being authoritative, this setting will have no effect.
2 n# X+ T! I/ } </description>: q4 t6 K/ M6 w4 D @
</property> |$ N4 e% X; W6 n3 Z! x& B
<property>9 v6 V$ d6 X6 H4 B7 c! c
<name>fs.s3a.metadatastore.impl</name>
( v) L/ t7 D& D& }* F) _( ?3 o <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
+ X2 o3 f9 g# u* \2 R2 Y <description>
6 m# q5 M# v( }, r3 F1 b Fully-qualified name of the class that implements the MetadataStore2 X* J* o9 W7 a. Y9 o
to be used by s3a. The default class, NullMetadataStore, has no
* b* f- |! b! `3 D effect: s3a will continue to treat the backing S3 service as the one: f* l6 q. X" {$ Q( d
and only source of truth for file and directory metadata.
) \2 w" J' n6 ] w0 ?! D& @4 } </description>
$ h# ^, ]6 _9 H8 F+ w8 u+ h7 ~</property>& l7 ]8 z" J$ c @# T# p) g* H
<property>
' p/ ?" y8 T0 Z3 o7 F <name>fs.s3a.s3guard.cli.prune.age</name>3 t8 }* W% N) j* q3 }. U6 M
<value>86400000</value>
$ R j! M% q' `4 u <description>/ H, G. Z; l( ^& q
Default age (in milliseconds) after which to prune metadata from the
8 z" J+ x( v# b+ v- k metadatastore when the prune command is run. Can be overridden on the# E/ h. I1 E& y% n' a& g- Y
command-line.
6 F% K/ D& K, e9 B3 j+ |# z; C </description>2 m7 J8 [6 M" v/ [( O
</property>
% C4 v( h0 `% o1 B4 Y5 {9 b5 l& p<property>
4 c& y Z0 e+ m1 o' F* N U: U2 g <name>fs.s3a.impl</name>
4 V9 f A( R4 Q9 [ <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>* e% \9 k1 A% x, k4 N5 N* J: s$ R. R [+ I$ A
<description>The implementation class of the S3A Filesystem</description>
2 [' f9 U+ Q0 U6 k* O+ U5 E: z3 J</property>
( g5 x+ z7 k. l% J- k+ F<property>; I4 F u0 A& G5 K9 m4 F+ M
<name>fs.s3a.s3guard.ddb.region</name>1 k# k; D* A& q5 C9 u( F) ] o3 j; T
<value></value>
; {: x" t! h5 n$ c6 N5 Z <description>
" E/ L7 q6 j9 Y+ { AWS DynamoDB region to connect to. An up-to-date list is" a/ U2 p6 X& Q8 a; P* P& p! d. V
provided in the AWS Documentation: regions and endpoints. Without this% V) D' Z; _! M* U" G* t
property, the S3Guard will operate table in the associated S3 bucket region.$ c; {1 x9 R: T' @/ Y
</description>
0 q( M O# k8 z; O' Y8 Z- R9 ^</property>6 q( K9 N1 h# L- r5 @, _
<property>/ u" h5 x) Q. q6 v8 L7 r( [' b
<name>fs.s3a.s3guard.ddb.table</name>
& X- [+ F: [! C% B. ~4 i <value></value>
/ U: p$ }7 m1 y <description>0 N, |8 L6 N2 n. ]8 A
The DynamoDB table name to operate. Without this property, the respective
! v1 [4 N8 Y# b( R1 H: K S3 bucket name will be used.% }# V M% K5 q6 ^
</description>
* M4 n6 T, x: f9 y4 N' [9 l c</property>0 g! _$ w. E; T9 M5 G. i' ^8 U5 V' J
<property>7 W" T4 ~& s7 X$ x8 J
<name>fs.s3a.s3guard.ddb.table.create</name>
6 j3 E* i2 r! Z <value>false</value>" i: t) G6 \2 X* s2 W: w
<description>
% z/ ]! j3 N7 L* b: b" O- v+ ]% A4 k9 @ If true, the S3A client will create the table if it does not already exist.
4 e: c& @0 U C+ n, q! j; | </description>7 h6 H+ A- |9 Z, x/ w" n- q
</property>1 S$ t% {# p& |1 x# t
<property>
! i( |) J; w9 s3 F9 w, y <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
5 a: q% u, Y5 ?' n2 C s <value>500</value>" F% s& m5 F* S* Y U9 ]
<description>
9 p5 Q+ z1 Z; y5 K Provisioned throughput requirements for read operations in terms of capacity
2 M7 l9 b! M* I; l4 M units for the DynamoDB table. This config value will only be used when$ B1 L% K+ T3 T/ Q$ j# |, ^
creating a new DynamoDB table, though later you can manually provision by/ C) |4 Z7 [8 d1 Q. g: X
increasing or decreasing read capacity as needed for existing tables.
1 i, F/ n" w) _) s$ `/ d# o# D See DynamoDB documents for more information./ \/ |' y+ P- A3 R- g' y ^
</description>: P+ k' s, A. r' }5 S- X7 T
</property>
" | ~; S! D3 D<property>
" U0 i7 _/ w' p- n <name>fs.s3a.s3guard.ddb.table.capacity.write</name>: f" B+ N1 s( d9 F1 i& h# J. R
<value>100</value>7 Q: x8 m5 |4 n+ {, A& w* H
<description>) a' I! U F/ o# }2 ?( Y& i
Provisioned throughput requirements for write operations in terms of
! p( y7 M5 k2 C' a7 q# I capacity units for the DynamoDB table. Refer to related config
' F) M, x; S8 [5 w' H! M1 k/ a8 \ fs.s3a.s3guard.ddb.table.capacity.read before usage.8 S7 t* c7 _9 D+ S- X* g& h
</description>
; A' `: i; q! R! p o% L</property>
2 U. { n: x$ a8 `; V<property>; I& A% E) |* r6 W% M) n
<name>fs.s3a.s3guard.ddb.max.retries</name>! m0 ]. E, M5 A# a
<value>9</value>
) }5 {1 y# t; d" C <description>4 }/ v4 i- g2 }( h% N
Max retries on batched DynamoDB operations before giving up and
! H' D" Y& L% L: d throwing an IOException. Each retry is delayed with an exponential) {5 e4 z0 y0 k6 E, z
backoff timer which starts at 100 milliseconds and approximately" J/ F$ i! _' ^* w5 G
doubles each time. The minimum wait before throwing an exception is! F* W6 T& ?0 v( R$ D( w
sum(100, 200, 400, 800, .. 100*2^N-1 ) == 100 * ((2^N)-1)
( l: y7 g/ A4 ~5 b So N = 9 yields at least 51.1 seconds (51,100) milliseconds of blocking
8 B1 I0 o. l& [% U% ] before throwing an IOException.
4 N% H7 |5 ?% g* c! D6 ]1 | </description>
5 i" @$ ?8 a7 a) H; e6 p</property>
9 a" f/ R6 m n! D3 I- T<property>5 A) A4 \4 m A+ Q
<name>fs.s3a.s3guard.ddb.background.sleep</name>
7 J* W& B9 J$ A. a8 ~/ ^+ P6 ? <value>25</value>
" M! `) p ]: V <description>
2 m$ \+ q) Y1 r/ l+ V5 Q Length (in milliseconds) of pause between each batch of deletes when" V+ s# y' [# g& Z1 N
pruning metadata. Prevents prune operations (which can typically be low
9 s( @2 c$ D2 H# f/ W6 W priority background operations) from overly interfering with other I/O
# m1 a: O `8 Q- x operations.
- d3 P' S( }6 A2 c$ }0 F! ]7 } </description>% p& Z! G; F* b- A% ^
</property>
# A* V9 C% R2 K3 v' T1 ~6 \+ I<property>$ e2 P1 A& ^# o7 z) o2 l) Y/ D! F
<name>fs.s3a.retry.limit</name>. d5 d+ a5 m2 F- l, _/ r0 I
<value>${fs.s3a.attempts.maximum}</value>2 ]" W" @/ J3 e& i" r* ^; b
<description>5 ^( ~; t5 H7 R2 P N$ q: M2 T
Number of times to retry any repeatable S3 client request on failure,
0 @9 Y; ^3 U( I5 [ G9 A( o! s0 y excluding throttling requests.
, _9 x7 E" p* Q4 W" q </description>
" W4 k# e3 `4 j& i, y/ Q( {</property>, F$ R% ?3 R" [/ O* q
<property>
& M- \% \! f9 l) q0 Z <name>fs.s3a.retry.interval</name> e& Q3 {" B u% P
<value>500ms</value>2 R5 [1 N! d- h4 c& u2 K4 y' J
<description>- G% p- m7 A3 {7 `; S" V! ?; X
Interval between attempts to retry operations for any reason other
5 R) y' r" v1 s than S3 throttle errors.
0 B# a+ I H+ d% ~3 B </description>
Y0 C2 F9 ?" b& [. r3 g</property>2 G7 p% e3 _& Q$ c- N4 Z
<property>) Y3 D+ |7 s$ {1 d7 v6 C
<name>fs.s3a.retry.throttle.limit</name>7 ~) g' k. ^) ^* ~4 Y
<value>${fs.s3a.attempts.maximum}</value>
% A; w- [" Z# Q3 X5 k' K <description>7 _! H. r2 _& B$ v0 I
Number of times to retry any throttled request.- ?& P, w2 z7 W7 F) X' S* b7 E5 V
</description>+ d# t6 b3 W5 ^+ p3 B7 d
</property>( w$ O* F$ |" ?" z" I# k7 G
<property>
6 ~; L, C9 y* n7 k& x8 k <name>fs.s3a.retry.throttle.interval</name>
8 ?3 g$ t, R! ] <value>1000ms</value>
. P( d6 k3 }) P" m. Q: o" \) p <description>
6 L; A6 Y2 q& T Interval between retry attempts on throttled requests., b4 t5 ~% Y. k1 b; b- n
</description>
& B% M, v# W, N, T</property>! ^6 H8 k7 x+ T5 I2 c* J
<property>
+ g! l4 C) e% p" \5 W <name>fs.s3a.committer.name</name>1 l& V# h. G7 ?. P# I
<value>file</value>4 E5 o2 J5 H" ?7 X
<description>% Z3 f1 \# H# U# o
Committer to create for output to S3A, one of:
4 M8 o3 O. ?" V0 k! U4 j% T "file", "directory", "partitioned", "magic"., B! m/ V- [& N+ B( e
</description>+ Z* c6 A: X/ \) S5 u6 W n8 \7 n. E
</property>; a( |; I" G) P8 B1 K7 T
<property>' v0 ]5 ?; l7 U% A/ J& |& q9 q
<name>fs.s3a.committer.magic.enabled</name>3 ?. ~( m7 k8 W% r& D
<value>false</value>" T0 P1 g* U1 m. t9 ?- R/ R* k
<description>
8 l r K/ }. ?4 F, C1 P5 Y4 X# z3 R Enable support in the filesystem for the S3 "Magic" committer.
# L9 y/ F5 ^' C# J# ~ p When working with AWS S3, S3Guard must be enabled for the destination# M( n$ q5 I# w- | T2 j
bucket, as consistent metadata listings are required.7 Z. C- f# j8 l
</description>
8 ~# F; w7 V0 k2 o3 f0 t</property>
+ _+ I8 A# c8 |<property>0 n6 |: ]0 }4 `$ U, x
<name>fs.s3a.committer.threads</name>
/ o/ }' z' T3 J4 ? <value>8</value>- M3 T4 r7 ]8 M0 g. b
<description>6 p/ I& i7 ?9 U4 t. S
Number of threads in committers for parallel operations on files, w, K, C6 P9 C1 M1 k
(upload, commit, abort, delete...)
7 {) ?) u$ f6 [( E </description>
+ X) g) L- L4 O9 H</property>
2 f9 M6 O2 c1 p/ W* c5 d' ^- R<property>- P; ]/ N3 F5 u5 o
<name>fs.s3a.committer.staging.tmp.path</name>
. b1 A: t. O2 y <value>tmp/staging</value>
4 h2 S* h9 s7 g6 {5 F- J1 k( N <description>
3 \' w x+ J4 {# R0 k/ n/ C/ R. r Path in the cluster filesystem for temporary data.
# D' M# W3 j* |+ W% j& x This is for HDFS, not the local filesystem.. ]9 W6 Z+ X. X/ M) I# G' f7 a
It is only for the summary data of each file, not the actual
* \7 M. W" K- N7 T data being committed. k1 R: H7 t& L( ?0 |2 N
Using an unqualified path guarantees that the full path will be
4 K' z9 s' k* {6 }5 F% C* M generated relative to the home directory of the user creating the job,
/ Z+ Z2 `, J6 E' O! q, _ hence private (assuming home directory permissions are secure).9 g4 P. ?+ P% n s$ D+ |6 H4 ^# t
</description>
, W7 _" ?3 U7 o' N0 H% T8 S</property>0 A! G+ z# _1 V; _, k
<property>
' }2 [* p7 H) _" C8 r <name>fs.s3a.committer.staging.unique-filenames</name>
$ p* N! W/ a3 \6 B <value>true</value>
. _+ \+ N9 s, y7 z6 X2 K+ { <description>: z. \% a& o8 W
Option for final files to have a unique name through job attempt info,
0 l# x: r$ b% G8 f! U or the value of fs.s3a.committer.staging.uuid
' I: Z) @ }" n3 x3 W! L7 Y: Q When writing data with the "append" conflict option, this guarantees
) j1 D4 ` n+ J6 {: q that new data will not overwrite any existing data.' B( i2 [+ [ b$ i& E+ S4 o; C# C2 [
</description>
2 u2 t6 W% [- K0 P( z5 v1 `5 d: N, f& |</property>
$ Y) _) E9 d$ M$ C/ [<property>
- O3 L. o i7 z <name>fs.s3a.committer.staging.conflict-mode</name>7 F4 P5 p) g n3 Q
<value>fail</value>
& \1 a7 R8 |4 Q& L( z ] <description>+ I. l+ ]0 `5 M( _' R
Staging committer conflict resolution policy.
. C9 y+ Q+ }7 K6 o4 o/ G Supported: "fail", "append", "replace".
# g# q1 i3 W$ i6 v& G+ Q </description>
1 ?7 C0 d8 n! P9 k' `6 w7 \0 d</property>
" w1 r$ D, ^) \: {" D$ h' y) O<property>
& a$ B, E# r6 B( ]( _, C <name>fs.s3a.committer.staging.abort.pending.uploads</name>& A% i" t' g( H; |2 B: S
<value>true</value>$ Z8 _# l8 d( ^5 i8 O
<description>
/ j8 r! n+ ^0 H; O Should the staging committers abort all pending uploads to the destination4 z0 O5 Q7 w: u
directory?9 p* F1 n& {1 ]8 d
Changing this if more than one partitioned committer is
: T' N9 e6 \0 I9 a) z writing to the same destination tree simultaneously; otherwise" y8 p$ V( d4 [# r
the first job to complete will cancel all outstanding uploads from the
$ y' m$ x& j H4 M others. However, it may lead to leaked outstanding uploads from failed) T. p( ?* `7 `8 K9 l
tasks. If disabled, configure the bucket lifecycle to remove uploads. t! h# N) h+ U/ x' \' i. G5 {
after a time period, and/or set up a workflow to explicitly delete
& c9 z; F; n L3 z entries. Otherwise there is a risk that uncommitted uploads may run up( z( d9 _; M1 x6 ]9 W! E5 `, b
bills.
3 i: {. O: [5 D: X </description>7 h% g+ }; I% x7 l) s
</property>
( O! k" T# G2 p8 T& {# c3 ]- O<property>
& j6 ~# m) o$ w; U B8 }3 y <name>fs.AbstractFileSystem.s3a.impl</name>
% t' Z6 g: p& B ]+ P4 z" l <value>org.apache.hadoop.fs.s3a.S3A</value>
# Y; k/ B8 D8 o7 ^$ t ?% { <description>The implementation class of the S3A AbstractFileSystem.</description>5 m9 C B& Y9 Z% N
</property>& `3 m6 y+ W4 W- b4 P9 V1 [
<property>
2 y' R, \# U: R <name>fs.s3a.list.version</name>
/ L& g; y1 Y1 y% Q( o <value>2</value>
$ O: \1 G7 a: J7 {% l+ k ]' ~ <description>5 O6 X9 Q$ _; V) s. v
Select which version of the S3 SDK's List Objects API to use. Currently
. E. [' P- j. q. @; @ support 2 (default) and 1 (older API).
/ b& s7 |+ Q& q </description>' z- E- `+ K3 K
</property>
0 a3 E n5 P( o- _: L7 Z<property>% `* M6 E6 m' m
<name>fs.s3a.etag.checksum.enabled</name>6 B) U' E% A+ i( d2 p
<value>false</value>
. a: c' G; n7 p4 M; q$ x1 A <description>/ c$ s6 S" T) S& @& T- s: x1 l
Should calls to getFileChecksum() return the etag value of the remote9 X9 m2 e: G& E, n# \+ p) l
object.
6 m! c2 o+ Q( Z: n3 e* W WARNING: if enabled, distcp operations between HDFS and S3 will fail unless
% E9 ~% p0 l* f* A -skipcrccheck is set.
( Z1 P5 [# a" k( m. R$ U </description>! g: J, C% r' B& m; A x% ]; X
</property>
$ Z- ]# B* l/ Q2 t. v* [% o<!-- Azure file system properties -->
' k# K5 {& A. u) `& i. Y8 E0 b<property>
" S4 s3 u# I! `; _, @ <name>fs.wasb.impl</name>, f4 v' n4 n) u8 }, G7 w' ~
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>8 P# u4 b" o7 p3 ^" t& S* ]# N
<description>The implementation class of the Native Azure Filesystem</description>
; @" n/ k+ j$ }6 ~/ |</property>6 A H) U6 |' X& y! L4 |5 \
<property>' L: {9 M1 G" L
<name>fs.wasbs.impl</name>2 I# y, y: s* Y0 I+ A
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure</value>
9 q# k9 m: Q# |" B <description>The implementation class of the Secure Native Azure Filesystem</description># i0 {' Z/ g! ^" V. E$ Q+ v; [0 h
</property>' \' ^# @0 A$ i1 i
<property>
2 m+ y* W7 D* D5 v1 ]% u5 J7 a5 ?6 v <name>fs.azure.secure.mode</name>
8 \' @1 @5 J+ `& A; y) J% f <value>false</value>9 L4 H" z- k$ `7 d0 S
<description> `% P. P/ E+ l# r. `
Config flag to identify the mode in which fs.azure.NativeAzureFileSystem needs4 t0 X- q! y0 x f1 U- c4 c) X
to run under. Setting it "true" would make fs.azure.NativeAzureFileSystem use
# }# G- k: a: m5 u$ |' T SAS keys to communicate with Azure storage.: j2 Q k- `3 r$ g3 o
</description>7 [$ K" r0 C$ c+ T
</property>
7 y' y1 U- @ I8 m( {! K4 @<property>5 l& P6 O4 E$ X4 A8 {" S' x
<name>fs.azure.local.sas.key.mode</name>
8 H( K1 L& j+ v/ C% Z4 i/ a <value>false</value>% O9 }7 } _0 b4 ]% M( E) }( h
<description>; P$ {' n! ]* b" \" o+ Q
Works in conjuction with fs.azure.secure.mode. Setting this config to true: M: h0 b" q, R) q, R; X& S g) n& H
results in fs.azure.NativeAzureFileSystem using the local SAS key generation; w- @ k$ v6 l
where the SAS keys are generating in the same process as fs.azure.NativeAzureFileSystem.
2 B3 D# ^( w, i( ^ If fs.azure.secure.mode flag is set to false, this flag has no effect.5 o. U" J# x0 z1 X
</description>7 h/ {$ ]) b8 M# s% Z# E. l( |
</property>
+ R& b1 ]8 ?. E5 j( }<property>: |& H. I% H; [5 `4 w7 c, Q5 x2 o
<name>fs.azure.sas.expiry.period</name>9 |& S! j3 V2 b; n8 V, n& u& q1 a
<value>90d</value>' @2 ~/ o2 A M% q
<description>
3 e u/ q$ U; _' K1 E: j1 R The default value to be used for expiration period for SAS keys generated." ^: J( \; W0 z- @" W
Can use the following suffix (case insensitive):
# b/ p8 _! d+ H$ L3 u/ T ms(millis), s(sec), m(min), h(hour), d(day)) Z# x! w# R3 K/ T8 f( F6 E' c
to specify the time (such as 2s, 2m, 1h, etc.).
! |6 w1 _8 ~' J/ p3 s </description>) m7 h0 U6 e/ D- X% @( C
</property>$ ]& I( V3 X6 |8 K0 A
<property>
; Y; i, u7 }6 W3 a) b" J; g <name>fs.azure.authorization</name>& k! V" x h5 H1 n9 m- x
<value>false</value>6 Z5 m& F7 j! s; n
<description>
( {2 ~1 P9 ~6 \( o% y; S Config flag to enable authorization support in WASB. Setting it to "true" enables6 W1 \+ W2 X" V
authorization support to WASB. Currently WASB authorization requires a remote service
' W5 H8 |! G5 i) Q5 N# F to provide authorization that needs to be specified via fs.azure.authorization.remote.service.url% e& w: L0 h1 H& z, F8 i
configuration
9 m x) H0 h1 Q# s </description>" p& {6 k" w0 h- r5 `
</property>
/ j+ a6 }0 }( T, G2 @3 d<property>6 ~- P& m* l. K& k% }
<name>fs.azure.authorization.caching.enable</name>
& ?: _# d S& k, h/ { <value>true</value>$ s% Y3 [6 T% ~$ S( e1 X4 z
<description>
& ?9 D' d$ d5 X; e; R" s Config flag to enable caching of authorization results and saskeys in WASB. a) t/ p8 q* C5 ^, q
This flag is relevant only when fs.azure.authorization is enabled.
7 Q6 X+ f- x, W$ h& V& t9 O! s: P$ F </description>
' }$ i; y4 x% k: Q- @</property>) N3 ~7 S) l, a$ a
<property>; u# Y8 {* O7 i
<name>fs.azure.saskey.usecontainersaskeyforallaccess</name>
" H. Q$ ]" V! |6 w7 d% y/ e7 s8 s2 d$ V <value>true</value>
. l4 y, A+ L7 N+ { <description>
, S( f4 l0 R7 `' G' g1 [- i Use container saskey for access to all blobs within the container.( S' r: N) J+ j
Blob-specific saskeys are not used when this setting is enabled.
# Q, E" ]! y0 N1 q+ B0 L0 Y This setting provides better performance compared to blob-specific saskeys.
' l: ^: h+ s8 E </description>3 V8 D4 p( t( ?. u) r) Y3 d) |
</property>; y, L4 h. H$ D# H; P. L) _. }
<property>' H C, v5 Z. f' ~' F6 Q6 F
<name>io.seqfile.compress.blocksize</name>
( c. |( A9 X& |) s; R <value>1000000</value>* J3 Y; C) i1 P
<description>The minimum block size for compression in block compressed. Q1 p8 o" N# C: H0 R9 y: l6 w
SequenceFiles.
! g& \1 c; Z3 a </description>" l! |$ U: F9 [4 c( l
</property>
- [$ S, Z; ]% j5 y2 C0 [* s' b) | <property>
& O5 I7 z) _2 B% r1 d/ q$ a8 N <name>io.mapfile.bloom.size</name>0 i! M9 {* W& j$ ]' m( T
<value>1048576</value>
3 a2 {0 T# f3 @/ t4 O <description>The size of BloomFilter-s used in BloomMapFile. Each time this many
# \0 Y3 C3 ^5 I5 e keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter).! o2 r7 [& C A" d# N
Larger values minimize the number of filters, which slightly increases the performance,! |$ P2 y- s/ X- z
but may waste too much space if the total number of keys is usually much smaller$ G6 @3 _* ?& j" }, ~
than this number.9 i: i4 H3 c, y# o* ]
</description>
* x( T; r# w/ B8 m6 ]& a1 j( x. r0 b</property>2 W1 e3 A. n/ y! P! l. |7 f* A( @9 D! ]
<property>$ K8 y3 t1 r. i; E6 U- i9 [7 j' A
<name>io.mapfile.bloom.error.rate</name>" K" L: t* X* ?! k
<value>0.005</value>
0 v& G! ^9 c8 ]% k& F <description>The rate of false positives in BloomFilter-s used in BloomMapFile.' q5 [8 {" _2 x
As this value decreases, the size of BloomFilter-s increases exponentially. This D* n4 c% P3 H+ j3 Q: P2 u
value is the probability of encountering false positives (default is 0.5%).$ m5 c* ?+ C% i: ]
</description>. r) ]: y& a* ?7 H& V: s% ?9 M
</property>! o2 t* A j: G; D- S! P
<property>6 H) s: C l2 w+ |1 e) T* L4 F2 J) J8 ~
<name>hadoop.util.hash.type</name>
$ O( d3 X! B6 ?) B( B" v p( l <value>murmur</value>5 R' H: b7 P. F' p1 i5 z
<description>The default implementation of Hash. Currently this can take one of the
# F' x, y, r" t% _8 w) |; g4 K1 v two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash.
3 ]1 o& l5 ]5 [ </description>% A7 o$ f" G& @$ l! i5 p
</property>2 }' W- g' C% j: _) _3 Q
<!-- ipc properties -->7 X0 D4 q4 S/ H" K
<property>& |6 g" e ~9 t$ W8 |
<name>ipc.client.idlethreshold</name>9 V2 z, `6 u! {* `4 P- }( w5 \" y
<value>4000</value>+ [! X5 Y% u! b
<description>Defines the threshold number of connections after which6 }8 ?& v, N( P" B& Q
connections will be inspected for idleness.: L! H# `% W2 H9 Z* z
</description>
7 j% f$ G- c4 S/ V. g. ?2 W. s</property>9 r/ h; r1 C8 b$ ]: ]$ h) _! r
<property>
9 L" Y* g) {' q' Q6 k+ h <name>ipc.client.kill.max</name>0 K! J& G- I! f4 `+ u% ]
<value>10</value>
0 g2 g+ C; R. ]' G1 | <description>Defines the maximum number of clients to disconnect in one go.
% n3 W: D; l4 k0 P </description>7 P# r' m* h! k) H. Q; g
</property>
1 {% m+ h1 ^5 P* ~9 k6 _8 e<property>( ~* [& S' @2 A" z5 Y& o: [ v
<name>ipc.client.connection.maxidletime</name>( [6 ?! w, }! H) U% B7 o: j
<value>10000</value>: k+ p3 f7 |6 w( C
<description>The maximum time in msec after which a client will bring down the
; g& q1 B1 I" e5 ? connection to the server.
3 F4 T0 Z$ ]4 p </description>
9 C: ~5 J, V* O- J9 y</property>
' n! A4 f! O3 t$ L& q<property>/ J* g1 W* U) M' X
<name>ipc.client.connect.max.retries</name>
- Y# B* R. M5 C <value>10</value>
! r- q- o& O$ L <description>Indicates the number of retries a client will make to establish
: Z$ @7 A( Z. A: A, f+ s a server connection.
4 h* ]' |8 ~4 _* r8 P1 \) s </description>* N4 B" \ t- b0 F! x; e
</property>; @2 t0 i0 X& ~" ^4 [* o
<property>
/ P* p9 y0 h F <name>ipc.client.connect.retry.interval</name>
% R9 m. j5 F: q5 u, ~ <value>1000</value>* r: z( {- Z {+ Y+ [* O! y6 W
<description>Indicates the number of milliseconds a client will wait for/ l# F% l9 b' a. y" P
before retrying to establish a server connection.% |9 U/ F# o% P" y* l+ `) e% g0 `
</description> Z# J! X8 `& Z
</property> _7 d* B5 ?9 Z2 m( X
<property>& {8 v9 e* I/ X! J' i* I5 x, \
<name>ipc.client.connect.timeout</name>2 Z# M! ~0 u8 Q& Z" b- ` N2 |/ h
<value>20000</value>7 M$ W1 X3 J3 w2 @
<description>Indicates the number of milliseconds a client will wait for the
' ]9 r# z/ i J# i2 z7 e socket to establish a server connection.: b2 g9 g2 F/ _ X/ m- A7 l5 q- O
</description>
9 I( t' [5 n) ] Q</property>
# |; u* D: T$ B" S% m<property>' R/ A/ ^( d0 R; Q8 ?
<name>ipc.client.connect.max.retries.on.timeouts</name>
' w2 f1 G g! A/ Z8 k0 M/ `" S <value>45</value>
6 @7 M+ z% J L/ L. V( E; ^( Q <description>Indicates the number of retries a client will make on socket timeout
4 G- j. |! \& z6 k to establish a server connection.
0 L) @6 X' Y4 Z* {. [5 e1 A/ b, Q </description>/ H8 |0 m* q6 S0 f0 }% S
</property>% E. f! N+ S. n0 C
<property>
( F7 @ g6 y. l- J& o" N0 T8 [ <name>ipc.client.tcpnodelay</name>
8 h( } g! e, p2 P <value>true</value>, k$ a [4 ?& g8 t4 t
<description>Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays.2 d$ r/ n6 _. u
</description>, Z4 A/ b5 f6 R$ [; S
</property>
/ B( k+ K3 O8 ^1 V3 ^8 e<property>% f7 W, O. L. ~7 g
<name>ipc.client.low-latency</name>
- D: ^" K2 @; _) \6 F* B$ { <value>false</value>
# v) B4 W0 [% l* {' i( Q0 C! v <description>Use low-latency QoS markers for IPC connections.
2 r- b2 t3 e6 }" t% D </description>
& s4 _" @0 T# n, n% N1 c4 W8 I</property>; V' x" x/ k# y1 ~* _
<property>
# N1 x: }9 y/ j' q$ S; N <name>ipc.client.ping</name>
7 ^/ M j M0 C# Q* ]$ r: ~6 u <value>true</value>
+ L' \* V& K6 D4 S9 R3 k <description>Send a ping to the server when timeout on reading the response,
! t7 Z2 W. S% [" h, _1 [6 D if set to true. If no failure is detected, the client retries until at least
# Z, {. E4 x9 @) J" p a byte is read or the time given by ipc.client.rpc-timeout.ms is passed.+ W1 i# t* V9 n r& }
</description>- K4 q" O. a1 t/ W- h& C* |
</property>- n+ b' K. `% U* j- i' h; @% r
<property>! q1 Y! J8 \/ b2 @/ X1 Z: g
<name>ipc.ping.interval</name>
( @* P d, Z4 l <value>60000</value>0 ^: |! G5 d$ h; V# o7 [! o7 q
<description>Timeout on waiting response from server, in milliseconds.) t/ M i2 x& @$ w% y" Y
The client will send ping when the interval is passed without receiving bytes,
I' N, Q2 Y' v4 o( z if ipc.client.ping is set to true.
0 M1 P0 U* j% Y, O. ^& u </description>
! c* g/ [% ^; S2 ]</property>* ^5 ~5 v. @) I% |6 t7 y
<property>7 e% ?' K; q" K5 h d' G) U1 C
<name>ipc.client.rpc-timeout.ms</name>
1 T4 \4 k" g" n# e/ K9 Y <value>0</value>
. |7 N. i X6 } <description>Timeout on waiting response from server, in milliseconds.
. p# G6 m4 l- G% h If ipc.client.ping is set to true and this rpc-timeout is greater than8 R0 f( {2 ]) G) r. Q% q# z
the value of ipc.ping.interval, the effective value of the rpc-timeout is) Z- K2 |* m, K% t" a8 W2 U
rounded up to multiple of ipc.ping.interval.
$ O& _7 A6 I3 O M& F3 k! y* v7 ` </description>
; N/ H4 q! T0 m</property>
: a$ i* q# | C0 g! b/ r4 ^<property>7 F, k# i/ A1 }; w1 Z' n
<name>ipc.server.listen.queue.size</name>$ y" ^& D1 s( `( B- G) Y7 Z, w5 K
<value>128</value>3 N5 J: j: H+ G9 U' B1 _ J! R
<description>Indicates the length of the listen queue for servers accepting4 w) [+ I; H( R( p/ A1 V
client connections.8 D) x+ ]) e# m
</description>
0 ]3 ~) l2 C7 \# p1 n</property>' t) u7 I4 _; S" v& V
<property>' [) |9 X2 d& F/ p# ?4 _
<name>ipc.server.log.slow.rpc</name>
! d! |0 F: m: E' ~2 H# \" k <value>false</value>
/ \) y, D% ^5 {5 u; Z/ s <description>This setting is useful to troubleshoot performance issues for9 t* S( x0 z% f& K
various services. If this value is set to true then we log requests that3 a* T9 R1 s+ R5 A; S
fall into 99th percentile as well as increment RpcSlowCalls counter.
4 I) B$ W2 l6 `# E+ U, u8 I& _ </description>) ^6 ~5 L K4 U; B5 J- o# L
</property>3 V! l/ f+ Y ^2 w: Q# r
<property>
$ c3 S4 `% y# d% o2 L <name>ipc.maximum.data.length</name>6 u2 ^; C9 f* q5 D! Q* X2 f5 \
<value>67108864</value>
. E1 j6 K& R! t$ v8 [ <description>This indicates the maximum IPC message length (bytes) that can be: B7 O6 v! m# O I, Q* o
accepted by the server. Messages larger than this value are rejected by the
% V) D6 @" Y" R h8 F+ | immediately to avoid possible OOMs. This setting should rarely need to be
3 ] {1 n8 M. G3 c( z k8 D6 n/ q( \ changed.
/ e- I/ p6 E0 w6 p3 f/ i8 p. q </description>
0 ^+ G5 ^6 D& p$ `</property>
/ }0 [3 u" u8 t5 R" o/ @<property>
8 s; C- ]" g3 L$ ~4 Q7 Z4 y! B1 H <name>ipc.maximum.response.length</name>
* n5 _- q I! F& M, V <value>134217728</value>7 {" E- N; h5 C& F; `/ y
<description>This indicates the maximum IPC message length (bytes) that can be/ M7 |& x! T1 H" l
accepted by the client. Messages larger than this value are rejected9 I6 X" u6 g) R1 \2 T
immediately to avoid possible OOMs. This setting should rarely need to be
0 g3 i8 I0 I$ A: J3 | changed. Set to 0 to disable.
4 `) N% g6 [. s </description>
% E1 i1 g8 p1 j& y9 ~& i4 M6 Q# e</property>$ k+ {. X; m+ A
<!-- Proxy Configuration -->
) @) \ |) |' k, \, g c" ?' T<property>
$ X" ?' C. J2 x% N <name>hadoop.security.impersonation.provider.class</name>$ T) o6 W1 y4 c$ B( |4 Z
<value></value>
8 _$ t( v0 d# t' ^# a <description>A class which implements ImpersonationProvider interface, used to
7 {9 w' A' F, a# E3 v9 x authorize whether one user can impersonate a specific user.
2 _4 j5 z! x/ i If not specified, the DefaultImpersonationProvider will be used.
/ `& P% q: u2 \+ {* e If a class is specified, then that class will be used to determine
G: z4 \2 o6 A+ [* P- m6 X0 q the impersonation capability.3 W9 W% w% t4 s
</description>
. `9 b$ n. y7 [6 N( j7 X</property>9 w2 O6 t# i+ }# ~( s
<property>' Q- t/ [" F. g
<name>hadoop.rpc.socket.factory.class.default</name>. H9 A# q# i) w+ Z
<value>org.apache.hadoop.net.StandardSocketFactory</value>
( d. m1 V0 ^8 Q$ S/ @* n <description> Default SocketFactory to use. This parameter is expected to be6 }/ h! Y7 x* N& L5 O b( g! Q
formatted as "package.FactoryClassName".
/ H5 v+ h1 R4 h( @ </description>9 P h: X2 m$ Y" u" N, K, H: E: }
</property>
. E1 m; P3 e3 R<property>
* {7 {8 f$ U+ p3 n* i <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>% A( @) }3 a9 r" E" X
<value></value>
* k4 ?( {2 B. I+ P- d <description> SocketFactory to use to connect to a DFS. If null or empty, use
9 t- }/ A/ D+ N! ]5 k c# ?- O) h hadoop.rpc.socket.class.default. This socket factory is also used by7 l- p" q4 [# p- C% L! z: }
DFSClient to create sockets to DataNodes.( k$ p( N9 D! P" r7 l, [
</description>
! }3 P% r+ @- X' t5 I% _& {2 c</property>
) p% j3 N6 k$ S<property># M$ P4 ~, N5 z: _ m
<name>hadoop.socks.server</name>$ L* K) P% t0 A0 ?7 @
<value></value>
' F! D" h- p, l8 f6 t <description> Address (host:port) of the SOCKS server to be used by the
/ i( _6 [ w& ]6 ] SocksSocketFactory. m( h6 X( l7 t: A& i' q
</description>" q& T+ |5 H; R( O7 O
</property>0 F% r+ {/ m4 ]+ U
<!-- Topology Configuration -->: ]! m, l1 H+ R; y
<property>
, ]5 V' s5 ^( u: {6 X- e$ [ <name>net.topology.node.switch.mapping.impl</name>
8 h @3 K1 h3 K4 u/ M <value>org.apache.hadoop.net.ScriptBasedMapping</value>
, v& r: f9 \6 f6 j8 N <description> The default implementation of the DNSToSwitchMapping. It C* w1 x/ N5 r5 T" s/ {
invokes a script specified in net.topology.script.file.name to resolve( T! `( y7 \% T1 O1 H& G
node names. If the value for net.topology.script.file.name is not set, the3 `! b( W( z% ?, U
default value of DEFAULT_RACK is returned for all node names.! p$ S) {; Q' _8 s% P0 j6 J9 o! X
</description>
7 _' X& _' t( t4 x( P% s7 p; {% O</property>
7 I$ ]: B; W T0 W- _2 D<property>" [( A6 Z( V& C
<name>net.topology.impl</name>3 \% H0 H0 f- V' j3 u, \
<value>org.apache.hadoop.net.NetworkTopology</value>( O$ E" L' d" m9 v: K
<description> The default implementation of NetworkTopology which is classic three layer one.$ j5 Y) B# c+ F9 z0 Q) J1 P% v
</description>* b' A7 o) U0 }: B N
</property>- j- y. ?, M2 g* f2 c( h
<property>& ^/ R9 x( r( Y+ `9 O u
<name>net.topology.script.file.name</name># D# O3 F7 L2 n- n/ _$ P
<value></value>6 n" b: _' ?) h0 n) i
<description> The script name that should be invoked to resolve DNS names to: g z( h f5 ~
NetworkTopology names. Example: the script would take host.foo.bar as an
" q- {$ {) b, a argument, and return /rack1 as the output.
- n3 h9 |- }# @0 d8 L" p" K </description>
, r0 h; y' g' D8 O$ \( e; R2 \</property>2 O/ E5 K3 P: ~( Z3 }2 y+ d2 l
<property>
2 A; z6 m' A4 b6 u8 `8 F& M5 G( T <name>net.topology.script.number.args</name>! k, A2 V$ I" Y* N2 V
<value>100</value> F/ P3 i3 O$ o9 o
<description> The max number of args that the script configured with
# k9 {1 i7 J6 C; n% P0 y7 M net.topology.script.file.name should be run with. Each arg is an
! P% |, r0 _" d& D IP address.7 [* Y9 J" i1 h! E
</description>
+ @4 {1 s, ]# P' `</property>( T# d: Y* K, [5 f& G8 H
<property>2 ]0 f/ R1 t6 U/ t
<name>net.topology.table.file.name</name>+ g0 o+ }0 o8 ]2 ~0 t
<value></value>
) u# W3 D+ W6 n, }- G <description> The file name for a topology file, which is used when the
, U/ b- L* t! ^$ m( P net.topology.node.switch.mapping.impl property is set to# U* n' U" p3 J+ M _
org.apache.hadoop.net.TableMapping. The file format is a two column text
2 W8 h# Q0 B- G) F; ^ A file, with columns separated by whitespace. The first column is a DNS or, Q3 n! k3 [- i
IP address and the second column specifies the rack where the address maps.
* e1 |9 [' L' H* | If no entry corresponding to a host in the cluster is found, then
4 \( Z) a7 H! ]! q* f! V* ^& J /default-rack is assumed.
3 y8 x! [( l- {( X5 U </description>7 U. l# `; ?: L' \3 y
</property>
$ K6 G0 h+ l5 C2 S<!-- Local file system -->. _2 P( u( Y: D" a/ f7 O) b: h# e H n
<property>/ t+ G0 B4 w! p) T0 C2 k+ n5 a# e
<name>file.stream-buffer-size</name>
- ?% h% s) z: \) {( m- g P <value>4096</value>
9 r3 ?. l) {2 \6 C1 x" X' |2 O <description>The size of buffer to stream files.
j; W1 a" a/ p The size of this buffer should probably be a multiple of hardware1 M- r' ]" _6 k3 w3 V2 t
page size (4096 on Intel x86), and it determines how much data is( E" u' @! Z4 ]$ Z- M$ m
buffered during read and write operations.</description>- q$ c8 s8 v- I/ [+ o
</property>" m/ ~ p' p: L' \7 i, C1 n7 i
<property>4 V+ P! h8 {7 }$ X! Z% a) l
<name>file.bytes-per-checksum</name>
4 o# B, F6 l, q' _ <value>512</value>- F9 z% V* |( }
<description>The number of bytes per checksum. Must not be larger than
6 `- R+ K4 `$ E' ?' H file.stream-buffer-size</description>. c% @! t, e8 ^% M! |9 E
</property>7 Q/ x% G W5 [2 t0 o
<property>( z' [8 \; K" ~! U3 Q
<name>file.client-write-packet-size</name>& [* i, f, w6 ?* {4 g2 N
<value>65536</value>
- ^; ^* L L3 v2 \- @, V4 q8 n1 z <description>Packet size for clients to write</description>
/ H0 ~" c; N8 x: {% s( ^( O</property>
5 T' H0 m( {5 l7 e' D& u! }% J$ h<property># Y( v+ ]+ Q! k" Z4 q7 U4 E) x" U
<name>file.blocksize</name>' n- Y# G$ _, v m4 N c5 H
<value>67108864</value>7 V# _+ C* g5 a' l- ]+ Q. o
<description>Block size</description>! Q* ?3 c! D& v# t
</property>) F" Z: k/ S: B1 q" m. r6 X
<property>$ o5 H8 i. {9 { c, F9 u9 {, G
<name>file.replication</name>7 v+ t1 U3 e; P7 z0 @& g- ?
<value>1</value>, W- O! N: d q
<description>Replication factor</description>( q+ f" ~- g5 a' p( w$ I( B' v; W
</property>
" f9 j2 G1 x& Q J1 ~' W<!-- FTP file system -->
; a- ~. A# }( f: g* Z2 Z/ Z<property>
+ S, w, [% l7 J6 T6 y5 a <name>ftp.stream-buffer-size</name>
' [. U8 }3 G5 Q4 _ <value>4096</value>+ i& A' }; ^! e3 m5 O U
<description>The size of buffer to stream files.
! P3 \& }& I4 v3 L, n+ t) z5 e. C The size of this buffer should probably be a multiple of hardware. f; a/ B( j) `9 ]/ H% i2 f& J. [
page size (4096 on Intel x86), and it determines how much data is
6 g5 z3 k/ s/ ^3 J4 i, @7 l; t buffered during read and write operations.</description>
9 G9 D% ^) p$ I& D% v' p</property>( B8 d, D/ J; k, z. ^
<property>7 K5 c; O$ E; i
<name>ftp.bytes-per-checksum</name>
5 ]4 w' z! i, E; O& C% M% V- M <value>512</value>: y7 A: V" Q* l
<description>The number of bytes per checksum. Must not be larger than
5 d6 c# q& W2 a. t/ S; t- E ftp.stream-buffer-size</description>: {( r3 l- N3 f Z- g- `" g& Z
</property>9 [" r6 u. o, ~8 m
<property>" G0 P( Q; l! [, N3 p# x
<name>ftp.client-write-packet-size</name>
9 t% q8 R7 d1 g2 k7 I <value>65536</value>
7 \4 C! ^7 o+ a <description>Packet size for clients to write</description>+ ~. N( b: b: _9 ?3 d2 J, ^
</property>
: u# C: P0 m7 R1 X<property>
3 V3 d6 a2 ^% F3 X8 V <name>ftp.blocksize</name>
# Z# P3 ? ^8 k H0 U7 l <value>67108864</value>3 Q8 ]5 C" {, v& y
<description>Block size</description>
/ B* z& A6 a- l7 |, U</property>; q; w! n. I, ^4 g* f
<property>
$ a' Z0 Y; M% H! [, l# k <name>ftp.replication</name>
* H% l% s- b* l- q! G <value>3</value>( `% C b# T, N* W$ `" @: n
<description>Replication factor</description>7 {- {: }+ k, p- v
</property>9 s& s7 Z1 h" n4 `) q5 p. Y, w& W. W
<!-- Tfile -->
# I4 g; @. c$ q Z! {" g' h<property>
9 T( m R6 W; h3 d# L+ G( u <name>tfile.io.chunk.size</name>
* F- f, q) }) w" ]2 D$ {2 u" a <value>1048576</value>* s; H _0 Q1 P# f
<description>; @% E( S9 P8 h. z3 L6 ]
Value chunk size in bytes. Default to/ q- M- R0 N7 [; \% P
1MB. Values of the length less than the chunk size is
/ s" C. f/ I6 Y1 p0 S guaranteed to have known value length in read time (See also
* r, L' Y9 Z7 [9 u! z$ I TFile.Reader.Scanner.Entry.isValueLengthKnown()).6 c# J8 U2 |& s$ t/ w. W [
</description>- c. s) L5 o6 s% l$ P' v8 M
</property>, z X- j1 U/ L2 G+ V6 R' x- L+ Q
<property>
5 \$ k% A3 Z% j% v <name>tfile.fs.output.buffer.size</name>
/ I' m& Q5 ~0 D& E4 D" \3 J. Y <value>262144</value>
8 E* Q0 t2 c( D `6 P/ I' {7 T <description>8 p- z S/ y h& E+ Q1 ~
Buffer size used for FSDataOutputStream in bytes.
4 M" I) g: P! i; P5 F </description>2 E) A- n( z( y9 C2 U1 R
</property>9 d, p" S1 D( {6 X' ~' I i g' P
<property>
4 e( S; E5 c6 B <name>tfile.fs.input.buffer.size</name>- K( M( n& f+ U/ J
<value>262144</value>% ~4 f. u8 K" ~% W% l6 O
<description>
( p5 M: ~! D0 J& y2 q9 R Buffer size used for FSDataInputStream in bytes.: t+ \* D6 J& U; \+ p
</description>
5 T! D5 k' b/ \) w: O</property>( r, k/ H) \7 |, h N4 D: g
<!-- HTTP web-consoles Authentication -->' L8 A, |. l) k
<property>7 e- S" ~% _, q; Z
<name>hadoop.http.authentication.type</name>, @& q; H1 r2 t6 B; r$ c$ b
<value>simple</value>
. p: `5 [0 A1 t7 z- w0 k9 W <description>& N" \, T# ~2 E* W2 y' `
Defines authentication used for Oozie HTTP endpoint.- o. }4 d! U9 E. Q; u
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#, i2 z& r# j, e" Y' K
</description>2 C) j4 m0 N m4 l6 y$ l
</property>
: G$ Q* T) r& ?# }" b<property>+ g* k- Z- w8 p% o5 U
<name>hadoop.http.authentication.token.validity</name>
( z4 e4 V: e' L: i7 b! y <value>36000</value>
$ [8 W2 h; P2 X <description>7 V! R2 I# g' y4 `
Indicates how long (in seconds) an authentication token is valid before it has
: Z) O; D, G) z to be renewed.' @4 v. Y r: o; x
</description>
" s: v3 M, N; \( ^6 ?( I4 M; o</property>+ i+ m( Z- Z# l0 K
<property>
* r5 _! j$ \7 V. n7 l s; Y <name>hadoop.http.authentication.signature.secret.file</name>
* A1 F2 ^3 A7 L) q. [' w <value>${user.home}/hadoop-http-auth-signature-secret</value>
1 Q7 E, X/ W# t6 L% J$ w" e <description>! r6 U E2 S: O3 s8 O2 v
The signature secret for signing the authentication tokens.5 ?* D: o2 C4 t2 D0 H; t, x
The same secret should be used for JT/NN/DN/TT configurations.0 K! A9 j! s9 N$ S! ^; y. _4 `
</description>1 F# N( r) A5 y) W1 `; Z" U8 V
</property>
1 B4 B9 t n! Q<property>
' p1 g. o/ m' T- S <name>hadoop.http.authentication.cookie.domain</name>
# A" _/ O( I/ @, O <value></value>
* @. V! H8 ], S% j$ B( H3 M5 @' k <description>
$ [6 t9 Y! f3 ?: I, u J The domain to use for the HTTP cookie that stores the authentication token.
* a& j' p% x/ p& \1 x In order to authentiation to work correctly across all Hadoop nodes web-consoles
6 ~# v v' r9 O6 f) m6 s the domain must be correctly set.
2 l/ o4 y: t; d( D& l! X- h IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings.* A+ N2 b3 l- Z( t& `
For this setting to work properly all nodes in the cluster must be configured8 x6 w: C, W8 j
to generate URLs with hostname.domain names on it.
3 i; z1 `2 \% K$ s, P5 h* V </description>
) k. s- ?. f/ h# |# z' p1 q: s8 M' R</property>) E0 ]$ d: G- W) W
<property>1 g1 ?: X" s; q6 J, \2 i2 s
<name>hadoop.http.authentication.simple.anonymous.allowed</name>
# a; p8 ^: r+ w/ H) _ <value>true</value>
+ @8 Q8 B x# Z" h7 U( r( |" r/ [ <description>) f3 X. P: S" |1 C9 @2 ?3 W
Indicates if anonymous requests are allowed when using 'simple' authentication.
! a+ ^% Q5 H) K </description>
# H9 W" z( Z$ z" {</property>6 d4 m7 c) X: I% q
<property>
2 P7 x' E8 F5 Z2 x <name>hadoop.http.authentication.kerberos.principal</name>
) E# d) x! ]/ ~ <value>HTTP/_HOST@LOCALHOST</value>6 D G% U9 B9 T T* Z5 T
<description>
& T0 A1 m8 h1 h Indicates the Kerberos principal to be used for HTTP endpoint.: s1 K* {% u8 x k* n' V
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
# T2 q, @- n6 K3 G </description>
6 m5 l3 J* N' x# N" H, f' E</property>
2 G4 R5 x, {4 d- v4 Q" x8 G<property>8 A1 E/ x) S" h; ~/ ]6 e. a
<name>hadoop.http.authentication.kerberos.keytab</name>& [& R5 q: t6 |9 T: i4 Y: b) `
<value>${user.home}/hadoop.keytab</value>
$ F% o9 h( n" Q( V, H <description>
& s# K' @7 w: r! w Location of the keytab file with the credentials for the principal.
' Y$ l2 n. x. W0 j" y4 s7 q$ S Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.$ f6 ^. D1 a* p0 d4 |* L
</description>2 F& ~. O6 ]9 U
</property>
- J0 l% C5 v3 B<!-- HTTP CORS support -->$ |8 o3 ?. `, ~- U
<property>
$ [' {( e8 K( R7 D2 H8 A% |% @ <name>hadoop.http.cross-origin.enabled</name>1 B- F' ~; r% W/ L
<value>false</value>0 |, F% e3 n% w& t
<description>Enable/disable the cross-origin (CORS) filter.</description>. \6 x. ]+ f, v
</property>; |% j+ Q/ s9 D7 m3 p2 `
<property>$ s, @4 F% `0 w
<name>hadoop.http.cross-origin.allowed-origins</name>) x* n/ S7 a. {' m8 b6 O
<value>*</value>5 n. W- ^: |; P1 F0 W
<description>Comma separated list of origins that are allowed for web services
% h! u9 ~7 o+ g6 { V5 {% z G needing cross-origin (CORS) support. If a value in the list contains an
! Q, \- |( @! q% A asterix (*), a regex pattern, escaping any dots ('.' -> '\.') and replacing y6 A7 s7 z1 ~6 H( [5 \
the asterix such that it captures any characters ('*' -> '.*'), is generated.
* a: |2 m- u' U- Q Values prefixed with 'regex:' are interpreted directly as regular expressions,
3 i# s \5 k {/ l0 X) L6 Z2 v e.g. use the expression 'regex:https?:\/\/foo\.bar:([0-9]+)?' to allow any
2 q( l2 U5 ?5 v2 |$ }! L origin using the 'http' or 'https' protocol in the domain 'foo.bar' on any4 ^; r9 H: j3 p. M) P
port. The use of simple wildcards ('*') is discouraged, and only available for3 O6 K( x# h1 J4 N5 ~
backward compatibility.</description>8 k2 e+ @" m" z% h* B0 ]% ?3 m1 Z$ V! Y
</property>
+ o7 U- A4 W* Z<property>
* ], Y* n+ x9 Q+ d <name>hadoop.http.cross-origin.allowed-methods</name>$ Z" x+ p- F$ n/ ^
<value>GET,POST,HEAD</value> `/ k3 M; b' \/ W m9 s) C
<description>Comma separated list of methods that are allowed for web
: X- _8 [( V) h* l# t7 \& J5 s9 ?6 ^ services needing cross-origin (CORS) support.</description>
$ n" i) p& ?. m" ]</property>
+ ]7 ^1 I1 |6 V; ]0 [1 U<property>
0 o5 e& G9 d1 g! U <name>hadoop.http.cross-origin.allowed-headers</name>! j6 M1 O9 m4 A( T
<value>X-Requested-With,Content-Type,Accept,Origin</value>7 D; [7 f7 g8 `: E
<description>Comma separated list of headers that are allowed for web
# @ b- b6 p; G/ l3 h h services needing cross-origin (CORS) support.</description>
" H1 i( t! ]2 ^6 k</property>' s$ K5 N/ o0 ^* s/ F6 N
<property>( @7 H2 Y- _+ g
<name>hadoop.http.cross-origin.max-age</name>$ X" Z% `/ Q1 H+ t5 ]
<value>1800</value>
9 e+ P* b* ~! s. ]$ L- j <description>The number of seconds a pre-flighted request can be cached6 L2 G: T! l0 Q. H
for web services needing cross-origin (CORS) support.</description>3 h8 {3 _5 L0 J0 {0 p3 r, o1 X
</property>
. R. }* d2 m4 u2 R3 ^<property>
1 e5 b/ Z3 C+ ]& a9 }" g <name>dfs.ha.fencing.methods</name>) Q$ X9 x, \5 x4 B
<value></value># h. U- P3 Z1 D; H u% R: I5 x0 \
<description>
% K7 x) K2 w# d List of fencing methods to use for service fencing. May contain J/ y1 ~$ ~% j/ a5 c
builtin methods (eg shell and sshfence) or user-defined method.3 q/ P+ T! P7 k3 N0 C& ~$ w+ @' _
</description>
* g1 A3 P9 V1 k</property>. }) U3 p7 G5 Q9 }5 l
<property>/ r& ~0 A1 ~0 @$ }$ J# i" z
<name>dfs.ha.fencing.ssh.connect-timeout</name>
* g- q0 T2 M- x2 F- ] <value>30000</value>: K, m' }7 G) I5 F& d
<description>+ X" N' [$ G. }' U. N8 P
SSH connection timeout, in milliseconds, to use with the builtin
% C3 x, j4 U- b6 q2 j' J sshfence fencer.
( E |" w- D5 `8 | f) Z: i) } </description>
3 z4 |5 G; {3 }; d7 w</property>1 z# L9 U4 Z" x a0 g
<property>3 z$ y9 Q8 Z! ~
<name>dfs.ha.fencing.ssh.private-key-files</name>
) I k* Q' J) i2 u <value></value>
4 D% i N, S- [6 F <description>
) z5 a% i( g" c' D3 D2 _ The SSH private key files to use with the builtin sshfence fencer.; U& f; o. b0 ^" i$ z! `: v5 R
</description> f9 }9 P" m- ~) [; u
</property>) `2 j& |2 A( p$ M4 J# R: c
<property>: J; X+ _3 a7 |9 y, G# R
<name>ha.zookeeper.quorum</name>" [9 h/ J* v& Q! h& l6 B# K6 e; Y
<description>' E* v1 i8 z \" o: f
A list of ZooKeeper server addresses, separated by commas, that are
- r% D, ]9 k6 n/ F4 _0 D* K% A/ D' \ to be used by the ZKFailoverController in automatic failover.
+ x9 y. s# A5 i- ] x </description>
0 g( {! j% `- b2 T</property>) |4 u1 N; C/ h$ I
<property>
' G. T7 f- \0 I7 r- C <name>ha.zookeeper.session-timeout.ms</name>4 z2 D1 H) g3 s' w7 W; \0 Y4 l4 F
<value>10000</value>
/ e; T3 y4 O* p/ G) G1 D <description>
! U7 }0 b2 @: v+ ]0 x3 p The session timeout to use when the ZKFC connects to ZooKeeper.
$ e8 P5 A5 v% X Setting this value to a lower value implies that server crashes
2 E; B. E: r- }! Z: k7 i1 e" [ will be detected more quickly, but risks triggering failover too
5 ?: o' R8 I- {7 M/ C' i( b: v* [ aggressively in the case of a transient error or network blip.
$ q. @, H& _& G3 P" b& u. H+ Z </description>
+ J% n; P. E6 f, a</property>
$ X0 Z3 K3 |4 T6 Y% ?% g$ y9 W<property>
: @& g: D: O% n# u' U) v: f' F <name>ha.zookeeper.parent-znode</name>$ z7 J9 {1 v& ^+ f% b
<value>/hadoop-ha</value>9 i' A8 J* E4 K6 s
<description>
1 G$ |/ p, e- ~/ M F8 { The ZooKeeper znode under which the ZK failover controller stores
% D* J8 |2 ~8 \+ @3 A5 N2 s: y its information. Note that the nameservice ID is automatically" E. g& @9 W( h, [$ C7 E0 j/ h
appended to this znode, so it is not normally necessary to
2 a8 g* t; ? P2 Z- H! y configure this, even in a federated environment.
. G' F: g1 Z+ I: ~6 m! @- b </description>
# `1 V2 z) d; r7 Y9 p0 Q7 y</property>
3 @0 g$ C) P+ z' ?7 P<property>) f, `2 y$ O2 p* [* b& H
<name>ha.zookeeper.acl</name>
2 L4 J3 e6 T b" f <value>world:anyone:rwcda</value>
q9 |0 b f0 q1 t1 T <description>9 `. m) x& e" F/ H
A comma-separated list of ZooKeeper ACLs to apply to the znodes# \; |% h( ]; ?, ?% A' x; M
used by automatic failover. These ACLs are specified in the same
! E8 Q/ l; i. O9 y9 U/ C4 `0 B format as used by the ZooKeeper CLI.
/ e% Y+ B. U# c& y If the ACL itself contains secrets, you may instead specify a
% ?) G% M+ B' D path to a file, prefixed with the '@' symbol, and the value of& L7 ^7 l3 c4 R& }
this configuration will be loaded from within.
; l) j7 z5 y- W5 m1 ? </description>3 s! X6 \4 ^, g) A, q2 k
</property>
9 Y+ l/ S, l, \: Q% r q3 U0 o<property>9 j) t s) w9 l
<name>ha.zookeeper.auth</name>
4 C% q& k6 |4 s6 o, f& d( R. |+ y <value></value>/ [/ s. `2 {, e0 J d
<description>
7 ?9 n* j# Q" f) J( c8 m. d A comma-separated list of ZooKeeper authentications to add when
* X4 } q2 n' w" k connecting to ZooKeeper. These are specified in the same format' J5 w& a) r7 y! B1 c' {6 L0 F
as used by the "addauth" command in the ZK CLI. It is
2 ?, s/ g, j7 p0 E: s7 }5 e important that the authentications specified here are sufficient
) F' t) o7 H# O! b/ w$ Z' Q- c" { to access znodes with the ACL specified in ha.zookeeper.acl.
$ z, n& ?9 l# i9 m& J$ ?1 S If the auths contain secrets, you may instead specify a# r D* B$ u0 @) x0 z9 ^
path to a file, prefixed with the '@' symbol, and the value of+ M/ z+ x7 n* I0 q- _; K% u) j$ `
this configuration will be loaded from within.
4 Z6 b' p0 A2 r" _3 Z </description>
) s, ?) ?8 R/ v f7 V3 n Z</property>
( ^; X, U T _. r<!-- Static Web User Filter properties. -->
1 k- O& I- _$ X" ]' a<property>: F1 j/ R- ? V2 p3 j
<name>hadoop.http.staticuser.user</name>
0 L) C: K4 ^7 I5 ^ <value>dr.who</value>. B5 t& o' i8 J) R
<description>& f! m$ O: |2 ^( B8 ]- J
The user name to filter as, on static web filters
" p* Y& J7 \6 Q while rendering content. An example use is the HDFS& M! X. {3 g' @' i
web UI (user to be used for browsing files).
# r. v; r( L g. ] </description>
8 n! d" i& A# F( x</property>
0 g+ G, b5 Q: S; }<!-- SSLFactory configuration -->/ e |! y" Q- {% i9 W
<property>
) a% R$ A! G& E, l9 n3 C <name>hadoop.ssl.keystores.factory.class</name>
) E5 ~2 I) x& z+ r' m5 `/ ^" Y <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
. u, M. p; k3 N6 `$ ~ <description>
% n7 V Z$ O2 p/ ] The keystores factory to use for retrieving certificates.: y3 x8 o2 R4 I$ _2 }( i9 O& M
</description>$ Y+ q" ~% p5 G# w
</property>
" [) U% q5 v. ~! P# z<property>
# q0 e, ~% y% y <name>hadoop.ssl.require.client.cert</name>+ Y& R1 g5 k' y0 h
<value>false</value>
- @( p7 ]4 \& t# J <description>Whether client certificates are required</description>
# k- }- J( m; _9 u+ {</property>; G, f# Y2 I4 M1 {
<property>
( c3 n6 ~5 A# g3 e& W <name>hadoop.ssl.hostname.verifier</name>! b$ `. h- x1 {5 U/ X
<value>DEFAULT</value>
/ I, e- h/ k) @5 h$ h. y. D* B$ Z8 I <description>
! P m5 L1 H4 C2 S; _' u The hostname verifier to provide for HttpsURLConnections.% t+ h ]3 Z# [# W$ K* D' U0 _
Valid values are: DEFAULT, STRICT, STRICT_IE6, DEFAULT_AND_LOCALHOST and" k- t) ?' _ U; A( `$ M$ Z
ALLOW_ALL
) J$ }9 j6 Y9 i7 v$ y </description>- G; i0 M3 \' m! \
</property>
' R! a, {* T3 \6 |4 n$ h# Y# k<property>! a9 r0 p. P8 I4 E5 K! F
<name>hadoop.ssl.server.conf</name># q& J! g, v @
<value>ssl-server.xml</value>
5 I# G, V' M+ l1 @% f, w <description>" F+ H$ {7 W5 s
Resource file from which ssl server keystore information will be extracted.( v: T7 Y8 l' o
This file is looked up in the classpath, typically it should be in Hadoop
) `- S) L9 f% ~4 n: v conf/ directory.
$ }) Y$ Y# H; T$ d) Z. D9 ?' j: Q </description>
9 x3 Y/ J0 g# B" O8 n0 w( Y</property>) k9 D9 D. R9 t+ |7 b
<property>
5 S. S3 P/ I+ W <name>hadoop.ssl.client.conf</name>6 b5 M+ Y5 b1 w6 v7 n: O5 l' a: }
<value>ssl-client.xml</value>
4 @' l9 ?7 S) ^- D" X <description>
* Y6 I8 O( j1 r# r8 |9 ~+ y& u& b Resource file from which ssl client keystore information will be extracted2 o ^, C! W& o1 A' r3 e9 h& F
This file is looked up in the classpath, typically it should be in Hadoop
) U( d% g8 s' [$ Y0 ?8 ~ conf/ directory.% O- b2 z/ c& n7 c
</description>
# S: v: N9 w% }) Y) |/ m& N</property>: P7 p, v1 l/ Y) o4 }+ |
<property>
s! E( ?6 j+ A Z% h0 B <name>hadoop.ssl.enabled</name>0 M: S/ \. r5 h- n! D7 E
<value>false</value>% x6 ]) }/ K: s6 \& J' D
<description>( s- k" h. w' s8 o. @. G
Deprecated. Use dfs.http.policy and yarn.http.policy instead.0 n3 |3 c+ ]+ X _; Q. {
</description>3 I( J: Y# \1 ?4 y
</property>
: n" s' Z3 ?, R1 t. z( g. L% e2 ~7 P<property>; Q) n2 O) ~$ ~" V
<name>hadoop.ssl.enabled.protocols</name>
7 w9 w( c; ]8 {$ a8 C' e <value>TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2</value>
- {% f) s! X g5 o- U, m' p <description>
+ }6 C1 h) F$ O7 @ The supported SSL protocols.
( x7 V7 Z8 ^4 q/ z </description>& b! G9 T2 i$ y- b
</property>: f4 }( }9 {, _1 H# i% O
<property>/ B( h9 P( [7 S# L2 N
<name>hadoop.jetty.logs.serve.aliases</name>
2 V2 u. \0 ^1 [7 C <value>true</value>
4 F* k7 O. m! L4 x9 R <description>0 P# v2 U$ m# W3 V: _( U+ ]
Enable/Disable aliases serving from jetty
5 C: m8 e: \8 }, q/ S </description>
# F, y% m3 X, v9 K" B4 a: p</property>- n/ @/ F# q- J# e
<property>
: u; @: ?- k- Y' {7 }; ^7 a7 ?- ^1 c# Y <name>fs.permissions.umask-mode</name>
9 f9 Q% S$ {$ n) w, r <value>022</value>
: ?$ S/ _ p$ b1 U6 [ <description>8 g6 m+ _1 W- S* {$ _7 V, w
The umask used when creating files and directories.% P# Y& F& m+ {; j' m: L; E1 w! x. q
Can be in octal or in symbolic. Examples are:
1 }# H8 f! N9 c6 v0 | "022" (octal for u=rwx,g=r-x,o=r-x in symbolic),( a8 M: @$ b( D# w+ ^- B
or "u=rwx,g=rwx,o=" (symbolic for 007 in octal)." h, [6 b. @3 b. V+ q
</description>8 `- m/ S9 p& l
</property>
; b% \4 `- d- ~$ M8 A6 o1 q6 J# r<!-- ha properties -->
* Y/ r! s0 f+ D- c% N<property>1 ?4 M$ | O5 C6 B/ ?$ T1 f# b: i
<name>ha.health-monitor.connect-retry-interval.ms</name>
+ x# K# a9 Y* d& Q. k, A( ? <value>1000</value>9 S# S- z) Q% A3 F( A
<description>0 ?7 c; _. q( L4 o- V* {* n
How often to retry connecting to the service.) z. T: N: k4 h2 R3 [- H% Y: E: }
</description>
2 \; O. v/ T6 U. J) J</property>/ k/ q" S6 g6 b# T
<property>
" ]6 f6 h h% P2 x0 S <name>ha.health-monitor.check-interval.ms</name>/ r4 H+ R# x8 @" c( {& `
<value>1000</value>8 a& s; n1 ~+ U) z, ]5 {0 V3 n+ Q/ ]9 z
<description>; e' g/ k/ K2 i1 |& f- a
How often to check the service.
4 v" Q% C7 m: L4 L# l </description>
}$ j6 m! v' d& y$ f) S</property>
0 U( t6 }$ p8 A0 t* O1 F<property>% g, n$ W9 u8 k2 {: @9 o
<name>ha.health-monitor.sleep-after-disconnect.ms</name>8 V1 g/ b G9 {* Y( v
<value>1000</value>
- ~+ i1 L# _9 i3 q5 y0 a6 q <description>2 g" n" v$ V5 R/ Y9 R
How long to sleep after an unexpected RPC error.! G4 b f6 o! n; s( {
</description>
7 b8 S8 A) s$ y% G% e+ \</property>9 H P6 u/ J7 t) E, b8 H- p
<property>
1 q( z* R5 i; S3 t0 u& ~ <name>ha.health-monitor.rpc-timeout.ms</name>$ \: }: e2 l7 K4 J1 u% g
<value>45000</value>7 _2 ]3 q+ L: p* q" \$ ^
<description>* T6 o2 |% [& W, a% z2 D$ m0 u
Timeout for the actual monitorHealth() calls.. O7 L. {: x% F( T Q- U
</description>
( L! X0 X2 q& C& a% f2 q" x0 H</property>
5 |7 @( @7 `) F3 ~8 J6 I9 n<property>4 [. O' z% G- w8 u$ p1 @7 @7 y
<name>ha.failover-controller.new-active.rpc-timeout.ms</name>
1 c0 Z }6 n' Q) r <value>60000</value>+ e. b9 T" p1 D' j, w0 J' Q7 i
<description>
% |$ x, w+ f- b4 H! P. h7 h/ X. C! k Timeout that the FC waits for the new active to become active
' k d# \/ R. U* k3 v e </description>. W7 q1 s4 e6 N- I
</property>
. ]# O+ [+ l* P a" f3 V' `<property> ?2 \( a. M7 l# X. Y% e6 w5 Y
<name>ha.failover-controller.graceful-fence.rpc-timeout.ms</name> ?9 b0 V v$ Z
<value>5000</value>" @! m% |/ t- x9 z8 |
<description>9 r7 d$ C" L$ U4 m
Timeout that the FC waits for the old active to go to standby
* n* O9 r* u7 [5 G </description>' a$ L. j$ n, o/ K$ c% a6 ~$ T, j
</property>
5 y N% B$ i$ W<property>
) W- W) S3 y9 K# A0 G$ p <name>ha.failover-controller.graceful-fence.connection.retries</name>
" x( x; j6 m3 N4 c2 F) V <value>1</value>1 {, r4 ^3 C$ W$ Y% u) p- e9 z0 ~7 b/ [
<description>
Z' n1 G1 @5 I8 Y FC connection retries for graceful fencing
. p$ U; T1 h/ O# Z4 B- u9 M5 Y </description>! w4 Z; H- L& ^
</property>
* ?1 M1 Z" C7 o<property>
2 }' K$ }9 U7 U% d <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>
- B/ P$ |7 ~: A6 V8 s <value>20000</value>2 h, _" b( k2 I6 x6 b% r
<description>
/ S! Q; `+ T, [3 \ Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState
- b: S! o. B; F2 x4 o; h) ? </description>; p9 r7 i0 g/ w- q5 K! J
</property>
) \! ?5 q5 V3 C% s- R7 z) U: z<property>$ A; r- y% g ?6 c& t: m
<name>ipc.client.fallback-to-simple-auth-allowed</name>2 a9 `* ^/ X( c3 k
<value>false</value>: w* }, `, r0 @' j9 _
<description>
5 J, k3 J- P! L: a When a client is configured to attempt a secure connection, but attempts to
( e; L6 p4 e7 z4 [1 i connect to an insecure server, that server may instruct the client to
' `% Z- H8 o2 h6 _1 \; N, O! } switch to SASL SIMPLE (unsecure) authentication. This setting controls! X- I# v: R0 u) W" O
whether or not the client will accept this instruction from the server.
# F9 ?, T9 O _7 F$ I# @& a When false (the default), the client will not allow the fallback to SIMPLE9 @" S! R! B0 r4 _, `
authentication, and will abort the connection.' f9 H: C9 b6 I) v$ ]7 ~- G
</description>
6 l" }0 m) V2 g+ T</property>) d( T: ?4 N6 G
<property>6 h% K# k4 M% b$ X2 r1 Z
<name>fs.client.resolve.remote.symlinks</name>
* H) J3 L" Q6 }2 I( K# K5 w0 f <value>true</value>/ E. c3 w; j# X% {: G
<description>6 y/ v" _3 M) D8 z" t o% @7 W- q
Whether to resolve symlinks when accessing a remote Hadoop filesystem.7 ]* `- q" W' U6 P5 u: c
Setting this to false causes an exception to be thrown upon encountering1 l- T$ S" y7 [+ _0 X
a symlink. This setting does not apply to local filesystems, which
6 R# j" q$ ^# ^/ E/ i automatically resolve local symlinks.$ Q# u7 r1 B$ J; m1 n1 N
</description>
6 X g6 N( B9 b9 V9 A% v+ T% X</property>
+ ?/ _: k2 N4 M) U/ t/ `" L$ G; G<property>9 t0 Q6 D) Z. q9 C, u! ^
<name>nfs.exports.allowed.hosts</name>
2 t8 g3 g% f2 Q9 c) R& a! ~ A. M <value>* rw</value>0 k2 j( H5 i% E& F5 U& ^# ?6 ]
<description>
" k2 t6 n# z0 _ By default, the export can be mounted by any client. The value string
7 Q4 W3 M% A7 Z5 H contains machine name and access privilege, separated by whitespace
" M1 n( `$ @; g U, \' _ characters. The machine name format can be a single host, a Java regular
. o0 I% K; J" h# @ expression, or an IPv4 address. The access privilege uses rw or ro to+ D$ P5 g& U% ]0 X) p
specify read/write or read-only access of the machines to exports. If the
4 [5 _1 X4 g+ k; e& X2 C access privilege is not provided, the default is read-only. Entries are separated by ";".
2 f7 B; |! J* j/ w. z For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;".& ~ V' C0 R' a# X d/ {3 U
Only the NFS gateway needs to restart after this property is updated.
5 t/ x: m- H- @+ [$ k* \4 ~ </description>' t( I6 R' q, J! `
</property>
# A9 j1 C) W4 z$ p( V' T% Y<property>
2 p$ W9 s& A" w$ ^( [% H; b* ` <name>hadoop.user.group.static.mapping.overrides</name>
2 H8 n6 `0 ?7 j0 \1 N9 w <value>dr.who=;</value>
& e" H! U( o! N8 [# T$ `1 G <description>$ |7 K( R5 C" C4 T/ T* c7 M
Static mapping of user to groups. This will override the groups if) P3 W( R+ x0 k D* R
available in the system for the specified user. In other words, groups
9 V \. J& S# m" G4 B3 D9 r% { look-up will not happen for these users, instead groups mapped in this: ?; F/ y8 @: E
configuration will be used. I7 U# l1 x$ ^$ ]' y) o1 T
Mapping should be in this format.
$ B* o( m5 [. a5 {) @7 @ user1=group1,group2;user2=;user3=group2;& Z9 }+ z- H8 Y0 r; m6 B' Z
Default, "dr.who=;" will consider "dr.who" as user without groups., l& d: V6 S, v, ]5 m# f3 P
</description>: `; C, Q" z$ r2 L
</property>$ n a1 |& |8 ]* n8 O* _
<property>
. L& G: ~0 X* F; ] <name>rpc.metrics.quantile.enable</name>
& F' \" R: a( t% c <value>false</value>: v6 g! m w' x' O: s- D
<description>
: b i$ U, G& T8 ~0 Q. t( x1 y Setting this property to true and rpc.metrics.percentiles.intervals
& l% |( [& R1 d P7 _ to a comma-separated list of the granularity in seconds, the
# M7 |: S3 O) A3 K1 @ 50/75/90/95/99th percentile latency for rpc queue/processing time in1 e6 n. V; }- B9 Q( x
milliseconds are added to rpc metrics.
; g; i, ?1 }( w8 `9 N7 V </description>
( b# n7 ]& s6 e2 X. h" i</property>
7 W. H Y q; u0 B" ?% X<property>) c( m' r' d, i$ y. \
<name>rpc.metrics.percentiles.intervals</name>
9 M7 |! @! P o- f/ U1 O }2 T8 { <value></value>
4 Y! y5 `- z3 y. @+ S; U; e <description>9 ]" b4 {1 b3 H* B
A comma-separated list of the granularity in seconds for the metrics which( [, ~+ |* r4 U/ D* d2 D
describe the 50/75/90/95/99th percentile latency for rpc queue/processing
4 t" @$ j+ S4 T% v7 s/ ~# }+ Q time. The metrics are outputted if rpc.metrics.quantile.enable is set to
7 s) m* k3 Q6 m7 W3 s3 p. } true.
" |/ j% m( @! \, o. ?0 k$ N </description>; ?: z! |% X0 ?# V) o
</property>* F0 \5 A0 I; d1 U; w6 N4 f
<property>: Y" P( L* J8 g. {( I: X
<name>hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE</name>! B$ _, ^. B& ]4 u, s
<value></value>& H0 D0 Z4 }5 u8 R3 T! d) _/ Y& T
<description>
0 O% i$ t6 ~- G& S: C+ V- ` The prefix for a given crypto codec, contains a comma-separated
* t: `) V! Q) H" m list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE).% n3 P+ ]/ o& x" u" {
The first implementation will be used if available, others are fallbacks.
) J2 [/ V" b0 u6 |; c x </description>: L& e) H$ v6 a/ K2 }- S7 m( ?
</property>5 Y5 [% Z/ O" Y3 G4 n, E0 L9 x
<property>$ o; ~ ?4 [. s% y6 g: ~4 y8 w
<name>hadoop.security.crypto.codec.classes.aes.ctr.nopadding</name>
+ o# I1 T+ N- i <value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>" ? H; P: m, M N1 V- ~ g. n
<description>
7 D+ F9 \! m1 e& v6 P6 C% Z: k Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.* |9 \: G2 w/ t7 w; c" s
The first implementation will be used if available, others are fallbacks.
7 I$ P) n# p7 t/ d </description>
# S( l. \5 o- I( @$ t</property> F6 k. B6 L( S% J/ P
<property>. Q# T+ ]! u, Z" `4 o% {% N1 N
<name>hadoop.security.crypto.cipher.suite</name>
- p/ x& ]: [! P# E) h <value>AES/CTR/NoPadding</value>
( I$ {$ B; D. Q <description>
- b, L2 q* m m) Q4 v! {3 [. P Cipher suite for crypto codec.
5 H- E, t& k; W0 I- p9 n1 l </description>/ l" d1 u) G4 A
</property># T- ?( A4 F! N% r7 {
<property>
; U3 T/ @/ `' x: Z- U6 C' R <name>hadoop.security.crypto.jce.provider</name>; |, o4 w4 f5 u3 r( W
<value></value>
* E7 u; K8 N8 h; O; L% Z7 m P <description>
8 k! j0 \) I2 Z The JCE provider name used in CryptoCodec.# d& N" D- z$ X1 O
</description>* D- k% d! D/ v% m5 K! K+ x8 d' p1 S+ k
</property>" n- [1 C. p a! D: y8 {1 j
<property>
6 b6 n7 B8 x/ t% @' P+ B <name>hadoop.security.crypto.jceks.key.serialfilter</name>" |" k; y$ q* Z4 P3 E. b4 O O
<description>
2 F; e5 I( D! A Enhanced KeyStore Mechanisms in JDK 8u171 introduced jceks.key.serialFilter.3 `( c9 |/ \4 R& \6 A: p2 G- z
If jceks.key.serialFilter is configured, the JCEKS KeyStore uses it during
# X+ z7 C" H2 ^8 E the deserialization of the encrypted Key object stored inside a
# `. O' r5 L% P( u( G SecretKeyEntry.& U. E' W* |6 C" Q3 `- h9 O
If jceks.key.serialFilter is not configured it will cause an error when) H6 ?( \9 }3 o9 O
recovering keystore file in KeyProviderFactory when recovering key from
4 @9 h! H5 _3 R6 G- b keystore file using JDK 8u171 or newer. The filter pattern uses the same) ~" e# v4 y* o& ?
format as jdk.serialFilter.& Q0 K3 M9 y: H& S) k+ M" C
The value of this property will be used as the following:
q( m. j2 m7 N- v- c. \ 1. The value of jceks.key.serialFilter system property takes precedence
9 z6 Y9 D( y8 s# t( H$ x over the value of this property.
5 `" { x$ A! J4 I ?; s }7 F" N 2. In the absence of jceks.key.serialFilter system property the value of
- K8 i" {' n+ U; n* U$ Y$ x this property will be set as the value of jceks.key.serialFilter.
9 D4 x2 N, J; F7 K3 h. g" J& X- _ 3. If the value of this property and jceks.key.serialFilter system$ i- O6 K* ]1 p8 K0 L! X
property has not been set, org.apache.hadoop.crypto.key.KeyProvider
* Z3 @+ H+ _: e4 h, Z( ~/ z6 x5 u sets a default value for jceks.key.serialFilter.
b+ V5 h8 |9 x2 m5 v </description>
$ ~: |: t& {1 F; V* B. ^( s</property>
+ V5 k4 R% ]+ }) h<property>1 A3 y! e% y9 x, m" M4 R" L0 v1 Z
<name>hadoop.security.crypto.buffer.size</name>
# W: g7 z/ X& b8 q2 m: _ <value>8192</value>& a- `5 }! x3 Z- `% s- Q+ q% H* o* I7 W
<description>% w: U0 c$ r. ~2 E4 Z8 b
The buffer size used by CryptoInputStream and CryptoOutputStream.9 ?5 \5 [4 U6 z; u
</description>; [. R. Q0 Q, K8 m% I
</property>
7 |6 j2 {7 e% g# k<property>
1 V% x4 D/ g* H1 I9 G" v <name>hadoop.security.java.secure.random.algorithm</name>
4 ?$ V; c3 H3 Y. Z <value>SHA1PRNG</value>! ?& A0 u2 `( h. b) I6 C
<description>' T8 N0 C* j7 i; Y6 C2 H" ^
The java secure random algorithm.& \ b3 P! D! {; |- [. f& e* A
</description>7 D* z* e2 V! W( {3 f% m
</property>$ }, _ g' U) x. _
<property>
+ `4 e9 q7 a# T; t! I, y3 N <name>hadoop.security.secure.random.impl</name>
- b" d2 J: e- k8 l7 N4 n' T7 @ b <value></value>
( P; h: X: B# [" E x9 L7 g <description>$ G9 x& Z1 }+ B% G2 _* V( ]
Implementation of secure random.
# k3 ^3 A/ s, s0 p </description>/ \) s8 `0 C5 u0 Q3 V4 U5 C/ e! H
</property>
4 l1 ]" f/ g) i. P( H<property>
# N+ b) L; c; V <name>hadoop.security.random.device.file.path</name>* e) _* z6 v) a; Z, E
<value>/dev/urandom</value>* \. k& T1 O) U) a
<description>4 P5 ?0 o1 L4 n" J% e( @8 y
OS security random device file path.! |' N/ N* l% ]& N) Z$ n3 f2 W
</description>
/ c7 S8 S4 ? t5 j% W3 y# }</property>
: a: q/ V8 G0 q+ K<property>0 ?" o4 F8 w2 r0 g/ Y7 `! X6 n+ e0 _# |
<name>hadoop.security.key.provider.path</name>; D+ a9 y$ L* i% a3 G
<description>
2 j) s: d! ^2 O. j The KeyProvider to use when managing zone keys, and interacting with
4 N3 i8 _$ }: B5 | d" `$ _, \ encryption keys when reading and writing to an encryption zone.) q) i) x3 ^1 v' r5 f
For hdfs clients, the provider path will be same as namenode's* N2 e+ T; [; R/ ^- E
provider path.
. y7 r. f& R* O# Y </description>
- W3 N4 z+ @/ Y</property>, P; R3 g @. r z5 {
<property>
0 ~5 h" r d) V+ Z <name>hadoop.security.key.default.bitlength</name>+ e# M7 T+ q/ {; j7 Z6 v1 U" q) x0 p. P1 `
<value>128</value>5 J8 j+ w( i$ @1 C" |' j. U
<description>
P: ~( f0 Z5 {5 Y1 ` The length (bits) of keys we want the KeyProvider to produce. Key length
6 o6 T6 l# n+ i+ ]6 j: l# C defines the upper-bound on an algorithm's security, ideally, it would! S" J8 @. O- m7 F) j
coincide with the lower-bound on an algorithm's security.
; Y: E: V" b4 v </description>: @- z9 l. I2 `7 a, M" t
</property>& \; F4 ^3 i; n6 ]
<property>( ~$ S: A5 @0 _* H" R
<name>hadoop.security.key.default.cipher</name>- b; f& Y7 `2 {$ G5 p- B
<value>AES/CTR/NoPadding</value>/ f( k4 }& Q' l% O
<description>8 Z0 i# R% I. h4 R/ c
This indicates the algorithm that be used by KeyProvider for generating
$ o8 S! e$ {' B) V1 G9 e key, and will be converted to CipherSuite when creating encryption zone.
; I+ G' Q/ V. h* |, Z5 N </description># U8 ~3 }8 C: m( U0 `# q/ s
</property>
3 Q4 e- e) |8 g& D6 W<property>7 ]/ Z( \' r6 P. Y1 C4 @8 K5 C4 d
<name>fs.har.impl.disable.cache</name>+ X2 g) f, Q) Q8 ]: o1 e' t4 g; q- [
<value>true</value># o# D8 S. J' @* l! i. i( H
<description>Don't cache 'har' filesystem instances.</description>: B6 S8 C% f1 N; @0 H: z2 c
</property>+ F Y) E9 P- w& _1 x
<!--- KMSClientProvider configurations -->' b L9 M' T. N
<property>
0 c' R5 i7 }+ x* H' K <name>hadoop.security.kms.client.authentication.retry-count</name>- W% N0 h6 ^. T, C3 ^
<value>1</value>$ \' U1 Y) }1 [4 e* d9 K5 s7 i
<description>
- t: y$ d/ R; W; R Number of time to retry connecting to KMS on authentication failure G" j0 B& s. `0 R8 X! g
</description>
! g$ |) \1 P3 K6 ~6 @+ A" V8 \/ ^</property>
Y. ^9 ^0 p/ X<property>; M+ c; w+ H9 S6 K7 W/ g
<name>hadoop.security.kms.client.encrypted.key.cache.size</name>* s9 k/ G5 o. b2 D% @
<value>500</value>3 ?6 q' ?5 x/ P6 x2 q( \
<description>
- J( S( q0 r4 F6 l! P; ? Size of the EncryptedKeyVersion cache Queue for each key
2 Q; X( Y6 M) H! q, v </description>6 }# S4 h% a4 J" c% U
</property>
" v+ S3 \5 L9 L+ c1 [& P% o<property>
6 c7 N2 S g1 Z/ Q* m- N <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>
6 i2 m7 h6 J5 [- r% K8 j f3 K <value>0.3f</value>
7 _2 n5 y0 b4 Y8 a% T% i <description>
! k# {8 K2 _" i( P* A% I5 ? If size of the EncryptedKeyVersion cache Queue falls below the
( A% b4 X6 K4 _ low watermark, this cache queue will be scheduled for a refill
' W4 K9 u3 o# e: |0 y( z8 j </description>
7 h; x1 H7 i5 h# L</property>
) D9 g0 h5 _: n0 r<property>
1 c) r: J2 l) L; F1 g <name>hadoop.security.kms.client.encrypted.key.cache.num.refill.threads</name>; `# O0 B6 {+ w8 i8 a
<value>2</value>
5 U6 i8 m7 P5 N' a" \" `' u" J3 K <description>
1 F8 Z) n$ p' c9 e3 ], m Number of threads to use for refilling depleted EncryptedKeyVersion
; _) F2 O& x& o8 W2 {! e N cache Queues. D( Y9 x% T4 n3 P+ M' \5 u9 H" d9 L
</description>7 a: P9 s& b/ l3 e1 X z8 w# K3 g
</property>
% f* `! I6 {: m<property>
2 M N7 J5 B2 c/ O9 W <name>hadoop.security.kms.client.encrypted.key.cache.expiry</name>
3 I9 T% Z) \* n" ]" ] <value>43200000</value>9 |; i3 q3 w( [& y: T1 ^+ |9 H
<description>( u0 {' n K" K& k; Y# T
Cache expiry time for a Key, after which the cache Queue for this
& y6 H+ a8 _' v key will be dropped. Default = 12hrs; Z$ k# G: X8 l5 f
</description>
. t; `# l4 N% ]( E</property>
% |: M) M4 w3 s% k. Q( s m9 W+ Z<property>
& q- o, K3 ^0 `" I* k <name>hadoop.security.kms.client.timeout</name>/ R% H) U( L, r8 c* L1 d5 h
<value>60</value>. z7 O- r" ~, s4 d) H" L- E+ w) r
<description>
& h$ `7 r8 f# t* ^1 s Sets value for KMS client connection timeout, and the read timeout
6 L% _6 b* [( k! [+ s/ h* r to KMS servers.+ z3 R& Z7 J+ X4 Y& e
</description>
% G& r' _5 d! ?$ ~</property>$ }' J5 s8 |- c" } }6 x4 c6 {$ c5 E* ?
<property>: y% G+ ~# s; e# `' J3 D+ ]8 L
<name>hadoop.security.kms.client.failover.sleep.base.millis</name>+ t3 B' q5 q7 o, E4 {* n* n* ]
<value>100</value>; Q2 Y! [- a0 g# ~0 {/ U
<description>8 ?/ A( X: V ]7 L5 S
Expert only. The time to wait, in milliseconds, between failover3 q$ z/ _$ g, J/ o- l( F7 Z
attempts increases exponentially as a function of the number of: l3 F X* b" I
attempts made so far, with a random factor of +/- 50%. This option% v3 H% `, H& V
specifies the base value used in the failover calculation. The4 A" s% Z8 u* l& Z3 I
first failover will retry immediately. The 2nd failover attempt
N2 [7 p# u Y will delay at least hadoop.security.client.failover.sleep.base.millis( c6 ^4 W- v ~, I) R- U
milliseconds. And so on.) w# g& w0 j% w
</description>
# a1 y. s1 E0 e' d$ \0 z0 g</property>% z8 V* S1 G+ c- X, O! ~
<property>
! [! M$ t' e/ D- M& [! [6 I <name>hadoop.security.kms.client.failover.sleep.max.millis</name>
5 [0 b0 J% A0 |9 M O3 I& j <value>2000</value>
% i( z% L- {" o0 ^" t <description>
! q+ Z0 V4 k; }. O Expert only. The time to wait, in milliseconds, between failover% A, s9 Q* I6 g5 C3 a4 ^' ]
attempts increases exponentially as a function of the number of9 q5 o' k% d3 V
attempts made so far, with a random factor of +/- 50%. This option4 ?* } o8 s5 A4 A M( C$ o
specifies the maximum value to wait between failovers.
& }; y( I2 B! P) L4 I Specifically, the time between two failover attempts will not
; W0 v# _9 R; ~9 n exceed +/- 50% of hadoop.security.client.failover.sleep.max.millis9 G* O/ b5 E0 z$ W' P/ c, R
milliseconds.) r7 w$ b. X4 }$ r
</description>
$ ?6 B. m& G2 L1 R7 _</property> e5 v C& n* k. m9 T
<property>
E1 J5 a# k5 q% p+ }* d2 Z <name>ipc.server.max.connections</name>
2 A; d/ i) C5 _$ X! n <value>0</value>
9 l/ O( _& O. C8 y0 N/ T4 e' ~ <description>The maximum number of concurrent connections a server is allowed* i- r- V( B O7 S' G. M
to accept. If this limit is exceeded, incoming connections will first fill& k" W5 m8 Y/ W; f. _8 F0 z
the listen queue and then may go to an OS-specific listen overflow queue.' V B% B* F. o8 L
The client may fail or timeout, but the server can avoid running out of file5 l$ V7 h. o) F5 ]$ [6 G) N
descriptors using this feature. 0 means no limit.
0 r; j8 ]- D) v2 f7 W& E) O </description>
) V8 R2 ^( k6 U, S, O& o% t</property>
. @5 O/ k* _4 u! |4 y <!-- YARN registry -->
4 I7 p. b3 J- d& w- g, Y- B/ ~ <property>
0 \6 o- {& y% L <name>hadoop.registry.rm.enabled</name>9 |: _# ~% o/ n9 `9 M( n
<value>false</value>) U: b# m- |8 z, C; G. {
<description>
( u# J. i! I b2 f R Is the registry enabled in the YARN Resource Manager?% S4 C2 }, P$ s8 P% h, v/ M
If true, the YARN RM will, as needed.
' M( V& u* H1 @6 x* ] b7 S& Q% C create the user and system paths, and purge
" |. U4 P8 S' H2 t( s3 z service records when containers, application attempts
' y4 Z( H# E- ~6 p; W/ \8 F and applications complete.% |% `) V* o# K. t f7 Y
If false, the paths must be created by other means,
& [6 p4 Q9 J% p1 f i* } n+ s5 w and no automatic cleanup of service records will take place.
0 \4 Y* l" R& o ` I# O. @6 K </description>
" z* n7 b6 t: P" q' h </property>% {3 a+ B' e/ ^2 P
<property>
$ E; I3 `0 ]% S <name>hadoop.registry.zk.root</name>
$ Y" B, ~2 j9 ?7 \6 y <value>/registry</value>/ ], f- N2 X9 S" }5 E X. ?; l+ n
<description>
% ?+ }; A& h2 j! O" m7 G The root zookeeper node for the registry5 f* [+ E2 @7 z ]
</description>
- X! L9 x3 t1 C+ J2 ?1 A </property>
5 G( H1 F5 `& e. ^+ y1 } <property>( H; [6 G# s) h. N, A; x: T6 T" R
<name>hadoop.registry.zk.session.timeout.ms</name>0 _" S i( c1 q. Z/ s+ S
<value>60000</value>
. U7 E4 L- e. e/ J <description>
" O4 D. I1 K- t' ?3 v! @: d Zookeeper session timeout in milliseconds
) l. A; O& v1 g% B& k, V </description>
+ B; H1 Z% }0 A </property>, f: l. D! z. Q: F9 b2 g! X7 y- B% T
<property>
) Y0 K% H. ?5 N! | <name>hadoop.registry.zk.connection.timeout.ms</name>
, {& y' n) I0 a+ y& t <value>15000</value>
( \8 e* p) D) i" O1 Y" I+ h4 [ <description>
$ P9 E$ D0 u2 b Y/ P Zookeeper connection timeout in milliseconds
$ f9 F x% Y% Z5 D </description>) d+ Z3 _7 g: }% e3 C
</property>7 \! G% h- G) G1 }4 L
<property>
. H' d. }8 b/ W6 _6 ^' ~+ Y <name>hadoop.registry.zk.retry.times</name>) U+ |0 Z/ g6 B- J! J& i
<value>5</value>
: R& f- L2 U' O, g <description>( p/ ?9 A0 _( ?
Zookeeper connection retry count before failing; z% D% _% P6 A( e, k" I
</description>- n9 I7 M) H' A) C
</property>
" D6 ~0 @! g- ~+ E6 ^) j+ q <property>
+ Y ^+ d4 B3 t7 U7 e0 H4 @ <name>hadoop.registry.zk.retry.interval.ms</name>' N' d& w/ k2 c# G+ l" H
<value>1000</value>" ^: _2 y3 W3 i% V8 i" M
<description>6 {9 ^+ ?' m" x1 _! j
</description>& q3 V* Q9 _3 x" F3 K
</property>3 C% c' k0 y) w, Z6 p! |
<property> ~, C0 l6 q6 x& N5 h$ y
<name>hadoop.registry.zk.retry.ceiling.ms</name>+ @; s+ M5 w8 t- ^
<value>60000</value>: M9 F9 A0 \2 }, y' ^. e
<description>
5 I8 ^" t- h5 {, [) F1 ]. G Zookeeper retry limit in milliseconds, during( b2 i* H) n4 o; m1 X7 z0 A
exponential backoff.( O. M4 s1 U6 C/ m+ V1 ~# } u
This places a limit even3 ~! x7 `7 k( d7 H3 K6 e( N! i
if the retry times and interval limit, combined
' x, z' G* d2 y% l with the backoff policy, result in a long retry2 b4 U! A" f5 E+ x9 c* E+ P
period
* x# P4 H; o8 \ </description>3 H/ j) R' u8 A" a4 \7 W0 k
</property>
' C/ s3 I, f7 _' [# A9 A# }; ]+ x, c8 @ <property>
( h1 M2 [# l& f! d. V/ } <name>hadoop.registry.zk.quorum</name>
# h! t0 W& ^& M <value>localhost:2181</value>/ X7 `2 Q4 Q( m# `$ l% t2 g* [
<description>- d' A- y4 E4 f( ]* W/ P i3 k
List of hostname:port pairs defining the
* S6 e- }" i5 f: _5 z zookeeper quorum binding for the registry
- i/ o$ z+ o- L9 n i; d( [ </description>- }# [! z' Z0 s6 y1 O6 H$ t
</property>
! _* S/ O3 n: C" Z/ F& M5 h/ Q8 f <property>
9 M7 z6 d* R+ y/ d0 \ u <name>hadoop.registry.secure</name>
/ ^+ }% t/ A! W8 _9 H <value>false</value>$ J; @$ R! A( ~9 n5 [
<description>9 H4 C+ L. }% U- L- [3 i X* A4 G
Key to set if the registry is secure. Turning it on& b2 t/ R: ?/ b) Y6 i# P" e- q
changes the permissions policy from "open access"
2 l1 d5 ?( t( D- q to restrictions on kerberos with the option of8 J) b( U- F [6 U6 P
a user adding one or more auth key pairs down their- s* [4 `5 i m7 p2 j# U. D
own tree./ {: V& v% E! D' T
</description>8 |. l( P, K# D$ c0 R
</property>3 B/ H3 w: z7 O; _. v! X& l
<property>) g* L: g' h( H2 X7 U$ N+ ?3 E
<name>hadoop.registry.system.acls</name>
/ t+ o* a: {8 W( w9 d6 D0 H6 t <value>sasl:yarn@, sasl:mapred@, sasl:hdfs@</value>
7 U) b& `2 \/ C <description>( q( u. C0 W8 `/ {* C7 `
A comma separated list of Zookeeper ACL identifiers with4 A& |2 u9 {; N7 b1 H4 B* ]' |; N7 R
system access to the registry in a secure cluster." z0 Y# T S1 \
These are given full access to all entries.
* L i, u# F# R* @7 e* v/ k If there is an "@" at the end of a SASL entry it
5 M0 R. y) g! {4 `+ R7 X instructs the registry client to append the default kerberos domain.0 g) [2 |) M7 i* d/ o4 \
</description>
$ Z' C) z6 ^/ j0 P </property>! _8 R$ M# z7 D0 g
<property>
+ v4 S7 S: |+ K <name>hadoop.registry.kerberos.realm</name>
' P6 i+ p3 }$ B% A( d- d <value></value>! K5 K! i: o- f3 _+ p7 c% `
<description>
. l, P1 s3 ~- }' s" [; B The kerberos realm: used to set the realm of$ p" x% P/ B& m& J- |5 P. C
system principals which do not declare their realm, t" }8 i" u( [! _( o
and any other accounts that need the value.
$ G% N& H+ ]! u$ H2 L6 g/ \ If empty, the default realm of the running process. O9 T% Q, P0 t. J$ P* O8 w1 J. s
is used.& a: m4 d: g$ w) c
If neither are known and the realm is needed, then the registry# a5 w' l( v9 r( \8 b; ~2 _
service/client will fail.
6 n: ^9 Z( {4 {% f2 [: a </description>/ @# ?/ P4 I% j, E
</property>
4 `4 a$ E) q, q0 A7 R2 x( `3 b, x <property>
/ ?& r- H; X$ K9 Z7 R$ \ <name>hadoop.registry.jaas.context</name>
$ |7 y( h9 T ]8 ]$ g" l$ R <value>Client</value>( p$ B7 R) s U* ~$ C6 S
<description>
# ?$ F# D2 Z/ a1 Z! y) c5 v Key to define the JAAS context. Used in secure
8 q/ ]. f' E _' ] mode( G- `4 f+ M+ C! ] L/ ]
</description>$ O4 ? D6 [% r5 o
</property>
+ d$ f$ a3 c4 N" l' G8 ] <property>
% L& `" k j: n1 C/ e <name>hadoop.shell.missing.defaultFs.warning</name>
" x$ ^8 m3 B; J) Z1 {0 |* F* M <value>false</value>5 G0 }' m4 Z4 z
<description>* q; g3 w, x. `3 t( f* D! ^: A
Enable hdfs shell commands to display warnings if (fs.defaultFS) property, U# j( X9 I; C6 y& w1 J S G, f& J
is not set.
3 P4 l$ X7 M! H5 [: `/ ?3 o </description>
) w- B; |- \* K: e4 C, J# m. y% ? </property>
* e J+ l C7 K8 N/ T( D <property>
0 i2 M( r6 P; A- o+ { <name>hadoop.shell.safely.delete.limit.num.files</name>
" I( c3 D! D2 b8 h' @2 c9 b5 s <value>100</value>
# _3 J- ?$ I; {/ n- ^" p1 H <description>Used by -safely option of hadoop fs shell -rm command to avoid
5 g: t/ ~: C( x! { accidental deletion of large directories. When enabled, the -rm command2 u/ t' F1 G* `; A
requires confirmation if the number of files to be deleted is greater than& U, k" r# Y7 h% x
this limit. The default limit is 100 files. The warning is disabled if7 ]0 C e2 L6 t4 l
the limit is 0 or the -safely is not specified in -rm command.
4 Q( A, F% f+ M+ D1 K </description>, }' [0 i; l) u% ]; j$ G! H, e
</property>: r1 S+ V: r" T) _+ S4 n6 P1 B) m- a
<property>
' [' E. W/ O1 f/ i1 T; e8 f <name>fs.client.htrace.sampler.classes</name>
/ @0 N: R( Q" @) h- e$ O <value></value>
8 w- u6 @7 M" S& P1 m" m3 ^# L' } <description>The class names of the HTrace Samplers to use for Hadoop
* {1 o7 I7 g6 f) H8 j filesystem clients.* w5 z! l9 a' Y: F9 C" Z5 E
</description>, A0 T$ h2 N5 P3 N. G0 t
</property>
* \/ S6 M$ R" h1 i' j/ I! w <property>$ r$ G9 k# j: d; j0 y
<name>hadoop.htrace.span.receiver.classes</name>
( \/ j2 h) E) ~$ e' O; T5 T <value></value>
8 ~. ], G+ v0 V8 D6 n% i @ <description>The class names of the Span Receivers to use for Hadoop.- {4 N8 y- A* W% s: i) O* {
</description>& o2 \* D, b1 B
</property>
+ D9 V% Z" ?; r1 V <property>
, L% }' z: N) O7 J! ]/ { <name>hadoop.http.logs.enabled</name>
( G. X: f- \1 Q% `' ~" F* l# T/ [ <value>true</value>* \& j! Q8 Z! |
<description>
4 L! [/ g" F* j1 j, O0 D3 j- V Enable the "/logs" endpoint on all Hadoop daemons, which serves local7 l4 m _% S9 [7 w: Y$ f; h, R5 Z
logs, but may be considered a security risk due to it listing the contents3 h6 c% M2 j6 }$ R- A o
of a directory.
& z2 j* i' P; h% G% W </description>* g- d9 e0 W; j! d
</property>
) J, \% g0 ?. {/ G( f. v& [ <property>. L- S) M# _' m7 X* w; t
<name>fs.client.resolve.topology.enabled</name>
( U' @6 O7 R, f) Z; P8 S6 |8 b ^ <value>false</value>
) l6 ~; n: C3 K8 |! K8 a <description>Whether the client machine will use the class specified by
' G! ^* X, Z: _* H& K property net.topology.node.switch.mapping.impl to compute the network
, @% a# j( k( y. M. F) Y/ S# ^- G distance between itself and remote machines of the FileSystem. Additional5 |* k, T5 `0 {# z* w
properties might need to be configured depending on the class specified
' Z5 g& H( d+ j+ e- u, w, `- _ in net.topology.node.switch.mapping.impl. For example, if
8 G$ m4 Z$ T1 x0 J/ H) P& \ org.apache.hadoop.net.ScriptBasedMapping is used, a valid script file, ]2 f8 q3 d, k9 ~0 U$ f
needs to be specified in net.topology.script.file.name.9 Z" m$ n# z [. X z6 j' [9 x! k
</description>
2 v" D P% t9 X' P: R </property>. `% I2 N0 W/ a7 a0 N8 d4 S
<!-- Azure Data Lake File System Configurations -->* ?- }$ t% U6 Y7 ?5 t+ d8 U
<property>$ {4 M+ l$ ^/ M, a- ?) M
<name>fs.adl.impl</name>( t! {$ E* T2 {& ^6 I
<value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
- P/ S: r9 z$ h7 A9 y$ Q </property>' b$ b/ l" h6 J5 m
<property>% b2 W% t. @, _
<name>fs.AbstractFileSystem.adl.impl</name>% I3 `- n, j! m6 x& [4 g L0 p9 P
<value>org.apache.hadoop.fs.adl.Adl</value>; B& n& I W; y1 P1 Z* \
</property>' t6 V% t7 k, c. F& N
<property>* ^: {. M. g8 S% x: E. L" R3 ]
<name>adl.feature.ownerandgroup.enableupn</name>
. f+ w/ j* d$ ~7 c9 ]7 W1 S <value>false</value>3 W( h6 x; G- [0 K$ a D+ l1 W
<description>, n9 N1 Z$ X* h \3 \/ a0 T: s8 w
When true : User and Group in FileStatus/AclStatus response is
2 J* f; V- j8 H! n4 O5 Z represented as user friendly name as per Azure AD profile.
" n* [; k- g/ J4 A9 r When false (default) : User and Group in FileStatus/AclStatus
! K+ Z1 R% A1 S3 R; o- q response is represented by the unique identifier from Azure AD$ t2 |* z. i% }" L' e
profile (Object ID as GUID).8 q6 l* ]; w5 X( Y
For optimal performance, false is recommended.1 m5 r/ L3 m) j0 Y4 _" [
</description>
$ w" x* I$ E8 c, c- L1 _5 n! s </property>
) y' \' Q% P% W( B; Y% P8 y <property>9 g6 @3 _& f+ }5 u4 X. i
<name>fs.adl.oauth2.access.token.provider.type</name>
) T& J; z1 \2 a* j$ a <value>ClientCredential</value>
5 L/ D$ ~3 r3 M: P1 z <description>
& M: g/ |" M9 @: N Defines Azure Active Directory OAuth2 access token provider type.( _9 Z* d! z* |% D( H; B
Supported types are ClientCredential, RefreshToken, MSI, DeviceCode,! m( ~' `3 |* _3 @6 n
and Custom.' l U3 G& H5 m
The ClientCredential type requires property fs.adl.oauth2.client.id,
0 n5 D0 j/ ^8 d fs.adl.oauth2.credential, and fs.adl.oauth2.refresh.url.
, `8 ^. w1 N$ J: k; K% M; C+ t The RefreshToken type requires property fs.adl.oauth2.client.id and0 F D' H5 a( a/ d; C
fs.adl.oauth2.refresh.token.
8 K S+ b3 ~8 ?; D! ~' O5 v7 k" W The MSI type reads optional property fs.adl.oauth2.msi.port, if specified.6 a% g# m* n8 H+ Q
The DeviceCode type requires property$ d0 M4 s! p. d: U7 C
fs.adl.oauth2.devicecode.clientapp.id.
4 C& h. k9 E$ @9 v The Custom type requires property fs.adl.oauth2.access.token.provider.
- j( s& c& u4 P3 j( L </description>' B1 c, m8 @! ~( ?. A
</property>6 x8 J5 _1 j; `2 ~6 p1 P
<property>& m6 H, E$ w( u/ X. U( ?% |! z
<name>fs.adl.oauth2.client.id</name>* i8 I% v, H6 j- P* \6 a1 w1 @
<value></value>
: f/ q' v4 \. m7 p, f <description>The OAuth2 client id.</description>4 S- O; c1 j9 f$ G
</property>
& [/ |9 H: ~9 @2 I9 t <property>) O& c6 c' y7 l' b
<name>fs.adl.oauth2.credential</name>
7 u( ?8 R* l2 L8 M <value></value># w0 z# A, E3 x8 r8 E
<description>The OAuth2 access key.</description>* U) ~% @" F3 c( ]5 k4 {6 |. W
</property>
+ J# p# q5 r1 O6 A2 j <property>1 B4 U7 E f/ m8 H2 e- w! A
<name>fs.adl.oauth2.refresh.url</name>/ Y0 ]7 `# M; o2 @! F
<value></value>8 F# T9 [ t/ E8 m/ P0 v
<description>The OAuth2 token endpoint.</description>
0 h* e+ `5 l" G8 c4 u U5 { </property>
2 t b. T c! v5 L <property>
) m! C" V8 l e7 f% M1 B6 I2 L, M <name>fs.adl.oauth2.refresh.token</name>3 e" u- I' J6 `
<value></value>7 S7 r- U* |4 r3 k
<description>The OAuth2 refresh token.</description>
9 t; B/ b4 v; l" ^4 k </property>. r7 w& N1 m9 A
<property>5 M; j: k3 e! C
<name>fs.adl.oauth2.access.token.provider</name>
$ j# _* g; \7 |+ d! f <value></value>
1 _4 ]7 ^5 ^* i <description>4 x: g/ O" ]$ X. H/ J+ Y
The class name of the OAuth2 access token provider.# m; J% _$ }3 c) e+ y" T
</description>
" w1 z. c3 t* N# ~2 g4 P& e </property>
: `5 [* l) H% q/ b5 c S <property>
6 v' u) P: r" d' T$ d <name>fs.adl.oauth2.msi.port</name>
' g, ?+ @' }5 z, } <value></value>4 \) [) L7 D! _5 c; O5 R
<description>2 c& C% W8 X3 t: T9 m
The localhost port for the MSI token service. This is the port specified
. h: h% [5 D3 O, Y1 x% V1 T when creating the Azure VM. The default, if this setting is not specified,
; Z% [" t6 z9 B is 50342.0 A* u6 U: [9 u7 X: r) Y
Used by MSI token provider.
% K/ H0 N J' w$ l0 T& V/ j- o </description>
# B; E* G9 j+ @% x' n6 ] </property>- f& V7 Z+ K' E; C( a6 J' H- j I
<property>
- R, }* [5 t6 H) O, K <name>fs.adl.oauth2.devicecode.clientapp.id</name>
3 t1 Q: j+ @2 d* _; L$ l <value></value>
1 n% R, x- H( L* z# a: q <description>. Z5 _* {7 T$ B6 ~
The app id of the AAD native app in whose context the auth request
/ D7 A8 b6 @' w should be made.
/ e+ e% y( ~7 i; S h5 { Used by DeviceCode token provider.: T1 D$ N& P: z [7 @! j
</description>) \. A7 M2 q0 ]3 x
</property>
7 N; h- i: ~5 ]( ^2 |8 { <!-- Azure Data Lake File System Configurations Ends Here-->
" S5 ] F: @9 e9 l9 Z8 p4 q <property>
u- x, H# U% s, I, r <name>hadoop.caller.context.enabled</name>3 U6 h) N' S% v1 x$ I; \$ l
<value>false</value>
6 _0 _6 v5 @! L$ b& B$ e <description>When the feature is enabled, additional fields are written into
+ q6 `/ c4 {- h$ k/ `+ F2 U name-node audit log records for auditing coarse granularity operations. H4 G( \# E6 y& T$ Q9 ?& O- v
</description> b- b$ k' ^+ e6 Q7 `' P
</property>
7 K- Y% H& y$ A( i% K <property>: M9 x. u* k2 U
<name>hadoop.caller.context.max.size</name>$ B! {/ w3 _# @/ o3 r# o9 i
<value>128</value>
" y2 D; @4 P b( k( M" \; R2 G <description>The maximum bytes a caller context string can have. If the3 V4 n8 r. P( d# \; D h
passed caller context is longer than this maximum bytes, client will
' R' x4 h' ~1 `% W9 h/ r; h truncate it before sending to server. Note that the server may have a
" \% X8 j7 L0 c5 n6 c: Y different maximum size, and will truncate the caller context to the
; F- `. }+ B1 U- ?9 S maximum size it allows.
: B0 l# r b( K; B9 X. k7 q* A5 Y </description>: u. h. y. |5 r& j. q
</property>8 r5 D0 v& t( O" F) a `2 p% q# X6 w
<property>0 \- `# N8 _) e
<name>hadoop.caller.context.signature.max.size</name>
" L! ]2 |4 B& G a! A1 \ <value>40</value>
* S8 @' \+ i, l2 c. x* T1 D <description>
: r! o4 B3 b, O& c The caller's signature (optional) is for offline validation. If the4 z: n% W! l7 c( ]* \
signature exceeds the maximum allowed bytes in server, the caller context
0 K) G; u! _" V+ x( _. h will be abandoned, in which case the caller context will not be recorded
" W/ |* K8 Z# a: z in audit logs.- F! C. |" |" h5 W" S7 e6 R1 L) k
</description>
' D8 z# `% l2 P </property>3 T* w3 D4 t* f& k. e
<!-- SequenceFile's Sorter properties -->
$ `5 ^; m$ E, ~. f9 o1 u <property>
/ u( R. |% ?3 U* Q/ ]8 x <name>seq.io.sort.mb</name>
9 x$ b! t$ b5 P3 k. h6 ?; D* A1 V <value>100</value>, {/ p) T, U i
<description>; U9 P& c, v0 {" m( k
The total amount of buffer memory to use while sorting files,
! s3 r+ q/ L1 e% G9 s$ B @ while using SequenceFile.Sorter, in megabytes. By default,
1 m& G7 E6 u* v. I gives each merge stream 1MB, which should minimize seeks.0 U c2 k" }- x# h) E, W4 p2 q5 D
</description>
- ?& C. `. W9 p; k9 t; S </property>
# t: T3 F0 _. ~( h4 C5 |6 j <property>6 {$ c" i) q% z( T |. H
<name>seq.io.sort.factor</name>
' W4 Y$ x- Z: o" k2 v/ ^, V: w <value>100</value>( D- ^7 f' ]# G3 h7 a# _
<description>' `& O6 N* h1 b( y8 L0 j
The number of streams to merge at once while sorting% d2 I- x3 F0 j
files using SequenceFile.Sorter.! a- f/ I) N9 u2 L% `- q
This determines the number of open file handles.
2 d# A- p# e1 S0 k& \ </description>8 Z; p) Z3 {- L$ b! }
</property>
5 z B1 A ^; D- r9 u% ? <property>
0 n) j6 ^- O5 f8 ]7 Z1 c/ [ <name>hadoop.zk.address</name>* P3 j5 Z; b* |
<!--value>127.0.0.1:2181</value-->+ t- N$ U6 H2 q; k8 s: N% U
<description>Host:Port of the ZooKeeper server to be used.( X: c* \, C6 O, r
</description>2 H/ {# H# p9 a7 n
</property>1 c6 \& o$ l* t( b7 b& |4 z
<property>
9 J: e7 T0 L' J1 T) O <name>hadoop.zk.num-retries</name>
" O; a6 L5 g' e* z8 _ <value>1000</value>" }% @. R% b) v. `
<description>Number of tries to connect to ZooKeeper.</description>
5 B0 G. J$ v5 i0 @ </property>, `3 }# h) s1 B, m$ N$ x h3 U
<property>
2 @: X3 b: d0 M1 O, D <name>hadoop.zk.retry-interval-ms</name>
0 R# T+ r; P: ~" Z& l) p, W; Q <value>1000</value>
8 I/ d9 K9 O# A! |* T) g! C4 F <description>Retry interval in milliseconds when connecting to ZooKeeper.# x) \( }3 J. u' N
</description>0 q5 {$ x m, J: j
</property>! W$ C3 ^% R9 Z! {2 h$ i3 T, D
<property>+ Z! y0 G8 B& x, L# X
<name>hadoop.zk.timeout-ms</name>" L! l4 q% F8 ]
<value>10000</value>) ^- u2 \ n9 P2 {& W" s
<description>ZooKeeper session timeout in milliseconds. Session expiration1 n- \; z$ Y$ R. E3 B
is managed by the ZooKeeper cluster itself, not by the client. This value is' f3 O& y; R# e0 a% k, v# k
used by the cluster to determine when the client's session expires.
, e' D2 h3 T6 ~2 I Expirations happens when the cluster does not hear from the client within
1 s1 `$ I1 H5 X ^, I: O3 ~ the specified session timeout period (i.e. no heartbeat).</description>
h% o$ f" e4 s- B" D: M6 L. @4 H </property>" @5 i. U6 g& R' {# ?( `
<property>
8 ~6 K/ ]+ N$ D5 c6 l2 L <name>hadoop.zk.acl</name>
9 K* D8 T: _% e- Z <value>world:anyone:rwcda</value>5 F. S% L$ L$ g \5 q
<description>ACL's to be used for ZooKeeper znodes.</description>
8 |) c/ K! [) W) c </property>
6 y5 k" N2 I0 z" _2 E. l <property>" O) q3 F/ U% L4 [7 s
<name>hadoop.zk.auth</name>2 p+ }" O2 N% J) P. }( u. h8 \. s& a
<description>
7 R8 G& v M3 t Specify the auths to be used for the ACL's specified in hadoop.zk.acl.
" M3 a) I2 i( |5 j This takes a comma-separated list of authentication mechanisms, each of the9 Y" _ B% A- _
form 'scheme:auth' (the same syntax used for the 'addAuth' command in
6 e+ X7 Z; X3 R N the ZK CLI).
2 @" W) o& Q9 _" K. M" j </description>
9 |! |3 b# ^7 k </property>
1 g6 ~+ e. |4 h5 r <property>* |7 K7 i; n+ y2 g- X0 G
<name>hadoop.system.tags</name>
, }* o0 Q# v$ U' G <value>YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT
% b3 A* ]1 G+ _ ,SERVER,DEBUG,DEPRICATED,COMMON,OPTIONAL</value>$ n! S2 c2 Y2 t5 d& Z: M( Z
<description># J* L* X2 m( O6 @/ U
System tags to group related properties together.
6 M0 Y( O3 B' R) S8 W+ m </description>% N! z# U$ p! ~( A6 b3 x
</property>2 X( u$ U5 ^ R/ p- Y: B
<property>
. x- j: R2 B+ r <name>ipc.client.bind.wildcard.addr</name>8 F$ N1 \: |$ r$ f
<value>false</value>) @6 d; N5 D& `5 ?* B
<description>When set to true Clients will bind socket to wildcard2 r; z. s. O4 {6 B# Q
address. (i.e 0.0.0.0)1 Y5 C" v: H: R6 ]1 D2 m, s3 {1 Z! ]
</description>, v: A) y! v; z9 C; T
</property>
- k+ x7 k& I( l8 X0 ]</configuration>; c4 P# P5 b4 A: U
2.hdfs-default.xml# W# B& ~, ^1 p" u1 X8 Z+ _
* @4 {, b: \; E- ~5 I
<?xml version="1.0"?># a; j; r. R4 {! G( R
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
" b1 X s. S+ A3 G$ R<!--% A* ?5 B' A4 L( b! g
Licensed to the Apache Software Foundation (ASF) under one or more; A+ r _3 M& J
contributor license agreements. See the NOTICE file distributed with0 E/ c- A" \9 P7 D
this work for additional information regarding copyright ownership.
! U# M% V; Z% j The ASF licenses this file to You under the Apache License, Version 2.0
& x5 _+ t8 T0 T, B( h* r! e) j% ` (the "License"); you may not use this file except in compliance with: n4 \" D: k& |% F7 W/ N7 u
the License. You may obtain a copy of the License at
1 m2 [" U4 j% X# L. H http://www.apache.org/licenses/LICENSE-2.0
3 S( e7 y0 L v+ f* Q0 N3 I Unless required by applicable law or agreed to in writing, software
) s( Q6 q: x9 {. q9 Y0 _ distributed under the License is distributed on an "AS IS" BASIS,3 }) U& f% Q& m( C- P; L% N
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
6 L- q: v9 p: T3 [; G( h See the License for the specific language governing permissions and, x6 Y5 `% o: _* j
limitations under the License.
7 [% D9 K% v) J' N( D3 T9 G6 w& T-->/ W7 k( T2 T0 R" Y0 O. m' u! c
<!-- Do not modify this file directly. Instead, copy entries that you -->$ b) \) f; B8 e, i$ A
<!-- wish to modify from this file into hdfs-site.xml and change them -->
. W- a0 e) Q+ y4 w" |3 r& X<!-- there. If hdfs-site.xml does not already exist, create it. -->1 t9 A" L# ~( z# e4 ]7 }! }
<configuration>9 H Q' z; q- B+ @$ z* j! H: Q3 W
<property>
! j# W0 f3 T/ Z9 f6 V8 M <name>hadoop.hdfs.configuration.version</name>
$ q8 R7 i$ r/ h <value>1</value>6 {. h5 ?5 v; H* ~; ^9 L" t8 Y
<description>version of this configuration file</description>
; I" m' p: b1 F" O' N G& o/ v</property>
' b/ b; l# T6 c! s, n3 X<property>
3 K$ N' h3 I' m& h4 Q3 E <name>dfs.namenode.rpc-address</name>
- A8 q5 D; V$ u5 a <value></value>) M; l( k$ k9 _& J K
<description>
' Z* v1 o; p3 j$ d; } RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,$ f0 ~+ }& ?" y5 C/ }, H
the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1+ F: a( U1 Y1 n6 ?& B% ^
dfs.namenode.rpc-address.EXAMPLENAMESERVICE! C+ y, h6 \# Y1 K4 l# W
The value of this property will take the form of nn-host1:rpc-port. The NameNode's default RPC port is 8020.
! l3 Z+ U; o: s4 I @ </description>' ^# \% |6 f0 J# R
</property>/ M6 J0 j- f+ p* g; J0 z* H+ E+ P
<property>! L( ]1 m/ Q1 p
<name>dfs.namenode.rpc-bind-host</name>0 |9 V4 P; \" F7 _; b, N
<value></value>% O# u* i2 k9 A/ |( x1 E% P3 y
<description>
9 ~' B. E& h/ F The actual address the RPC server will bind to. If this optional address is" B1 R4 Q- z. D9 o; \/ H6 h
set, it overrides only the hostname portion of dfs.namenode.rpc-address.2 x$ F1 x5 I8 C# K8 g. q) T; e$ ?$ {
It can also be specified per name node or name service for HA/Federation.
2 C4 @# V8 Q6 a6 D, `% p% e This is useful for making the name node listen on all interfaces by
4 X, {6 v7 m* |- ^7 z: P/ b* O* d setting it to 0.0.0.0.
7 }* b5 ^' J' C </description>
1 o5 j1 i/ n' i</property>% r g; @/ {+ h2 c
<property>8 b. n4 r# R9 l% T
<name>dfs.namenode.servicerpc-address</name>" ~) y! c; o" d6 v7 T
<value></value>
) A0 k- Q' P: J9 H U <description>
6 C) U$ G! f' d7 P; k- ~ RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be
) R' d- I7 x& {% T# p, I; l; P6 }4 f& {/ ~ connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist,
: V5 s+ ]. B9 a: V% b& f; i. k* N" y the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1
& M- q9 l; V3 W: I dfs.namenode.rpc-address.EXAMPLENAMESERVICE& c# w, W& A$ u& o! n6 [9 C: `2 W
The value of this property will take the form of nn-host1:rpc-port.
& j6 r# D1 k# ^0 p: s* X$ D. } If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.3 U/ b& b4 i/ v
</description>
0 o+ O) L# V; D+ ~</property>
" o/ q; y( P/ p! o7 m<property>5 i+ h+ @$ [+ C% H: Z) R
<name>dfs.namenode.servicerpc-bind-host</name>
9 @, s) y! g3 s' n0 Z <value></value>
% G! r- y# a. q0 V7 X2 \ <description>
. Y1 c. X) \/ C0 Q+ w/ [8 ^4 H The actual address the service RPC server will bind to. If this optional address is
& w6 u' ` W. `/ g8 b set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
9 V+ o2 L- ]1 u# W0 t0 @" q, R It can also be specified per name node or name service for HA/Federation.* Y& U9 y# [$ j3 @( g. w
This is useful for making the name node listen on all interfaces by
2 p+ q5 t8 i6 w& I% ^: N! G, L setting it to 0.0.0.0.2 s, ^/ w( v- r: O- G
</description>
e2 D9 l+ D0 }3 B</property>
* ` e8 J9 R5 w9 x# B<property>* r, |+ Q; S, d: S" Q! E0 {
<name>dfs.namenode.lifeline.rpc-address</name>
* @6 v& w8 }8 }( A4 ^5 C# p, { <value></value>; y3 D0 Y, K1 D" L" c- D
<description>
$ H. D" q! {9 R5 L& m7 Q# t' v NameNode RPC lifeline address. This is an optional separate RPC address* R" \; r+ k7 O; ?& }2 K( J
that can be used to isolate health checks and liveness to protect against0 K A1 O8 Z+ B, a
resource exhaustion in the main RPC handler pool. In the case of
' G! m9 ^3 g( s" ^7 k, p& {6 x HA/Federation where multiple NameNodes exist, the name service ID is added
) F& B. j$ }8 g to the name e.g. dfs.namenode.lifeline.rpc-address.ns1. The value of this
- K9 [5 r, C' s5 }; n; I4 c5 L property will take the form of nn-host1:rpc-port. If this property is not
% b% `0 U0 j! J2 {0 r defined, then the NameNode will not start a lifeline RPC server. By
% U& ^: F0 Q* W' z: l5 U default, the property is not defined.) n) D4 c' U& D
</description>
" x9 k4 c$ [' k5 o( g5 @/ G</property>' Q2 ~* ^6 Z/ W: O
<property>, K" i. X3 o! i: t% A. j, |' [
<name>dfs.namenode.lifeline.rpc-bind-host</name>
/ v" o4 l7 n% I8 A <value></value>* U* F) w4 \- Y( E% P6 s
<description>
5 k6 k' ~0 R6 g) o The actual address the lifeline RPC server will bind to. If this optional' E+ _( f* d; p9 f9 l6 l
address is set, it overrides only the hostname portion of
5 d4 Y; K4 r& S1 p dfs.namenode.lifeline.rpc-address. It can also be specified per name node
p1 j& ]+ }3 @8 T/ @9 o f or name service for HA/Federation. This is useful for making the name node6 d' W% x7 Z" q, R
listen on all interfaces by setting it to 0.0.0.0.
5 V5 i2 q4 f0 C! g( h </description>& X, q& r6 H$ o1 `# V
</property>
7 w5 h- g( O& ~2 P7 w0 ~' m. R, p$ [<property>
\* Q% q' r; M& q <name>dfs.namenode.secondary.http-address</name>
9 U6 W3 V7 z2 M <value>0.0.0.0:9868</value>
6 ~* q" R5 ]* _5 x: h! @ <description>
& l) g4 m; ?, d& d( B& C, ]7 ~ The secondary namenode http server address and port.7 @ X; g2 M9 b7 |5 Y4 H
</description>
% e7 z" s e. j( D w' A</property>
- f2 j$ ?/ g3 O) D+ w, r) N) r. q<property>
( }0 e# I% K. F1 j- u <name>dfs.namenode.secondary.https-address</name>" z6 m6 w; [! [* X
<value>0.0.0.0:9869</value>
5 D4 z! ]" \+ y5 r4 Z) d <description>
8 A! Y6 f7 s7 `2 Z! K u The secondary namenode HTTPS server address and port.0 N# E" `; P* p1 V. |6 B
</description>* S9 P8 I, S9 S( B% h) k W L, ~
</property>) h, N' h. M' S' Q9 Z# Q' z
<property>* C. B3 R% l& J0 ^& _
<name>dfs.datanode.address</name>- L5 N* ]6 C5 C7 G4 W
<value>0.0.0.0:9866</value>" z# t# t9 G2 t- G& d
<description>
4 A: j$ f1 B# I) b The datanode server address and port for data transfer.& c8 f+ z: l% o/ H* I
</description>
4 M' h) H9 l9 A; q& C* _- I& d</property>2 x* A; `% f9 d4 B; `( K4 }
<property>
O7 P6 L8 a* O/ N: g9 u <name>dfs.datanode.http.address</name>: P0 G0 Y* @* @- D. p! s$ k
<value>0.0.0.0:9864</value>
/ I+ ? S Y5 M p <description>
4 x: |4 i% p' L4 L2 t" | The datanode http server address and port.
. n I f4 c. L9 C5 ~% D6 ^; H' } </description> S! `" {! m, }6 E$ h: j/ @
</property>
# O/ X5 h+ T% b- Q7 t4 O1 K<property>0 W7 E; l) }* A9 d- c. U+ t: m9 d/ M( g
<name>dfs.datanode.ipc.address</name>4 H4 V& M. i' X. e+ j! Q
<value>0.0.0.0:9867</value>4 ~! S8 s, v' B: ~) o& x' Q/ [
<description>
$ r5 Q& S$ j& K0 p. M- O' E The datanode ipc server address and port.8 }! E9 a; P2 F+ Y' i1 D7 C2 K" Z
</description>
+ l/ J8 }8 ^9 x% Y</property>. j9 j V8 `' C6 ^# h; W
<property>
: m# G9 r. _6 d' V$ F8 } <name>dfs.datanode.http.internal-proxy.port</name>
0 s" n/ f+ Q3 C% r) T: s. D <value>0</value>
# A! l% y. R3 q% d& K8 ?* F6 t <description>' I* [& S0 o$ ~
The datanode's internal web proxy port.
! m8 g6 h1 x2 M" h; e By default it selects a random port available in runtime.
( @# l' h5 U G( y3 O8 d# t4 S2 J </description>
8 v$ A0 Y- ]% [5 w ~</property>
- R6 d' T# `4 S6 w1 N' ]) x& S% Q' c<property>8 B9 k- v# t P& {
<name>dfs.datanode.handler.count</name>' \. m: \6 o9 k/ g9 n0 J. ~( K
<value>10</value>; W* I0 r. F6 Z3 x4 H6 U; g' N! x
<description>The number of server threads for the datanode.</description>
% w: n* s; n* a$ S- T* ?</property>
9 [& C2 U3 G, {" ?( i<property>" L. ?* Y. X6 V1 d0 w, \1 {
<name>dfs.namenode.http-address</name>
$ l6 F( f( ]& q# E3 _ <value>0.0.0.0:9870</value>
) }7 F' |2 Y M$ L9 O$ `0 v6 g n <description>
. e% b9 R) k" A! ~ The address and the base port where the dfs namenode web ui will listen on.
( o9 f; A# |! D, T3 ~6 N </description>7 ^; I$ R, g% N; ?
</property>) Y7 O4 r& a, b& D
<property>
. y, u& E7 d# q <name>dfs.namenode.http-bind-host</name>
' ]5 @+ A8 i. q6 G <value></value>3 [) S0 v0 s( c& X" p# Y$ }! V9 ^
<description>5 ` D3 r5 Y. B% O$ r1 E, m' P
The actual address the HTTP server will bind to. If this optional address5 s: ^; t8 v& k
is set, it overrides only the hostname portion of dfs.namenode.http-address./ \5 s" ~ q" w# }2 [7 T; l& Y
It can also be specified per name node or name service for HA/Federation.
3 a9 y4 z/ p( d" X) C t+ _/ _ This is useful for making the name node HTTP server listen on all; P8 f$ }6 _9 |" e
interfaces by setting it to 0.0.0.0.
+ l9 m+ M' P* w% u6 j$ _ </description>& D" W# s. v( m s
</property>6 h! Z, Z3 [- [! h; R/ ^
<property>
: P+ B. p7 [0 C- M, X& t <name>dfs.namenode.heartbeat.recheck-interval</name>6 O6 Y& Z' @. ^5 L
<value>300000</value>$ B: L0 M2 Y& u2 a. S& D M% ^4 d
<description>
2 z: \. g- n$ S) @7 q1 G/ p This time decides the interval to check for expired datanodes.0 }0 A7 x- i. v* F \6 S0 R) P
With this value and dfs.heartbeat.interval, the interval of3 a' j/ D% Y' v6 I; l
deciding the datanode is stale or not is also calculated.
$ N( P$ g; C! Q8 b7 x9 j The unit of this configuration is millisecond.; d) }% ^7 X' k5 v0 v4 X8 M
</description>
( d- c8 w; Y6 g0 g</property>
& i) S6 k% D4 k/ Y% d0 G3 M3 O<property>3 ~! p& c$ G5 h8 y. }
<name>dfs.http.policy</name> m8 N- `2 m; w, L) G
<value>HTTP_ONLY</value>
( y& D, F# I& t! o <description>Decide if HTTPS(SSL) is supported on HDFS2 X1 e2 G! V+ V
This configures the HTTP endpoint for HDFS daemons:% O9 b$ I" v' \& ~. \; O% k
The following values are supported:3 ?' ~5 ?' j* a5 n% e C* F, S
- HTTP_ONLY : Service is provided only on http! c. R) a) U0 B, s
- HTTPS_ONLY : Service is provided only on https
: l) Y1 U. Z/ m& v, q6 x* i: U - HTTP_AND_HTTPS : Service is provided both on http and https1 l3 `- R, ~+ y: \1 b- G. {' G
</description>3 v4 v7 D4 h% L% j: J
</property>
" [7 A7 Q O3 i- ?, q<property>
! N3 o$ ]8 j7 _5 p* W8 i <name>dfs.client.https.need-auth</name>! @/ `% C+ g# c* o# O+ T( p
<value>false</value>! t4 b& Y7 Z- x# ?. v
<description>Whether SSL client certificate authentication is required- i# @. _1 W, j! f$ |, _: m
</description>
2 R/ m9 V' \3 Q- ^</property># s5 m5 d3 W) _2 y- z2 c7 p9 z* Y
<property>1 M8 E* O. `8 o% V
<name>dfs.client.cached.conn.retry</name>
! C1 `5 v( X' L4 X o <value>3</value>3 k M* n6 `" g3 B
<description>The number of times the HDFS client will pull a socket from the
+ p: X) e5 `; `6 l" O cache. Once this number is exceeded, the client will try to create a new
; b/ V) j- y' [2 z. H socket." Y. a: M6 t2 U# P, a z
</description>
- j2 Z. b* h0 u9 ]</property>
# B: W* S7 `6 H# x<property>
: s5 K& q; {" v5 X2 [ <name>dfs.https.server.keystore.resource</name>( h1 p ^, t1 h) e
<value>ssl-server.xml</value>
7 Q3 x7 Y$ S- c8 U; ]: g6 n+ n <description>Resource file from which ssl server keystore5 Z8 d1 F b/ }/ G1 i; B& {( P5 I
information will be extracted
8 t/ @$ q8 ~' i6 M- y0 f6 T1 o </description>
- P7 z& v5 k5 P5 B</property>6 j/ m2 _+ ~8 r) B, }4 X! j$ @8 R
<property>
3 l5 L7 U f2 v5 q+ J <name>dfs.client.https.keystore.resource</name>: O q3 C, F* r; e" e: X* q
<value>ssl-client.xml</value>. Z* f' Q% V: l9 y0 L9 _
<description>Resource file from which ssl client keystore; g) ?6 x3 K1 l5 V4 s0 C
information will be extracted
" y% {& {4 `2 P+ y b6 x </description>, Y i7 l- D( z9 P& [
</property>) U" V$ D9 G- l( Y: \ {
<property>
1 v! s, [1 F+ _$ g <name>dfs.datanode.https.address</name>1 L) u7 Q& k: z, ?! d% y
<value>0.0.0.0:9865</value>
: d c) K4 w! n7 n' L <description>The datanode secure http server address and port.</description>* \; U3 Q" V+ a" x5 S7 b7 ^& d
</property>6 b3 w& `6 Q2 s0 R) M9 |+ F
<property>) P: k5 h- l+ V) y) O! t8 E( X
<name>dfs.namenode.https-address</name>
. e8 x! l) o, ?' g3 @ <value>0.0.0.0:9871</value>
+ U8 g8 [4 g/ W. O <description>The namenode secure http server address and port.</description>! t) \8 l( w9 L! a0 d( m* E) t0 S
</property>
& H/ @8 W: ^) O" k$ y<property>
& q" y- J2 R8 h( w1 X: B <name>dfs.namenode.https-bind-host</name>' \" X, |! ^+ E; \) x0 o. q6 S
<value></value>
9 R5 ~* B/ d9 Q! s <description>2 F5 d( N& }4 Z9 C# H! H
The actual address the HTTPS server will bind to. If this optional address) q# U O9 X; ^( U% _, Q
is set, it overrides only the hostname portion of dfs.namenode.https-address.6 v9 N+ `: [# M# S5 ^ u& f
It can also be specified per name node or name service for HA/Federation.0 A" D, \! Y$ M; o6 }
This is useful for making the name node HTTPS server listen on all
% @9 b! X. z: _8 G' s* K interfaces by setting it to 0.0.0.0.8 F; `6 A9 f9 I( i
</description>4 h: Q x2 w+ B: a! k
</property>
`5 c* L; D2 @: I0 Y" D! B1 [* ` <property>/ b6 x' ~$ l, \" V7 E+ q
<name>dfs.datanode.dns.interface</name>
/ D4 q' e7 \; \, i, c, V$ H <value>default</value>
. e7 k% t9 _! q3 Z+ T' [: w <description>! e& n( M% y1 T- \( T: s
The name of the Network Interface from which a data node should
8 p$ m |' e9 E3 `5 A report its IP address. e.g. eth2. This setting may be required for some
( D4 g! _3 ^/ J4 Q multi-homed nodes where the DataNodes are assigned multiple hostnames
* d, i; r1 \; V and it is desirable for the DataNodes to use a non-default hostname.
* R; P2 d; M, n0 y7 r7 D( P Prefer using hadoop.security.dns.interface over# l: L8 O: Q/ @
dfs.datanode.dns.interface.
/ v/ p4 K& w8 T </description>7 m3 M% R8 W( J, k( ^
</property>) c7 H9 k/ W' ^ S! R, l a
<property>
- f# C+ t3 s1 N <name>dfs.datanode.dns.nameserver</name>- Q, I1 T& z0 L9 v2 }
<value>default</value>
3 Q6 q) U. E, w: _" ?) h <description>
. A. s# H8 j- P& j The host name or IP address of the name server (DNS) which a DataNode" F+ S, C2 O1 @4 m* Q4 e: v
should use to determine its own host name.0 H3 D$ M! x; v* k6 x) K% ^4 T
Prefer using hadoop.security.dns.nameserver over# M+ Y+ @8 E3 ~6 s) e# @- U
dfs.datanode.dns.nameserver.- P N& l; x: v N+ q; F6 h, q+ a
</description>
& o( r5 ?& \; W: }$ z5 w: b3 ~. \ </property>
2 u- r7 _5 _! r! H5 a- d& O <property>
! {9 _) l- B! N9 O+ ]' Q4 s <name>dfs.namenode.backup.address</name>
& v. W1 |3 l6 p9 q6 V1 k% _ <value>0.0.0.0:50100</value>
, G4 b3 t0 I9 X9 L <description>' T) h+ G! q, J: m
The backup node server address and port.
! f; u" I) S/ G If the port is 0 then the server will start on a free port.
1 \2 w: e3 S! C& O t* L" \; I </description>
6 ~1 \; }6 ?0 J</property>9 C$ q, L! C0 p+ J; [* {
<property>
% {' C1 T/ j; |: S9 T <name>dfs.namenode.backup.http-address</name>' P7 C0 U4 G: [2 D: x8 C7 o
<value>0.0.0.0:50105</value>; q* f: b; r2 n, |
<description>8 C& S- N9 ^1 m: c$ c* h- v
The backup node http server address and port.
. {+ a4 M3 k! w$ ^7 s2 ~# [' a6 g+ @ If the port is 0 then the server will start on a free port.) i, N' ]0 X# U/ C# v, u
</description>
9 T/ d5 c7 D. D$ e</property>9 f2 [5 I4 H2 M; {# X) I m
<property>
7 q/ q( S. x P I. S) F& K, P <name>dfs.namenode.redundancy.considerLoad</name>8 s# V; o4 j, W+ m& K1 J
<value>true</value>
: s4 h) U% ~7 Q- e <description>Decide if chooseTarget considers the target's load or not, j8 j% I8 ?5 v) `" Y+ s
</description>3 u, B& p; { [4 {+ P2 N" l
</property>
2 G& o* `3 a& z7 S9 z: f <property>
5 Z6 z% F; f' i5 E: X6 P <name>dfs.namenode.redundancy.considerLoad.factor</name>$ N% H& b. g" W j a$ Q
<value>2.0</value>
8 E; t" L9 l* \! z' X7 u <description>The factor by which a node's load can exceed the average
" k% V% \$ V9 Z# `4 Q) c before being rejected for writes, only if considerLoad is true.
; I$ k. n3 B9 @* d/ x </description>
! R7 o) R* P, Y, d9 t' _7 ^4 N </property>; i2 S$ J% N( b* d
<property># a' g4 s9 X0 A+ `/ ~4 q* |" s# F
<name>dfs.default.chunk.view.size</name>6 e8 v0 M% k4 ?/ Q( r
<value>32768</value>9 |( B' ~! _1 g7 I4 i
<description>The number of bytes to view for a file on the browser.8 _/ ?, f0 |! D1 C9 B
</description>6 f: i" W$ l: @ a
</property>4 @& M9 W5 L" k2 F2 ?% \5 Z
<property>
3 Q2 ?) X; E6 z3 T, K5 [ <name>dfs.datanode.du.reserved.calculator</name>% \* x& t/ I9 q" t) G% X
<value>org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorAbsolute</value>4 I# G) B/ |" o' r" T/ N6 }
<description>Determines the class of ReservedSpaceCalculator to be used for9 E- |7 H7 _5 U2 |7 ^7 o$ s7 X% i
calculating disk space reservedfor non-HDFS data. The default calculator is3 I8 |: g7 v7 d* V: t- ~
ReservedSpaceCalculatorAbsolute which will use dfs.datanode.du.reserved* E& ?; h+ Q4 ~' g- P" d
for a static reserved number of bytes. ReservedSpaceCalculatorPercentage, t, }8 O" S3 t" d% P7 j! i
will use dfs.datanode.du.reserved.pct to calculate the reserved number) d% ?7 p; `$ q+ E
of bytes based on the size of the storage. ReservedSpaceCalculatorConservative and2 c/ d0 s+ _+ }7 ^
ReservedSpaceCalculatorAggressive will use their combination, Conservative will use
; ~: h+ K$ E. ^ k- S# W! H8 G3 e$ z maximum, Aggressive minimum. For more details see ReservedSpaceCalculator.. z! f( l) m! l: a, H" o; B
</description>7 O/ ~% I9 R( d0 }
</property>; [6 Q. z3 E" ~
<property>% D. p% d* ^6 z9 w; X q, ~
<name>dfs.datanode.du.reserved</name>. C! U1 V, P# ?
<value>0</value>" Z1 z3 i' [" N; t% ?# j
<description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.2 \7 B, A) \9 V7 V+ I( P6 ]* N
Specific storage type based reservation is also supported. The property can be followed with5 o$ e# x0 d( v1 w1 a" R
corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.3 Q# a$ N8 |" j( I2 f/ a9 `% C& c* V
For example, reserved space for RAM_DISK storage can be configured using property
7 M/ E2 i# ^) F/ P) B 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured2 i# }, d+ k( e, E) d% K4 q
then dfs.datanode.du.reserved will be used.! t c" v3 ~! ? g' w$ F$ a( L- ~
</description>
) f7 A, c' Z) G+ M) Q3 L- c</property>" X* }( I" g1 c! z" d; c7 ?1 ~. g
<property>
# {1 V; y! G" {- s# X <name>dfs.datanode.du.reserved.pct</name>
8 u' d# V, C& n+ g8 V' a <value>0</value>
7 i' t8 \' J* E+ J1 S" K <description>Reserved space in percentage. Read dfs.datanode.du.reserved.calculator to see# x" j: T* F; d m/ |- z% P
when this takes effect. The actual number of bytes reserved will be calculated by using the
: U; Z8 J' G) Y. J1 L @' T total capacity of the data directory in question. Specific storage type based reservation: z+ I5 m& V# |
is also supported. The property can be followed with corresponding storage types
+ f& o/ E+ w J- |5 G" I# k6 a ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
1 [, b/ R! b( P0 k For example, reserved percentage space for RAM_DISK storage can be configured using property
0 o) p5 {$ z/ [+ ^$ v8 @ 'dfs.datanode.du.reserved.pct.ram_disk'. If specific storage type reservation is not configured# R: }( D1 V# Q$ ]9 f& j
then dfs.datanode.du.reserved.pct will be used.
' X* b; H/ l9 ~, d </description> ^: }4 P6 @' X* {0 L1 D2 r# `
</property>
; Z0 z4 u( V- U+ H. G<property>6 W u& U$ @' F5 P9 F$ L1 n& J
<name>dfs.namenode.name.dir</name>
! }3 N: A, |; j0 n/ a <value>file://${hadoop.tmp.dir}/dfs/name</value>
, |* D: |* S' _# L <description>Determines where on the local filesystem the DFS name node
+ E7 c7 C U6 B+ k/ F should store the name table(fsimage). If this is a comma-delimited list% P& y8 D1 c9 f9 x) \* y! J
of directories then the name table is replicated in all of the
, ? Z7 J" E) L! J' {1 w2 J directories, for redundancy. </description>
2 s9 A3 L9 G `* U7 A</property>) d9 L- ~7 T( O
<property>+ f% q% ]1 Q* _9 N
<name>dfs.namenode.name.dir.restore</name>
! ?% l9 y M' E1 i, G <value>false</value>
) V, j5 q$ }5 T! V' b4 c <description>Set to true to enable NameNode to attempt recovering a$ W, G% M5 p6 e( B; }) @
previously failed dfs.namenode.name.dir. When enabled, a recovery of any
+ J7 T/ T* d# p1 t: u failed directory is attempted during checkpoint.</description>
% H9 y- [7 |1 K& w7 w# d</property>6 I" D% h5 H0 l, H) M. @+ X
<property>
$ m' q' E9 X, v' t <name>dfs.namenode.fs-limits.max-component-length</name>
) q$ d$ f* g! `4 \- d! |# h- v <value>255</value>
, j/ F+ q( w+ E; H2 {# r. h <description>Defines the maximum number of bytes in UTF-8 encoding in each
% ]9 p. O9 p. o3 a" ^ component of a path. A value of 0 will disable the check.</description>+ v6 Q" A4 f) X* U+ y/ O# @% h
</property>
% P. \( l2 h' g& [# N6 ~<property>
, v6 \; h$ k) _6 B5 B" ]2 [ <name>dfs.namenode.fs-limits.max-directory-items</name>
2 }& p/ }; |. F, N( ~ <value>1048576</value>
! W% A* o2 h9 t0 z- b! B' A% d# W9 N <description>Defines the maximum number of items that a directory may" K7 L9 | C9 v3 }4 c1 [8 z
contain. Cannot set the property to a value less than 1 or more than
7 Q; {" l" f1 J: h2 o 6400000.</description>1 E; T+ c! F( O
</property>: S5 g" S) H( u( \8 l
<property>
! @5 D7 ]) T! |, a8 B/ ~ <name>dfs.namenode.fs-limits.min-block-size</name>
& w) e- _ o7 D9 g8 ~- a- ^/ L <value>1048576</value>
" B( p. g2 E5 n. L+ z <description>Minimum block size in bytes, enforced by the Namenode at create! y6 ?/ L: Y+ ?* }4 W+ L
time. This prevents the accidental creation of files with tiny block' f) X K4 H( J5 [
sizes (and thus many blocks), which can degrade2 \- R' P" p1 x @
performance.</description>0 c' r3 ]8 V8 V5 e% {& O9 a3 r
</property>
# u4 W8 ]5 I4 y: m. _<property>
& J5 e; U! ?# K6 x \3 I) B8 E <name>dfs.namenode.fs-limits.max-blocks-per-file</name>
# \* j, n$ w, T E( o, Z: E. l <value>10000</value>
1 F' Z' M& X4 j$ I' j, c% c <description>Maximum number of blocks per file, enforced by the Namenode on
+ I N' I1 L) E% m. o. u& N2 ` write. This prevents the creation of extremely large files which can0 c8 r, O& R: g3 C l% Q4 E; g
degrade performance.</description>
) T, |0 S% l( i; G* [8 o0 A% R( q1 d</property>$ C2 E* Z# l) Z9 H, B/ C
<property># [& E7 H+ y7 k
<name>dfs.namenode.edits.dir</name>1 W( B |# V$ n: |8 A
<value>${dfs.namenode.name.dir}</value>6 c6 Y( R! G/ {' X1 O
<description>Determines where on the local filesystem the DFS name node4 C8 D7 T6 K9 ]" X" P
should store the transaction (edits) file. If this is a comma-delimited list
3 ~& r1 \* b# |7 I of directories then the transaction file is replicated in all of the
) _; y/ S' I' \" k9 Q directories, for redundancy. Default value is same as dfs.namenode.name.dir( }& _5 z' H6 q0 V
</description>
) T# ]# `& R" G! K" Y</property>
4 k$ J! x0 R7 O8 E<property>5 t8 j) c( l# O3 [. K% P+ W* J4 Y6 N
<name>dfs.namenode.edits.dir.required</name>4 G P% ]7 j1 O
<value></value>: t: W/ a& F, i0 i
<description>This should be a subset of dfs.namenode.edits.dir,
2 K. Z' j) f8 R4 t' \ to ensure that the transaction (edits) file" D9 B n, y- H5 T! V' W0 K+ n' P
in these places is always up-to-date.& Q* J8 r: J6 Q" g) y8 S8 ~
</description>
% k' U' W# O5 d: R4 n- l</property>2 i' A; N- o* o9 T, _& b) }. j
<property>
w9 u$ e. Q( }, l( d6 B" ^ L b: m, ? <name>dfs.namenode.shared.edits.dir</name>
& x9 b/ H$ T* p0 D, b <value></value>
! n3 {5 b0 ^) B <description>A directory on shared storage between the multiple namenodes8 n. P$ ? e" [
in an HA cluster. This directory will be written by the active and read
# P Z/ U' t9 T4 f- ^ by the standby in order to keep the namespaces synchronized. This directory
/ _$ w' J3 _, Q. F$ a. `6 O does not need to be listed in dfs.namenode.edits.dir above. It should be
2 Z4 A: T# ]3 r) r left empty in a non-HA cluster.
, ?8 u7 D' V2 V2 A </description>2 {8 H$ q) g" ]+ f
</property>
( N/ C* V6 @% t8 @) B. g<property>
; K4 R: v/ p* Z, J2 n! h, s, t <name>dfs.namenode.edits.journal-plugin.qjournal</name>4 }1 J) [$ C; @/ |! K9 q
<value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value># Z3 }8 r& ]. ?
</property>3 u# @" h' N+ D( P
<property>4 D& p8 W+ f# l
<name>dfs.permissions.enabled</name>
1 S& s) w8 _7 ] |! Z; r+ b: W" D <value>true</value>& v* J7 ^" q5 j
<description>: c3 K1 k5 H* T {7 y. _3 l- g2 M
If "true", enable permission checking in HDFS.: o2 _ D6 v2 ^( \, |
If "false", permission checking is turned off,
0 L( o) I0 V$ d1 e$ j4 }. _ but all other behavior is unchanged. L1 m( i' ]* }& h# U: }! e
Switching from one parameter value to the other does not change the mode,
. y/ w2 f& H- ?: J4 V: R2 z owner or group of files or directories.
9 u! r9 i7 H$ Q </description>. B; a' g8 o, ?$ h) D- L
</property>8 i! X8 j6 V& s7 w
<property>
+ D) b: r' H+ M) B0 V' c <name>dfs.permissions.superusergroup</name>9 a a* e, h. S P
<value>supergroup</value>- D5 R/ n: B, r0 X
<description>The name of the group of super-users.1 ^' L( |( r: m& j: a$ R: X
The value should be a single group name.
) f' k3 ~* T$ F4 j </description>3 @6 C6 l* [/ ?$ ^9 X* o
</property>* u: N* b4 y& Q+ }2 P
<property>$ S4 E1 c3 w0 \: k/ f6 U
<name>dfs.cluster.administrators</name>
) h, F/ T! J, I& n2 E: H <value></value>
8 D! B+ _' D) Y8 q. x& k <description>ACL for the admins, this configuration is used to control
3 D+ W* w( {& i) ~& ?* l0 ~ who can access the default servlets in the namenode, etc. The value
& z* d7 i( J8 b should be a comma separated list of users and groups. The user list
( G* ^: u4 t: n2 I, ?7 e: X, F$ S& A comes first and is separated by a space followed by the group list,$ \0 \! W: h4 H& \3 r0 U: ~! R% Y
e.g. "user1,user2 group1,group2". Both users and groups are optional,
4 O" P# T% I7 M* S2 X so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2"& R1 U1 V, Z9 Y& S/ y
are all valid (note the leading space in " group1"). '*' grants access, Q6 U* J/ _; O$ |
to all users and groups, e.g. '*', '* ' and ' *' are all valid.
* S2 h$ d2 q6 F3 u </description>
0 Q2 x) P- B; r* E</property>
) U3 p! ?" q& A: S<property>8 d3 X8 R& M1 |# V) l4 @
<name>dfs.namenode.acls.enabled</name>
# q- X# J/ r; y9 `, j <value>false</value>
. {3 Z. X& k( y3 Z' c4 g! R <description>6 t `' Q1 C5 K0 F
Set to true to enable support for HDFS ACLs (Access Control Lists). By! ]- F9 E, |: ]9 L1 I! T7 g7 _
default, ACLs are disabled. When ACLs are disabled, the NameNode rejects7 W' q E6 X, {. g Z2 _ T
all RPCs related to setting or getting ACLs.$ G, r$ h- q" T6 u' E [7 \
</description>
9 {+ ~& h# L9 w6 f, X2 Y</property># a: d% r. }) \9 X: t# X
<property>* v( T9 I( ~8 n
<name>dfs.namenode.posix.acl.inheritance.enabled</name>
# U- o3 \ [. A1 d9 o! s <value>true</value> l6 y' t( i5 d. A4 |
<description>
) J+ o- A* v+ r/ ^ Set to true to enable POSIX style ACL inheritance. When it is enabled
1 u0 V- ^% u# ? and the create request comes from a compatible client, the NameNode
& h1 E. _" e% m. ~ D: i X; S will apply default ACLs from the parent directory to the create mode
# E' s, g, s- M8 R$ ~- { and ignore the client umask. If no default ACL found, it will apply the9 o8 S3 @ {) E5 q' q l8 B
client umask.1 y7 w8 }- P, u# a0 Z( }
</description>' a* _( |. b `* H- S0 c r8 U
</property>$ Y* L2 `& y( [, u1 T. l
<property>
% j% s! `& b7 c: \7 c' R <name>dfs.namenode.lazypersist.file.scrub.interval.sec</name>
; B) j2 q) ? G$ h6 S3 Z5 X <value>300</value>- V& p* N5 Q4 r+ d! }
<description>
9 h3 g) k5 c& u The NameNode periodically scans the namespace for LazyPersist files with
6 U/ S, }. n& _' m% u missing blocks and unlinks them from the namespace. This configuration key. U; i [& ^* v. h
controls the interval between successive scans. If this value is set to 0," x- E( t* `9 n: V. x
the file scrubber is disabled.4 ~, P, l$ D4 a; k
</description>+ M! {8 ~5 z, R- V6 Q3 V
</property>
' @: U! H4 C2 |<property>
- ^ A t$ C* {' D* z$ k <name>dfs.block.access.token.enable</name>
m3 }0 V% X6 L! p <value>false</value>
% c$ {" A7 U# \: O+ p1 H0 E# z& C <description>
l" ?- }+ O; H/ q' \; s If "true", access tokens are used as capabilities for accessing datanodes.
M* G- Q. Q* z l If "false", no access tokens are checked on accessing datanodes.
! \" {& c, e# Q2 |4 @ </description>
8 _$ Q% q: ^% f; O</property>
, J6 q0 P' c( {4 C1 F: ?8 S<property>
+ }; ^ P6 Z5 z <name>dfs.block.access.key.update.interval</name>
- \8 p$ b3 f5 e i9 M$ d9 G <value>600</value>
) \; y9 r# m; c <description>8 @7 B7 R" B2 j9 M/ _& J
Interval in minutes at which namenode updates its access keys.
6 n8 p6 ^ z' ]/ h I4 B/ D </description>
' D% Y$ D" |% T) E# l</property>$ `0 n; \7 H# f7 i; ^! ~
<property>
# |- o( e" Q2 g; i! }; `# @ <name>dfs.block.access.token.lifetime</name>
) B& P" ~, e! _* D+ _ <value>600</value>) }4 j9 m- y5 t! k! h' A% ?: |3 W
<description>The lifetime of access tokens in minutes.</description>) U8 N+ l# E; @; J9 K1 n" D. x7 ?
</property>
4 B: S+ Z! m) e4 Z% g Y$ i<property># \4 D7 n; ~ j" t; _- u' k
<name>dfs.block.access.token.protobuf.enable</name>: f* O# R4 `" X1 q* s
<value>false</value>" [$ ?! w( u) F. ~
<description>" B# M: H9 S$ w0 u7 y
If "true", block tokens are written using Protocol Buffers./ z0 Y2 |; [6 l! x
If "false", block tokens are written using Legacy format.9 ~ N% `+ l$ O3 ?$ _. J! X
</description>
, d1 D* \) W4 t& h</property>4 c% E: C" h+ T/ _5 X+ F4 ^0 S9 a
<property>
# @' w( e+ d& ?5 C( s3 }* Q <name>dfs.datanode.data.dir</name># ^+ q' z. q' {
<value>file://${hadoop.tmp.dir}/dfs/data</value>
" Y# ?& T: a( @9 R( s) E1 ] <description>Determines where on the local filesystem an DFS data node; |0 V" M- n7 c: \) E) ]0 n
should store its blocks. If this is a comma-delimited( @6 {6 Z) L4 I' c
list of directories, then data will be stored in all named w- j. C: V% Y9 W0 @
directories, typically on different devices. The directories should be tagged
3 X+ U$ y8 @+ R$ t1 s with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
) Z* W& Z/ r% \$ w0 E) A storage policies. The default storage type will be DISK if the directory does
- _6 s2 ~, s( ^/ O$ F not have a storage type tagged explicitly. Directories that do not exist will" N0 ?+ R/ `3 f- v
be created if local filesystem permission allows.
* D$ }' d# _. D5 r </description>
$ K! p8 M! n8 f2 E0 y5 P' n. }</property> I' K# L" K* P/ j0 \0 `/ D
<property>
: c+ q* l% O* I <name>dfs.datanode.data.dir.perm</name>; s% z6 j6 f+ t8 n
<value>700</value>% g- `' o( D6 F8 a
<description>Permissions for the directories on on the local filesystem where4 h2 h0 m5 Q) W: H: j
the DFS data node store its blocks. The permissions can either be octal or
, R- \6 B0 H& U7 B3 e symbolic.</description>+ q: O# e, _9 u5 i
</property>0 l5 n3 u4 t4 M5 Y8 _3 n
<property>; I( x% h1 T1 {- A% T. W
<name>dfs.replication</name>
- a% i; Q2 \6 `+ @5 v2 Z <value>3</value>
: Z. [* q6 [3 Y! ^3 c z <description>Default block replication.
: N4 E" Q2 |3 g, d' l! a4 X' L) ?* l The actual number of replications can be specified when the file is created.- F. o4 H3 k q! J, X
The default is used if replication is not specified in create time.
% Y% z/ P9 k. o& A </description>, C+ `8 @0 `* B |3 w& D" S7 q
</property>
9 T) M0 K/ J4 B1 q" `<property>* F* Z+ ] \* X" I! Q! \
<name>dfs.replication.max</name>2 S, l+ I/ S8 P- @5 b: d/ N
<value>512</value>% l, U5 d t1 M* Z* C) K
<description>Maximal block replication. & C9 E* Q5 x: L. a6 Y: t
</description>+ C& a) m$ A' H! S O7 X# L
</property>, [" c h/ i9 r1 L- o+ F
<property>- t! s4 Q5 Y/ B7 M* N0 R$ w5 Y6 ~
<name>dfs.namenode.replication.min</name>* Z- R% @( E V( V1 v
<value>1</value>
7 U6 Q0 O. R9 j6 L# S [ <description>Minimal block replication. # s8 h) b' R, N8 D
</description>
! d. m$ _7 J0 o+ @# [0 ^</property>
9 `( \8 B& A2 T! V8 z2 j5 C! S+ N. t" D<property>
( ^0 m; H' e! h7 d9 Q2 G3 j: w <name>dfs.namenode.maintenance.replication.min</name>
' }7 m% f8 D9 x, v1 B1 z <value>1</value>
% M- k$ M0 b" v9 o7 X <description>Minimal live block replication in existence of maintenance mode.0 x# Z: K$ g4 _0 M$ O/ R
</description>
3 S( z/ _1 x/ v& g! H# K8 @" t</property>% L O/ F% M1 A4 [0 {
<property>
( b- A$ |" x' D+ m <name>dfs.namenode.safemode.replication.min</name>
, n! I: S( @& R1 _* j9 {) V# T' A- _ <value></value>
2 b, \( v# p( F <description>
6 r+ D# d, |1 y2 J/ I. ` a separate minimum replication factor for calculating safe block count.
" _2 n. J7 f* T0 D This is an expert level setting.% U- n3 }9 x: ?5 t2 a, b! h
Setting this lower than the dfs.namenode.replication.min" c, s, _6 ?0 X$ Z3 H5 }
is not recommend and/or dangerous for production setups.
0 N1 L: ]0 @& d7 Q5 E4 c When it's not set it takes value from dfs.namenode.replication.min( Y( E8 \% S4 z/ A7 g$ w! K
</description>/ n; o' Y, b% D( |8 l
</property>
) a+ A' V$ g) ^( Z. n% V<property>
# Y+ J- m' s7 L5 g" W1 X7 w <name>dfs.blocksize</name> B# }& G& v& ^8 s; P* [
<value>134217728</value>
1 k+ s* s- o+ V, P1 C7 k <description>2 e* E* S/ {. f- E; e& a# h
The default block size for new files, in bytes.
! h/ v! n" Y1 Q You can use the following suffix (case insensitive):) B! v7 \% a: O% C7 _
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
* B+ e) Z/ w8 a2 r. }7 j$ w Or provide complete size in bytes (such as 134217728 for 128 MB).; h# P& K% n6 ~& y
</description>
3 }" v5 M) }3 ]$ |</property>; ?$ l W. z5 R, B' m: f, e! W% \
<property>3 `* L6 k# e3 o5 O5 A
<name>dfs.client.block.write.retries</name>- h" L# g; r3 x. g" m+ {
<value>3</value>. X3 Z5 V; h% S- j5 ]
<description>The number of retries for writing blocks to the data nodes,
2 L2 E4 J5 f5 U* T5 ], W1 L5 [ before we signal failure to the application.
2 f. J5 ~+ `( ?1 L4 v* t </description>- ]# h* y. b) m2 _9 U2 N1 v
</property>4 M1 b0 D: ]5 ?5 R4 M
<property>
[+ b9 A# @, t7 z o <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
& I& Z! U3 H5 q, r% f8 e8 K <value>true</value>" s0 y J6 p$ v I5 U* h
<description>0 ~% P- n4 f# @1 C+ f9 b: j% A
If there is a datanode/network failure in the write pipeline,
+ G: i' f" X9 n: V- s2 _! w DFSClient will try to remove the failed datanode from the pipeline$ f q6 U, M6 P+ U/ a& ^9 F
and then continue writing with the remaining datanodes. As a result,
! r2 P! W' x4 S+ e the number of datanodes in the pipeline is decreased. The feature is7 }, i0 Z5 \! i' G- {! \
to add new datanodes to the pipeline.
, S- x4 ]& @" | This is a site-wide property to enable/disable the feature. u6 C( w$ A- z" s% o* k( ^4 M
When the cluster size is extremely small, e.g. 3 nodes or less, cluster) w" {' r! T5 z: H
administrators may want to set the policy to NEVER in the default
4 Z/ h; V2 ~5 y9 z! a, P, P! d# u configuration file or disable this feature. Otherwise, users may
6 i* T) i3 R; e X experience an unusually high rate of pipeline failures since it is' d$ U; k7 q" ^7 ~7 V
impossible to find new datanodes for replacement.
- ?3 o. A( [, D4 N# D See also dfs.client.block.write.replace-datanode-on-failure.policy- _" D4 L9 i( C3 i+ {
</description>* H' _1 ?; G+ G+ f: U* ^0 d
</property>
7 d- e' V7 v0 D; y1 K, d8 @<property>" n: n, G4 E% {! ^2 l; H
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
6 f ?% z9 p; P0 S1 d7 _/ j: N <value>DEFAULT</value>
& y2 `4 w% {: l; T0 r <description>
; \; G9 Y6 M, q) [6 l4 _ This property is used only if the value of
! q' r) T7 W1 L- f# s0 P$ \ dfs.client.block.write.replace-datanode-on-failure.enable is true.
" k9 F- A' r" J$ ^$ n ALWAYS: always add a new datanode when an existing datanode is removed.
# ]1 z) C/ `) h: b NEVER: never add a new datanode.
( F9 c* P0 R8 f% ` DEFAULT: 4 x1 e. j6 y- m8 b4 ?
Let r be the replication number.- T1 A% C$ N1 n& Z- V
Let n be the number of existing datanodes.
7 X3 ?! u& H0 t) z! T Add a new datanode only if r is greater than or equal to 3 and either" x# h6 E! R3 ]+ E" Z }
(1) floor(r/2) is greater than or equal to n; or% T f9 N) Z2 j/ Z9 d# `
(2) r is greater than n and the block is hflushed/appended.
0 y9 A8 C x& X4 j; a. z* r* v </description>% g& j1 e$ J4 \# n
</property>
5 T2 o# b: _- J+ w! r<property>9 N% r. @2 Q, f" D
<name>dfs.client.block.write.replace-datanode-on-failure.best-effort</name>
2 J b; X0 }% `2 ]) e+ p <value>false</value>
$ P5 G. |, T# Y b6 b6 o# H <description>1 g, W- V7 O Z( y
This property is used only if the value of
/ I9 {- m% f6 {# R dfs.client.block.write.replace-datanode-on-failure.enable is true.
2 I- S! A0 u4 }! T Best effort means that the client will try to replace a failed datanode
2 V; c0 ]! @ K in write pipeline (provided that the policy is satisfied), however, it
1 P: _/ F5 N, b/ e continues the write operation in case that the datanode replacement also
+ o! Z5 i9 N/ t% h ^: a$ w fails.9 u# |% F% X! k" r" [
Suppose the datanode replacement fails.! P: V! p& @: Q4 p" \! N N( t- [% g
false: An exception should be thrown so that the write will fail.
. N( n% e% j3 Y8 E4 m. q3 F true : The write should be resumed with the remaining datandoes.2 _0 Z1 i* P3 k: X; I/ g7 H6 s) {
Note that setting this property to true allows writing to a pipeline
/ Z& s- d- u, o0 L5 Q7 P. \8 P6 M with a smaller number of datanodes. As a result, it increases the
6 |& o1 r7 g" r# N7 ]7 X. w- K probability of data loss.) ?* M& |9 k$ n6 q3 f$ `. c& O& r3 ^
</description>5 [5 Z% j5 A2 _( C! n0 E' q: \
</property>0 c) ^. }# O1 G% M1 o
<property>: c: _* v9 \! T
<name>dfs.client.block.write.replace-datanode-on-failure.min-replication</name>/ y' [$ L& |9 C: Z& ~4 M0 N+ R
<value>0</value># f @6 ~5 \ B z0 F
<description>
9 }3 Q" X4 ^, u& F+ O5 ` The minimum number of replications that are needed to not to fail! X: [6 S( f* i U6 k
the write pipeline if new datanodes can not be found to replace
; h. i/ I+ y& q! z failed datanodes (could be due to network failure) in the write pipeline., ?" U) g8 s* v2 _) r4 m {& V
If the number of the remaining datanodes in the write pipeline is greater
& j. ?# C6 E1 C) D6 u" C7 D than or equal to this property value, continue writing to the remaining nodes.9 d' ^$ a& z6 z$ x7 O7 R
Otherwise throw exception.
) K) V. h7 Z1 ^ If this is set to 0, an exception will be thrown, when a replacement* d& m: Q( v2 Y9 \* R) a5 C
can not be found.3 v1 R- z) l) d
See also dfs.client.block.write.replace-datanode-on-failure.policy9 W& w& l9 @, `: i0 A
</description>% L7 m& h& ^4 _% @0 l2 t
</property>
* f2 |' u; u4 w4 d1 t1 Y<property>. U* F+ Y" ?5 E
<name>dfs.blockreport.intervalMsec</name>
" _7 e6 O$ L4 X. T3 R1 } <value>21600000</value>/ |) d7 E; r( G& x# s$ p& L
<description>Determines block reporting interval in milliseconds.</description>
) p/ R, i( h+ _: \: b</property>
) j& ~# [, G: Q" @: r<property>% O9 W7 V9 a& Z' }
<name>dfs.blockreport.initialDelay</name>
" P+ K2 i& {7 B1 L9 R/ r- V <value>0s</value>
4 d( `/ c. j2 E" ]" ? <description>1 A/ S1 o3 @& _: ?
Delay for first block report in seconds. Support multiple time unit
' o6 \' z! L; @2 _ suffix(case insensitive), as described in dfs.heartbeat.interval.$ h: l0 ?5 v9 N a2 G
</description>
; \4 Q. V) j* v% l</property>. @; i. ?9 i0 k
<property>
( _* H' f; Z. h( _" B <name>dfs.blockreport.split.threshold</name>
7 X" K* x& ]! r/ g$ | <value>1000000</value>
* V8 C" E/ i0 B <description>If the number of blocks on the DataNode is below this
. F. Q* d8 c+ } k$ ~ threshold then it will send block reports for all Storage Directories6 y2 K/ k5 w8 G9 j3 {
in a single message.( M" {4 h/ @/ I/ b4 T
If the number of blocks exceeds this threshold then the DataNode will6 @# U7 U* m( O2 t. _" c, h& Z
send block reports for each Storage Directory in separate messages.
( o9 T% x: n" A O/ `9 J) V Set to zero to always split. U( P6 M: D9 P8 F
</description>
4 i- s# \4 M1 C; [</property>
3 \9 H% J: L8 ^0 i2 L<property>; q# K, }# ~( ?# R" c3 l6 f/ b. m
<name>dfs.namenode.max.full.block.report.leases</name>9 `# s, M7 f& W9 ~' C. Z* a. _
<value>6</value>, D* Q. E& o1 K1 X
<description>The maximum number of leases for full block reports that the& s8 d) j# M) r2 _+ } a6 a
NameNode will issue at any given time. This prevents the NameNode from
x! y+ @ B i7 b: R being flooded with full block reports that use up all the RPC handler
' t% m5 [' K% g threads. This number should never be more than the number of RPC handler& b* N5 u3 L1 N9 x: K
threads or less than 1.
4 E/ _* G9 _& }# v; A( Z5 a </description>; s: {$ Q& L4 `' V
</property>
5 P/ a; `, K# ~* L2 v# t<property>& f! y' ?% _7 T5 Z4 X) I+ i! T
<name>dfs.namenode.full.block.report.lease.length.ms</name>4 ~8 I; z6 K) J& {& O8 R3 F
<value>300000</value>) D+ A D: y+ g% i
<description>
0 s6 f L- j* ?5 H The number of milliseconds that the NameNode will wait before invalidating( i$ d% s3 D* b' v' ]
a full block report lease. This prevents a crashed DataNode from
" S4 v% O* P& ^, _3 ?6 T | permanently using up a full block report lease.4 g0 Q/ o2 z3 }. n' V" |& X
</description>& s2 T7 W; W1 P4 @6 B' i: t
</property>
8 u, E q* X, B2 _) v, Z: C<property>
! Q3 b/ h8 `$ r) o <name>dfs.datanode.directoryscan.interval</name>6 r7 l) I) ~5 e+ v6 _
<value>21600s</value>/ `* Z- D3 m, N8 p
<description>Interval in seconds for Datanode to scan data directories and" A7 r1 X4 B- Y3 `/ D
reconcile the difference between blocks in memory and on the disk.
0 S% m$ r5 m$ _ Support multiple time unit suffix(case insensitive), as described
4 Y) R; J5 S1 f3 D- O* q in dfs.heartbeat.interval.* g3 \( @5 S3 R& N3 o4 i
</description>
' n( H$ V) }- k C0 ^/ I/ T</property>6 z5 X8 P# A' P
<property>
" m. Q/ @6 d Y/ p( H; j <name>dfs.datanode.directoryscan.threads</name>3 C+ @7 V: g3 g1 o' f
<value>1</value>
3 I5 k$ f+ B+ a( _" S1 l <description>How many threads should the threadpool used to compile reports
5 E$ ]6 t3 _' Y! f for volumes in parallel have.+ w$ v# |/ ^$ O1 A8 M S
</description>) a4 s" y$ \7 I T. {1 H8 u$ f
</property>
1 N* |" l( D% S<property>
, e/ L; u& B( y <name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>8 j! ?* m1 m( \- m
<value>1000</value>+ U$ ~5 [- X6 H, t% d d
<description>The report compilation threads are limited to only running for
& z7 _) r c) G a given number of milliseconds per second, as configured by the9 U/ s9 W- m; |( g* Q
property. The limit is taken per thread, not in aggregate, e.g. setting6 E) E' p/ g7 x% C( U1 Q3 S# H
a limit of 100ms for 4 compiler threads will result in each thread being7 ~7 J5 `, s2 ?) F+ x8 i: Y. N
limited to 100ms, not 25ms., l. S! o2 s; d2 o G$ c
Note that the throttle does not interrupt the report compiler threads, so the0 N4 J: R! `* W6 \8 b8 Y
actual running time of the threads per second will typically be somewhat
$ H8 X- _5 H0 ~' [% n3 r higher than the throttle limit, usually by no more than 20%.
4 ~4 H3 S0 _: E" K Setting this limit to 1000 disables compiler thread throttling. Only6 J6 V+ b& h8 c( n; J
values between 1 and 1000 are valid. Setting an invalid value will result
8 Y0 Y2 \+ M. R& t in the throttle being disabled and an error message being logged. 1000 is
* S+ j7 N8 G5 g2 m the default setting.
# ^: m( f, M2 Z1 A+ K' Z </description>
9 l6 D) z9 ~& x</property>
! b8 `3 N: M E# _* j<property>
. X, ^/ u q/ G' q6 X" E' } <name>dfs.heartbeat.interval</name>
. k$ X9 b. ]6 j: k8 @ <value>3s</value>
0 `. B' y) H. t <description>
! y8 H4 R, \2 c9 E" b( V* W+ ~" B) r Determines datanode heartbeat interval in seconds.
8 U0 H' k' X0 T) L ~* |& R: v Can use the following suffix (case insensitive):; U4 h' B4 Z5 V
ms(millis), s(sec), m(min), h(hour), d(day); f( {8 }9 L; E1 p2 s7 N a
to specify the time (such as 2s, 2m, 1h, etc.).
' |, T8 p% q# X+ ?5 b/ _ Or provide complete number in seconds (such as 30 for 30 seconds).5 x! P; o# c3 ?9 D3 {
</description>
- V% R0 G0 Y* R/ R) T. A2 w, u9 y</property>
% L1 t1 n g k: B3 U4 J* V<property>7 l- R( ]8 }7 c
<name>dfs.datanode.lifeline.interval.seconds</name>
- U* i* X9 b" Z: k, w <value></value>( Z$ g! d3 ]2 a& P/ F
<description>
9 [% }* g. q/ L$ h Sets the interval in seconds between sending DataNode Lifeline Protocol
, Y2 s1 P2 `8 ~5 i9 `) l( q messages from the DataNode to the NameNode. The value must be greater than
* Z( H Z- @) D the value of dfs.heartbeat.interval. If this property is not defined, then
5 x* T' V6 g/ Z& L5 {1 n, K the default behavior is to calculate the interval as 3x the value of
' Y, ^! f( ~4 |, l+ r3 g1 U _ dfs.heartbeat.interval. Note that normal heartbeat processing may cause the
0 X2 b Z2 U6 ]6 e DataNode to postpone sending lifeline messages if they are not required.
& s7 @0 q5 C* f4 s Under normal operations with speedy heartbeat processing, it is possible
4 Z( v. N( X$ r& y* U& I that no lifeline messages will need to be sent at all. This property has no
- k- g3 c: D, ^ effect if dfs.namenode.lifeline.rpc-address is not defined.% M8 E Z+ y; J% X
</description>
# F+ C2 b. G, ^" c</property>
% b' e' J c3 r<property>' `8 ~* I7 d( m0 |6 Z
<name>dfs.namenode.handler.count</name>& B( X9 ]2 O7 S! `4 a% i6 `& f
<value>10</value>
( K2 B/ `* D y+ n2 I: Z5 O1 Q% P <description>The number of Namenode RPC server threads that listen to+ R# R$ H3 a) L c
requests from clients.1 n6 I1 |2 a0 ?( j# q- a
If dfs.namenode.servicerpc-address is not configured then
8 A1 q8 I) G5 ]. B# F0 y" a- K Namenode RPC server threads listen to requests from all nodes.
% w/ r- j/ w# }/ }5 B2 H </description>1 ^6 J+ [: s5 \# D
</property>2 a; Z1 L2 b& K- M( M% q! p
<property>. f+ T. C1 X$ _1 I0 S/ o/ I
<name>dfs.namenode.service.handler.count</name>
2 e8 p' e r j: } z# r7 T+ f0 Z <value>10</value>
6 ~( p9 T% j0 T, L% } <description>The number of Namenode RPC server threads that listen to
8 ?- ]% G# V! O R* F( \; l requests from DataNodes and from all other non-client nodes.+ _& X! V9 ]* p
dfs.namenode.service.handler.count will be valid only if
. C' U* _% i) P dfs.namenode.servicerpc-address is configured.
/ M0 L, L' r6 W3 | </description>9 s- t* {5 m1 g. T3 M; g: a
</property>
# ? R: y0 P7 ~6 @8 l) P<property>$ W& X9 K) o: w* {
<name>dfs.namenode.lifeline.handler.ratio</name>
+ f) H* N* Z9 D1 P <value>0.10</value>* j1 V* l: y1 s& u7 {# R7 K
<description>
; |1 t' C; r6 P. S/ m( ~ A ratio applied to the value of dfs.namenode.handler.count, which then
2 @6 ^. P7 L. j) \ provides the number of RPC server threads the NameNode runs for handling the5 i0 l2 k: x% k8 y' _& r4 M" f- [
lifeline RPC server. For example, if dfs.namenode.handler.count is 100, and" w0 ^! F: ?9 W; r
dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts( l% t) t8 P2 X# @; S
100 * 0.10 = 10 threads for handling the lifeline RPC server. It is common
) K* l' ]4 O d7 n: U to tune the value of dfs.namenode.handler.count as a function of the number' V5 i$ s4 h2 e" s+ g- r- b1 z+ ?' {; c
of DataNodes in a cluster. Using this property allows for the lifeline RPC {% [1 k' w8 [0 N; A; h/ d
server handler threads to be tuned automatically without needing to touch a
7 U* E9 P) K+ \1 P* D separate property. Lifeline message processing is lightweight, so it is
# E; W$ l7 N3 }6 c3 B/ `7 b expected to require many fewer threads than the main NameNode RPC server.* Z% R' k9 }; b. [
This property is not used if dfs.namenode.lifeline.handler.count is defined,
* A- N, w) d& E1 L- K/ I which sets an absolute thread count. This property has no effect if
/ k8 d( z, {: D7 t- D# k dfs.namenode.lifeline.rpc-address is not defined.4 e0 ]$ b& x5 R, |) G! _' Q
</description>6 d; P) b4 l0 v8 U. O& f
</property>
- `8 h; k7 t/ @3 t" V) {1 d- A<property>
. [9 v; g6 g6 k% H <name>dfs.namenode.lifeline.handler.count</name>
" R+ g' j3 D$ F. H4 O3 c8 X: O) b <value></value>
8 {. E5 |5 C" `+ z <description>- A/ E# o- s* {/ i% m; g3 _8 L
Sets an absolute number of RPC server threads the NameNode runs for handling
/ b! D D1 Z5 n2 P the DataNode Lifeline Protocol and HA health check requests from ZKFC. If
$ c9 }7 L. c0 v* S: X: Z" H+ o/ L this property is defined, then it overrides the behavior of, X( T, Q2 Q L) A5 F+ j4 d3 F5 I o
dfs.namenode.lifeline.handler.ratio. By default, it is not defined. This# o! e9 }2 J0 B8 ]! H4 H8 x
property has no effect if dfs.namenode.lifeline.rpc-address is not defined.- W0 j2 ~# R4 ^
</description>
! Y' H/ V6 Z* i ]</property>
$ f6 D4 S4 N( |+ N<property>8 I C( p! r- P" H+ R* K. z9 m
<name>dfs.namenode.safemode.threshold-pct</name>
0 U/ [) F1 Y2 }% {- u' ] <value>0.999f</value>
7 i1 ?6 y F7 T5 t5 _ <description>
& E" Y- j: W1 n# p6 U Specifies the percentage of blocks that should satisfy 7 C E* `: h% C P$ C
the minimal replication requirement defined by dfs.namenode.replication.min.8 i7 w7 S1 Z0 n C0 F
Values less than or equal to 0 mean not to wait for any particular
6 l$ t5 d+ C' w; [0 o( x. q percentage of blocks before exiting safemode.: L8 V6 @$ y% d5 @+ d$ p- c' L
Values greater than 1 will make safe mode permanent.3 @% U2 m, V* z, ^
</description>2 j% n2 e6 U' r# s" k! R( u
</property>3 \! M: A$ U7 s! } L9 F" ^
<property>
$ R- \! K" B3 Y. K( r <name>dfs.namenode.safemode.min.datanodes</name>
9 n, E3 L4 n+ {( \9 t <value>0</value>: u8 D: S4 c; \& b- o& ^% o
<description>
3 H6 E- d9 d8 _ Specifies the number of datanodes that must be considered alive
6 I$ s. I8 H2 |+ r. f before the name node exits safemode.
z r n0 N; u) ?" f S2 h Values less than or equal to 0 mean not to take the number of live5 j0 E3 e% ~3 y# J" Z7 |
datanodes into account when deciding whether to remain in safe mode
9 M- G5 H0 ]' y; b w6 [+ L during startup.
2 V% M2 y. x2 i X% W Values greater than the number of datanodes in the cluster
, M8 I# a* u" p7 z# } will make safe mode permanent.2 A/ e: w1 m( S% m! e" ^
</description>( a% h- t& p9 p; m8 l7 G& p' }
</property>- N% t7 [9 o0 W) ^" Q/ n
<property>
, g' K7 j3 o# D% z. B' C* a& e6 y <name>dfs.namenode.safemode.extension</name>
; S6 ]1 x2 c" W <value>30000</value>1 |2 W' v3 R* {7 K1 p+ Y/ J! Q( I, g V/ K
<description>
( q% V9 g% O; I2 m' Q0 A0 m- [ Determines extension of safe mode in milliseconds after the threshold level" T) C( e" H8 p$ V- I
is reached. Support multiple time unit suffix (case insensitive), as
0 U6 W; J2 X! ` b1 y) ]* { described in dfs.heartbeat.interval., d ]& f* d( s6 t) l
</description>- ~# l4 v! b; S0 g! `8 [7 G
</property>2 b# v r0 J) z4 _6 q) b. a$ _" H# o
<property>( Z/ s) `! n* W3 W, C1 K- v! A
<name>dfs.namenode.resource.check.interval</name>
+ C* c" E! C7 p% A* D <value>5000</value>* [0 m5 d2 ^4 J
<description>! [+ x$ v4 F. ? F8 ^( x9 t
The interval in milliseconds at which the NameNode resource checker runs.
8 ]& A i& M) R0 F" L, b, Y The checker calculates the number of the NameNode storage volumes whose
2 x& u; o# b( J, o" p+ ~9 g: z available spaces are more than dfs.namenode.resource.du.reserved, and
- l2 ^; r) |9 W, M& g* }; ]" r enters safemode if the number becomes lower than the minimum value2 P( o+ b: R( i
specified by dfs.namenode.resource.checked.volumes.minimum.
' M9 w" i( @/ k$ b" P </description>
' d% C: Z, D( B" Z</property>
# ]& R, F \% R" e# O5 e5 @" K- f<property>
5 f- z# n" c3 W# ^0 Z <name>dfs.namenode.resource.du.reserved</name>
1 ]; h x9 o5 I* _4 b' i4 K; M <value>104857600</value>4 _9 s6 N& J2 t c/ ?* X' G$ @7 a
<description>
; h2 Q( ]7 t0 \5 A$ j The amount of space to reserve/require for a NameNode storage directory6 l) V7 d c- k' x
in bytes. The default is 100MB.
6 v7 l& ?$ u7 L: Y! d* S5 b/ i. c' N </description>; z6 }8 |# H! \
</property>
2 { u* {! X4 M [$ C; D+ S<property>
" t: Z& ]( O {' V <name>dfs.namenode.resource.checked.volumes</name>. r( ^4 V) U# _
<value></value>
. s: J9 f8 ]+ j <description>' b5 W; m0 ?; v- Q# s# s# x* u0 i
A list of local directories for the NameNode resource checker to check in
$ ]3 Q1 B, P: X% G" W' J addition to the local edits directories.7 B! o$ H! ]/ ~$ X- L
</description>
% _! V$ u" Q4 _) c- A</property>
9 X7 ?" ~: j; C<property>
. ?$ T5 M# X4 z# K& k2 U <name>dfs.namenode.resource.checked.volumes.minimum</name>
" I" n; h8 ~3 u& @1 x3 S1 e# [ <value>1</value>
' J# k- X* H# i <description>
. q6 i. Z V0 `" `/ i/ N& U4 u! u The minimum number of redundant NameNode storage volumes required.+ J5 m( X6 H) @4 @+ e
</description>1 z* y- \& J6 o2 l' c7 F5 d
</property>; B. i- k+ J# i( o
<property>9 }0 o+ K/ c! P6 T/ L, {$ J0 ?/ w3 M
<name>dfs.datanode.balance.bandwidthPerSec</name>* ?9 s# E0 Q1 b; c/ m2 f
<value>10m</value>) e7 x9 |9 ~! x) R! C4 F
<description>6 _0 t" S* ]; a- n' g! Y7 A |/ E
Specifies the maximum amount of bandwidth that each datanode
4 G7 M( F3 V) g/ t; b0 [% z can utilize for the balancing purpose in term of u* v) f: Q5 [0 V9 i5 H1 K5 l
the number of bytes per second. You can use the following
+ M6 B* o0 v# Z6 O suffix (case insensitive):" ~. k+ P7 E( a4 B! S
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size: q0 V+ w6 V9 W% x
(such as 128k, 512m, 1g, etc.).
, N4 W6 W6 c6 n0 p& Q* C+ T Or provide complete size in bytes (such as 134217728 for 128 MB)." h8 Q0 c9 l/ F" e" J. t) P! Q
</description>
7 B7 S* a' p: k0 U& o3 N9 A</property>
5 K/ e- ]+ \3 @; ~4 f<property>( J# t9 J9 K. m) a( O* O
<name>dfs.hosts</name>2 ~$ m7 N/ K1 B6 n( G! H
<value></value>
& r1 D1 T* V; g/ T; F <description>Names a file that contains a list of hosts that are; h; ?. _) i0 n( O. X# n \
permitted to connect to the namenode. The full pathname of the file
! ?1 p9 L% m# |: g; {& c6 a must be specified. If the value is empty, all hosts are: L* Q" r& i% e( a/ L4 l4 q9 T
permitted.</description>) G+ M' M" U% C, K+ t+ h
</property># a& _- C/ m- @3 X: N9 e S1 F( K
<property>
4 |% W: B# |3 j0 C( G6 t; [& _3 x- P <name>dfs.hosts.exclude</name>7 M' Y- R( @2 U% |' x% m
<value></value>
: c: y- I4 D; ` <description>Names a file that contains a list of hosts that are8 z9 y" L" v$ g% a( t5 W3 z3 e
not permitted to connect to the namenode. The full pathname of the
/ H$ s3 q& W" y% E file must be specified. If the value is empty, no hosts are
7 y( [6 r8 z5 E* U5 p excluded.</description>
( I* p' B* S1 v* g</property>
1 J7 x/ |# S. w<property>
7 B. {: Q- a E5 L8 b/ i <name>dfs.namenode.max.objects</name>1 v4 d+ G0 ^. ^0 H4 K0 x
<value>0</value>" L; R& L6 f$ [
<description>The maximum number of files, directories and blocks$ |' [* u" R+ w6 x8 \6 O
dfs supports. A value of zero indicates no limit to the number
; q- { A6 A1 R5 R+ p% c, ~ of objects that dfs supports.9 \6 O6 [7 X# L& [1 [# R
</description>
4 E4 \! Y) M5 l8 R1 {% a" a</property>7 d S! M" B" U
<property>2 ?2 k1 T1 a' ]' e
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
+ i7 f8 D' _# {2 ?! Z* ?: W2 d" o* @ <value>true</value>
' O1 v' S7 Y- N# C0 A* s( p& S) J <description>
- ?& O" K$ L) `) i" v5 }# u If true (the default), then the namenode requires that a connecting
7 E2 H3 S* X% @2 R; y- R datanode's address must be resolved to a hostname. If necessary, a reverse1 Z3 B3 n3 Z. z M
DNS lookup is performed. All attempts to register a datanode from an
3 w1 q! Y. K6 A% L1 j) x' ] unresolvable address are rejected.8 N7 z% B" E$ Q' ^- x
It is recommended that this setting be left on to prevent accidental
8 G4 [, Y2 f4 s. ~1 T( G registration of datanodes listed by hostname in the excludes file during a
: ?* m: g( ]7 |* m9 D) d0 H, r DNS outage. Only set this to false in environments where there is no
# z6 a! h' i/ D) s1 }2 v$ J infrastructure to support reverse DNS lookup.' T' P! [- \/ L2 q! A: S
</description>
3 y, X( Z3 N: l s</property>
8 Q" e/ `$ v5 V ?$ N<property>3 `7 V2 v+ a _0 ]; a) U, ~
<name>dfs.namenode.decommission.interval</name>! {% E1 m4 K( r5 E0 H2 X, `4 d$ a
<value>30s</value>
7 S# a% u& c. _ <description>Namenode periodicity in seconds to check if$ N, b+ h T/ {9 D7 n$ y9 \/ S T8 v
decommission or maintenance is complete. Support multiple time unit" m% J5 i, R/ @+ }- D
suffix(case insensitive), as described in dfs.heartbeat.interval.# `# }0 J* n8 h1 F3 I3 a
</description>
3 L% r2 f0 @& r7 ^, \/ z; A& ], k</property>" ^9 B& T9 N; i8 E: I
<property>
' I1 O/ m4 [3 @# _# g5 @ <name>dfs.namenode.decommission.blocks.per.interval</name>) X" D5 h: r: f& ]9 @
<value>500000</value> Q) G2 J/ Q4 c# P
<description>The approximate number of blocks to process per decommission8 q8 {5 _4 S" R+ i; [/ R
or maintenance interval, as defined in dfs.namenode.decommission.interval.1 n. R0 {/ U% w: V6 Q. w3 E
</description>
( P" Y4 y3 E2 s# R0 `- F8 p/ h</property>% G, ` p) h+ l, a) B
<property>
( L! F* g- l$ ^. ~ <name>dfs.namenode.decommission.max.concurrent.tracked.nodes</name>
7 T- I/ R R _! j( E' b" ] <value>100</value>
4 A# T6 D2 `8 X6 c1 ^2 _ <description># u) ^. u/ Q# \" i- ^4 w. M2 |
The maximum number of decommission-in-progress or) u# E2 I7 O. H) i6 h
entering-maintenance datanodes nodes that will be tracked at one time by2 i5 _- p% U; J1 p
the namenode. Tracking these datanode consumes additional NN memory" ~+ R1 J. ?/ w7 S. F) K
proportional to the number of blocks on the datnode. Having a conservative
+ O& u% a, Q. N, @" V limit reduces the potential impact of decommissioning or maintenance of
) ^. N1 i" q4 | a large number of nodes at once.
1 z" N5 f) g1 K& U6 J A value of 0 means no limit will be enforced.
, h! {1 q2 @" M& _, h; f </description>; U6 L" }, ~+ [; O( W
</property>
0 g* W1 @2 X" ?. e, m) V7 b1 Q1 g1 O<property>/ k2 y; r; a+ Y
<name>dfs.namenode.redundancy.interval.seconds</name>
8 W4 P( e; p9 |4 g. M <value>3s</value>8 O3 g+ }0 u. H* z
<description>The periodicity in seconds with which the namenode computes
1 x0 n% W' ?: y low redundancy work for datanodes. Support multiple time unit suffix(case insensitive),1 `* R% T1 v1 Y# x' Z
as described in dfs.heartbeat.interval.' @9 T# a! {) O
</description>" C; d; W+ O5 \
</property>5 }* |) f* i8 K9 U7 r
<property>
5 g8 l. U3 m/ D <name>dfs.namenode.accesstime.precision</name>
* }8 p" t1 t X <value>3600000</value>
: e& j$ R+ [$ \4 m G1 q <description>The access time for HDFS file is precise upto this value.
6 Q% P9 ?0 d" @2 ?8 [% T The default value is 1 hour. Setting a value of 0 disables: l. @/ G2 r% q
access times for HDFS. f1 b/ O1 M$ I5 ~- e
</description>
& C+ J$ ]+ e) I</property>! x$ ~ g: _6 g& _9 q+ P9 _, c
<property>& E; m+ i' Z# f! i- s: h8 E7 ^
<name>dfs.datanode.plugins</name>
: r% w8 {% q) P! F, Y <value></value>/ k- c4 D" a7 J/ L! ^$ d( S7 x
<description>Comma-separated list of datanode plug-ins to be activated.
& Y! u" d+ y% A( R, D4 H8 O </description>
+ M5 ?+ _% ]9 Q! i" |% I% c</property>
* ^. d2 o1 I( }<property>. c H3 V0 @# M
<name>dfs.namenode.plugins</name>- {- x+ o6 U. T- w! H, F
<value></value>! z) Y! W: H' w7 v
<description>Comma-separated list of namenode plug-ins to be activated.8 `: q- R. N: E7 f1 q+ H
</description>
0 }; W$ ~0 o2 |# b3 |$ F3 h9 L</property>
0 ~% V% a2 @" O S<property>8 d# w9 w" E9 T% f
<name>dfs.namenode.block-placement-policy.default.prefer-local-node</name>
8 W3 J, b1 D' T" K) J/ w% M) v <value>true</value>
, G4 O W7 B4 G' F, [; s <description>Controls how the default block placement policy places3 F: t$ ?5 M2 H) Q' n) Y
the first replica of a block. When true, it will prefer the node where7 ~1 o" T: G; |' p7 W4 A
the client is running. When false, it will prefer a node in the same rack
" K0 F% _. y9 a) O+ J) r k4 K0 } as the client. Setting to false avoids situations where entire copies of) ?) `$ ~1 I6 A$ [& w& `
large files end up on a single node, thus creating hotspots.7 Q& K# K6 g( y8 w9 r. e& V2 z" A
</description>8 H, Y) W- m) a# d8 f3 l
</property>
6 d7 ]) V% ~5 E7 C. G }; h<property>
" @- m6 C- {& N <name>dfs.stream-buffer-size</name>6 [+ G$ i& O! H0 M6 C) W- U4 i
<value>4096</value>
0 E; I+ q4 W8 M; _4 D <description>The size of buffer to stream files.
/ P! U6 i. |* O, [" q4 ^9 |9 U The size of this buffer should probably be a multiple of hardware
6 E7 }9 C* Y) p, Z page size (4096 on Intel x86), and it determines how much data is
4 @: ?* C! Z, g3 s buffered during read and write operations.</description>9 `$ _1 [* \& m, N* W; P
</property>) C( J+ b! z( d+ O( a- r% o# M3 ^
<property>
/ N7 C- q# y- B. ^ <name>dfs.bytes-per-checksum</name>
3 i [$ Z( L/ |* @, y <value>512</value>4 i' ^( h- `8 T" B2 g0 z
<description>The number of bytes per checksum. Must not be larger than& X+ u: w) w0 H7 h3 ]
dfs.stream-buffer-size</description>! O8 I8 A4 q7 ]/ R7 C7 u
</property>7 Q; V2 _- T% J* J. q R3 w# x* s" b
<property> D9 p @$ b3 N8 x- N# D! [& [3 G6 O
<name>dfs.client-write-packet-size</name>
' x/ R* ]* W3 N. @ <value>65536</value>
: i0 c$ K5 U6 K& o <description>Packet size for clients to write</description>8 |6 m$ j- J: _2 A" x' [4 m L
</property>$ x( X8 b/ h" M2 c( k5 ?2 N3 t9 h
<property>) g3 R; z, `6 b$ u6 K
<name>dfs.client.write.exclude.nodes.cache.expiry.interval.millis</name>) ^$ b ?0 ~& M+ ^1 ?" A/ Y; g+ l
<value>600000</value>
M" e. E" H$ v3 G+ x <description>The maximum period to keep a DN in the excluded nodes list, l- l d+ y' a& b4 x V
at a client. After this period, in milliseconds, the previously excluded node(s) will
( G& }6 ?1 q7 ~9 d: k* W% ?( O be removed automatically from the cache and will be considered good for block allocations8 e4 R0 {6 m, E/ Q" \% B/ N/ T- b1 T
again. Useful to lower or raise in situations where you keep a file open for very long6 U" B ^4 J, H! {. V% |
periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance Z1 p7 e5 t0 K' N% G$ b! I
restarts. Defaults to 10 minutes.</description>
5 S- [/ ?( _7 X</property>
' @& n! [% Y! G4 n5 x6 @<property>2 q6 g2 O. {* U6 k- r' |, Y0 @
<name>dfs.namenode.checkpoint.dir</name>9 F* x+ R$ F& _
<value>file://${hadoop.tmp.dir}/dfs/namesecondary</value>
2 J% `0 \' x5 i: \ <description>Determines where on the local filesystem the DFS secondary7 Y8 ^7 l# L" n+ w+ v
name node should store the temporary images to merge.
- N( D' Q& ?4 p/ a6 u If this is a comma-delimited list of directories then the image is
, i1 b# _5 v; F( F. m$ i replicated in all of the directories for redundancy.
: w2 g6 k- f+ P6 `3 Z </description>0 z4 E. H/ N9 V5 A( j8 ^8 [" v
</property>
" D! r0 e9 }, J<property>- {. i5 i, ]8 o4 v3 r7 y) O# Y% W
<name>dfs.namenode.checkpoint.edits.dir</name>' N+ _/ {) L, Q8 s g
<value>${dfs.namenode.checkpoint.dir}</value>
9 z8 ^0 w9 U; d% l9 H6 ^, i <description>Determines where on the local filesystem the DFS secondary
* C) @/ t, C% E+ K% H! _ name node should store the temporary edits to merge.
7 O; q6 B1 ~8 u. N If this is a comma-delimited list of directories then the edits is" k# q) [0 {9 g. t; x3 [
replicated in all of the directories for redundancy.
$ T- `4 A) I8 i5 C7 i0 ^% [ Default value is same as dfs.namenode.checkpoint.dir
4 v+ }4 U8 v) W </description>$ d) c7 C0 p( V
</property>
7 [" R/ Z( |! O% a0 J<property>; \ u5 E+ g# ~: g0 ^
<name>dfs.namenode.checkpoint.period</name>1 t- g+ a4 [" t! \" T
<value>3600s</value># i+ I5 J l/ l& M' n/ r/ @
<description>: S w: P/ L, D* e q3 Y2 v V
The number of seconds between two periodic checkpoints.
2 g; G. q* \ ?2 d; N; D# @& i Support multiple time unit suffix(case insensitive), as described
- W8 w j( x' S+ l$ T in dfs.heartbeat.interval.) W5 V& M, U# j, z
</description>
{- Q- x: u& x$ z$ H6 r) V</property>+ r: x1 x- Q7 C# {* Y
<property>
2 v- i7 W/ w; Q <name>dfs.namenode.checkpoint.txns</name>' e9 D* b, ]5 _1 w
<value>1000000</value>
- T3 }# V8 J( u9 l2 C0 s# X0 H <description>The Secondary NameNode or CheckpointNode will create a checkpoint4 ]4 S+ L* ?; X8 i" N
of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless: O$ Y5 R9 Y' N b; v9 e; I! h
of whether 'dfs.namenode.checkpoint.period' has expired.9 Q6 Y4 T, F K: n& y2 O
</description>
* m+ k; s0 L- @# _0 q1 S8 L" d% s</property>
' p, |" e9 r9 ]% {1 u<property>+ k! S( e/ w$ z- W" m
<name>dfs.namenode.checkpoint.check.period</name>
- e& q$ g' S- y2 B <value>60s</value>
: o# e/ Q# E5 R+ {7 e4 m( z5 w% _8 N7 v <description>The SecondaryNameNode and CheckpointNode will poll the NameNode+ o* p, k1 X* H5 M/ G
every 'dfs.namenode.checkpoint.check.period' seconds to query the number
, o3 f& T8 w0 A/ D) f5 f of uncheckpointed transactions. Support multiple time unit suffix(case insensitive),& b' p, B' `$ D- t6 ?/ D
as described in dfs.heartbeat.interval.( y( j8 j4 m/ t. x, P. G
</description>
4 H' {! h a& w5 v) T; D</property>0 q9 k2 g* v+ b C$ Q- e- G; p
<property>4 y5 @, K! C2 I$ t9 D: b
<name>dfs.namenode.checkpoint.max-retries</name>
# w, ^( g: E7 k5 q8 P: y& X <value>3</value>2 |9 X3 N3 I6 i# N( Y
<description>The SecondaryNameNode retries failed checkpointing. If the , u' d8 s A3 l% J8 z" p9 z2 o1 C
failure occurs while loading fsimage or replaying edits, the number of) e. e2 ~2 r; e6 n, e; ^& V
retries is limited by this variable.
/ u- E: [, }% F( R </description>' e. K( k" p3 p9 l% Q8 j% l
</property>3 y5 E) A8 Y- ?
<property>
+ {+ P- G! `0 v+ u) r4 O' v6 k7 l <name>dfs.namenode.checkpoint.check.quiet-multiplier</name>' m8 Q; z& j e% T2 _1 `7 L8 U* M- M4 ^
<value>1.5</value>
0 V, M2 d) a1 n5 {+ o7 m$ i <description>1 b7 e. z! l+ k3 ?0 T+ }- P
Used to calculate the amount of time between retries when in the 'quiet' period7 M6 F, G" i/ v! D( ^6 K
for creating checkpoints (active namenode already has an up-to-date image from another! i: b. p c, t/ s$ M
checkpointer), so we wait a multiplier of the dfs.namenode.checkpoint.check.period before
0 n. O' a& c1 p/ |2 d9 C retrying the checkpoint because another node likely is already managing the checkpoints,
0 t M5 e$ f0 L, Y( C: H0 v allowing us to save bandwidth to transfer checkpoints that don't need to be used./ q: ?- C6 x3 n) U1 C; q9 W
</description>/ f7 H# d/ [4 z7 R0 m
</property>+ i* S8 }3 a: s" B0 Q! n
<property>
" ]; v8 y4 p6 q7 r4 ]$ k5 U! C <name>dfs.namenode.num.checkpoints.retained</name>
8 o) W3 j1 w/ N3 C+ `2 Y' x <value>2</value>% j; L6 J) ~7 p" N3 u
<description>The number of image checkpoint files (fsimage_*) that will be retained by
2 z! s8 x; a! ` the NameNode and Secondary NameNode in their storage directories. All edit5 d' Q J+ g# F; ~, D. O
logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
. q; z* p- |# n' t) b checkpoint will also be retained.& c$ t2 ]' Y/ _% n) C4 n7 h1 H
</description>% B0 s" o- Z6 H* [+ s' X
</property>. ?% M6 D5 Q3 A9 `6 J' I% x. e
<property>- l7 h( X9 \9 m) ~' V
<name>dfs.namenode.num.extra.edits.retained</name>
) \! E I! m( l0 R% M1 U8 ~ <value>1000000</value>
1 K& y2 B! I q' R2 K, X. M; | <description>The number of extra transactions which should be retained5 J; j1 Q$ z" X Z
beyond what is minimally necessary for a NN restart.
4 v' [. _& p- f% J- j- E It does not translate directly to file's age, or the number of files kept,
. ]( \3 }+ a4 E0 f h but to the number of transactions (here "edits" means transactions).
; s' \' D3 _) i& q0 K9 ~ One edit file may contain several transactions (edits).
- p" ?* u `# [# p+ c During checkpoint, NameNode will identify the total number of edits to retain as extra by2 T5 f& r c- @0 @
checking the latest checkpoint transaction value, subtracted by the value of this property.
j Z9 N, J8 k2 w# ]) r+ c( z Then, it scans edits files to identify the older ones that don't include the computed range of1 a) t" ~2 U$ p: M! H
retained transactions that are to be kept around, and purges them subsequently.
/ D0 g* P ?0 T) D( c The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have
# L5 W$ O" N# s% p6 v# Q! ? been offline for some time and need to have a longer backlog of retained
; y& O6 g% X6 ]2 @" p edits in order to start again.0 l& }& _& w0 |) t3 U2 \' {
Typically each edit is on the order of a few hundred bytes, so the default
7 }, h! l( Q+ {# C& U1 H of 1 million edits should be on the order of hundreds of MBs or low GBs.
7 ?+ u0 S& K# w6 \7 z2 }. C% ~ NOTE: Fewer extra edits may be retained than value specified for this setting/ [/ S( R3 m$ R2 Z2 n0 Z+ U
if doing so would mean that more segments would be retained than the number5 j' ~5 @: z. V, g8 E+ S
configured by dfs.namenode.max.extra.edits.segments.retained.5 ?: C" {" Y! s$ z
</description>7 H3 P3 v) c0 R' x _
</property>
+ ~& a2 c5 l5 h! r<property>; o3 k* R: _! t1 Z
<name>dfs.namenode.max.extra.edits.segments.retained</name>
d( v' q$ v/ V6 x6 Z <value>10000</value>+ T7 ?" I1 {0 g9 q% g) T! s. a; x
<description>The maximum number of extra edit log segments which should be retained
: \/ b& z8 U" s c- Z beyond what is minimally necessary for a NN restart. When used in conjunction with/ N9 y% ?. n; \: p
dfs.namenode.num.extra.edits.retained, this configuration property serves to cap
" X, N0 Z# W7 y% s# k6 _$ C the number of extra edits files to a reasonable value.
4 p4 G/ n8 W/ S R3 a% m </description>" ?: r; R8 h9 H* j. y: ?
</property>
$ w& i* d$ l% h6 B# V<property>. d- L* s" L9 ^1 ~! f
<name>dfs.namenode.delegation.key.update-interval</name>' C- j6 G9 U8 m& g( l
<value>86400000</value>
; f. x( Y- Z/ d2 `2 b <description>The update interval for master key for delegation tokens & e! z9 q2 ?: J
in the namenode in milliseconds.
8 h1 q% G# h$ g+ a* T2 D# S </description>
% W1 v( `. \6 } ~3 Z* v</property>
l+ O8 E- L( F9 L<property>
0 ]1 i2 v9 E8 ]5 \/ X <name>dfs.namenode.delegation.token.max-lifetime</name>
. m3 x: a9 ^- c2 \! q1 H( h: q! I <value>604800000</value>, ]: f" y6 [+ \
<description>The maximum lifetime in milliseconds for which a delegation
2 Q p9 N/ _ ]( k' v/ c token is valid.
5 n6 o5 t( \8 V# c0 L </description>9 w- z$ f; t7 p1 X0 _# G4 _3 j
</property>
0 ?# n; x# }! R9 S' x( d4 t& ^7 g1 H<property>
. g! N' g1 u5 a7 M3 P <name>dfs.namenode.delegation.token.renew-interval</name>* C* I& A" J! ~7 |9 e
<value>86400000</value>
w& u& X. D" |1 Q9 W- i <description>The renewal interval for delegation token in milliseconds.
( }) ^5 O0 g' \9 M/ Z) g) H4 e2 k </description>
, h+ b) t- l. t# ^</property>+ d3 D$ e$ |- i$ X X# d
<property>
, m3 ]9 _3 i% P+ \ <name>dfs.datanode.failed.volumes.tolerated</name>
* z t* q. y, ~& q <value>0</value>9 c1 ^! [7 D$ [- @
<description>The number of volumes that are allowed to C$ {5 c7 c6 C/ Q$ F
fail before a datanode stops offering service. By default
5 ]% h- d' [" @$ P) R) _ any volume failure will cause a datanode to shutdown.
6 E' a! x- E5 j1 R0 e! I </description>. \& f3 q: g; b: I$ c+ G6 o
</property>
0 a/ l/ [6 M+ l, _9 F<property>. i& t2 i6 K4 l5 s: A! z
<name>dfs.image.compress</name>8 }' G" H% X. i7 H" t" b/ e% H, b
<value>false</value>- ~1 k. a7 t2 u: H/ R
<description>Should the dfs image be compressed?
; J) a5 f5 X8 c </description>
- U, k1 ^" ~9 F</property>
0 H: t1 ^" Q! @5 n1 q6 K4 o' f<property>
; Y0 a" M3 S7 L: B; t* v, n9 T* D <name>dfs.image.compression.codec</name>
9 q. o& H |2 j; O; t* l% E <value>org.apache.hadoop.io.compress.DefaultCodec</value>
- n; N( K. {6 f$ ?: r. d1 e) L1 N" s <description>If the dfs image is compressed, how should they be compressed?$ B7 L. W0 W4 P, p6 J
This has to be a codec defined in io.compression.codecs.
' A0 }) [, v& s8 e. E </description>
+ u5 l* P8 m. k2 w2 z( z, F</property>
- }; r+ a8 S5 g j0 k# a7 g<property>" {, I9 }4 t/ F: w
<name>dfs.image.transfer.timeout</name>
: x) x. j2 T1 e( ?- G" t <value>60000</value>2 `7 O7 x) c5 u$ T3 H
<description>6 |$ h& K4 p* C' W! X7 n3 Q8 e/ d
Socket timeout for the HttpURLConnection instance used in the image
& p9 Y, o$ V3 y transfer. This is measured in milliseconds.& G6 t9 V5 X4 R" Q$ l" c" L. f& F
This timeout prevents client hangs if the connection is idle
; m% t5 \0 {% f1 `& N0 P+ ]& K! e for this configured timeout, during image transfer.2 E. N1 k! N; m! ?
</description># j0 V, u4 G s7 R
</property>& f0 G: E/ l7 u% l' h1 J8 Q; A5 ^9 w
<property>( N; c( h4 T3 i
<name>dfs.image.transfer.bandwidthPerSec</name>
& O4 I& F; F- p1 \. O+ c <value>0</value>
+ O- B. u( Z2 B$ s( g- W <description>! o* p2 m u, i9 b$ s8 @
Maximum bandwidth used for regular image transfers (instead of6 E i' f& X9 R2 E$ ^
bootstrapping the standby namenode), in bytes per second.* w4 a9 { f4 e) B: k
This can help keep normal namenode operations responsive during+ W( s! M/ N) B8 F1 ~
checkpointing.
, }4 r) a) Z: N. {- ^$ u* L H A default value of 0 indicates that throttling is disabled.+ n7 k4 L/ V7 J8 L
The maximum bandwidth used for bootstrapping standby namenode is. B7 K6 \2 n2 T5 Q
configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.& N2 Z& `8 Y% r4 R
</description>' K" D2 k+ [# c' {4 E
</property>5 A, e, B! K* Z3 i: s
<property>2 D- P" y8 b( w5 U6 J
<name>dfs.image.transfer-bootstrap-standby.bandwidthPerSec</name>
: i$ H2 N" V. |+ [" G$ v' u <value>0</value>
3 y) a. P! g& w7 j5 I4 J( u <description>
! y1 E7 T- I: D$ D: V; Q s Maximum bandwidth used for transferring image to bootstrap standby
% y! a" L) e; c1 r" f namenode, in bytes per second.$ x0 r7 ` i8 }; f7 K) t
A default value of 0 indicates that throttling is disabled. This default$ a+ F2 j C4 D0 x1 X6 s
value should be used in most cases, to ensure timely HA operations.: r- O6 e) ?/ w1 r
The maximum bandwidth used for regular image transfers is configured
' d, T' Y# w5 U with dfs.image.transfer.bandwidthPerSec.
0 a, Y+ N, s! e) ^* W </description>* j( ?4 \+ _" y5 P
</property>% P) x: ]1 C- S0 Z
<property>$ A; K2 q7 ]2 P; a
<name>dfs.image.transfer.chunksize</name>2 o8 S$ Y& l- ~/ ?
<value>65536</value>
8 t) K; q" i! z/ W <description>& `; n" r- Y3 x& K# B4 s
Chunksize in bytes to upload the checkpoint.8 L* r4 M+ S. d* Q z9 k
Chunked streaming is used to avoid internal buffering of contents+ W% D3 q8 Z9 S5 @1 C
of image file of huge size.
" x+ A, n3 Z4 v% @4 z! v, b6 c$ i </description>6 H/ t1 Y5 ?9 z# f) s
</property>
0 T7 A) s: @& {- L( \# M<property># L9 E3 I8 {. l/ d8 W; M
<name>dfs.edit.log.transfer.timeout</name>
9 ^2 E- q% s5 T0 P <value>30000</value>
; Z* }7 k; m- F. c% C# Y7 K <description>; `5 X& ?/ g: L2 r8 K2 I4 \
Socket timeout for edit log transfer in milliseconds. This timeout
; t; V: u# }* p: R' Y- o# U# K0 W should be configured such that normal edit log transfer for journal
* U% \* A/ I3 t# c4 E* f node syncing can complete successfully.5 a* ~! U3 d1 ]1 ?9 z1 u- a
</description>+ G7 h% k+ c* v3 x1 e% I; O g
</property>) u% i4 w' I$ u# Z; Z4 p) L
<property>! D* e& i. L Z: A7 P9 }. ]( n
<name>dfs.edit.log.transfer.bandwidthPerSec</name>
0 v+ \% x+ m" q- K* z/ w <value>0</value>
; A: U7 s3 d' ` K5 B" K: i# K2 g <description>
* L5 g* k( W$ H# z8 v Maximum bandwidth used for transferring edit log to between journal nodes c. F* O! n r" q. b2 i/ n" D
for syncing, in bytes per second.. ?- Q9 z2 T- y
A default value of 0 indicates that throttling is disabled.( Z0 R3 f2 x: j) r
</description>
& H/ Q/ F2 ~5 v# g</property>
4 F" c' |& _- j! E% {<property>! n& H; h0 t/ t
<name>dfs.namenode.support.allow.format</name>0 U+ _4 J4 r& q4 R8 a+ U
<value>true</value>
5 A- X2 Y' H4 \ <description>Does HDFS namenode allow itself to be formatted?1 M- w+ o( p! S a% V; K6 O
You may consider setting this to false for any production
8 z( n. y2 `# n2 g9 V# b7 N cluster, to avoid any possibility of formatting a running DFS.
# S M! w8 R, g b </description>) G) r" y8 L, m$ j i- E) c8 w/ ^& T
</property>9 p( U8 S- o& H$ _" @
<property>
8 O F0 k" v; k <name>dfs.datanode.max.transfer.threads</name>
+ v9 w2 g/ j3 T! P <value>4096</value>
. r" S' ~( j2 g! z <description>/ s. m# J0 y6 C% R! L2 \; j5 q8 h
Specifies the maximum number of threads to use for transferring data b6 @+ y4 W$ |
in and out of the DN.
) ?6 |1 {8 X4 W: S4 c7 R </description>
) D, W; @: c+ _! X, L</property>+ a6 B; I2 t' L/ h1 t1 C" D
<property>
0 d9 P: G) {7 r2 G& U! i- w! L5 t <name>dfs.datanode.scan.period.hours</name>
+ q' L) R/ G6 N# g+ F <value>504</value>
1 x+ D" k/ d4 t% P; k# K <description>3 t! @$ \4 y# e- K
If this is positive, the DataNode will not scan any
6 x4 Y! y: [+ o individual block more than once in the specified scan period.
& D; j! j, {6 `3 ]4 u/ w# r$ Z1 h/ O# ] If this is negative, the block scanner is disabled.
/ k7 l, M' `4 s q; x If this is set to zero, then the default value of 504 hours9 A+ C+ U1 c) d) M
or 3 weeks is used. Prior versions of HDFS incorrectly documented
w/ ~3 K2 c+ r* t( g6 P1 o that setting this key to zero will disable the block scanner.
% X; Z) |) I) s+ Q3 p </description>2 T! x. }7 ^+ [: q# M. v) K! [! i/ \: m
</property>9 k( R! b( `$ ^: F x9 D
<property>
% ]9 B d* ?1 x7 s/ R- q% H2 y <name>dfs.block.scanner.volume.bytes.per.second</name>0 v* t- t [$ a* q2 A
<value>1048576</value>
, f+ f. \$ [# N1 w5 q( P% O5 j# I <description>+ z, S1 a0 }* T; l, z
If this is 0, the DataNode's block scanner will be disabled. If this3 U$ @* v" ~. W9 e! B
is positive, this is the number of bytes per second that the DataNode's
2 W5 U" h4 ^4 l! N, U( f" v [ block scanner will try to scan from each volume.& f8 e4 |. N- k+ e% R4 ?( x6 X
</description>! [, f! f% r, H( Q$ s1 u% x
</property>+ E: z% O& _( y6 u# D0 |
<property>
! Z+ r* H- W' b9 U <name>dfs.datanode.readahead.bytes</name>
9 W" d$ p' A/ |0 X <value>4194304</value>8 U I; g- k9 C4 K5 L2 O
<description>
7 Q2 H) a! z. G7 w) l6 {- m8 ^3 U While reading block files, if the Hadoop native libraries are available,! e! [0 b# r+ @6 g9 S
the datanode can use the posix_fadvise system call to explicitly0 G0 D. t- _0 u0 V/ p
page data into the operating system buffer cache ahead of the current
1 D1 l. n; ^8 A1 G reader's position. This can improve performance especially when; j+ I( t2 b& |
disks are highly contended.
+ d6 D- _% V" ?4 E This configuration specifies the number of bytes ahead of the current
6 y+ }' v" x. b) d6 b; A read position which the datanode will attempt to read ahead. This. ]' s7 V# Q/ _3 N
feature may be disabled by configuring this property to 0.
; a, O! a/ H, P) a0 N% f1 L& D If the native libraries are not available, this configuration has no
1 e0 w6 n4 \9 p; q: ^* p# T effect.+ e C1 D! e) T9 q& u2 P$ q2 q( I T$ J
</description>* S: _. a+ Y, h8 b N
</property>; A$ B8 e# r5 y3 `7 L5 I
<property>/ u. z8 ^ f+ a; w& v, R- e: R
<name>dfs.datanode.drop.cache.behind.reads</name>
- M6 n7 m$ w! S1 h) E2 a <value>false</value>3 O8 L% x% i G- \6 n
<description>4 D' z( E$ l+ q) `; A
In some workloads, the data read from HDFS is known to be significantly5 E; y* u+ M$ U: Q/ q' l% U2 K
large enough that it is unlikely to be useful to cache it in the
1 j2 g7 s/ A2 |% ` P( J operating system buffer cache. In this case, the DataNode may be' |- Q* Y4 N. {1 O) _+ c* ]
configured to automatically purge all data from the buffer cache* j" i, t3 a( O. D! ~
after it is delivered to the client. This behavior is automatically: N* w' i5 w( v; M, n! D/ x
disabled for workloads which read only short sections of a block& K b+ J2 j+ {& \; d2 o* v
(e.g HBase random-IO workloads).) g' U+ ~2 _/ ~6 U2 Z% }
This may improve performance for some workloads by freeing buffer
" U# e- x& R% Z; p/ q; S& R cache space usage for more cacheable data.
, Y# c1 p$ z& a, q- t+ o; W6 ~. g3 M0 X If the Hadoop native libraries are not available, this configuration
# o' O% J' E0 F1 |1 h/ U has no effect.0 \) G4 P4 j) l$ X$ O' G
</description>0 a4 L: s# N& ^/ l. J4 Z/ N4 s3 A+ Q
</property>+ S2 [% `/ {1 o9 @1 z: S) E+ N
<property>" e8 O; g5 ?- I# C. d) K
<name>dfs.datanode.drop.cache.behind.writes</name>
2 n3 Y3 G6 Q% o4 Q/ H) j2 ` <value>false</value>
% _3 U; L. z1 s1 E6 ^, N9 H" x <description>
$ t( C2 S( w0 y) P) m# u% w) l In some workloads, the data written to HDFS is known to be significantly
3 ~$ p, J( ]8 c4 @3 b# v: s# K large enough that it is unlikely to be useful to cache it in the
, |$ y* _% w5 r% d operating system buffer cache. In this case, the DataNode may be* T, _7 f Q2 `4 S
configured to automatically purge all data from the buffer cache
- Q7 P R' A3 k after it is written to disk.$ V1 [1 _. A' x/ `1 Y8 t* g
This may improve performance for some workloads by freeing buffer
0 E6 }9 r) [! @8 z cache space usage for more cacheable data. j3 m4 ]; B3 }/ F: e0 S
If the Hadoop native libraries are not available, this configuration' e3 j& L; | i `1 M# V5 E3 X
has no effect.+ U, }2 v$ g/ V+ I- G: S1 Y H5 r
</description>
* s5 J! X" U) q- t, Y</property>9 O P. l2 g- `( O8 ?6 Z3 y: b
<property>7 o) y8 \. D: \
<name>dfs.datanode.sync.behind.writes</name>
; P8 B# e( H/ c, [6 v <value>false</value>2 {4 m$ k' R. |0 ?- @0 x
<description> K8 y6 _0 @# }, I v4 A
If this configuration is enabled, the datanode will instruct the
2 l" r( Z5 G# b operating system to enqueue all written data to the disk immediately
% ^. _' U& S' R/ q& ?0 ~ after it is written. This differs from the usual OS policy which
3 X7 Q( `7 a. V# N b9 N may wait for up to 30 seconds before triggering writeback.
3 _3 j+ H1 J& \9 M This may improve performance for some workloads by smoothing the
. G: i7 k4 j. ^: T$ n+ ` IO profile for data written to disk.. H% m# K, `2 v* ~/ _4 V
If the Hadoop native libraries are not available, this configuration$ u b4 ~3 x0 F! }
has no effect.7 u' y" W& y0 l4 }' S& G3 k2 _% t
</description>) f- x! e8 k8 B- {3 o' q
</property>
/ y" @' Q/ f6 T! ^' ?1 t<property>
/ l7 y. w, z, c; [7 L3 E <name>dfs.client.failover.max.attempts</name>
/ K: B4 Z3 X! W$ q, B5 X+ c <value>15</value>
0 z6 ^$ R) I+ m; m8 B2 c7 x <description>
0 ^* k+ W' t( u4 Y; {% a! N" b Expert only. The number of client failover attempts that should be
" L2 E& v0 t( g m! j% B6 o made before the failover is considered failed.# x# U. X: M' w- w8 N" w: h9 k3 n
</description>
7 P# N, `8 u+ M4 X1 m7 U X</property>; r$ N, w' D8 D' k: s, j
<property>! u b7 k' H$ l- S! j
<name>dfs.client.failover.sleep.base.millis</name>
) Q P7 h" K& g p* Y <value>500</value>
& [! l- s0 b: h& ?9 |# k( W <description>) U! S. ?% B, y. ^
Expert only. The time to wait, in milliseconds, between failover
0 x8 g* r. _" I- d$ N; l attempts increases exponentially as a function of the number of# X/ R/ G$ j( t' `* m
attempts made so far, with a random factor of +/- 50%. This option
$ b: N+ Q# _# [6 _0 i2 n5 j specifies the base value used in the failover calculation. The0 v, h& `" E. ]+ V" u. h
first failover will retry immediately. The 2nd failover attempt. c: T( G# P1 {! L8 F
will delay at least dfs.client.failover.sleep.base.millis1 {2 [0 n: K6 ? V9 x
milliseconds. And so on.2 ]# @1 O# {" j
</description> v8 y& G6 c; Q
</property>4 G0 \+ ?5 z! I8 O6 u7 j" m
<property>) x/ C0 ]4 V; _) C
<name>dfs.client.failover.sleep.max.millis</name># t) \1 Q& I+ e1 O& C [
<value>15000</value>) P% o+ i( ^6 V6 l& u
<description>* v4 ?. E5 B3 n/ O) d7 H4 J+ r2 B
Expert only. The time to wait, in milliseconds, between failover" Q r% B Y9 r/ ^1 I
attempts increases exponentially as a function of the number of5 W/ @" c" j) j j
attempts made so far, with a random factor of +/- 50%. This option+ t7 C, X: _$ f3 b5 d
specifies the maximum value to wait between failovers. & O) Q" ~ s' D' n* ?) P
Specifically, the time between two failover attempts will not
+ J; o% i( `8 B% Q- }, }9 t exceed +/- 50% of dfs.client.failover.sleep.max.millis4 V- E9 h! S$ `6 y- z
milliseconds.
; j2 B \. M5 K+ C; D' M! N </description>
" q. W" N! }* m+ R) O5 K4 Q</property>
' Q( g( m+ Z0 P* x8 j; D<property>8 }; e1 [, A% s0 v1 v& `& J
<name>dfs.client.failover.connection.retries</name>
9 N1 W6 Z6 U) b5 Q# ~% [. ^ <value>0</value>
+ a2 o3 P2 o8 h0 g; q <description>
' t! h+ t+ N7 h5 p8 c Expert only. Indicates the number of retries a failover IPC client% k% }6 d1 a! |+ u9 L
will make to establish a server connection.
- k# g7 }& S* J/ O: j: g </description>* A; D( |& |' q1 g- m
</property>
$ V. x' @4 F- ^) ^<property>
0 }. m" F* `) {$ } <name>dfs.client.failover.connection.retries.on.timeouts</name>" e3 l/ r9 ]- J/ i' @* l
<value>0</value>% Q! g* ` C F
<description>* ]5 {: P- G; h* k# i2 Z
Expert only. The number of retry attempts a failover IPC client
/ D# A* r% m; [, f. f will make on socket timeout when establishing a server connection.
# C1 I* q; A3 w i6 _/ [% ~/ t8 t; u </description>
1 e* `+ T' X- @& k2 ?7 g! `; G</property>
* k6 |% ?6 O( f. T* q6 K0 f8 K<property>$ N8 B: N/ p% v$ f& [
<name>dfs.client.datanode-restart.timeout</name>
* _* b8 f! T0 Z2 r <value>30s</value>
; b1 h3 R" H, w* D3 H6 B <description>
4 \0 |4 C& K4 p& F5 i3 J! z Expert only. The time to wait, in seconds, from reception of an; q' U" F, @: N9 X8 j: h1 E
datanode shutdown notification for quick restart, until declaring
$ B \9 s' F( r6 b; l- g the datanode dead and invoking the normal recovery mechanisms.
( Y( S; a, n8 y+ f, R; z& l1 X' Q The notification is sent by a datanode when it is being shutdown/ L' n: d1 j7 A7 w$ w
using the shutdownDatanode admin command with the upgrade option.
2 r3 S8 ~4 |2 r0 S$ v- {, i& i- E: @ Support multiple time unit suffix(case insensitive), as described/ C. B9 L! }: V2 h, e# @
in dfs.heartbeat.interval.' {1 O8 ~" q4 R6 y7 F/ o7 q
</description>% D! P7 @3 u% w7 T( c& X
</property>
9 k* W$ d2 V7 I; d2 N" N<property>0 H9 H$ D- `! k
<name>dfs.nameservices</name>3 D) O7 q9 j# {4 |( v8 R
<value></value>9 ~; d+ N$ W4 s, r
<description>$ v5 b9 X. K- a
Comma-separated list of nameservices.
" K3 G2 e7 v9 p6 `/ d5 ?( H% _ </description>
7 X" E; T9 v1 _+ C. Q( n</property>
4 C, n5 f0 k. x0 k7 Y<property>
# @0 c, M6 B" a9 A: `& D <name>dfs.nameservice.id</name>
9 P: Q. U+ |3 @' l. L0 Q+ @ <value></value>
8 P0 t$ |8 a k! w <description>
/ P- ^% c3 P$ C+ h; O6 m The ID of this nameservice. If the nameservice ID is not
& P4 I( |# M( T# @4 b; o1 B configured or more than one nameservice is configured for
! _- }- z- F: J' z7 }1 u& e dfs.nameservices it is determined automatically by
! k) d6 O8 M0 @* { i2 T2 Q3 y! [5 u matching the local node's address with the configured address.
5 n0 Y: b1 \: p' T </description>
0 j, S1 L8 u/ J# s</property>
# ^2 y+ n6 Y( H0 n1 }<property>
# R5 D/ _& F3 B/ H7 s# G <name>dfs.internal.nameservices</name>. C7 ~( O% Q- e E
<value></value>
( p) y2 {9 k+ a1 ?# n <description>
' y* s, G2 r1 r; X3 @0 e Comma-separated list of nameservices that belong to this cluster.5 w% y4 t! K# W9 `2 ` P
Datanode will report to all the nameservices in this list. By default* ]$ p" F/ ^9 w
this is set to the value of dfs.nameservices.) i7 v( B" b0 e4 m+ e+ N
</description>8 L4 y+ k% }7 |" V
</property>( f( m; M2 ~+ ?6 C% H6 V
<property>" t& m$ T M x& Y; q
<name>dfs.ha.namenodes.EXAMPLENAMESERVICE</name>
7 E X; [2 S) o- d <value></value>- g+ K8 ?2 E4 [2 }0 J
<description>
1 e% S; _- t) O2 {( d The prefix for a given nameservice, contains a comma-separated9 [' U' Q5 c1 |6 g
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).: ?2 t- w5 V( q' y- X
Unique identifiers for each NameNode in the nameservice, delimited by
8 n* ~5 _1 M/ B. B" F commas. This will be used by DataNodes to determine all the NameNodes
' U5 w* E/ n' H% v1 F, D! F; o in the cluster. For example, if you used “mycluster” as the nameservice
2 i$ e; R: y7 i9 J ID previously, and you wanted to use “nn1” and “nn2” as the individual7 A: i: {0 U* ^& u% j1 V7 z
IDs of the NameNodes, you would configure a property! h$ E1 r" V1 o3 E
dfs.ha.namenodes.mycluster, and its value "nn1,nn2".
; Q; O4 w6 \) U+ I3 { </description>
% o+ a3 }3 Y9 n</property>! w. Q7 {1 u. t3 [" j- _2 L( h! l6 H
<property>6 q b+ x( ~& e/ z6 I4 r" a4 N
<name>dfs.ha.namenode.id</name>4 P* t0 t- j/ F$ z$ M
<value></value>6 J" D% c6 p6 K. y' ~
<description>
% I! G% N1 j4 _! n The ID of this namenode. If the namenode ID is not configured it+ \9 P' p) n5 q2 R0 v* ~# g' L
is determined automatically by matching the local node's address
/ x' o; A- _! n/ ?1 b9 ^" l with the configured address.
7 z' q2 W" H& z' i, |3 ]1 ?3 O- ] </description>
7 J$ n( |: ~$ b1 f! [</property>. _0 G: K$ L% i; Q6 @& B8 P8 e# i- E
<property>; V. [. J' G7 C9 Y& p- N: l
<name>dfs.ha.log-roll.period</name>) f' S& C/ U- `! X" M0 r
<value>120s</value>
% k% v5 D+ S5 D: X- A- [ <description>
4 T+ E7 u- [# h! ]! u1 ~. z; ? How often, in seconds, the StandbyNode should ask the active to
5 B/ W) ]5 K) i: u: Z, C4 B. `( z7 F( X roll edit logs. Since the StandbyNode only reads from finalized
. p1 a2 H0 u: R6 n } log segments, the StandbyNode will only be as up-to-date as how. l0 J( M/ |- R {0 O- \
often the logs are rolled. Note that failover triggers a log roll4 Q+ B3 m0 @9 R6 A6 e5 n3 [. H! K
so the StandbyNode will be up to date before it becomes active.! `. V# W" M( ]1 P
Support multiple time unit suffix(case insensitive), as described
& t+ P: T1 \& g/ w/ q3 ]6 H in dfs.heartbeat.interval.: |4 N. @/ M* [
</description>4 U* H7 H8 V; C. t1 f
</property>0 K ]) e" q9 n V/ \
<property>7 {/ Q9 [+ {$ @$ J* L
<name>dfs.ha.tail-edits.period</name>7 S }- u* W8 L* [: P$ b
<value>60s</value>
( E: @0 Z! h' M5 v1 j, k- B% ^ <description>
4 a0 W( G: W4 d( q2 h How often, in seconds, the StandbyNode should check for new
6 b2 Y5 E8 d- X: K. o$ [ finalized log segments in the shared edits log.
$ M4 q) g5 v; y L Support multiple time unit suffix(case insensitive), as described
$ M% V- v4 B2 ^ in dfs.heartbeat.interval.
. X7 q( [: |, Z; n, R8 { </description>
4 q) ?$ o# z' v, H& f</property>
R' v7 \3 _1 X, Y0 u$ o# }. g0 Z<property>
/ e! p G* g& H2 S0 S <name>dfs.ha.tail-edits.namenode-retries</name>
: q* B+ Q5 x5 q4 ?5 ~' l' _6 u <value>3</value>
0 `8 I9 ?0 O7 G <description>% o! ~8 T) F, D7 M, s
Number of retries to use when contacting the namenode when tailing the log.
) w' ]% N! Y$ a </description>2 @8 Z! C8 | e2 X0 n/ h+ M# }( K
</property>- t) H8 K2 b0 M
<property>
8 j! }3 x! {$ R& o) t1 \+ @ <name>dfs.ha.tail-edits.rolledits.timeout</name>
+ u& H P `' m* q1 F: A; X <value>60</value>
* Z0 |+ i' J0 H" D+ B <description>The timeout in seconds of calling rollEdits RPC on Active NN. l0 ?# D- O& K' j& c% B; {
</description>
1 T: T, ~5 l. W+ J6 L f' M* ]3 U; M</property>' a( o+ ?) |5 m, ^% \, ^
<property>% R3 R7 R, n( }) n
<name>dfs.ha.automatic-failover.enabled</name>
: c( o/ [9 T, T6 y9 @. R6 Q' t! P <value>false</value>
- c& X( m* d: x! B" d8 q# `9 O <description>
3 v. `# J. t* ?. X, r Whether automatic failover is enabled. See the HDFS High
3 v8 p4 K1 a" n; I/ M J Availability documentation for details on automatic HA% ?, _5 Q: h, s3 ?$ j
configuration.6 j) Q" |8 p2 E+ a2 f
</description>5 H/ W) a5 C( @6 ?: [
</property>
; m; u/ W7 l/ G' h0 ~<property>
# x) ^) a0 {& i& I% I# B: p <name>dfs.client.use.datanode.hostname</name>3 k5 Y# {- {. n) f1 y8 x
<value>false</value>
# y& m0 s9 d. M2 b <description>Whether clients should use datanode hostnames when
- J4 o* D/ y& T3 j* s# w+ b connecting to datanodes.4 g" {) A+ o. C# Y7 Y |
</description># m7 ?$ g( r! s- p
</property>, v/ S; X4 \2 |# [! u% R
<property>
" g' Y1 i8 e8 i& q; d6 A+ d <name>dfs.datanode.use.datanode.hostname</name>
# a6 {6 b1 E, c% v R <value>false</value>
0 B8 w5 {( z ~/ e5 Z8 B8 N/ n <description>Whether datanodes should use datanode hostnames when4 k2 C- e. Z6 o$ D/ M: D
connecting to other datanodes for data transfer.2 C( |7 J& e) |/ M- G) Q% ]1 `2 z
</description>/ X% C+ I6 T: A: s; S% I* I
</property>
& @( N/ K w) g<property>7 j9 K4 _7 j9 J. O
<name>dfs.client.local.interfaces</name>
; c+ s# C* Z ~% v8 F" @ <value></value> M( D" T) r8 w$ ?) h+ [
<description>A comma separated list of network interface names to use0 @/ M; n: z8 f
for data transfer between the client and datanodes. When creating
9 T h0 P% \. [% u$ q0 ` a connection to read from or write to a datanode, the client
, f! H" b* `$ n4 {3 `, n chooses one of the specified interfaces at random and binds its
: p" Q6 Q) z9 Q0 H. y& S socket to the IP of that interface. Individual names may be1 X! l3 V% s2 g' ]
specified as either an interface name (eg "eth0"), a subinterface, O' o( {6 I2 X: |$ G/ T
name (eg "eth0:0"), or an IP address (which may be specified using
. z& w8 {% a7 ^7 y% Y$ H. e+ O) _* u" h CIDR notation to match a range of IPs).
" e& H. J, \1 H, X+ r4 @+ F. p </description>
/ ^+ I" w6 w; c/ L</property>
) s# O4 }( N5 q8 d8 p<property>
% [ B1 @9 C8 C5 N8 ^3 a0 i <name>dfs.datanode.shared.file.descriptor.paths</name>6 Q) q/ Y% J' Q
<value>/dev/shm,/tmp</value>
1 Z& k, L! B3 I <description>' @" V% }( \/ P9 v/ @! O4 F
A comma-separated list of paths to use when creating file descriptors that
) c; b5 w, \7 q& [( Z1 A( v will be shared between the DataNode and the DFSClient. Typically we use
9 p) C& u/ B' H1 ^+ y! ^( O /dev/shm, so that the file descriptors will not be written to disk.% Q r7 |/ W3 o) K8 L( R' h
Systems that don't have /dev/shm will fall back to /tmp by default.
2 T6 s l6 e$ y, Z! p </description>; m. \8 Y {" i+ M' Z" n& ~. V
</property>
0 I; O4 m* Y/ D u0 E* t<property>6 P% @5 J( O V2 ?( J' A( S' j
<name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name>6 z$ @9 W7 q6 L
<value>60000</value>
: c) V% K) k4 Y; F <description>
+ e+ b2 S8 Z3 P3 v9 d; ? The length of time in milliseconds that the short-circuit shared memory9 | ^$ K; {3 D9 W7 R' R
watcher will go between checking for java interruptions sent from other2 E$ q7 x: s9 l- E
threads. This is provided mainly for unit tests.
$ M4 U: D; l8 ~ </description>$ G6 [; G' k, E) {8 J) b7 m" @
</property>
+ D: F, v, e6 T$ B6 R& [<property>
' s% l! c) W5 R- l: g <name>dfs.namenode.kerberos.principal</name>1 j9 G) }: q8 X
<value></value>! c3 }, v9 V( u
<description>, h0 I. b u. k: x) l1 ]* J
The NameNode service principal. This is typically set to
6 Y- u( i1 ~9 I( t) E5 I: Z nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its
* \2 E* D% X( Z1 E2 d own fully qualified hostname at startup. The _HOST placeholder
+ o# l/ S" d# J! g( g allows using the same configuration setting on both NameNodes
) J3 z; j) X& z% T in an HA setup.
4 W o) |1 f) A9 O- }" o5 x </description>1 A- @4 ?1 S8 q6 G+ \7 ? p" o: n
</property>$ E( D5 q. I7 a: N8 _' D6 L1 g" `0 a
<property>
, p) X- A* I- b T( X! y0 P. ~ <name>dfs.namenode.keytab.file</name>% ]+ Q1 I. v' A
<value></value>% p; M3 R+ u4 C
<description>
3 \6 t7 \% ^, q9 T3 Z The keytab file used by each NameNode daemon to login as its+ T. I9 ~1 F' J, O
service principal. The principal name is configured with. Y% L8 j2 `* _
dfs.namenode.kerberos.principal.
! W( f: ? S% `7 T, i& q6 o </description>7 v3 k* x4 Z2 v! q
</property>
& E. @) W7 z; ]! e7 d6 H6 W$ e<property>
8 O1 r( h' f" I/ ~* ~. g. R <name>dfs.datanode.kerberos.principal</name>! R# X3 k9 `& M; c
<value></value>
- _. T7 `" j4 g' z! o <description>+ R# f6 _/ @: A: [+ D% H
The DataNode service principal. This is typically set to n! ?. }$ K# B' j2 N+ q
dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its
& k% e. G/ Y. y c1 g$ u own fully qualified hostname at startup. The _HOST placeholder
2 E t9 x: U1 a. s! p8 ` allows using the same configuration setting on all DataNodes.
7 k. c, m; T! @' ?! g2 H </description>; j# M& W/ j) g
</property>
9 C$ J3 Q4 o2 y4 p* {3 |. y<property>
4 [6 h: ]# s1 w3 I% ?& C( o$ k$ V& p <name>dfs.datanode.keytab.file</name>
9 `; Z/ j& I4 X+ E8 J <value></value>
: i3 Q4 P1 N# h" e <description> }, W8 l5 {8 f# } _: S- X
The keytab file used by each DataNode daemon to login as its
/ d' m% ]/ q& P7 A) V6 [ service principal. The principal name is configured with
5 H0 K' F" g3 `* h6 G. } dfs.datanode.kerberos.principal.
! e# Z( ^4 v0 ]1 Y* C- b9 C </description>
" y* v) M" J" a7 G0 `, ?</property>2 n4 \- r: j( K U) u* O$ J
<property>3 M8 \, U: ~0 `6 r* t7 }) q1 r
<name>dfs.journalnode.kerberos.principal</name>
9 V; y% ]5 w4 _2 J7 Y: u <value></value>7 R; \% F3 j0 n, N1 v5 B0 J
<description>
0 J- x5 O' N+ y! C# B: Q" C% ? The JournalNode service principal. This is typically set to: u8 c' K, i2 D6 p0 P, j
jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its
. l. T+ y9 L) S4 M4 i! q ~9 r own fully qualified hostname at startup. The _HOST placeholder
. }0 a6 K3 y' I7 N, |# K allows using the same configuration setting on all JournalNodes.
2 `' R* t, V" W2 Q3 T. _ </description># V3 u; Y3 c# O& o
</property>
6 \7 z" i/ v+ N, T<property>2 ?" J6 J& H8 `& M5 \6 ?3 T' H
<name>dfs.journalnode.keytab.file</name>4 V8 I3 I4 k7 |+ d* a ^
<value></value>
! B" T+ e3 i) S7 m- ` <description>
! `, w$ q; j" X& O9 z" T The keytab file used by each JournalNode daemon to login as its
: m7 u1 d! X$ {2 c service principal. The principal name is configured with
' }9 L% H/ n1 K+ r$ W5 p dfs.journalnode.kerberos.principal.) u! d, Z4 X( V6 h
</description>. b; l, ]& Z0 K2 g% R5 }* p& }
</property>
' t6 f( T2 M2 b' `* {! Y) ^<property>9 R r" N6 f3 Y. h* [6 V- Z
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
1 R _* Y: |' c! [ <value>${dfs.web.authentication.kerberos.principal}</value>+ d9 l2 `' |" d9 U. g& p
<description>& ~- v2 ], s# j' P
The server principal used by the NameNode for web UI SPNEGO
3 J3 C% c, v8 ~. j' c. w! c3 o authentication when Kerberos security is enabled. This is
( H- C; z) `2 E: k! M3 ?0 c typically set to HTTP/_HOST@REALM.TLD The SPNEGO server principal
g! o/ d( g; z5 L9 b begins with the prefix HTTP/ by convention., `7 [! F1 E1 u& Q" r/ J
If the value is '*', the web server will attempt to login with
4 K6 `& S' @8 d, i every principal specified in the keytab file
' w" H. p% H# a! e dfs.web.authentication.kerberos.keytab.
# T) k( x$ j. {: o1 b</description>3 r- I6 P3 C& v" F5 l8 {* {9 H3 E
</property>
& m% {4 v" l5 f! M1 A0 e<property>" j9 T% f" Y" o7 I! U" w! J
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>3 h Z' H" ?# X' C [: ~; U/ f
<value></value>8 O/ d6 |: \) x+ W/ x
<description>% j9 I3 E1 a3 x6 X, J
The server principal used by the JournalNode HTTP Server for
0 z' a' n* y+ I; `. u3 G/ a SPNEGO authentication when Kerberos security is enabled. This is
. ^2 D/ v3 }; B6 v0 d; M typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal- R+ w: k* j6 Q5 Q2 P
begins with the prefix HTTP/ by convention.# Q% r( y2 h3 X$ D
If the value is '*', the web server will attempt to login with
& ~9 {. \/ d/ \/ D$ ?6 _ every principal specified in the keytab file
% I! @/ s4 k, {9 ~$ A8 g U+ O- e dfs.web.authentication.kerberos.keytab.5 _& {$ E2 T; H2 K: o
For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}. i* M8 E& t: g2 e8 N( @4 S
i.e use the value of dfs.web.authentication.kerberos.principal.# O3 d+ F0 y5 [/ ?& B2 r$ ?
</description>
$ C$ s1 a0 A6 Z# g3 t</property>6 e2 o- F2 }& }
<property>
- n% o$ k) |% I <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name># _& l/ S0 i8 @7 s( d+ h0 @
<value>${dfs.web.authentication.kerberos.principal}</value>
* A1 Q. q8 C! A2 e. c <description>
" T. r' t; q# G5 c3 f! ~3 l The server principal used by the Secondary NameNode for web UI SPNEGO6 R3 W ?/ _% u7 P
authentication when Kerberos security is enabled. Like all other
* u' d$ A% B3 o8 A M9 s% N Secondary NameNode settings, it is ignored in an HA setup.4 R3 D* q- x) l1 W" x8 T; l
If the value is '*', the web server will attempt to login with- N# V6 b6 e2 E+ _+ M# j& J
every principal specified in the keytab file% U- A, m( X/ i2 \7 s( s0 i
dfs.web.authentication.kerberos.keytab.
3 t" u% a+ @: e( H4 P9 E1 M; e1 O* m </description>
" ]& v# [- T0 _. i</property>
/ I1 X' f/ ] E, y' P<property>6 ]+ {3 e ]' V6 N6 N7 G% e
<name>dfs.web.authentication.kerberos.principal</name>. }2 b" I4 n2 T
<value></value>
+ G6 t3 b R" b- y" D6 h <description>
% U9 J6 ^# i9 E' v8 L0 i The server principal used by the NameNode for WebHDFS SPNEGO
- l( a, G! g& \2 i authentication.
5 m0 L: G1 b* v! ~8 t Required when WebHDFS and security are enabled. In most secure clusters this
, [9 I1 E3 k- D p6 b setting is also used to specify the values for
6 v, v, [1 |+ Q* j4 q- l% W dfs.namenode.kerberos.internal.spnego.principal and
n0 w" Z" T# C' d1 W dfs.journalnode.kerberos.internal.spnego.principal.+ [( P1 b9 J2 T) O0 ^
</description>$ q& H( B3 N0 A1 j7 @
</property>: L" l7 h9 x& X2 c3 i7 E+ c% K
<property>
: z7 F, s3 j: I1 ^" | <name>dfs.web.authentication.kerberos.keytab</name>+ a3 r# f1 Q5 w0 c9 u0 I# z1 o
<value></value>) j7 L5 n4 O' o0 D5 I/ l
<description>/ {2 n1 X' S! N! }: Z; f
The keytab file for the principal corresponding to
3 r9 m( h$ M2 p. | dfs.web.authentication.kerberos.principal.- e+ q( ^. |9 u; J: S% M' S
</description>
& I$ d* B- k, e8 h2 P</property>! U5 a& w% J2 I8 A
<property>
0 ~: @/ D: ]0 Z# E' s <name>dfs.namenode.kerberos.principal.pattern</name>! h, A, J! T# L0 T3 C0 c* b
<value>*</value>
& m7 \% W% X0 t" M+ N$ z. E <description>
]$ v+ W P6 ?& Q' R( V$ M A client-side RegEx that can be configured to control* Y4 G2 k8 t2 f R- \
allowed realms to authenticate with (useful in cross-realm env.)
/ f4 {: ~ O* `/ a </description>
! i* Z. H# r) l" U2 s</property>- x, z- n& F2 p+ [8 B! Y7 {) a5 b2 J
<property>
' V1 e7 d# u, L3 w; J: c" n <name>dfs.namenode.avoid.read.stale.datanode</name>& z; O7 q: d5 _* S" V0 f
<value>false</value>2 e2 M1 h: ?4 D, {$ C& r: b B M& d
<description>+ b( M# [; g" d6 F4 j
Indicate whether or not to avoid reading from "stale" datanodes whose
$ B) I; J5 z- T# R& y5 }* B8 u3 x& w heartbeat messages have not been received by the namenode
B% `3 \7 z% j" M+ q for more than a specified time interval. Stale datanodes will be8 _. s2 A7 M% r6 P. U
moved to the end of the node list returned for reading. See
! U- Z3 @& x" F4 _0 _2 U- q) Z) n" {8 m dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
$ P/ P& @! {2 z1 d; D </description>, j. A0 V0 r2 B$ Z+ O
</property>% A# }2 Y& h) |# g; i$ m
<property>
5 V8 G: i* X; p9 ~7 F0 A1 Z" U <name>dfs.namenode.avoid.write.stale.datanode</name>
- ~' m& w# r% a/ g1 y, B <value>false</value>* @1 X( q+ a9 b; }$ ]! \
<description>
2 C. N5 L5 y9 I- d% b Indicate whether or not to avoid writing to "stale" datanodes whose W9 M! R% V* P* r
heartbeat messages have not been received by the namenode
8 n8 V' X) o: E) I# o2 F" h for more than a specified time interval. Writes will avoid using # T$ Z ?4 r0 e' O
stale datanodes unless more than a configured ratio / e$ U! E0 N0 i- F
(dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as
5 d$ o. i5 Y( \: e+ c' ~ stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting8 A$ Q4 R6 m# g/ U: z8 p' Y
for reads.' ?+ k! e' i3 Q; q( A* x
</description>* F5 I* M9 N: E9 I, ~
</property>
D2 v2 E g. S1 T2 v+ K<property>
7 P3 C; V5 K: X u8 ]& W, s' H; J6 n <name>dfs.namenode.stale.datanode.interval</name>
4 g! { U. s9 k1 }" z: X$ ^# \( z <value>30000</value>
- j S; b; `. d0 w. n$ w <description>* V, R3 o* n/ X( N% ^4 U
Default time interval in milliseconds for marking a datanode as "stale",, s7 b# S! L* @! W6 L
i.e., if the namenode has not received heartbeat msg from a datanode for. {1 |! Y' o' q% X# p1 B2 y8 s0 L
more than this time interval, the datanode will be marked and treated
( Y4 B0 i( k/ \+ ` as "stale" by default. The stale interval cannot be too small since , g) S& u* Q1 J
otherwise this may cause too frequent change of stale states. , F" `* n/ M, `; @9 O
We thus set a minimum stale interval value (the default value is 3 times
* {8 A- {7 w, l% W of heartbeat interval) and guarantee that the stale interval cannot be less+ x' L4 Y- i1 _2 R4 E$ E j
than the minimum value. A stale data node is avoided during lease/block i, D5 _5 `$ t1 T g0 ]- u
recovery. It can be conditionally avoided for reads (see
: W' g! D0 r4 a$ T5 t dfs.namenode.avoid.read.stale.datanode) and for writes (see6 e5 _6 }! E, n
dfs.namenode.avoid.write.stale.datanode).& h1 w9 B4 M4 M( V" O
</description>9 H+ J0 b9 P& `
</property>
! y* t$ ~. N. l9 V7 D/ ^<property>
2 U# c+ |; ]* D3 J <name>dfs.namenode.write.stale.datanode.ratio</name>9 m2 a% V1 w) r- S- W; O
<value>0.5f</value>
! c& c5 i2 K4 W T2 { <description>
, X: }0 |2 d. P9 E When the ratio of number stale datanodes to total datanodes marked/ P/ }' b9 Y% J" c
is greater than this ratio, stop avoiding writing to stale nodes so
! X# n2 a% Y9 O: \" H' t as to prevent causing hotspots.
8 H6 q. w4 k5 e# z8 M, i: U; X- v </description>, |7 l& L# s6 b- Q7 D6 {$ A, x7 g
</property>3 M# A: K( H& z5 O! n; ^) G
<property>
5 _: e( P& Q( w3 ?; `/ G3 ~ <name>dfs.namenode.invalidate.work.pct.per.iteration</name>
& h% w& s. ]0 F/ Z <value>0.32f</value>5 Y) u( F4 i. U+ Q/ o
<description>9 g+ Z% A+ L h$ F
*Note*: Advanced property. Change with caution.
F* ~9 W& H6 u( ? This determines the percentage amount of block9 g# B+ d& F; u+ f- g+ t
invalidations (deletes) to do over a single DN heartbeat
* p; t: l) S9 K2 i, t @ deletion command. The final deletion count is determined by applying this
" w( u/ B2 V# r8 P( W percentage to the number of live nodes in the system.
$ H6 K K4 v6 M+ a4 |& c" ^ The resultant number is the number of blocks from the deletion list
K8 z4 R" H) y {' n chosen for proper invalidation over a single heartbeat of a single DN.
2 F/ a+ m+ a$ a( a( T( C Value should be a positive, non-zero percentage in float notation (X.Yf),* X2 N$ j, k6 s6 n; R
with 1.0f meaning 100%.
$ ?2 R1 d/ J% x: Z) T* @, D( v </description>
. K" J/ r2 F. K$ S; x</property>8 x6 r% y7 T3 e# u+ I
<property>
3 P5 I, y' C% p, U' U* F8 M <name>dfs.namenode.replication.work.multiplier.per.iteration</name>
: E2 P4 f5 B% }, m9 [0 e5 Q' g <value>2</value>
1 X& [) P/ f$ M' y! E6 i6 I( b2 W <description>
1 M. n+ L) s# y! k H *Note*: Advanced property. Change with caution.
; a5 C) \; U! H3 d9 M This determines the total amount of block transfers to begin in7 g9 v# |1 r* w- v- W! v* \/ j
parallel at a DN, for replication, when such a command list is being6 U: Q# {7 L$ D( m9 w* j* l
sent over a DN heartbeat by the NN. The actual number is obtained by
9 y0 g. s. H7 c% g" \8 H multiplying this multiplier with the total number of live nodes in the
6 J; g, i& O: N( e cluster. The result number is the number of blocks to begin transfers
- E: q# N6 E, b5 d immediately for, per DN heartbeat. This number can be any positive,5 q. F! F% ]6 X2 ]
non-zero integer.
2 N" `& T$ z: d) o' v5 G </description>
8 \+ l; c: o% @1 R( N( p</property>8 r" Y1 v( r9 B- n) j5 \4 x& r# U
<property>
! c( _2 R5 E' G, d6 m <name>nfs.server.port</name>$ u! Y5 z! j+ H3 i" D) W
<value>2049</value>! h& T& @- _' m
<description>! `' L& u. d+ Y2 [& }- ?
Specify the port number used by Hadoop NFS.+ M$ R5 Y4 I) Y8 E6 o
</description>
B& `; X! F5 ~4 a4 o( E</property>; E$ h2 l* m3 B4 z$ U- `4 o! _$ ?
<property>
1 m& @5 |- G1 d0 [' H: j0 F D$ x <name>nfs.mountd.port</name>) _. x$ N1 a1 \, A* t R* c j' q
<value>4242</value>
7 M# ]/ H; R7 }- a$ R <description>
2 n1 ^8 W9 g. A$ o Z4 u Specify the port number used by Hadoop mount daemon.
1 P( r; w4 O" f* ~ </description>
. @) R. p5 @% K; C# ?4 G2 }1 h</property>
( {+ F2 e `0 P' ^% [# r9 h4 ?, [, W<property>
6 K/ o) B0 y+ N7 r <name>nfs.dump.dir</name>( R3 M- Z! ~% O$ Y; N8 }
<value>/tmp/.hdfs-nfs</value>: I: r* H3 K8 Q$ V; j% c/ r
<description>
3 {8 y% {- V% C: G* n This directory is used to temporarily save out-of-order writes before# _! [& m. P; a2 M1 K) w. c
writing to HDFS. For each file, the out-of-order writes are dumped after
9 Q& {/ x1 @* E* t# ~- U% E they are accumulated to exceed certain threshold (e.g., 1MB) in memory. ) d$ ]! y n9 A5 a4 A" a- R
One needs to make sure the directory has enough space.
9 T4 J9 T$ H2 A0 _1 M </description>
5 q% ^7 I4 B9 |: o</property>
. E9 c4 T. Y8 k8 c! M3 E<property>
- i/ e1 k" Y. a+ m4 Q <name>nfs.rtmax</name>
2 q5 z7 A0 A- w& A8 c <value>1048576</value>
+ i4 |' t$ f. @; v7 u ]! d. z- R <description>This is the maximum size in bytes of a READ request
% g8 C! k. W8 N/ p1 T supported by the NFS gateway. If you change this, make sure you0 h8 g, s5 l& l0 o8 v8 I
also update the nfs mount's rsize(add rsize= # of bytes to the 8 r- S+ h9 ]4 ~+ ~3 Q6 C
mount directive)." ^2 j7 [- V( ^5 k9 K3 f
</description>, ]0 Q7 H7 I& L7 i
</property>
' m9 S$ P9 I! a7 F<property>
! K: R- e5 \* f9 d0 Y6 o <name>nfs.wtmax</name>2 ?/ ?6 F7 b0 a; i, F. {
<value>1048576</value>5 R A0 {: v. _) \! a* N
<description>This is the maximum size in bytes of a WRITE request- J) z" j: `( W5 t
supported by the NFS gateway. If you change this, make sure you
- ]: S! z- g m! \ also update the nfs mount's wsize(add wsize= # of bytes to the : U# M0 r& V) U' t8 F. D
mount directive).
# r$ w6 y9 v& B: ?# ~ v </description>4 E, G0 W8 B' w& l
</property>* O, O3 W# [% q4 o7 G0 B% y
<property>* O9 a2 l8 v/ S
<name>nfs.keytab.file</name>
) m$ V0 G$ [% A+ r9 E* B2 r <value></value>
& q" A0 q2 F) W <description>, J$ U: Q# p# e) w
*Note*: Advanced property. Change with caution.0 M. b* z8 Z& U
This is the path to the keytab file for the hdfs-nfs gateway.
6 R9 Q) X8 q4 i# [' W) r This is required when the cluster is kerberized.
- T1 d F% ]! A& I% _ </description>
h% W/ g2 O2 y6 C2 f2 J, Y5 P</property>5 V7 s" h$ z- ^/ V
<property>9 L7 m' V) ` n! h$ j+ ]$ s3 h
<name>nfs.kerberos.principal</name>
4 o+ V1 M* T1 L <value></value>6 D* {' |9 e4 p9 H- O/ |
<description>! R& C( g. h- f/ w5 {& G! e
*Note*: Advanced property. Change with caution.$ F. l3 Y, }$ t
This is the name of the kerberos principal. This is required when* y; x1 T! }8 x5 g9 P( v4 Q0 t- f
the cluster is kerberized.It must be of this format:2 X2 [3 W0 O f( V
nfs-gateway-user/nfs-gateway-host@kerberos-realm! L* ]' d* ]- \! ~' Q4 f
</description>
' T6 v9 V& Y2 \* }/ f6 I7 E1 i</property>& O8 s; ?! [; v: a
<property>% D+ O+ ` T$ p3 H& z# [: h& ]' y
<name>nfs.allow.insecure.ports</name>/ Y% a) i3 e" P) ]; ]) c# X
<value>true</value>
( m" j% R) d+ C, n6 L: C7 _ <description>
% z" h+ D4 l3 h1 _+ U+ _+ n When set to false, client connections originating from unprivileged ports6 F8 Q, u( @4 g$ R2 _6 l/ M
(those above 1023) will be rejected. This is to ensure that clients$ [* ]! E$ }% o; A1 i6 l0 u
connecting to this NFS Gateway must have had root privilege on the machine
+ ?! O* }- }- W7 x3 H% |5 V9 b1 ]( H where they're connecting from.
- U3 X/ V: W2 [$ y2 ?( i! n: S* R </description>
8 b8 U& x+ c9 N' C</property>
d( `; B* G# b; b% p8 A/ f/ j7 A- K<property>
& v" @& S- ^8 T6 Q# o- S- L <name>hadoop.fuse.connection.timeout</name>
- \+ q* N4 z: b9 y <value>300</value>
" B5 r5 H4 \' | <description>& ^4 T2 T1 w/ l v( o
The minimum number of seconds that we'll cache libhdfs connection objects Z1 f: Q( F5 F6 r9 h7 G' X
in fuse_dfs. Lower values will result in lower memory consumption; higher
3 ]+ N8 z) h5 E; X T" r values may speed up access by avoiding the overhead of creating new. X0 G% a: U+ D7 B+ F% w
connection objects.
- y$ V0 @; V5 [& Y$ }. i/ E, ] </description>
6 Y! G% U1 g1 O# `$ C* m7 Y A</property>
: ]3 l4 ]) Y- Z% b" |0 n0 b9 [<property>
: u* d6 N, R( ]0 w! p* v <name>hadoop.fuse.timer.period</name>
3 h4 c) L6 H9 q( h: K* }% L M4 } <value>5</value>, V* v/ m) I1 X4 E2 i! O% t
<description>
7 r7 P5 B* a x" w7 D$ e The number of seconds between cache expiry checks in fuse_dfs. Lower values
{- T/ @9 Q: X will result in fuse_dfs noticing changes to Kerberos ticket caches more* _: O( K* b9 j
quickly.' \/ H& X9 j4 L- ^( X* e3 `
</description>; y) s* i# b" x; s' z' v0 Z
</property>
; p' S2 h# y' w<property>
$ t% X' `" s, B <name>dfs.namenode.metrics.logger.period.seconds</name>0 m: g5 J' S8 t2 g
<value>600</value>
* c/ m1 J0 z% ~7 ~3 Y! g3 x: F6 _9 K <description>
4 }# x$ `9 |& p3 j This setting controls how frequently the NameNode logs its metrics. The
& P! e/ y/ E' {! j& V4 ?4 W" c |3 [ logging configuration must also define one or more appenders for! O( V: V" K/ t; I% g# v D) G P
NameNodeMetricsLog for the metrics to be logged.
% x$ f3 l9 Q7 G2 @" L8 h0 D NameNode metrics logging is disabled if this value is set to zero or8 o3 O/ q; m3 L; y" V6 z5 z2 O- C
less than zero.; ~/ }& d6 S7 P# Q$ d
</description>; y+ w0 G5 B2 a( ~) S- h
</property>
) l- \. H1 c l& S" }7 F4 r<property>0 w$ X5 V$ ]' `0 }
<name>dfs.datanode.metrics.logger.period.seconds</name>: E+ i; z- v7 }; r) }0 I( |$ u
<value>600</value>
; ]6 n& u4 v/ Z, \' C6 E <description>
6 x" ^7 e% `+ Z- [ This setting controls how frequently the DataNode logs its metrics. The! `6 ?0 x/ O; ^* `& m7 Q4 E
logging configuration must also define one or more appenders for* I0 u1 K0 D' T2 u5 [# q, t, l
DataNodeMetricsLog for the metrics to be logged.
" r2 ]; p% R# W5 o. X2 ?! N6 @ DataNode metrics logging is disabled if this value is set to zero or1 S6 ?' J$ j+ m( O" D P: n
less than zero.3 l4 }! d5 \9 f. j5 u9 K
</description>
' R- T# v. p4 `5 s3 v# \$ e2 S</property>
+ Z/ h$ P' H7 E) v( V<property>
5 v# q/ J3 n1 b9 ] <name>dfs.metrics.percentiles.intervals</name>' C" N; j- T2 Q9 K! y* F# }1 [8 K
<value></value>8 {$ m: `0 ^2 g- f9 V% m
<description>
& _' [4 o8 n5 z6 u7 J Comma-delimited set of integers denoting the desired rollover intervals
. C6 y t" }% K0 I( y. s( j) N (in seconds) for percentile latency metrics on the Namenode and Datanode.3 f) O' y0 i6 R7 o$ i
By default, percentile latency metrics are disabled.
0 M' u* |) x, z </description>
$ k( |$ n- p& g</property>! g) Z0 a" ^( |% |9 h! K9 d2 u
<property>
7 `; ?/ c* W' m4 M3 }! U1 j2 f <name>dfs.datanode.peer.stats.enabled</name>5 O7 H4 d: |, @- v* e8 J
<value>false</value>7 n, Z' Z L7 w8 m1 d' z
<description>
( f$ F, G3 q, k7 G' r: K3 S; R A switch to turn on/off tracking DataNode peer statistics.4 j3 g0 z$ e% }
</description>5 c# ~; w) V0 A
</property>4 B7 J" @9 O# V5 f1 ~( W& h g. _
<property>
9 j4 e# a* X- W4 C- p. [ <name>dfs.datanode.outliers.report.interval</name>
% W* k w: u# \% O' \3 t3 }4 y <value>30m</value>
( g, |) T# b ^8 U6 R' V9 {( ^ <description>
$ m0 ?5 ]8 O; `. c" t# F This setting controls how frequently DataNodes will report their peer
( [0 x) l; h+ L N% g latencies to the NameNode via heartbeats. This setting supports
* x0 m) ?/ L1 K multiple time unit suffixes as described in dfs.heartbeat.interval.
/ H$ T/ |$ g9 F3 M& m3 R If no suffix is specified then milliseconds is assumed.; w* j; c" u6 u9 P- j+ Q0 e) N% T
It is ignored if dfs.datanode.peer.stats.enabled is false., V" B3 H& U& ^3 ?1 A
</description>
$ F" P! L4 N [" I! h% h- }</property>1 L- |% M0 W5 x6 P3 O+ j. T7 h
<property>
# b& D1 p3 x5 k2 q5 C6 U" \: l# E <name>dfs.datanode.fileio.profiling.sampling.percentage</name>
: S/ o) Y* \& @4 J# t& J <value>0</value>8 C3 b/ q: }0 z
<description>
3 l' G! ^& G% W0 S% L This setting controls the percentage of file I/O events which will be
% v" `( R, q8 c: c K profiled for DataNode disk statistics. The default value of 0 disables
5 o9 X( a, v1 y$ L7 ?) _* {( J disk statistics. Set to an integer value between 1 and 100 to enable disk- u3 C. z3 ^0 J) i0 J
statistics.8 i b$ e. n5 b' N ]
</description>% O& P- |9 U" w' u
</property>
0 U( v, u; ?, ?+ c. j0 a- R<property>7 O& p- X$ F) ~, U
<name>hadoop.user.group.metrics.percentiles.intervals</name>
u9 v# z$ T4 P/ G2 s. h; @9 o4 G' b <value></value>
- [3 o- Y c4 U2 h1 s5 _1 z* G* u, Y <description> M* F' O$ a+ E. L" |
A comma-separated list of the granularity in seconds for the metrics, E; r5 L# O( i- o |4 \/ N, u
which describe the 50/75/90/95/99th percentile latency for group resolution
) d$ J T2 I7 A: c1 E3 B8 O- ]3 w9 O in milliseconds.) a8 z0 W, ~3 C# d! k7 C
By default, percentile latency metrics are disabled.- b: L5 ~6 G7 Z; [* Z. s
</description>/ c) b, m( W6 ?! N$ i1 f; a e
</property>! S, C- k( @1 }. R7 ?! j
<property>2 K# y# }! |2 @
<name>dfs.encrypt.data.transfer</name>; @7 k* ]$ g0 u& J# P
<value>false</value>
1 V/ E! I g( i0 g7 w' X <description>3 P% ]' ^9 a: u$ r& I) d; T
Whether or not actual block data that is read/written from/to HDFS should
$ U" L' {, `8 L5 L9 n+ L/ } be encrypted on the wire. This only needs to be set on the NN and DNs,
! B4 S1 M t' M" \ clients will deduce this automatically. It is possible to override this setting 3 m& d# I6 x3 E4 U. [) c5 \/ |
per connection by specifying custom logic via dfs.trustedchannel.resolver.class.
( ]* I5 L/ K& H$ a- t O( z </description>
/ [/ _# y8 ]' M7 y& F2 l</property>2 I; B# y0 l; H. j, {
<property>6 x0 p$ l; u6 U0 v
<name>dfs.encrypt.data.transfer.algorithm</name>* w' v e) s+ z9 n4 Q
<value></value>, q- V/ ?! W) G! }& C
<description>
8 K) L8 m; e6 ^/ M$ n( p This value may be set to either "3des" or "rc4". If nothing is set, then
, f J1 l! V8 H the configured JCE default on the system is used (usually 3DES.) It is& L4 ?- D! d R# X2 s: M
widely believed that 3DES is more cryptographically secure, but RC4 is J* m/ F7 W: F$ u9 u( }0 _
substantially faster.# i, w( d; a2 r3 }* h: Y* }
Note that if AES is supported by both the client and server then this 5 J: ^& t! n' o& w
encryption algorithm will only be used to initially transfer keys for AES.3 t* k. P% V, |$ y: j2 L
(See dfs.encrypt.data.transfer.cipher.suites.)7 b! Q- a9 }+ U( [9 C
</description>- @2 N: [6 w8 p% m7 \) v2 {
</property>$ V$ Q3 q7 r6 {0 X2 ]9 W' ]6 r4 G# I
<property>5 v6 ^& g3 Z7 w2 J: c7 n
<name>dfs.encrypt.data.transfer.cipher.suites</name>! J6 _& F8 Q/ ?5 n$ L/ ~
<value></value>. E P( C j4 a4 J
<description>! @6 ~$ ^9 n/ K9 d2 M0 k% K
This value may be either undefined or AES/CTR/NoPadding. If defined, then
! m) Z8 t x. E0 _8 b dfs.encrypt.data.transfer uses the specified cipher suite for data
) e, ^& y" x( i. z" u z encryption. If not defined, then only the algorithm specified in% s0 i3 }; Q: G. S4 z
dfs.encrypt.data.transfer.algorithm is used. By default, the property is
$ @$ V) g [! N: U not defined.
$ A* s: s4 [: `: q: @. x" |4 w3 Z </description>
) Z# t! V% D" n* a</property>
1 h0 h5 [- R9 j! m# e$ K# w E<property>
u6 y, x- U: B2 A" M6 u <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
+ T3 v: o$ \8 Z* j. b3 b4 s O <value>128</value>
* ~* }1 v' A& b- C' m5 ^. d <description>9 x6 h/ y% j4 Y! v1 Q
The key bitlength negotiated by dfsclient and datanode for encryption.
( S* l' b1 n/ w: `8 j6 g5 l This value may be set to either 128, 192 or 256.
2 \! ^* G* o9 \9 t </description>! h$ {& i9 u d: F1 {" N
</property>
% d3 {3 j6 L9 P- r! w4 ^) A6 _<property>
! ]* [* r6 l/ P! R <name>dfs.trustedchannel.resolver.class</name>5 A. X, J9 Z0 C: r
<value></value>
, E$ [" Z/ |# x8 R# t8 G# A <description>2 a' D& A- B8 {
TrustedChannelResolver is used to determine whether a channel
- y2 s4 R0 x5 b is trusted for plain data transfer. The TrustedChannelResolver is4 G# u# |' H) e- `8 H; T; s$ ?6 m
invoked on both client and server side. If the resolver indicates ) v. P! Z! _7 j
that the channel is trusted, then the data transfer will not be 7 c: y j8 a' m4 b( p
encrypted even if dfs.encrypt.data.transfer is set to true. The$ t6 ^9 W D2 z: n& u! B% @$ D
default implementation returns false indicating that the channel 7 J1 b+ ]# y+ q# S
is not trusted.
0 ^# Z6 N+ t9 n5 u8 c </description>
" j. v5 q% {! o5 S- `</property>
& f1 v( U. |- E1 y; u9 I<property>/ L3 g# o7 @! K3 K, ?" N
<name>dfs.data.transfer.protection</name>% n8 A* t% G y4 Z% p
<value></value>
4 E. X; |7 e/ ^3 U4 W U+ m/ E: V: _ <description>6 o C, X6 z6 H0 `- L0 g+ x0 q( u
A comma-separated list of SASL protection values used for secured0 e- \' N* k0 p8 P% p7 o& a
connections to the DataNode when reading or writing block data. Possible
3 T: \. v* l6 ^ values are authentication, integrity and privacy. authentication means, c# E; m- k/ ?. z! P
authentication only and no integrity or privacy; integrity implies
+ N1 H' f. q+ I+ N+ g authentication and integrity are enabled; and privacy implies all of1 I) q( @: [' C2 c
authentication, integrity and privacy are enabled. If5 A) F$ ~) g( p) u" x+ k
dfs.encrypt.data.transfer is set to true, then it supersedes the setting for* u; I8 ^2 J0 H* T* O
dfs.data.transfer.protection and enforces that all connections must use a
; { W1 D) |, c0 I9 V3 b( b specialized encrypted SASL handshake. This property is ignored for
4 a5 s, g7 e+ G. r8 e; B connections to a DataNode listening on a privileged port. In this case, it
( { O0 g' x( W: `9 a is assumed that the use of a privileged port establishes sufficient trust.! V Y3 ~& H1 Q* x
</description>$ O& U3 E; i! O H* C# D% G7 m
</property>* t& y4 h; ?- u D: p5 ]
<property>
, n8 u1 H1 C$ j8 k& ?9 a <name>dfs.data.transfer.saslproperties.resolver.class</name>' [. W7 d; x8 H
<value></value>
* u) H, J1 h. l9 g/ d <description>; g: i& }/ C" h$ r O' O+ F
SaslPropertiesResolver used to resolve the QOP used for a connection to the) b" G6 _2 D( g* }0 l b+ P* g
DataNode when reading or writing block data. If not specified, the value of) k# ?, q$ Q, |
hadoop.security.saslproperties.resolver.class is used as the default value.
4 ^# ]& S; E& u' L5 g# M- K </description>; g# q3 e0 K. r) D' y* A+ p
</property>
6 J) D0 L6 A4 ?; d8 S* _- u" t<property>
- S5 F; v6 J; p) M- J A/ ~1 z6 ^ <name>dfs.journalnode.rpc-address</name>
1 D2 h- j; Q3 ~ <value>0.0.0.0:8485</value>! \$ I4 g( W( w6 _" K$ H3 q$ v
<description>
! V! m6 M: @: r+ ~' W4 u The JournalNode RPC server address and port.
# l: L$ ^5 B! T' ]) V </description>$ L! J) a$ k. {$ ^ t3 T. k
</property> j8 N9 N) W4 N. M: o1 x8 K
<property>: |5 t6 Q5 r1 g `
<name>dfs.journalnode.rpc-bind-host</name>. Q6 T/ d- \3 x9 R
<value></value>
( L- Q2 O& }* R& {' A8 S/ q1 B/ y, M <description>$ }- j. R" |: `$ G: q4 |0 y5 p, O
The actual address the RPC server will bind to. If this optional address is4 s# r$ h9 w h# ^
set, it overrides only the hostname portion of dfs.journalnode.rpc-address.
9 `& ]. H) A% X. L4 C This is useful for making the JournalNode listen on all interfaces by, S, O0 J9 X2 J- D1 p, P
setting it to 0.0.0.0.
6 m' M) W" x A4 z9 \! C </description>3 `- T6 u1 w0 N4 D! @* H6 [$ d
</property>
$ V3 Y+ f8 i8 y0 O+ I' @<property>& U" b9 l- _, y* K
<name>dfs.journalnode.http-address</name>5 l# F9 Y% s1 m4 L H4 d1 N
<value>0.0.0.0:8480</value>
$ ?4 q: `5 g. E6 {: D5 V& f <description>
: r* a4 g8 X/ U, E The address and port the JournalNode HTTP server listens on.
5 F* U+ S3 H8 X If the port is 0 then the server will start on a free port.& B; x9 U5 R1 e- w/ F
</description>7 G% h( S4 S) B Q: V; S) t3 O
</property>
# G9 a4 j6 X) w5 G9 H- @<property>
+ w# ^ }( y; f: v7 ?$ q7 J6 c <name>dfs.journalnode.http-bind-host</name>/ P9 ?( ^ y& i7 V* h
<value></value>! O2 J# B; z+ \' R4 J& g
<description>
; A: {, I; f) f2 H% Z The actual address the HTTP server will bind to. If this optional address+ _( {3 S, p& }' ~* ^% d
is set, it overrides only the hostname portion of
. [9 L: P; C9 d" k1 Y( ]8 G dfs.journalnode.http-address. This is useful for making the JournalNode
' W0 K8 o6 S3 G/ j/ B5 D HTTP server listen on allinterfaces by setting it to 0.0.0.0.& S* s) O% p- ?" S5 L" |) Q' C
</description>6 T& i! @# R8 y3 O: P) t
</property>
# N5 `# ~, ^* W1 ]<property>
; @. z" o* ^& h( @3 t <name>dfs.journalnode.https-address</name>8 k# c Q' ?" X( ?+ E
<value>0.0.0.0:8481</value>
: M; V7 m5 I, v0 r8 f: g <description>
: b' |( l* S N The address and port the JournalNode HTTPS server listens on./ _; U% W s3 M! y R
If the port is 0 then the server will start on a free port.9 ^& P/ { d5 i+ Y2 H2 n Q3 S
</description>' P* j9 q4 g! [$ q1 }
</property>1 W' V, N4 R- O. C$ H& h9 K
<property>' |! m& E, E% V, F0 e6 y
<name>dfs.journalnode.https-bind-host</name>
! i8 O2 | S- M/ \8 | <value></value>
+ q& Q; G' Y; k' d) E <description>2 w( h; I( _, z! {3 @3 j
The actual address the HTTP server will bind to. If this optional address
) _" P% N, `0 C is set, it overrides only the hostname portion of/ O. Q4 t& z+ Q( c K
dfs.journalnode.https-address. This is useful for making the JournalNode# C, Z; t1 G: L1 O3 Q4 g& x' N: l" m
HTTP server listen on all interfaces by setting it to 0.0.0.0.0 X5 Q8 F* }* ] i0 D% f+ p
</description>! Z8 b4 @8 F, m1 ^: |9 [
</property>5 Y1 V" c1 D) i& e9 T
<property>8 u E# v8 A4 v v; |4 U
<name>dfs.namenode.audit.loggers</name>
' R7 s6 u9 d& g <value>default</value>8 Q/ ]6 e2 U" E
<description>
' ?# {* }0 G) N, a8 H List of classes implementing audit loggers that will receive audit events.! x3 _4 H4 ~" v4 Z8 E9 q: v, E K
These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger.! ` Q _& J- V; S- H" R
The special value "default" can be used to reference the default audit
: T' m3 [% h8 v" Q. ~6 P' F" y logger, which uses the configured log system. Installing custom audit loggers
2 g4 N( y6 T2 x0 j$ N) X may affect the performance and stability of the NameNode. Refer to the custom7 }1 t# z" I/ w& N7 h* R x- ~
logger's documentation for more details.& A6 a, }& x9 P% C. j( c
</description>' F8 n2 f U% @' X! y- s
</property>
( ~" {2 c# u5 q; s- w. e<property>' _$ J7 ]/ {8 p u7 j
<name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name>& B( c( F% S* T2 I3 u3 M
<value>10737418240</value> <!-- 10 GB -->
1 ]: D- x* {/ w <description>$ n7 F$ _% R6 o
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
8 n, s% B) z9 K org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.3 @# X+ E/ N. p0 W1 X& G
This setting controls how much DN volumes are allowed to differ in terms of
2 c; E, A$ p9 }* Y: J9 @ ` bytes of free disk space before they are considered imbalanced. If the free$ b; z- v# `- u
space of all the volumes are within this range of each other, the volumes
I9 o; \- |- y4 E. ^3 x will be considered balanced and block assignments will be done on a pure, h1 x7 R; Y6 I: ?7 n E# C3 t& k. r2 f
round robin basis.
% u! D$ `6 {- V2 [. h </description>
+ n# D# f9 D; Q# D f1 H</property>- E7 J& e& p9 t- ~6 u. K& @
<property>: U0 t) D1 Q Q5 Y
<name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>- R% W% O9 D% E
<value>0.75f</value>) P0 T* R/ N' K, n
<description>
+ e2 G: s; ^! v$ _( V0 F. H' _ Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
3 W _7 s, a1 W$ s org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.( D0 i2 f+ y: p" g4 c% g) Q8 P
This setting controls what percentage of new block allocations will be sent1 H5 p0 {/ y% E9 _5 t8 g- w9 z
to volumes with more available disk space than others. This setting should
7 w# r# S. n4 ?* `9 t/ r; Y3 P be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should
- h7 F, T) U: Y- R6 P W be no reason to prefer that volumes with less available disk space receive$ @! w, y% B0 t" t2 F& r4 @
more block allocations.6 p+ L. p% Q3 C
</description>+ g) H' ? x* T- D& K6 A$ Q* E
</property>4 c" x! n% p6 M4 J( l' @7 D! s9 y
<property>
( E: b' A6 J. R$ s: [2 ? ^ <name>dfs.namenode.edits.noeditlogchannelflush</name>8 B8 n5 |2 \# q- G- C# A0 E
<value>false</value>
7 F6 k3 x4 X: v; F5 I <description>
' s/ r( u* w2 ~' _ Specifies whether to flush edit log file channel. When set, expensive6 g9 @5 e# I) e/ `4 l$ W# ^: Q
FileChannel#force calls are skipped and synchronous disk writes are
, Y- l! S% D+ Y- H! H8 r0 U4 V enabled instead by opening the edit log file with RandomAccessFile("rws")/ ~3 s) N+ x( M& z1 v
flags. This can significantly improve the performance of edit log writes
( V% W8 Y% A3 [0 l% g) B on the Windows platform./ n9 d( m7 ]4 L7 [% q+ b
Note that the behavior of the "rws" flags is platform and hardware specific
1 e" U/ F. D3 c and might not provide the same level of guarantees as FileChannel#force.( O/ d3 L* d1 X, Z5 B0 ]
For example, the write will skip the disk-cache on SAS and SCSI devices3 ]; s7 b) A$ a; ?% j( {
while it might not on SATA devices. This is an expert level setting,2 v( w. h/ r: k5 a& H( q. Z6 K
change with caution.7 D# U, Z, a4 v) h1 V0 R
</description>0 B6 }: I' W5 a9 X/ X# W3 y1 y
</property># Q& F" Q9 R5 o+ I" f4 O
<property>
4 u. Y$ a- s9 Y% g. h- q$ X <name>dfs.client.cache.drop.behind.writes</name>0 } W8 z! C! w6 m/ M4 a
<value></value>
3 S# ]& Q$ I. U. }% [ <description>1 Q4 }* `! F0 h! o3 _4 w2 l4 N
Just like dfs.datanode.drop.cache.behind.writes, this setting causes the
/ m' l; i/ a J7 s page cache to be dropped behind HDFS writes, potentially freeing up more
5 C: B l' A5 F* V8 M; K! Z9 l memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this
9 S' y0 J0 A; Z9 n% o! ? N is a client-side setting rather than a setting for the entire datanode.. b0 ]* V9 k$ C+ r D& Y
If present, this setting will override the DataNode default.$ h4 m! J) V( G% H: f
If the native libraries are not available to the DataNode, this
7 v* n) n3 P' P6 p$ J configuration has no effect.. f0 s \% h' ^/ o3 i4 h, W
</description>
9 O, N% [/ B3 j; _8 ~, w; {</property>
6 p: G$ U b. l* ?<property>
, ?& V# _& q# P0 S& ?, D7 A$ [5 }+ P <name>dfs.client.cache.drop.behind.reads</name>
- N; y8 G, \5 M* N* f! s <value></value>
: M, L1 Q0 c8 k6 K <description>) X$ S+ S% ]- P8 Z R' g* ]
Just like dfs.datanode.drop.cache.behind.reads, this setting causes the
6 u W4 J' [. k* w" h page cache to be dropped behind HDFS reads, potentially freeing up more# u5 t5 Y) \2 Q9 m% o) @' A# l
memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this
1 `* g+ I2 w9 R* m' H is a client-side setting rather than a setting for the entire datanode. If
5 F" H8 e( p1 x, G/ M present, this setting will override the DataNode default." E4 B8 n! ~ t
If the native libraries are not available to the DataNode, this/ I6 X1 C; ]& p. `6 P
configuration has no effect.3 N% ^" Z9 f7 k* d. s2 z6 x- n
</description>, S, B K9 k7 b# f
</property>: N+ f- G% f' q3 A& U0 |/ V) w
<property>3 m4 @8 C+ G: f1 F; _
<name>dfs.client.cache.readahead</name>; H9 v; } r1 o% k% r( U7 f
<value></value>& r. n0 B' E) a2 H8 C" B/ l
<description>' d( r5 ]% S# j0 [$ H
When using remote reads, this setting causes the datanode to0 p! `/ {% C6 U: W$ w5 Y
read ahead in the block file using posix_fadvise, potentially decreasing
+ V6 G$ T% B/ O( Y* {$ Q+ A I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side
! u4 Q1 \/ h* b c setting rather than a setting for the entire datanode. If present, this
8 L8 j# |$ M4 ^5 q, q5 Z setting will override the DataNode default.
4 ~5 j9 @/ J! A9 q) C" p* I! n When using local reads, this setting determines how much readahead we do in
+ {, y/ H1 z9 I% U, k# C" ^ BlockReaderLocal.
: G9 t) }; S5 W$ m. t/ e If the native libraries are not available to the DataNode, this& U5 ~5 n9 q, ~) F" M
configuration has no effect.
H |2 V5 W) |8 c$ [ </description># V0 Y( [3 Z3 _( ^
</property>
; W( A4 D- P5 v# [% o) y* s<property>' G" A. [' l; o" x$ E; X9 F$ ?
<name>dfs.client.server-defaults.validity.period.ms</name>" b+ @0 ?! i! V+ s
<value>3600000</value>
' N! ]) p* H0 `- f8 q# J5 `; L <description>. f/ t( D- n8 y7 ~
The amount of milliseconds after which cached server defaults are updated.
& k) d1 U, M/ a |8 e' E By default this parameter is set to 1 hour.
( c( F& p( ~# q6 W) _& \8 M </description>7 |0 q+ d" f' a! I6 l7 L
</property>
; p7 O- n5 t# Y# N" c<property>
* n1 w. J/ h+ Z/ l* S <name>dfs.namenode.enable.retrycache</name>* I9 B& g9 q R: v$ N: U( `
<value>true</value>
% @" N* r k0 a& F+ k; T) }, a: W* w <description>
" o2 `3 x. ?8 R This enables the retry cache on the namenode. Namenode tracks for
* h" ^7 S' P3 s `; V7 } non-idempotent requests the corresponding response. If a client retries the/ D5 v% B4 a' B$ D/ f, N6 v
request, the response from the retry cache is sent. Such operations; v- b3 F$ y; ^; w
are tagged with annotation @AtMostOnce in namenode protocols. It is! W G4 G% P4 l4 L$ [
recommended that this flag be set to true. Setting it to false, will result
6 t- {1 e8 Y4 y# } in clients getting failure responses to retried request. This flag must
" t, y. f) ?& X8 f% F Q7 _ be enabled in HA setup for transparent fail-overs.. |( s1 x4 N, |: s- z, h
The entries in the cache have expiration time configurable0 H d8 j+ }- Y( t
using dfs.namenode.retrycache.expirytime.millis.
7 r6 |, _4 H& P7 q </description>
& g- Q+ J: l- u5 F) q7 S p</property> `6 }8 e1 r7 j, f# T/ `5 e1 @
<property>
4 \. {$ J- v' n$ G1 {2 m2 ` <name>dfs.namenode.retrycache.expirytime.millis</name># P1 K' c+ v8 g2 Y' u6 ~
<value>600000</value>6 f8 p! q! ~, e5 Q
<description>) q' C( O6 l6 O8 s
The time for which retry cache entries are retained.4 N! d9 a, {- |, `
</description>! V8 |( W- ~7 l6 s8 T
</property>$ N% r2 e4 z. F, a
<property>% N0 D/ E# r+ z2 C
<name>dfs.namenode.retrycache.heap.percent</name>( r" d9 c Z0 {8 Y. x v
<value>0.03f</value>! h6 @9 |( W c. U- H
<description>
& m0 C d0 w. L5 k+ r This parameter configures the heap size allocated for retry cache
8 m3 S) B) W# T' {% r9 Y8 S+ } (excluding the response cached). This corresponds to approximately
" N6 { o; |" t, @ 4096 entries for every 64MB of namenode process java heap size.+ p% s2 J- T. S7 L! v
Assuming retry cache entry expiration time (configured using3 C4 w: B* o; ]
dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this
& O3 `% `1 T( ~0 ~' B3 i: I enables retry cache to support 7 operations per second sustained8 R9 c+ H9 x: s7 p: O( J3 K
for 10 minutes. As the heap size is increased, the operation rate3 q5 Q! v G3 N& E
linearly increases.9 f J2 R6 S$ c; q9 l
</description>
6 Y r+ P3 V) x2 y! y5 w</property>/ v2 W& R9 y0 Y6 B7 E
<property>
$ k- L/ Z' i G7 m6 y' h <name>dfs.client.mmap.enabled</name>' M# m" X- X& C% d/ j$ U
<value>true</value>
& Y; M& c5 l2 P* g <description>0 k* m% |2 Q8 y+ ~/ i" Z
If this is set to false, the client won't attempt to perform memory-mapped reads.
# L+ u/ `+ c1 X" K0 f: O </description>
2 N* u. W2 i# [+ b* G</property>
, I- V* p3 S; G9 u& O<property>1 i8 _6 J( [" V( e! [4 A
<name>dfs.client.mmap.cache.size</name># u/ h! a. J& v. |9 x
<value>256</value>7 s: s+ l! C5 q/ m
<description>7 g F p/ L% h# J/ f! [1 I
When zero-copy reads are used, the DFSClient keeps a cache of recently used
4 ?6 h2 f6 E8 n' R p) p$ t memory mapped regions. This parameter controls the maximum number of
$ E) i* s R T# c+ g9 p! Q9 T entries that we will keep in that cache.1 }( E! ^9 S4 l: @/ o" T3 t+ c
The larger this number is, the more file descriptors we will potentially: K2 d Y" E1 E
use for memory-mapped files. mmaped files also use virtual address space.0 K: b# g4 z" L# t+ s5 K0 ^/ i9 w
You may need to increase your ulimit virtual address space limits before
* u8 Q7 [5 Y$ z w! K increasing the client mmap cache size.
! S' w' F; h' |% C0 E3 Z) m+ r Note that you can still do zero-copy reads when this size is set to 0.( p6 C6 A; R; v/ @0 ]' R. o Q( Y& X( d
</description>; w0 s3 x: k0 N4 c0 r
</property>8 ^" ^9 v! W6 Q0 F
<property>
" j7 `. \: }( N' i9 k- @% U- d <name>dfs.client.mmap.cache.timeout.ms</name>% {* z8 y% h* Q7 B, H
<value>3600000</value>
/ d; m4 k1 ?# y( c" V6 D! X <description>
+ n! x2 \" V1 @& T The minimum length of time that we will keep an mmap entry in the cache; I- o1 {0 J) h+ P% V
between uses. If an entry is in the cache longer than this, and nobody
1 {+ Y! d; T2 X4 i' O% S/ T0 { uses it, it will be removed by a background thread.7 Q5 e* H3 G! A9 m2 L+ |5 q% I1 j
</description>5 |/ a7 Q7 {2 ~( Z- r
</property>
" A( `* C q( C* o<property>, Y- j* g3 v7 m
<name>dfs.client.mmap.retry.timeout.ms</name>
$ r- D" s3 g2 D# q1 _7 ]& f# Y' m <value>300000</value>
- e! k5 ^# L; } <description>1 g/ \* n$ f9 l2 G# [' A% _1 O* B
The minimum amount of time that we will wait before retrying a failed mmap# O4 P; s' }& d8 N3 j5 O% o: g
operation.4 c" }3 y# ~, {( N8 P
</description>
6 r, \) A4 J6 i: m5 n</property>. r% j7 |4 G% O& l! E! f
<property>& E+ ?7 |1 C; g
<name>dfs.client.short.circuit.replica.stale.threshold.ms</name>
2 g% v ~# |7 v. s- O/ u <value>1800000</value>
( G/ I: i% L& h4 t3 R <description>' i v8 D5 m! e( z; A. ~
The maximum amount of time that we will consider a short-circuit replica to4 k: N9 M$ m" r+ l, m' h7 N5 P
be valid, if there is no communication from the DataNode. After this time- Y4 k- h7 T x9 W6 U/ ]0 n' l
has elapsed, we will re-fetch the short-circuit replica even if it is in5 z2 a' S/ s' R
the cache.
) i7 m9 u0 \& \: E+ } </description>" }+ F) i4 f3 D0 Z
</property>
1 E: G. G, M* z1 h% s<property>
4 r2 @! Y6 S# |8 q. B <name>dfs.namenode.path.based.cache.block.map.allocation.percent</name>2 g2 U t* _+ ?
<value>0.25</value>
" _0 _+ \4 M/ J" D1 B) C/ N, K5 C <description>1 H3 m; ^" x, y# T. D
The percentage of the Java heap which we will allocate to the cached blocks/ k D1 h( t/ u7 ^3 O
map. The cached blocks map is a hash map which uses chained hashing.
. G! b( Y- r7 C. S( o Smaller maps may be accessed more slowly if the number of cached blocks is
5 ?, O* N* h! I9 |+ \4 v5 d large; larger maps will consume more memory.1 W1 F$ k" w& J; \; g* r6 c- H' r2 Y
</description>1 P4 ], @: u9 U. {4 j$ Z
</property>: ~$ `( l4 a$ Y! c+ f. C
<property>1 U# B6 ]7 s6 K' r* M2 h+ k! p
<name>dfs.datanode.max.locked.memory</name>; o/ F& @# p- o% e* M
<value>0</value>
$ H; w: h' ]! o <description>3 }. G W3 V, Y1 B
The amount of memory in bytes to use for caching of block replicas in) {! R4 @8 |( U9 q- q- p
memory on the datanode. The datanode's maximum locked memory soft ulimit+ {, I* N1 _6 S7 K; j
(RLIMIT_MEMLOCK) must be set to at least this value, else the datanode
( z3 c3 |; o1 W( l% F% o will abort on startup.# t9 M+ F5 h9 O) L$ P H5 n
By default, this parameter is set to 0, which disables in-memory caching.4 B9 J# |+ ^: m* a
If the native libraries are not available to the DataNode, this, U4 h4 I& u5 Z! Y
configuration has no effect.& K* k8 {. Z' O" g) j
</description>* e- ]3 R2 n6 x9 m$ \
</property>6 _4 K3 g( B' d( X% {* m
<property>$ Y4 ?, e0 d, A5 Q# l4 [
<name>dfs.namenode.list.cache.directives.num.responses</name>
' m) D8 k' B& `/ d; ^9 h! ]% h, \5 P <value>100</value>/ m8 d; M( J9 P3 I+ ^1 Q
<description>* O! e7 A$ R/ [7 Z
This value controls the number of cache directives that the NameNode will3 H4 ?; k+ G& V9 N# K4 b" I. t
send over the wire in response to a listDirectives RPC.
2 U5 H" I% q4 L% s </description>
- ?) ^- |# Y" \- [/ G& W# K% ^% @+ u</property>5 O, @6 D, g; c+ G# |
<property>2 A! w& _/ t7 l/ G. J' c
<name>dfs.namenode.list.cache.pools.num.responses</name>) ^3 Y' N" [ {5 u0 [
<value>100</value>, [, t. k' [* T. y6 h3 A. y
<description>5 g1 _7 K4 k, ]; B* z% Z
This value controls the number of cache pools that the NameNode will0 l1 @( a9 ~/ T. ^+ j: M. F
send over the wire in response to a listPools RPC.
* e7 |: q, ~: B+ p ? </description>9 H0 g: w' _4 b
</property>$ S" k8 I9 ]5 i, h7 l' t i" r
<property>
$ e" O3 C1 \% m <name>dfs.namenode.path.based.cache.refresh.interval.ms</name>2 B2 H' K0 e Z9 N9 R0 j
<value>30000</value>
f- w# a8 n2 H/ m2 t <description>9 Q" H- Q3 p% u
The amount of milliseconds between subsequent path cache rescans. Path
" b- C6 p$ w( o R9 k cache rescans are when we calculate which blocks should be cached, and on1 f+ I' M4 K% f1 {+ k. b
what datanodes.
% Q6 J# V z v; Q2 `- G& l By default, this parameter is set to 30 seconds.2 n" D( X4 r0 i3 |
</description>" X- ]( d3 q) O, `5 f1 g- s
</property>: h! _, D! ~1 A, R1 i% h" D$ h
<property>/ y: H. V+ W. B/ K( J
<name>dfs.namenode.path.based.cache.retry.interval.ms</name>
% w4 P( b. J& t" z, {! \ <value>30000</value>
1 I. s0 R8 v; x' k2 K <description>" e& ]( t9 p0 d/ M( F7 f
When the NameNode needs to uncache something that is cached, or cache. D" w6 n/ p% |: P
something that is not cached, it must direct the DataNodes to do so by4 v. t R! A9 e: T$ E
sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode
. m' j& a4 s+ S; b* R( _6 u- U heartbeat. This parameter controls how frequently the NameNode will
* @9 x: V) r5 E) e4 D- @ resend these commands. L) `# P9 B* K) H4 R. N G
</description>; r; g) @# k$ {7 ?$ i& o
</property>
" P5 b3 o1 [$ v5 E2 A<property>' j9 |0 U! {4 X5 ~
<name>dfs.datanode.fsdatasetcache.max.threads.per.volume</name>
0 F: f6 e7 b( a <value>4</value>
0 ` v7 v' a6 z: K; n1 i4 l <description>- O4 j! p ]+ g! [
The maximum number of threads per volume to use for caching new data: K$ K2 n6 Y! A' _- B9 Q7 r
on the datanode. These threads consume both I/O and CPU. This can affect5 l3 K. j0 v) q0 A) P2 ]
normal datanode operations.6 w! A; `. L: c$ n. X
</description>. e1 \: j5 v' A1 S6 T0 `
</property>4 n0 S& m; c' f. X3 W& k) M8 ]/ Q6 l
<property>
( ], k E# l7 Z9 J1 V+ r/ A5 F! P <name>dfs.cachereport.intervalMsec</name>8 L; m, p1 B5 X# r- z4 c
<value>10000</value>
, v4 h! @* }- a <description>+ s/ ^ A1 y4 z
Determines cache reporting interval in milliseconds. After this amount of
2 n$ b+ w1 R$ f% d5 y. |" F time, the DataNode sends a full report of its cache state to the NameNode.
1 x- @- z6 @8 ^' f4 g, o' q The NameNode uses the cache report to update its map of cached blocks to
# i7 v3 j' g2 ^, x5 N4 T: W! }8 { DataNode locations.* O& q2 D4 \) F
This configuration has no effect if in-memory caching has been disabled by" B8 k7 {: r# S% W/ k
setting dfs.datanode.max.locked.memory to 0 (which is the default).7 N5 f+ p$ z/ v3 p% Y8 x
If the native libraries are not available to the DataNode, this8 Y8 P L$ {% J' G8 ~# o8 K
configuration has no effect.5 {, E$ W/ _- i9 S# A; n2 E! A
</description>
! m6 v; x( U( D7 S( S</property>9 w5 W- l. v; W( p2 I# f: n: y
<property>& R I% ]0 t+ {4 ?0 Z
<name>dfs.namenode.edit.log.autoroll.multiplier.threshold</name>
- D& G+ v6 x/ a: r' s$ v <value>2.0</value>: o, F, o' u1 `% j
<description>+ b5 C# v8 |* E1 |# k6 r
Determines when an active namenode will roll its own edit log.
0 n% D* X. C! G0 r The actual threshold (in number of edits) is determined by multiplying+ ?0 U( ^9 A0 w# x- u1 U# y3 S+ W
this value by dfs.namenode.checkpoint.txns.
( K1 l5 M* @6 h4 Q1 d This prevents extremely large edit files from accumulating on the active6 f/ M6 _( n3 ^
namenode, which can cause timeouts during namenode startup and pose an
E5 {( c; {- ]' R5 m administrative hassle. This behavior is intended as a failsafe for when5 w2 @' Z+ k6 _- e
the standby or secondary namenode fail to roll the edit log by the normal
9 [/ D" h" k5 D8 o checkpoint threshold.
8 J. g$ b7 d3 e2 h6 \$ |1 s </description>8 r- K. f+ ]7 p- Y6 c
</property>
) ]/ V, O( w: I/ T<property>
) h1 f8 P! q* V* D! } <name>dfs.namenode.edit.log.autoroll.check.interval.ms</name>
" v! C7 e# I3 S2 [- C) _/ p' _4 M <value>300000</value>
% ]. p5 g/ R; u- E6 z1 g <description>$ _; U: X8 Z4 ~1 h. L1 o5 x
How often an active namenode will check if it needs to roll its edit log,
" u4 n5 N1 `$ \! Z V a( N in milliseconds.+ ?" [! u0 D. p7 i
</description>
; t* N* A6 P( x Z</property>$ i% \& u- X* i4 \/ p
<property>6 b6 U" S5 t8 B2 ?# l* ]! f
<name>dfs.webhdfs.user.provider.user.pattern</name>9 L S; z$ ]5 J- V
<value>^[A-Za-z_][A-Za-z0-9._-]*[$]?$</value>
- O$ F3 R6 r- g. v <description>
3 l4 k; j2 ^( x G8 D Valid pattern for user and group names for webhdfs, it must be a valid java regex.
# s1 |% Y1 [& E$ m1 c" s5 y </description>
; I E. V) y' d2 `+ ?- z: D</property>
& H, K: N; H. F% ]0 d0 ]- O. }<property>
5 v% L" K: a( R. H) { <name>dfs.webhdfs.acl.provider.permission.pattern</name>7 h, D% ]9 F" d' X5 \" n' ~1 b
<value>^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$</value>. B9 x/ E" g( c: ]% x1 }2 ^% X' u
<description>
9 U6 S( x( Y+ ~3 @* W5 @+ _ Valid pattern for user and group names in webhdfs acl operations, it must be a valid java regex.' S* \* {( a. C9 A) h+ e! C* m# ?
</description>
0 `" b3 l2 V( M6 n) G ~+ l</property># ~. p! t" Y. H' R3 f$ Z b% u
<property>
% S( C }+ a+ l5 B0 f6 W <name>dfs.webhdfs.socket.connect-timeout</name>
. U4 {+ U& W& w. T9 N <value>60s</value>
# Q+ r2 M; m, x) |3 p <description># j7 O# Y, f: M* y; q4 v5 U- {
Socket timeout for connecting to WebHDFS servers. This prevents a
" u5 r& v& ^- k: x: y3 W# S: ? WebHDFS client from hanging if the server hostname is% N8 _& O6 i5 v& {
misconfigured, or the server does not response before the timeout" f3 }% G6 K" n- s, ~
expires. Value is followed by a unit specifier: ns, us, ms, s, m,
# ?; x# X/ G( l0 J8 F h, d for nanoseconds, microseconds, milliseconds, seconds,
" v0 Z( u0 w) g. s3 H minutes, hours, days respectively. Values should provide units,! U! [, b3 s7 l
but milliseconds are assumed., @5 N+ e- b) e- t+ k% P
</description>
3 w; X$ l: x# D1 \! f</property>
7 ^- O- ~5 ^+ Z/ t$ f3 w$ p<property>
2 D9 W' F/ g+ j3 Y1 B/ q2 D <name>dfs.webhdfs.socket.read-timeout</name>' i/ Z5 d0 W6 x+ v; ?4 l }
<value>60s</value>3 Y! \1 s" n4 v5 D6 Z% t) U# N
<description>
( j0 }0 M* l; Z8 K" q7 q5 Q Socket timeout for reading data from WebHDFS servers. This
2 x3 Y8 a @# c! r4 _ l$ v prevents a WebHDFS client from hanging if the server stops sending% u) ]; K, _* ]# \+ y/ p3 T) s3 M
data. Value is followed by a unit specifier: ns, us, ms, s, m, h,( m, x/ g5 d+ P7 f! J$ V
d for nanoseconds, microseconds, milliseconds, seconds, minutes,7 x' p& L l- K9 l- O3 |! q
hours, days respectively. Values should provide units,
+ Z2 m" J6 `% P1 D8 i but milliseconds are assumed.
4 c3 z* a& x3 @7 A* _+ P </description>
. E) u: [2 g7 B3 f5 I/ m" w H</property>5 y3 O. Y+ c2 b+ N
<property>
. H; ]6 [: X* g/ Q7 I <name>dfs.client.context</name>/ ?6 q4 Y( }' z
<value>default</value>) ~( W$ Y. _% E( R/ _
<description>5 n5 x0 D, x3 j5 V! U% }
The name of the DFSClient context that we should use. Clients that share; [5 x/ q/ V2 m- M$ K- R* ~) N
a context share a socket cache and short-circuit cache, among other things.; V! U/ m# Q- {
You should only change this if you don't want to share with another set of6 N6 J! F/ i4 Q; |$ S; I
threads.
4 E' F& y, c3 t& ^: `3 m </description># }8 m* g: v8 c% Z' o
</property>
( o3 L. h* L: W<property>, F$ I* I: o* l0 Q- B" Q
<name>dfs.client.read.shortcircuit</name>
^% a: y6 e9 l: e9 E* \( F" P <value>false</value>, N, o. G# u2 J9 W( A
<description>2 S# Q3 s% C' V% q
This configuration parameter turns on short-circuit local reads.
& G: D) N6 j H </description>
( ^) z" l9 B( {. T* A</property>
; l( ^0 d9 a3 U<property>% Q3 K: y5 c5 N0 S5 X* A* u( b
<name>dfs.client.socket.send.buffer.size</name>0 ^; X) O; o# Q) @. n
<value>0</value>
! E; G3 n; C5 t, R. V G <description>
" W# T) G- a) w: Z Socket send buffer size for a write pipeline in DFSClient side.; J X' k q3 f* `
This may affect TCP connection throughput.
8 k( ?$ v& h2 H! I: b If it is set to zero or negative value,1 j7 ^, e2 o! Q) j$ ^
no buffer size will be set explicitly,
# ? K" y9 v* ?$ {# ?6 h* f$ }( s( ]0 d thus enable tcp auto-tuning on some system.) \/ J7 M# r8 _& u# B! v* r
The default value is 0.8 R) ~, c* N; P( }2 o3 z
</description>
. h; U8 V: ?1 J7 R9 L</property>
0 ^, R( ~: L2 O<property>7 Z5 ?# ~! \- e9 t( t. R
<name>dfs.domain.socket.path</name>2 W8 D! l3 G% l4 i+ A
<value></value>6 k( f: O1 p4 @7 j& X
<description>
/ [) ?3 z5 V4 X. j4 s4 S Optional. This is a path to a UNIX domain socket that will be used for
1 v3 J, j5 y8 Z, r, y4 d. d communication between the DataNode and local HDFS clients.
# H. h1 b$ j& [, u If the string "_PORT" is present in this path, it will be replaced by the3 ~' A J1 _7 H" |6 e1 {" q. f
TCP port of the DataNode.
2 Q, B3 K1 Y4 P& D9 {0 A </description>
" [7 d0 U' b# K& @* v" R+ }3 U6 j</property>2 m3 s/ d0 o9 |' d) x1 ^
<property>, z. N7 q8 P3 t+ j$ [3 ?
<name>dfs.domain.socket.disable.interval.seconds</name>) s6 X" y8 q+ e
<value>600</value>
7 ]! f4 @: T& g% w9 I <description>- T: U& I; A5 f1 h* k
The interval that a DataNode is disabled for future Short-Circuit Reads,9 @, H6 ~4 c6 P5 _! Y3 W
after an error happens during a Short-Circuit Read. Setting this to 0 will2 X4 A* f* F% K$ @' A A2 F
not disable Short-Circuit Reads at all after errors happen. Negative values, G$ E) ]* f& K5 L
are invalid.
) `; Q4 R9 I1 d/ K: R3 m </description>' V/ a4 N% u3 `5 W2 z: K
</property>
7 ^3 L/ ^8 [8 u% |7 A) {2 L+ ]<property>
0 D6 u4 O/ N9 g: z- O <name>dfs.client.read.shortcircuit.skip.checksum</name>
! R a% ]' w/ r8 Z, [ <value>false</value>% {$ E; Q7 q; X+ {% W; {
<description>
2 B4 k1 z: B2 t$ |' x6 R If this configuration parameter is set,& U5 m8 V5 k7 |1 d1 R9 Z4 k
short-circuit local reads will skip checksums.& ]9 M# L( E8 W" i: @7 s
This is normally not recommended,
% L$ ]( T( t4 N7 E' Z7 W but it may be useful for special setups.
, B& b) n7 c# {# p) H8 _ You might consider using this
3 v( e# O6 j5 R; [- y- N if you are doing your own checksumming outside of HDFS.
8 a" P* O* \2 w o5 Q </description>0 C+ a* T, E' B3 V4 j5 f
</property>+ B' V H8 g k- Y6 ?
<property>* }) g5 K2 e4 p( _( H
<name>dfs.client.read.shortcircuit.streams.cache.size</name>. G# @: @- J& N1 T# B/ ^" ?5 B
<value>256</value>
- b9 s R6 A' |$ ^3 z <description>
7 v i! }: v. i# Z The DFSClient maintains a cache of recently opened file descriptors.
+ j8 q/ p, a6 L9 M6 V. r This parameter controls the maximum number of file descriptors in the cache.
' f# H# P+ }% Q Setting this higher will use more file descriptors,
& D' h+ x( J& O% u3 ~. s" e but potentially provide better performance on workloads; M) O! a @1 e+ {- A, I
involving lots of seeks.+ |# {' Y& M0 _
</description>
! O* L, z/ C! k2 `</property>
: n& Z& X& G/ z<property>9 x |% W, I7 |1 ]5 g1 \
<name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>) ]( R' F8 N! T
<value>300000</value>
# h- B4 ~1 S) t3 s5 F6 f <description>, k4 q' O+ }# }0 F8 z" K) x% Y/ Z$ ^2 N
This controls the minimum amount of time' }; D1 w- F+ j& w
file descriptors need to sit in the client cache context& v" O5 u! _' H4 s9 J+ {
before they can be closed for being inactive for too long.4 ]) w8 _4 G/ g) q2 ~4 A$ n5 G) `" w
</description>- { R; n! _% Y; p
</property>
( o( h" w/ t3 p. H* Q7 j<property>
/ F, y4 k3 R" p/ r4 l3 n/ o <name>dfs.datanode.shared.file.descriptor.paths</name>- G# k% b. J. w) W9 ]
<value>/dev/shm,/tmp</value>7 e3 o( e9 D) [
<description>
! d a! Z1 Q+ g; K5 |4 l t Comma separated paths to the directory on which. ?& }8 E! j' t; v# U3 L
shared memory segments are created.2 L# }& d: c. B
The client and the DataNode exchange information via
0 z* o2 v; n: C* @) `. y! v this shared memory segment.
& w' `, x3 l8 L3 ^ It tries paths in order until creation of shared memory segment succeeds.
8 N5 _ e. |. c) t+ V </description>
; w4 E! o! W; P0 X7 K# o6 s9 L</property> D2 k% L L; n
<property>
, ?! J y! Y0 d# R <name>dfs.namenode.audit.log.debug.cmdlist</name>; Z8 V7 k: V( S
<value></value>; N: ]" i! m% {$ w
<description>
# s5 ?. N& q" d- D/ r A comma separated list of NameNode commands that are written to the HDFS8 W! b" r- F' g, u4 i9 ]& E
namenode audit log only if the audit log level is debug.
& t+ o; _& s) ?. l( d! V </description>* C* x/ x: H9 e, y( @$ ]
</property>
6 t. \ ?0 Z9 I& j7 Z<property>$ Y' b, B: ?4 I- v, d5 w: D
<name>dfs.client.use.legacy.blockreader.local</name>9 `) l- _/ R* O6 |; D$ G
<value>false</value>8 f: c" f; c2 y! w- }- q1 W% g
<description>
# \- y- R+ N1 b/ |$ V Legacy short-circuit reader implementation based on HDFS-2246 is used
' s- n: r7 F1 d1 `8 H0 g if this configuration parameter is true.
" k& m* |& Z9 A4 Q$ f This is for the platforms other than Linux( ?; \% f3 g, c- v. C
where the new implementation based on HDFS-347 is not available.
9 s, J$ K* h( ]0 a7 f! U& ]( l </description>
; J3 {! X" v6 T! ^& L) N. y</property>
+ v( t* s* r2 H& s5 I<property>& Q$ R* Z( c0 V3 m
<name>dfs.block.local-path-access.user</name>( u) A, P2 l2 o, O9 \. H) |- ~
<value></value>- U2 _ u: H7 C1 \5 ^- g, U o
<description>5 ?( K( w1 |9 H( }8 ?4 a, S
Comma separated list of the users allowed to open block files
# ?( A0 `( z+ k' }; ~ on legacy short-circuit local read.% Z v, \- v& S: B+ S# V6 A
</description>
7 o; E1 g( V9 `4 W m# `</property>
3 O2 ^" {% y( N" W! \3 Y<property>4 T8 t3 u s) e2 Q- m5 S
<name>dfs.client.domain.socket.data.traffic</name>
+ P* I: w, u- Q0 ], x6 U/ W2 n <value>false</value>
7 G+ O; U& T1 p* C' R; g <description>
- ~/ X2 [0 T# n' t2 S This control whether we will try to pass normal data traffic
# _" D9 {" F; k5 A over UNIX domain socket rather than over TCP socket4 f. D' v( [5 v' U
on node-local data transfer.; B/ I6 V5 ~7 r( _
This is currently experimental and turned off by default.4 q$ a( ?& y1 N x; a6 b
</description>
5 J4 Z/ _5 D5 s</property>
/ a/ y: k5 \8 e; [! O) x<property>
6 l2 E3 g5 T( A7 q; l <name>dfs.namenode.reject-unresolved-dn-topology-mapping</name>
8 L& \% W' u# f9 x1 J) {# r* r <value>false</value>: E" C3 D" K# A2 m ]4 o. V
<description>
# ~! O% [' W* R$ _9 V If the value is set to true, then namenode will reject datanode
* p; j- v" P/ n) ^- v' t! Y( k registration if the topology mapping for a datanode is not resolved and
2 X( O0 J& g* c% ~/ @. \5 W$ m9 o NULL is returned (script defined by net.topology.script.file.name fails
3 y5 ]- P0 J& H2 G9 S7 s4 O1 G to execute). Otherwise, datanode will be registered and the default rack 8 G( l% J; T" B, t. t( ^* @7 w
will be assigned as the topology path. Topology paths are important for ' N9 z* d7 ~. [" P' X
data resiliency, since they define fault domains. Thus it may be unwanted
# g5 S S( a3 e$ e* \; n behavior to allow datanode registration with the default rack if the ' f2 V) B" X% @, K- r
resolving topology failed.
7 i; j7 r6 Y+ a. V! x/ z </description>: S6 X* U5 }6 k+ ]- m/ F" |9 _* c
</property>. [: w9 G# @; I) R! D" s6 C
<property>
& ]' z7 d- p- d! X2 {2 u <name>dfs.namenode.xattrs.enabled</name>
2 ]+ E h7 a: R <value>true</value>7 U5 D2 j0 L* V& p7 E5 V
<description>+ x/ B t. y' Z* r7 P' `) ], Q- Y
Whether support for extended attributes is enabled on the NameNode.: d& }5 q# l" C* Q6 G7 h% { e A
</description>
5 Y5 J& j' u' f! ?4 H# [$ ?</property>
. f: g F l" o2 V' d( B6 K6 z3 O<property>' f" s# M p+ T& `. ^1 L
<name>dfs.namenode.fs-limits.max-xattrs-per-inode</name>
5 u/ c4 R- @; A2 @6 x8 v <value>32</value>
8 y) X, o: C, G& S- y+ L! q <description>
+ M4 p4 b$ q8 k o# g( ?" ? Maximum number of extended attributes per inode.8 R7 O& b r. n7 K- |
</description>, V6 J! H/ y0 o! R+ l
</property>& Q; ]- B; O: w4 u4 J
<property>
/ M+ w! m# N4 t6 x+ ^1 O) I4 |1 c( R <name>dfs.namenode.fs-limits.max-xattr-size</name>
) V$ h0 @$ c) y3 \ <value>16384</value>
1 L. c0 d$ _5 L+ h1 w <description>
: n x7 z. O B The maximum combined size of the name and value of an extended attribute) i' _5 f- P P) F4 ?6 M q( \
in bytes. It should be larger than 0, and less than or equal to maximum
8 `& Q0 |. H4 H9 X' }0 z size hard limit which is 32768.3 z! Q8 \( N% U* r; r" O
</description>% t9 x8 n6 S: b
</property>
0 L0 s L! L! i; o5 K<property>* P& s/ o; @' F0 d4 g
<name>dfs.client.slow.io.warning.threshold.ms</name>
3 I4 W5 {9 V$ A: ]3 C <value>30000</value>; H6 V' i/ X9 g3 x- t; K
<description>The threshold in milliseconds at which we will log a slow
+ K8 y1 L3 v8 m5 b/ s io warning in a dfsclient. By default, this parameter is set to 30000! V* W& V) |. f7 P! S5 F
milliseconds (30 seconds).
5 J3 G- l5 n2 \2 U J </description>9 z3 O: w. m+ Y- ^2 c
</property>
6 W& T8 v, d, \/ u<property>. {; M$ G1 J* l7 m
<name>dfs.datanode.slow.io.warning.threshold.ms</name>) l C/ {( S- G+ y k3 C$ |
<value>300</value>
2 p0 w+ I/ s* m+ S' N <description>The threshold in milliseconds at which we will log a slow
! W" @' e( O/ u9 }& P' a( k io warning in a datanode. By default, this parameter is set to 300
, T7 E# Z! U# i* _ milliseconds.3 S `9 A# B- y' w. P3 \2 z
</description>
2 s$ K) K/ c K( H# g h</property>4 Z# k; }" c# Z/ m0 W' R* M( C8 E
<property>
) Y) a/ h; N, v4 N9 Y9 w <name>dfs.namenode.lease-recheck-interval-ms</name>
7 i( I( L% S7 y; E <value>2000</value>
, p& z/ M) I. u1 i# ~6 S- _4 d <description>During the release of lease a lock is hold that make any8 S2 ^, R1 ], Z7 A/ }
operations on the namenode stuck. In order to not block them during
7 [. u: C" ]7 b% j' F a too long duration we stop releasing lease after this max lock limit.
) D3 T. l. R, h# Z+ U! p& T </description>+ q! j' j0 M! i3 p P
</property>8 V; w% _5 A& N* a9 d
<property>
1 x/ v0 }: z3 p* ]7 } <name>dfs.namenode.max-lock-hold-to-release-lease-ms</name>
/ D [9 L r# N( \$ ?& ] <value>25</value>
. Y# @: g9 s6 N0 k <description>During the release of lease a lock is hold that make any o3 D# \7 f( P8 `3 _4 D9 T
operations on the namenode stuck. In order to not block them during" r* t$ ]; J$ X6 B# V
a too long duration we stop releasing lease after this max lock limit.: P9 s2 P- m! k7 `7 K; k
</description>- d1 V+ _6 k) Y7 H1 o: W
</property>
5 [- y: j" }* d3 c, x8 r+ j& B<property>* N) C4 n" @+ Q+ {3 j# ]
<name>dfs.namenode.write-lock-reporting-threshold-ms</name>; }& v3 l- O% h& ^
<value>5000</value>9 k/ e8 d6 S1 y: H2 y9 E! i
<description>When a write lock is held on the namenode for a long time,
; c5 I K" z9 n4 t2 F$ Z; h% F0 q this will be logged as the lock is released. This sets how long the# c) U; f& L1 g9 f. ^
lock must be held for logging to occur.
' y) F7 k4 x" L: Z( g" _! R- @ </description>8 I4 e# }4 f+ a7 O
</property>5 z% N! ?! x% i' @
<property>
1 t1 a+ C2 \; E; L( ` <name>dfs.namenode.read-lock-reporting-threshold-ms</name>" } [# ]1 I+ C2 |! n5 j! q
<value>5000</value>, a k( q" a/ \! e
<description>When a read lock is held on the namenode for a long time,- Y! F( D; ^- |0 Q! i* R
this will be logged as the lock is released. This sets how long the0 |5 ]& e( d3 Z+ u9 X& ?& z# I0 P
lock must be held for logging to occur.
% ?, Z1 r6 z5 O) l% S! { </description>
! B' y" N- c5 z9 f: p6 P \</property>% w7 g' n2 u2 N% H- \
<property>6 u5 b) A! t# k, j/ y9 p
<name>dfs.namenode.lock.detailed-metrics.enabled</name>
U& L/ O/ n! X& O# w <value>false</value>7 l$ ^& `/ H) L+ j7 ?6 C
<description>If true, the namenode will keep track of how long various
9 o; q+ d* }7 | operations hold the Namesystem lock for and emit this as metrics. These. n" _3 T/ I4 S, ^! g1 Z( v- \
metrics have names of the form FSN(Read|Write)LockNanosOperationName,
3 t7 r) H" a& N/ Z( R9 Q where OperationName denotes the name of the operation that initiated the
7 s: E. H% K7 \& B; I% u lock hold (this will be OTHER for certain uncategorized operations) and6 i+ t% X/ h6 E1 U
they export the hold time values in nanoseconds." h5 ^( E. ?% h6 V. t! w
</description>
4 {1 N* v+ p# Q' N: d- E! U; [9 @</property>
7 {5 l1 [) J* g {% {<property>
# }& x, j$ Y h" s <name>dfs.namenode.fslock.fair</name>
& N4 Q. l$ h7 S! W/ t" B9 O <value>true</value>
- v& b. H" q) p, X' y- H2 p& o <description>If this is true, the FS Namesystem lock will be used in Fair mode,
( d3 C2 w m! m6 B$ N! Z7 ]- g which will help to prevent writer threads from being starved, but can provide
8 ^2 p% j1 Z* K+ {& |; o; {& f lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock' |8 I8 S) y& _% n
for more information on fair/non-fair locks.7 {6 |7 G1 m0 e- V6 N7 k2 m
</description>, q3 X" _4 i* `( U9 m+ }: [
</property>4 w2 |. A, [" C9 @& M: q5 Y. w) _
<property>
9 L/ U; h, C3 k8 c5 f$ g; }. R1 X <name>dfs.namenode.startup.delay.block.deletion.sec</name>
W3 {. q; W. _& p <value>0</value>: _8 p, p, F7 x$ \9 E' c$ P
<description>The delay in seconds at which we will pause the blocks deletion
* p6 _0 ?/ @, K2 ?1 u: _ after Namenode startup. By default it's disabled.
7 d1 g' n* L5 T V9 S' R In the case a directory has large number of directories and files are
2 A1 A/ w6 x8 ^, E* C% b5 ? deleted, suggested delay is one hour to give the administrator enough time. R4 Q0 r; f4 D6 G# N0 S" X6 R. n7 @
to notice large number of pending deletion blocks and take corrective" K" m0 C& r6 S& h
action.7 B- C& I( ~9 { ~5 z6 v
</description># s- C3 [3 Z" @
</property>
$ T7 k5 Q* U9 d<property>
+ z! C- X; x, K$ l( k" h <name>dfs.datanode.block.id.layout.upgrade.threads</name>3 d1 v0 O! T0 [& k" y" @
<value>12</value>8 Q# [% z# U* \
<description>The number of threads to use when creating hard links from7 |. M8 E" V& E6 h
current to previous blocks during upgrade of a DataNode to block ID-based: g k" e; r( ?6 H( p3 E/ y
block layout (see HDFS-6482 for details on the layout).</description>$ a! ~5 z; V! U3 A3 P
</property>
2 ?. N2 r. R% T, M% D$ F0 M<property>
$ M7 ?- z5 Z0 T& C* `0 X! Q# t <name>dfs.namenode.list.encryption.zones.num.responses</name>
3 j' f; V' ^# i! ?% {9 ~5 C <value>100</value>
/ b' |$ P# c5 W1 T <description>When listing encryption zones, the maximum number of zones( S3 {& x9 r& v7 ~$ h4 o& f3 L
that will be returned in a batch. Fetching the list incrementally in" O7 q: {" ~3 n# c
batches improves namenode performance.6 e! s" A! W) X, l; ^
</description>
% p. C$ _4 X8 P- ]) X5 @- b8 I</property>, P5 J/ S3 N6 z
<property>! R* O' R' Y! z. r
<name>dfs.namenode.list.reencryption.status.num.responses</name>
, ?1 U" I7 q6 {( r <value>100</value>8 b* ~ b6 Q5 Y: H
<description>When listing re-encryption status, the maximum number of zones$ M3 E& K7 x* N8 K) O) S" d/ a! I
that will be returned in a batch. Fetching the list incrementally in
1 C5 s, p; z, c1 O" Y1 h) W batches improves namenode performance.% X3 ]1 z9 I: e; p
</description>
7 `6 Q8 r: q! W! P: c7 G</property>9 E$ }( |8 V' g4 t, u4 c
<property>/ X: d, @6 P7 ]: L& }( s
<name>dfs.namenode.list.openfiles.num.responses</name>4 m0 g' ^) b0 \, d
<value>1000</value>
. j0 s% x r8 V4 @9 } <description>6 u$ O5 r. ~8 }; k* {9 n3 _
When listing open files, the maximum number of open files that will be( M9 l/ `9 l; v- S0 F5 T% \
returned in a single batch. Fetching the list incrementally in batches
) y/ |7 {- [8 x3 z7 j improves namenode performance./ f$ o6 x% z5 D& c; [9 _& }. D
</description>
* X2 c/ N. ]5 L9 R. e3 } </property>
' `; c7 X5 c8 Q0 D<property>
8 `, F) r! I' w+ w$ I <name>dfs.namenode.edekcacheloader.interval.ms</name>
' h& g" U+ Z# e! G) S$ l <value>1000</value>/ ]% |1 s W3 ~
<description>When KeyProvider is configured, the interval time of warming
' H" g, l5 A. ~/ ^' I up edek cache on NN starts up / becomes active. All edeks will be loaded
% U$ U9 _6 w$ a B from KMS into provider cache. The edek cache loader will try to warm up the! A: u. k0 P& {* W2 o- w
cache until succeed or NN leaves active state.
& P. @$ B/ y% R- H8 | </description>7 h ]; T: h) m- u
</property>
- ~( M0 a5 \* \ y<property>
+ E9 y/ ^2 R4 V3 n! W) _! z6 k: ? <name>dfs.namenode.edekcacheloader.initial.delay.ms</name>
4 [# Q1 |! e$ w1 ?$ a( V <value>3000</value>0 d4 a: m A& R+ ~. V; V) D
<description>When KeyProvider is configured, the time delayed until the first
$ t, ~: ~5 h* g attempt to warm up edek cache on NN start up / become active.' w* _3 i/ \0 h3 H, O
</description>; k2 x& @* H" S F1 s
</property>
& Q: t$ w0 F$ y+ L @<property>
' s- _& d+ X) A+ [7 x <name>dfs.namenode.reencrypt.sleep.interval</name>3 ]" m* _* L6 K5 d( {
<value>1m</value> h ?9 }+ i5 `3 y1 Y* a
<description>Interval the re-encrypt EDEK thread sleeps in the main loop. The
& h) `& s- v- a1 q/ J+ { interval accepts units. If none given, millisecond is assumed.* K$ Z1 U5 n5 q( N2 W
</description>
9 ~! O5 l( [0 W( h. L</property>
' C+ n+ w" q, d/ J: T" d<property>
; ~6 k" y8 k: ]9 Z <name>dfs.namenode.reencrypt.batch.size</name>- F( \9 o) n$ N6 _. k/ R4 G
<value>1000</value>4 |: c" \1 R+ J" B- {/ ]6 m) |
<description>How many EDEKs should the re-encrypt thread process in one batch.
( H7 c+ k7 C$ h </description>
# Y N$ @: ?# N+ P4 W2 Z& [* w</property>9 i5 d3 L3 h! r6 N& u* p/ P, \
<property>6 h' g! Z- ^$ ^$ p) z6 y
<name>dfs.namenode.reencrypt.throttle.limit.handler.ratio</name>8 C+ j' E, d4 R) [' {2 u
<value>1.0</value>0 ~' L, q" S7 f, K7 P# p
<description>Throttling ratio for the re-encryption, indicating what fraction
3 e! Z! D2 a, [0 f of time should the re-encrypt handler thread work under NN read lock.
Z/ n4 q& ?# A8 v y Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are
/ o# A5 i3 d, W* m invalid values and will fail NN startup.% Y% c% a! i! N* O0 g: Q
</description>6 H; m$ U' ]& P, E/ p* S, P/ D8 d8 c1 s
</property>- h: o& o# ?* R2 Z- \0 X3 D
<property>. z+ j+ A8 I: {# p
<name>dfs.namenode.reencrypt.throttle.limit.updater.ratio</name>
$ V: U t6 |' W4 }# L/ Z <value>1.0</value>
, I$ a, s5 W: ?0 J( F X <description>Throttling ratio for the re-encryption, indicating what fraction
. \- g* Z/ `) L) ~7 W2 H of time should the re-encrypt updater thread work under NN write lock.
& V0 j5 Y: [' Z V% n Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are. Z) V$ C5 R) [5 O% T9 w
invalid values and will fail NN startup.7 I; I! \7 A8 Q8 D
</description>$ u5 k( l U3 Z y' ^9 i
</property>
9 Q( t& [& Y* `8 r6 p<property>' E3 m# l3 A2 _# V
<name>dfs.namenode.reencrypt.edek.threads</name>
+ c. [/ w S/ j <value>10</value>
" |. G) P$ _( b$ H, Y* t <description>Maximum number of re-encrypt threads to contact the KMS4 s- X( O' m% |* \% k! Q
and re-encrypt the edeks.+ H7 ] o" {9 r+ f$ p; O% t
</description>
. u2 k( I. G. F& l5 d4 c7 ?! i) {# a</property>
2 }- |/ z5 A1 S- |& j<property>) o' h3 K, `" O1 K
<name>dfs.namenode.inotify.max.events.per.rpc</name>
- o- l" |0 b9 W9 ^0 z' o <value>1000</value>8 g) G4 ?/ O0 t, v1 I: x0 W0 R
<description>Maximum number of events that will be sent to an inotify client
* r! F6 I' e" U in a single RPC response. The default value attempts to amortize away7 b# Z* q" q+ D! y* W2 B$ t' g
the overhead for this RPC while avoiding huge memory requirements for the' y- y, Z2 a9 ?" N
client and NameNode (1000 events should consume no more than 1 MB.)6 l2 V1 G4 u$ i1 O- F; r& v
</description>
. Q- M7 _/ K/ F) x+ @</property>5 A. u% m0 n/ N* ]7 r. }5 i% _+ U y
<property>; S. o2 |0 I Y2 I q \/ E
<name>dfs.user.home.dir.prefix</name>
. e. j# G/ p/ m+ Y8 Q- R: }2 Y <value>/user</value>
- q# h( O- [' g5 i8 n! b/ L <description>The directory to prepend to user name to get the user's
: e2 h3 w. A7 ~ home direcotry.; O4 F/ Z0 Y/ t2 i+ A& K
</description>
& }3 F& B+ D, O8 Z( I) @</property>
0 l+ L. ` R( k<property>
6 ^; B3 f6 G, S( K' S+ } <name>dfs.datanode.cache.revocation.timeout.ms</name>
+ a' W. ~0 y+ k/ f) l0 h% F/ a <value>900000</value>2 x5 ]: R0 V2 f' v( T
<description>When the DFSClient reads from a block file which the DataNode is
6 _1 l4 w. e) O+ o caching, the DFSClient can skip verifying checksums. The DataNode will. Y8 ~ o4 v& T/ N8 w2 J+ d
keep the block file in cache until the client is done. If the client takes
) k# _5 }. p- l; S( |. Y/ C* ^ R2 i% o an unusually long time, though, the DataNode may need to evict the block
+ Y" n2 V, X! w4 G5 Z file from the cache anyway. This value controls how long the DataNode will. u7 x9 a, ^4 ^0 e2 C! P
wait for the client to release a replica that it is reading without! M% [( N. q& X' Q
checksums.5 u' h9 ^$ a" k. \! C) G9 O
</description>
2 G3 G, y3 s: s# l</property>
' z5 g6 u7 _8 D4 F<property>
7 H/ r" P/ b+ i$ s5 @; B# i+ [ <name>dfs.datanode.cache.revocation.polling.ms</name>
: p. `" F0 O( [3 n7 T <value>500</value>
% s5 @$ s0 _$ P+ g# Y P <description>How often the DataNode should poll to see if the clients have
: b& }6 g7 W+ Q+ @: }$ K7 c* l1 a stopped using a replica that the DataNode wants to uncache.
5 i' F& r# M3 s% D, \ </description>
* z5 [- d8 U z1 \</property>
: A, |& L c3 @) G<property>$ O. K$ W' S6 o4 L8 ~" G
<name>dfs.storage.policy.enabled</name>% o& v( b0 f% q; L4 X! Z
<value>true</value>
' c; f" I5 T: n+ u. L$ T <description>: T5 O$ B' h( r; D a! x8 m4 \6 C1 D, g
Allow users to change the storage policy on files and directories., F6 ?/ C2 |( m6 ]0 |9 p- m
</description>% D+ S" x3 M! w! K
</property>
7 u7 N/ t# D4 M9 e$ _. V<property>
) M1 {6 L5 e S' R/ t" g <name>dfs.namenode.legacy-oiv-image.dir</name>: y% x @9 `# S5 E6 B
<value></value>: s7 o& y. [& v
<description>Determines where to save the namespace in the old fsimage format
/ U- S* Z/ M/ A4 j+ O" i" W during checkpointing by standby NameNode or SecondaryNameNode. Users can& E; z2 g m' X; o! G: ?) Y
dump the contents of the old format fsimage by oiv_legacy command. If' e$ L4 b0 p4 [+ i; M3 u
the value is not specified, old format fsimage will not be saved in, _3 H6 o! I% w. }
checkpoint.
7 N" k; j& ~: ~. ]8 {5 Y% y$ b </description>) m# q8 }4 \! i$ K) ~) a4 P1 A
</property>% u# H% w* n9 l- s) W
<property>6 l3 P% C2 s+ W" h! [6 j) r
<name>dfs.namenode.top.enabled</name>
# `4 y7 ?7 s3 l) ~' f' L <value>true</value>" x" O* j% h6 T; \) l+ l! ]
<description>Enable nntop: reporting top users on namenode
9 Z" D" H2 z3 |" A& M4 ?5 j4 n+ j8 G% y' k </description>
) J1 \( V9 o/ Z7 A6 r</property>2 ^, [/ G4 _2 j4 o
<property>
7 r& _) X; o0 k <name>dfs.namenode.top.window.num.buckets</name>
, N1 O" N7 A9 b# T! M <value>10</value>: y/ O6 O4 a( I6 f( B& N
<description>Number of buckets in the rolling window implementation of nntop
; P; e7 `$ u4 y& ^ T </description>& }5 P+ q5 _7 S8 f u7 ]2 I
</property>
7 n/ U/ F) e% e7 _' J Q<property>
0 j, ] v' [0 `- e7 B <name>dfs.namenode.top.num.users</name>
1 w" B$ d- T! a- k$ R" Q <value>10</value>; S4 O7 {, c. W0 p* M0 r& Q
<description>Number of top users returned by the top tool
. r* ?! u& b; N K# z A. F </description>
' h; v% |/ i( W6 U6 @+ {) e</property>
8 s4 I: [; S5 a0 o: y) k9 R<property>
# I+ u9 G0 N D( E7 G- O <name>dfs.namenode.top.windows.minutes</name>
2 K/ g% F8 f6 U+ u1 w A; O <value>1,5,25</value>
; H% [. {& R" U <description>comma separated list of nntop reporting periods in minutes
/ m+ R* N6 V7 M. R </description>, N' V7 b0 ~) K+ E( d6 d+ b) y
</property>5 T+ n7 _8 G: D; w' o
<property> W5 W' I9 k' |0 k$ z4 Q" E
<name>dfs.webhdfs.ugi.expire.after.access</name>3 J. b2 a5 Z' \# t* S
<value>600000</value>8 z7 l9 t! F5 g
<description>How long in milliseconds after the last access
% {' q( y1 @5 t: D/ X4 I3 {$ X the cached UGI will expire. With 0, never expire.
. e- j) C) T5 n: `/ |9 H! X6 }8 K </description>+ v& N1 c- W* @8 H5 t3 z+ U
</property>) s. _4 c. ~' ?# Q0 i2 p
<property>8 h. F; S. s9 j0 p# A
<name>dfs.namenode.blocks.per.postponedblocks.rescan</name>9 q! V' s7 g" j+ y) U" M
<value>10000</value>7 l/ I2 U5 f* S$ v
<description>Number of blocks to rescan for each iteration of
2 Y! c- b- [; ?) y postponedMisreplicatedBlocks.
9 {- P0 h" S4 v! y; D, e; J </description>
) c2 R1 V! k+ O- M1 U% C: A</property>5 t0 n& h0 ^9 |' R2 ^$ A
<property>$ _/ K" o7 v$ r. K0 ]+ c0 Y% x1 c
<name>dfs.datanode.block-pinning.enabled</name>8 h0 V5 W ^; `3 P5 v- g# s6 z9 `
<value>false</value>
5 T8 u. e: n; c, t2 G <description>Whether pin blocks on favored DataNode.</description>/ l# Z2 C y. R' @" J" h7 G
</property>
6 D. K3 T& y% f4 V: E<property>9 y# M, X' j% t y0 k
<name>dfs.client.block.write.locateFollowingBlock.initial.delay.ms</name>
/ @6 d' u8 ^( a' G6 R <value>400</value>
; U$ e/ I4 n/ N& q5 u: A& l <description>The initial delay (unit is ms) for locateFollowingBlock,
7 M! L: N o. O; S9 k, S the delay time will increase exponentially(double) for each retry.
; |5 r+ t3 S @3 y8 d2 ] </description>
4 T# L3 x+ k# @3 ^3 y</property> \, X% k5 d0 O3 i5 l U5 w
<property>, Y5 x. S; C1 [" r' S
<name>dfs.ha.zkfc.nn.http.timeout.ms</name>
2 G$ z7 S4 f% N* l5 l <value>20000</value>8 f6 K/ G( `: |: ?. L3 u6 o
<description>
* ]% J- I+ }! T% U% S The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC
- s* O5 t6 z5 G$ g# i: ~8 b7 l/ n tries to get local NN thread dump after local NN becomes" f' }& C, g. |- G* ~0 T/ M
SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY.$ t$ Z! Q4 e7 W" [" ]
If it is set to zero, DFS ZKFC won't get local NN thread dump.) U4 ?8 ] u* q5 p% X# [1 U' b
</description>1 z. r: Z1 T2 B" J. n' B; i
</property>
- y. g8 b! z/ H' o<property>
) b6 S* Q4 K+ G. P$ m# ] N1 } <name>dfs.ha.tail-edits.in-progress</name>
$ R) q1 ]( }7 F <value>false</value>. k# M' @# J. j1 c4 A' x: C% A
<description>
+ Q, M' q8 p3 {) C Whether enable standby namenode to tail in-progress edit logs.5 l [9 ?% B; u, {" N7 Q5 Y: Q7 S
Clients might want to turn it on when they want Standby NN to have9 Q$ j( A/ `4 A
more up-to-date data.2 z! {: D/ ~- y& C" ?( x7 z& }+ @$ ^
</description>
/ }% n1 ~, Y1 [</property>
0 D! k/ L& W! z& I5 g# x<property>* j* t& B L6 w# `
<name>dfs.namenode.ec.system.default.policy</name>2 `9 q0 P8 L4 I: x! ?
<value>RS-6-3-1024k</value>
# q4 }1 o/ Z& c0 P$ q* M! X' B <description>The default erasure coding policy name will be used# c L( a4 W: P9 u; L9 _
on the path if no policy name is passed.4 s) S! }4 Z2 y# ~" F0 d$ f
</description>
$ d l* s( n$ q' a r- _</property>$ K2 v7 [! ]0 o
<property>
0 c: _0 K# N% L; U1 f <name>dfs.namenode.ec.policies.max.cellsize</name>
1 W4 B, m+ r. y5 v/ G* H2 C3 N& | <value>4194304</value>+ d# k5 a! |- O5 H
<description>The maximum cell size of erasure coding policy. Default is 4MB.
% g. p# n) a5 Z% L! B3 W; P+ c </description>+ f& w4 Z1 p' `% [8 |4 D/ \. q
</property>
; v: u$ P/ U0 h3 e4 [* G9 ~<property>
* ]) _$ E9 ~" M, r& V3 K& M <name>dfs.datanode.ec.reconstruction.stripedread.timeout.millis</name>
6 ~$ C7 [% Y$ X% T9 w9 j0 `: j <value>5000</value>
# B& t5 |0 ^3 |, ]4 f, D <description>Datanode striped read timeout in milliseconds.
! Y( o, U n$ N/ x1 _/ o: O, u </description>( q3 K- x) Z8 u3 A% N T
</property>
' A' K# a7 s/ m<property>
' g6 ~/ ]0 ~8 o. }4 n( ? <name>dfs.datanode.ec.reconstruction.stripedread.buffer.size</name>
. r, x Y' ~8 T4 G7 w- b+ K. G* G( e <value>65536</value>1 c# n! \( q6 W. d1 F
<description>Datanode striped read buffer size.
# e; N/ _1 H& J3 ~' c </description>
4 w, r6 S/ z6 ~% ]9 j, B</property> J/ {3 `" a* E: x' i' z8 h$ B
<property>
" u! h f# n; z5 j& ~2 T <name>dfs.datanode.ec.reconstruction.threads</name>: l. w8 I! d( l7 W
<value>8</value>6 e3 N- h* d8 f
<description>
. q! C! ?; \" S# n% {# J Number of threads used by the Datanode for background
; f! y% _' c$ N9 L9 x reconstruction work.& }6 d( }% V+ n! }3 l/ K
</description>
5 o/ d! I/ {, U3 Y1 @# B m( C</property>/ @, P7 Z2 ^. | \/ |
<property>
4 g* p9 U0 d* n <name>dfs.datanode.ec.reconstruction.xmits.weight</name>
3 r( Q6 p7 a) F. k0 [9 b- p <value>0.5</value>. R0 W, d- L+ |
<description> ]" x- k6 z# i$ w/ L
Datanode uses xmits weight to calculate the relative cost of EC recovery8 @8 S- B: l- I' V% o6 a: o& w
tasks comparing to replicated block recovery, of which xmits is always 1.
; j7 _; w- u; { Namenode then uses xmits reported from datanode to throttle recovery tasks5 @! X _ P4 j" P
for EC and replicated blocks.
1 b7 B/ p; ?# t The xmits of an erasure coding recovery task is calculated as the maximum" X+ `4 r% y1 w# b/ a; k+ C
value between the number of read streams and the number of write streams.$ p, {) b2 d- y( f B/ |7 P
</description>' h8 S' s" D/ y
</property>
8 t# H6 E! M) K* V6 s<property>
) Q+ f/ Q2 }' r <name>dfs.namenode.quota.init-threads</name>! q5 F" ~1 B& Z; `
<value>4</value>+ s- |2 X% ?% ~7 W( S! b
<description>
% w* `& ?3 @. ~- f/ o7 s The number of concurrent threads to be used in quota initialization. The
. |7 x5 A# ]7 f) G speed of quota initialization also affects the namenode fail-over latency.6 L# {; K* ]" V4 c
If the size of name space is big, try increasing this.( b. ]$ ?0 X8 D: h3 w: \2 X
</description>' `! \0 M6 g3 ^/ w7 t, o
</property>
$ `! X F& F- L( z! \$ o3 `<property>) M4 }$ Z8 t( n& o) [
<name>dfs.datanode.transfer.socket.send.buffer.size</name>
' Z- A Q4 J M8 X <value>0</value>
1 t; q5 d5 |& X/ g( n <description>4 O" c6 J! |1 ^' s2 Z3 }
Socket send buffer size for DataXceiver (mirroring packets to downstream( x5 b/ p) w- }" R1 D0 |
in pipeline). This may affect TCP connection throughput.
6 k, H) }3 o5 h: N5 H; e) u If it is set to zero or negative value, no buffer size will be set5 |$ y9 u; R! `
explicitly, thus enable tcp auto-tuning on some system.
8 d5 u1 A. O' T5 O+ q8 a The default value is 0.
5 d, ], T5 \6 j7 x" f4 ]$ y( X </description>4 U4 y- P* L$ S/ V- y
</property>* p' r+ y% s2 h6 K3 ~. A
<property>
h+ G4 ^+ ~! u7 y' ^& ] <name>dfs.datanode.transfer.socket.recv.buffer.size</name>
- b1 V, ?. C7 v% \- x2 E. w# v <value>0</value># E5 s& ^4 ~8 V M
<description>, D- l4 Q+ \6 x8 x& T( p
Socket receive buffer size for DataXceiver (receiving packets from client* q: z, U5 d2 d8 O$ X$ P
during block writing). This may affect TCP connection throughput.
. X( C4 S) z( Y H, f/ W; y If it is set to zero or negative value, no buffer size will be set
2 \% B3 @4 h3 I! c i( \/ `" V8 b explicitly, thus enable tcp auto-tuning on some system.
* j. J1 v1 s! R/ D1 o The default value is 0.
: M0 G9 _4 p" ^ </description>
" [. ^" H% O) J I& q# C5 c</property>
9 Y$ m9 k' {4 X<property>8 X4 z9 B% e8 v5 k
<name>dfs.namenode.upgrade.domain.factor</name>
! ~0 |4 i6 v3 m3 A8 f& }: V <value>${dfs.replication}</value>
: s# n6 d/ p- Q6 [9 _# d+ |$ Z <description>3 V; b8 Z4 d: r+ R$ p
This is valid only when block placement policy is set to
6 q0 W% _" K7 R; c. ~. D7 ]2 E* [ BlockPlacementPolicyWithUpgradeDomain. It defines the number of. p1 f2 s7 e! G. ~$ L
unique upgrade domains any block's replicas should have.4 J# ?7 i5 @+ f) ^
When the number of replicas is less or equal to this value, the policy
/ o E% l; B; L+ w: U, \+ i ensures each replica has an unique upgrade domain. When the number of e: N4 W1 ?- k# T/ C
replicas is greater than this value, the policy ensures the number of
+ D+ D" ~+ V4 ~! J1 a" ^' l) P unique domains is at least this value./ q( O, p$ B% j3 H4 j
</description># Q; d7 w! P& l' K; g
</property>3 o D4 h. h# o6 o* F
<property>
1 S9 l" F, Z m; x6 e$ K( Q <name>dfs.ha.zkfc.port</name>
) R, o; F7 j5 Z. A+ J1 E# Q <value>8019</value>. }8 E8 |8 r D
<description>& P# A- ?6 M/ J1 Y; p6 H6 W K) [
RPC port for Zookeeper Failover Controller., G5 R3 N! f& Q+ e m0 B; L
</description>
$ g" |# m. v9 R$ J( n</property>
\- n' g7 E( y4 e& d<property>
+ |3 ^6 z* T+ B <name>dfs.datanode.bp-ready.timeout</name>
( q! [+ N7 @3 A <value>20s</value>
% S5 W9 ~3 e% ~7 N% C3 E <description>
$ u. t, C% M! \' Q5 J The maximum wait time for datanode to be ready before failing the! d4 P8 x% w# `7 Z5 E
received request. Setting this to 0 fails requests right away if the
- P! S4 f" g& E7 }2 e1 F1 ^8 i datanode is not yet registered with the namenode. This wait time
! y( p. b- b2 a/ ^! F reduces initial request failures after datanode restart.8 u( D' F: |! J/ W- @
Support multiple time unit suffix(case insensitive), as described
! S0 ]& O$ L5 M( @5 [/ B. E in dfs.heartbeat.interval.* ]$ `0 l/ J9 M
</description>. \7 u; u/ [+ v9 _1 Q; `' B# ^1 \
</property># i/ `% L4 d% p, n7 b- Z
<property>5 w$ E2 T: |1 K9 R, ^, z6 }' |: a4 e
<name>dfs.datanode.cached-dfsused.check.interval.ms</name>
4 u! h: p. \0 A1 ` <value>600000</value>
# t c' W- }0 C/ d. a <description>
1 q$ D7 Y- B7 l5 U+ F1 ?1 H The interval check time of loading DU_CACHE_FILE in each volume.
) J" I- [* H* M$ P# E5 Y, x# b3 y When the cluster doing the rolling upgrade operations, it will' k4 E: u3 F5 R; v' q
usually lead dfsUsed cache file of each volume expired and redo the# H6 L8 ^6 n- o) {+ h# W
du operations in datanode and that makes datanode start slowly. Adjust
6 u8 S7 }% I3 I2 A3 d this property can make cache file be available for the time as you want.; G% @8 m% J% m
</description>
4 ~) }+ c( n5 {& T3 \</property>
) j+ r! [7 Z1 C/ e' n" \<property>
8 m7 s$ t& ?( k <name>dfs.webhdfs.rest-csrf.enabled</name>
: o6 A$ y: S w6 z* N <value>false</value>! g1 P" }8 o3 k& h
<description>8 x4 r2 N# j7 S" C
If true, then enables WebHDFS protection against cross-site request forgery
# T, c6 w$ d T/ K (CSRF). The WebHDFS client also uses this property to determine whether or' s( d( x: i7 Z( V6 n+ {) r# n4 _) I( U
not it needs to send the custom CSRF prevention header in its HTTP requests.. [7 L7 o$ n9 G& p7 ?. u
</description>4 E3 d. ^4 u& A7 ?" o7 f4 `$ W
</property>% c% G- Q. ~- G1 Z5 {2 u/ K
<property>5 \: R5 m# i6 d2 f( d6 ?8 _
<name>dfs.webhdfs.rest-csrf.custom-header</name>6 @+ A% }5 v, k( Z) V
<value>X-XSRF-HEADER</value>
2 r; \; q, A" c8 W <description>
# t# S$ P) `6 ? The name of a custom header that HTTP requests must send when protection# T% ^9 ^" H3 i9 S0 @
against cross-site request forgery (CSRF) is enabled for WebHDFS by setting& r0 E w& F4 `! U' i# `" o
dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this
8 {: W3 |7 \: _' Z2 ?0 J0 D property to determine whether or not it needs to send the custom CSRF
/ g4 S8 h" X2 X# ~7 P prevention header in its HTTP requests.
" ?. ~6 r" M6 ?, n& _ </description>& A, d& }* j5 ~: c* }9 M8 V
</property>
& [2 o; @2 P3 h; s, L! [<property>
5 s/ h( K5 e( L; o# N <name>dfs.webhdfs.rest-csrf.methods-to-ignore</name>
C) V# Q b4 J5 q! u <value>GET,OPTIONS,HEAD,TRACE</value>
5 t _5 w8 m5 X+ X. g <description>% M4 c2 p# R4 Y
A comma-separated list of HTTP methods that do not require HTTP requests to/ ?& O0 s: a, D( L$ A4 @* e) y) B
include a custom header when protection against cross-site request forgery
. q9 U) Z' c f (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to, m. [5 g& L, U4 e, K5 N
true. The WebHDFS client also uses this property to determine whether or: u: z2 f5 n% M& ?$ P4 ^
not it needs to send the custom CSRF prevention header in its HTTP requests.
2 N4 H/ S, u; y9 o7 [' N; {% T: N$ l </description>
, d/ ]0 W+ P5 U4 t# a</property>+ p, g' r! U; z& C- a0 I
<property>+ n; Y& P1 x5 R7 |/ T
<name>dfs.webhdfs.rest-csrf.browser-useragents-regex</name>1 \& j) |- _) Z2 Q5 }5 t5 g
<value>^Mozilla.*,^Opera.*</value> r q% Z& _! B! h
<description>0 e+ c5 v& X$ H: b3 O
A comma-separated list of regular expressions used to match against an HTTP
5 h' Z% ]* o8 s) B! y8 J request's User-Agent header when protection against cross-site request
$ v8 x' b% p: }. C- l forgery (CSRF) is enabled for WebHDFS by setting
O0 {* b7 m) s% c$ [ dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches! N, P* f+ R. S, a4 T( l
any of these regular expressions, then the request is considered to be sent
& b1 d3 ?9 ?# \1 i' @ h by a browser, and therefore CSRF prevention is enforced. If the request's
8 s1 y+ R0 b% j8 V2 O0 k User-Agent does not match any of these regular expressions, then the request
8 ~2 D" [# [3 c0 `) @" v is considered to be sent by something other than a browser, such as scripted
2 C$ w+ K3 q' k6 X+ P) E automation. In this case, CSRF is not a potential attack vector, so
9 ~) X* r! m% Y% t- m& B6 A" z f the prevention is not enforced. This helps achieve backwards-compatibility& e" B' r: |: K" h0 o) y
with existing automation that has not been updated to send the CSRF
5 F6 S5 N% s2 @ prevention header.! c3 g( f0 J5 S6 v) T4 G& H
</description>
* W# n) g0 @% |% e$ Q( n) E. Z3 P</property>9 n) [$ w5 _9 a" ?7 j
<property>/ L P, O' T4 f1 C1 o+ a, |
<name>dfs.xframe.enabled</name>
2 Q6 I1 l! X' l( _/ P+ E <value>true</value>, {7 W. n! b& n* W
<description>) k, {0 }: Z' N# A3 q6 h
If true, then enables protection against clickjacking by returning: {" v2 i2 V2 M- [; L
X_FRAME_OPTIONS header value set to SAMEORIGIN.9 f) I9 p% h, V1 G1 n: V
Clickjacking protection prevents an attacker from using transparent or
* F- R( H* y: j# i' { S opaque layers to trick a user into clicking on a button
) x* Y+ B+ F* C/ u or link on another page./ B* E- Q0 P/ l% z/ s
</description>, c+ Z* R7 d) A. ~
</property>% i. S# w7 \3 K
<property>9 R: |* y/ E9 O3 k. h t
<name>dfs.xframe.value</name>
1 H3 Y! ?7 j+ U6 g+ U <value>SAMEORIGIN</value>
; {4 U. X- @' q9 ]4 i0 h <description>
, @- W7 j! y9 e4 \4 n6 @% f This configration value allows user to specify the value for the3 M/ `4 r$ J! I3 [# e3 M/ s0 s
X-FRAME-OPTIONS. The possible values for this field are
3 F9 u& [& X2 Y3 H DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an
- u. e6 s1 U$ \5 ^# A! }& U$ N exception when namenode and datanodes are starting up.
( Y% Y; V4 H2 \8 N& A </description>4 b6 D! X2 Q* Z2 y
</property>
, |4 E( d- L3 W) F% M6 X<property>
, d* }! W1 a% K( f9 k- Y+ F <name>dfs.balancer.keytab.enabled</name>& X4 R0 o- k a. m4 b
<value>false</value>
' A# l0 O$ C& \6 g; I- x <description>
0 a9 X) M [! E Set to true to enable login using a keytab for Kerberized Hadoop.& `$ R: I) |) K/ P$ P
</description>* z8 y/ W! C/ v; }
</property>
6 R5 [" P* }8 z6 [<property>
7 f( }7 `4 H! c. J# F+ E. w <name>dfs.balancer.address</name>- R! H3 v. R& a* ^, v* P. X J' K
<value>0.0.0.0:0</value>
k! H( \9 a! l0 n7 _7 a G4 J; k <description>( [1 m/ b" t8 ?- F
The hostname used for a keytab based Kerberos login. Keytab based login
- _* r: `8 h* b2 K can be enabled with dfs.balancer.keytab.enabled.% @# G0 M( y% n9 [: b
</description>& E8 l; P: i3 M& o) T
</property>
, a" ?, @: j+ ~<property>
2 i3 H/ F8 Y4 @' _ <name>dfs.balancer.keytab.file</name>
- G8 }$ r$ y3 O* t, ]$ V <value></value>8 }- ]+ {/ @* u# v1 q/ w" b* F1 h
<description>
; a4 D5 G. `- }5 }. G# V0 W( u% \8 g The keytab file used by the Balancer to login as its
; S, g4 x5 N; v* y' {/ V service principal. The principal name is configured with
% v$ X4 Z; ?2 {9 d- w dfs.balancer.kerberos.principal. Keytab based login can be0 w) H; U: ~/ h) Q3 K8 J
enabled with dfs.balancer.keytab.enabled.5 R1 Y8 f! O( n
</description>
8 T* Y2 k. O( X( ?6 h</property>
) B6 ~ I. ^! S- |9 T, F; H4 ]2 P4 T<property>* Y! O4 b% [6 T4 @, j5 R$ A( P# k
<name>dfs.balancer.kerberos.principal</name>+ Q) r. B. Q) s: `* o
<value></value>' P- A6 b; t- Q" E: k" s" t
<description># j' T6 g$ a$ }( a( Y! B+ @
The Balancer principal. This is typically set to2 G" }. ]4 h" p- S6 `- d& r
balancer/_HOST@REALM.TLD. The Balancer will substitute _HOST with its
6 i1 C# E; t' ?$ x# N own fully qualified hostname at startup. The _HOST placeholder
, U- o2 K {8 ^) ?" \& ?# J allows using the same configuration setting on different servers.- ?2 H% ?" f( ?9 _% m
Keytab based login can be enabled with dfs.balancer.keytab.enabled.
' m2 t# Q3 b5 p* S5 m$ I </description>
8 @; @. r `) E7 r+ c9 T</property>
" A0 L/ p X+ @% V9 U% j$ G<property>2 B$ M3 x$ y% @/ E3 y. `
<name>dfs.http.client.retry.policy.enabled</name>9 L+ h- s# G& _, p5 o) X \
<value>false</value>* ]+ J/ l! E3 @6 ]2 ^, |/ w
<description>$ f: t( Y$ _' F
If "true", enable the retry policy of WebHDFS client., [. b$ ~) x; J6 ]& [) [
If "false", retry policy is turned off.
" l* h& U* \- ^1 W! t Enabling the retry policy can be quite useful while using WebHDFS to& ?2 }6 r r) t1 p; \
copy large files between clusters that could timeout, or5 a' ~. F: F! J; d5 }
copy files between HA clusters that could failover during the copy.
' W- \3 J& _" ]& ?, `: p </description>
! A5 @$ r# }4 I7 u/ `) j</property># l6 q* W/ \* @: t
<property>
. R, E+ @: N& W8 G! O; {/ ] v3 f <name>dfs.http.client.retry.policy.spec</name>" V. F: V {" N c& B. e' ]
<value>10000,6,60000,10</value>
2 ~1 G+ V5 t& n/ I x7 S <description>7 E' E- t4 K8 h7 b+ X; I' Q! M
Specify a policy of multiple linear random retry for WebHDFS client,9 ~0 c5 b2 r1 J
e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),
) A- t/ ~7 I# e+ R ..., the first n0 retries sleep t0 milliseconds on average,) h9 x1 `8 k* l# U* X n5 g: k* r
the following n1 retries sleep t1 milliseconds on average, and so on.
H: G9 U7 B! U: m7 N3 p </description>
- ]* z8 R+ N, Z- f J; ?</property> v8 V/ ^9 f" j/ d, w
<property>
$ Y, o7 \: D( z, i: ^ <name>dfs.http.client.failover.max.attempts</name>& Y" ^8 i. i( }% B
<value>15</value>
4 P* O' n6 p. h5 N6 w4 @- M! D, | <description>! q' u- O8 q/ f. h" U* W' M* T
Specify the max number of failover attempts for WebHDFS client
, F* Q, a. X$ p+ i9 m$ ] in case of network exception.
) G& z: ?; l) Q# a: ~) x1 L) X </description>/ T$ U D% Q( e" I0 @5 A- A8 V6 _
</property>: H& m5 O6 _8 s/ r2 A% v- g8 H
<property># j+ B$ [( g$ F( m4 N' k' d' {5 T
<name>dfs.http.client.retry.max.attempts</name>
3 M# H6 c: v5 s) T9 _% D <value>10</value>$ a9 ]$ c+ X- x& V
<description>
/ w* l2 x, ?: w6 T) l/ x/ B Specify the max number of retry attempts for WebHDFS client,
) G6 f# ~4 ^0 D; S# E9 ` if the difference between retried attempts and failovered attempts is% G+ c$ t& n, W1 I
larger than the max number of retry attempts, there will be no more$ b7 w" G6 Q1 C6 ?. i4 O; N
retries.7 B' |) j$ G% @4 }, F
</description>
% b o* {/ v: e% H( t# F: Y( S% l</property>
U+ j ~) K& t- O( b5 G<property>
# y& c3 t( f$ O$ x6 W <name>dfs.http.client.failover.sleep.base.millis</name>) H3 i- R7 c( J. D$ F. X
<value>500</value>
2 B+ `) ]3 T% M- a' l/ i( c C <description>
, Q2 j! w7 \" l# L, d7 B/ c/ J% f7 k2 B Specify the base amount of time in milliseconds upon which the4 W' A3 S2 K2 h6 |" R, Y b
exponentially increased sleep time between retries or failovers4 ]- y' C. a5 e# A2 {0 r
is calculated for WebHDFS client.
( H9 G7 ]! U$ p% ~ </description>
) {& N( S: R) C- C0 y* x</property>4 x6 Z& ^9 E D o
<property>, \" @. |5 n3 g& Z9 o+ k& _1 v
<name>dfs.http.client.failover.sleep.max.millis</name># x# |8 y6 H; k( U* N
<value>15000</value>
# S& ^2 z; c: H8 @8 O/ e9 T <description>
1 j6 {5 i3 d5 s% \ Specify the upper bound of sleep time in milliseconds between, _" Q+ _; K8 }& Y5 d9 ?9 H
retries or failovers for WebHDFS client.. p9 V/ ]$ P. X% i
</description>/ |" @' r& X. V0 H: t8 F8 c
</property>. V! ~ s6 W4 {9 L3 H9 Q
<property>( N; U! l; O3 D9 t, H+ D- h7 a
<name>dfs.namenode.hosts.provider.classname</name>8 ^% i8 I* _9 Y) i5 K+ x, O8 P
<value>org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager</value>; e, R: o+ Y# ]; h1 y
<description>
" ]$ p$ P: I! [. s, p2 i+ \ The class that provides access for host files.9 z* l! v5 B, L+ k$ k7 i3 j
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used& w X. F; O e. u* J8 Q" f
by default which loads files specified by dfs.hosts and dfs.hosts.exclude.
! Q/ ^: t5 i8 ~ n2 \ If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is3 Q }3 ? ]8 h
used, it will load the JSON file defined in dfs.hosts.
: M9 m% J6 j" w To change class name, nn restart is required. "dfsadmin -refreshNodes" only
% U: X3 B9 v9 z8 l% A refreshes the configuration files used by the class.
! V/ B( Y0 z8 z </description>
) i$ F0 U+ n7 |</property>
, p) s& M3 Y, H/ \- V; P" X<property>
/ T" d8 _3 E l( ?9 m <name>datanode.https.port</name>
/ Z2 m# f8 m$ U! p( x: L- G <value>50475</value>. X5 U e8 v- l7 d i ]# n
<description>3 c- a2 q7 r- \1 M# C8 k
HTTPS port for DataNode.4 b% S( e/ l9 e
</description>
- q0 O L0 Z3 z$ N9 m7 n</property>
. `9 V: O! g$ ~5 M<property>
9 I" Q- a/ w& Y7 h+ J2 J( U <name>dfs.balancer.dispatcherThreads</name>; s6 W1 g' ?- T/ n
<value>200</value>( \, Y) i1 K2 C5 ^3 c0 ]
<description>* a q+ m* g6 N1 C* t( T: H
Size of the thread pool for the HDFS balancer block mover.
6 v5 ~' G, ~# y! ?' m4 ]5 G dispatchExecutor, A; H; a- Z! E) z" G( }* D
</description>7 f7 x) k- [- s, B b
</property>
/ v7 x1 E9 O' _, x3 m+ G<property>9 Y( W9 |" q& W9 L, |! q
<name>dfs.balancer.movedWinWidth</name>) l, M0 p3 [. _2 A
<value>5400000</value>" I; z7 T0 n& `
<description>! f$ d1 N R6 [1 Q
Window of time in ms for the HDFS balancer tracking blocks and its( a+ R; [7 C* ]/ c/ I
locations.
( Y" E% Y/ N6 u* F8 T </description>& F7 j5 I d+ u) z& j3 a! \
</property>* B+ w# @5 }6 P* f+ q+ q
<property>
% B( M, u( a' @. A2 x, W <name>dfs.balancer.moverThreads</name>
; g `5 G* P4 L# F* R9 {# @6 H9 F, C <value>1000</value>* ]$ i( E* R2 J: V8 N( L
<description>. k9 R, R4 s( }3 S, i$ ]$ Y* l8 d7 r- ]' b
Thread pool size for executing block moves.
5 T0 U# V$ Q. `# g5 h ^1 F- `- b moverThreadAllocator! ?2 N9 o2 R/ U6 d9 j# i1 X
</description>
8 |: Y+ E" G0 E3 e</property>
o1 h, d9 C; E" q) E2 ]" h4 j6 u& K+ ]9 y<property>
( C1 ]$ h5 i7 n5 m6 P# t5 r <name>dfs.balancer.max-size-to-move</name>1 L" I7 C6 ?5 D* K0 X6 ]
<value>10737418240</value>
0 U* C: y: M. q# D, O( O+ v3 C <description>
0 ~# J" S* t3 }9 @, j Maximum number of bytes that can be moved by the balancer in a single* j2 B/ C! C* Z# u6 e. Z, X' {! n
thread.
/ a2 M7 E0 Q% d/ s3 b2 d' y+ V- i </description>
+ r. u4 J; i6 A+ a* d</property>
2 M" Z! w; u- [<property>6 V# J0 G5 X, ^2 I
<name>dfs.balancer.getBlocks.min-block-size</name>( N# K% e0 \& g( x
<value>10485760</value>+ L/ T$ M7 `$ @3 X* ~# W
<description>- A% c$ F2 Z1 O+ K. q/ ], R( X
Minimum block threshold size in bytes to ignore when fetching a source's6 q) q% {( L" z$ T0 L- ^
block list.
4 q. b8 T) Y0 Z0 o0 M </description>* e6 m- H) `* \, H0 M: G
</property>
1 m4 b7 J0 e! s, D7 q<property>
3 T+ h4 c1 \( O% C) V2 M <name>dfs.balancer.getBlocks.size</name>; v8 ^! p. l' W1 I5 W0 c; J% a' [
<value>2147483648</value>
- j8 v& t9 k8 ?1 l% f <description>
0 S q' G6 m4 r: U8 W6 I Total size in bytes of Datanode blocks to get when fetching a source's
! ^3 T1 b) _( L( d block list.
8 T' k0 N8 R/ R2 j) T% a8 E </description>$ _; t. ~) z" f7 w5 e% e2 h6 n: S
</property>
! C, h1 o" ?' T( {<property>( i) I! x |5 e5 u" P1 S( m
<name>dfs.balancer.block-move.timeout</name>5 _( R! F7 c% C( q5 a8 ]" |7 W
<value>0</value>" m+ @0 d9 o' Z0 W
<description>. U- l- w5 S# b9 Q. j) d" N
Maximum amount of time in milliseconds for a block to move. If this is set
* W7 c2 v. q$ p" c, c B4 i7 G; n3 W6 M greater than 0, Balancer will stop waiting for a block move completion2 ]3 [2 m/ e. E0 g
after this time. In typical clusters, a 3 to 5 minute timeout is reasonable.
6 f3 T% D1 E& R6 V4 [3 }4 L If timeout happens to a large proportion of block moves, this needs to be. N( T" O9 t. n8 I
increased. It could also be that too much work is dispatched and many nodes8 _5 Q5 x, h9 |% G* Y2 u
are constantly exceeding the bandwidth limit as a result. In that case,
& X: k* g1 _1 k& @9 ] {$ [ other balancer parameters might need to be adjusted.3 J5 F e; K7 x6 @+ ?
It is disabled (0) by default.
1 J$ |6 G ?5 ~+ ^ </description>7 o% n0 K$ t0 N f$ b& Y
</property>
1 A$ F% X* D" n2 Z7 ~* J/ m<property>* P K- V+ f8 k, @ A
<name>dfs.balancer.max-no-move-interval</name>
+ Q* j* i8 M/ @ <value>60000</value>
E U W6 m3 |3 M( y' `/ N' f <description>
7 p. z0 i2 F) p( ^# G# U If this specified amount of time has elapsed and no block has been moved
; ^+ k0 }. u6 ? out of a source DataNode, on more effort will be made to move blocks out of
! N9 L1 V( Y: s2 u7 ^$ j this DataNode in the current Balancer iteration.0 H0 s/ Q: f$ k v5 X5 B
</description>
8 f0 y. B/ {6 V% u</property>% i7 V2 o5 [) _7 M
<property>
! D$ I- T+ f5 J* x2 ^2 B+ J <name>dfs.balancer.max-iteration-time</name>) {! m0 W h* f! w5 ^0 z
<value>1200000</value>) F% F H4 ?. O' W/ {9 x
<description>
! i( ~( r; ?& a9 T* c4 U$ q4 R Maximum amount of time while an iteration can be run by the Balancer. After+ |5 v& k& V0 \
this time the Balancer will stop the iteration, and reevaluate the work
8 s% w! G# h" e needs to be done to Balance the cluster. The default value is 20 minutes.; @+ U! @! l- I2 r: ?
</description>
1 i. e D& O' }, p4 j: f" h7 k) f- w</property>
0 W2 q( `. L7 K4 O<property> E2 I% d9 g3 J1 |- t
<name>dfs.block.invalidate.limit</name>6 z0 G* z$ i3 K2 K& s3 x. @. ~
<value>1000</value>
* R& G, u3 |+ i+ H, O- E/ m <description>
) j' L/ b1 K6 }3 P- [$ w The maximum number of invalidate blocks sent by namenode to a datanode
- v% v4 e8 | F per heartbeat deletion command. This property works with" Q: e3 h, w4 }
"dfs.namenode.invalidate.work.pct.per.iteration" to throttle block
( {% ^4 e! ?' e8 D% X, \' M: Z deletions.0 _8 T1 @& n- m2 B
</description>3 k8 }' P. l. w4 D6 S$ n) _0 `9 v
</property>
& o1 r4 z+ z @! j* C( ]<property>
$ _9 y0 L: h* W" g. r <name>dfs.block.misreplication.processing.limit</name>5 r: e% L2 ~' u. u% L' R
<value>10000</value>& T* \' [5 t+ l5 ]
<description>( s. U5 N% P7 l& S( m; r
Maximum number of blocks to process for initializing replication queues.
7 p7 h/ s1 R C6 t. x$ B# { </description>
5 x; J6 N1 U1 m" F* Z9 l4 o+ @</property>
9 }2 ^& J, N9 E& _7 W, `<property>
0 y; o+ c* q8 W, A <name>dfs.block.placement.ec.classname</name>
" q v8 D0 k# I: w% y <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant</value>
1 C, p% [$ y4 U& i <description>
) N A' F( M0 b0 o/ D* U' d Placement policy class for striped files.
& F% C9 W3 B. t Defaults to BlockPlacementPolicyRackFaultTolerant.class, ?" q, y6 G: |0 ^: {+ r" l+ H
</description>: n' b+ e9 N: O6 b* R
</property>! F! d1 D$ M5 x- T' C. e7 w% ?4 O3 l
<property>/ ?. C8 T( {1 s( c! U; T# S
<name>dfs.block.replicator.classname</name>8 i) G6 K6 ~ r% ]* H6 r6 S+ \+ [
<value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault</value>
4 j2 v& o* W) a t. i0 v/ q/ X <description>
$ @* o) D, O# R0 `! [ Class representing block placement policy for non-striped files.
% X3 J+ _( T) ]$ X4 t There are four block placement policies currently being supported:" @; T/ h0 p3 i% m' _- j
BlockPlacementPolicyDefault, BlockPlacementPolicyWithNodeGroup,
8 {: x. n$ e* s2 l BlockPlacementPolicyRackFaultTolerant and BlockPlacementPolicyWithUpgradeDomain.
* _2 O3 P8 A$ M/ F* c% m BlockPlacementPolicyDefault chooses the desired number of targets
) G( y% O9 A4 q3 ?/ [, Q/ g for placing block replicas in a default way. BlockPlacementPolicyWithNodeGroup
$ D- @" | T+ a: P) l; Y2 d) d places block replicas on environment with node-group layer. BlockPlacementPolicyRackFaultTolerant# r0 k" ~' X R! ^
places the replicas to more racks. i% N1 v% t/ ?7 M; t: c
BlockPlacementPolicyWithUpgradeDomain places block replicas that honors upgrade domain policy.7 }( O( H: S2 j2 `
The details of placing replicas are documented in the javadoc of the corresponding policy classes.
! b- U4 a. x! m, K: P The default policy is BlockPlacementPolicyDefault, and the corresponding class is
% c$ c0 {" [5 P# @ org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
8 h1 f! ]5 f6 J </description>2 t) j* |+ L" p: r2 h* I1 n" `3 R+ |
</property>
) w% A- L8 q5 ]5 Q1 b& J" Y<property>1 A% ?! X1 N# Y
<name>dfs.blockreport.incremental.intervalMsec</name>8 Y2 ^( m, Z2 O6 E
<value>0</value>
0 K: m: j' [6 i6 O <description>. U3 o# z3 B; i- C- X" }* s
If set to a positive integer, the value in ms to wait between sending9 `7 i* `: R, C, C
incremental block reports from the Datanode to the Namenode.
) h% I, E7 t5 A# p! v </description>
' [( D1 ~) Y8 j" v% g</property>2 F- B5 q# B5 V9 H+ H& ~6 q: i
<property>
2 N: n& y& `& w" [ <name>dfs.checksum.type</name>2 D+ v2 }( ]' D' \% b1 m- @
<value>CRC32C</value>- k8 L) @& H5 X0 L5 u& F2 K* O/ p
<description>
( ?. @$ Q$ y& d' ]6 E Checksum type/ S) k+ C& c- R8 Z3 J
</description>- |/ u3 F5 q' C. t, p# w6 O
</property>
1 M& t9 L: P; P<property>
: Q: m/ z0 R% T1 ~: k* I <name>dfs.checksum.combine.mode</name>
* H8 Q* E! h5 J <value>MD5MD5CRC</value>
7 ~2 |' M2 F0 k) G, p <description>" V" z I( a7 `. ?! d" I/ f% h( O
Defines how lower-level chunk/block checksums are combined into file-level5 G; U0 s" A- ~3 k
checksums; the original MD5MD5CRC mode is not comparable between files! L4 n! B( K6 i0 T) M! U
with different block layouts, while modes like COMPOSITE_CRC are
$ d- m5 o0 u X- D comparable independently of block layout.$ A) q Y0 _0 e
</description>
, O0 q% W# l( l/ D4 D: P/ S* e</property>/ L" c4 a+ Y+ \: D' r7 q
<property>8 d) `" z2 l( \6 u1 J; j3 \
<name>dfs.client.block.write.locateFollowingBlock.retries</name>8 a0 u9 k: [: r: p4 U# i
<value>5</value>7 Q& j' z h f& b( A$ h6 S& ~
<description>
; Z$ A( E: I" m4 J4 N Number of retries to use when finding the next block during HDFS writes.1 _/ L! e+ V4 d L: S9 L) q
</description>1 ^& _$ a8 Y2 w9 j5 ~
</property>! H2 F" e# l/ M) ?) B
<property>
# q5 |6 L9 N" r$ A8 P <name>dfs.client.failover.proxy.provider</name>+ T4 ^4 r& o! G5 N# U+ ]. c
<value></value>
# U! N9 L, k+ ~/ ?1 U <description>
7 |3 \3 E0 v9 [: O7 J The prefix (plus a required nameservice ID) for the class name of the
) o" v) F& O9 L9 t$ ` configured Failover proxy provider for the host. For more detailed
4 N2 }0 ]/ c% ?! C/ a' w2 f" Q2 y information, please consult the "Configuration Details" section of
* ]1 d5 ] g5 V; D8 Z; Z: p, m8 ~ the HDFS High Availability documentation.) W/ M9 c( |: n' c9 C# f( ?
</description>2 ~, O: e8 N$ H
</property>
4 P9 g& z! P' F A4 d6 u<property>
& N5 C0 j3 q) E3 f4 d5 U2 _" C' D <name>dfs.client.failover.random.order</name>9 }, @# ~- z" k. b, o/ {
<value>false</value>
0 z) r6 P" Y3 p+ m T <description>
( H3 G% J1 ]% e/ p Determines if the failover proxies are picked in random order instead of the
! v& J8 U* g3 }* e, T# |4 S0 u configured order. The prefix can be used with an optional nameservice ID
' J2 R. ^' ]' _- P (of form dfs.client.failover.random.order[.nameservice]) in case multiple% b9 p }* w8 [* Q5 I
nameservices exist and random order should be enabled for specific
! X, _1 J1 F; f6 T nameservices.# L3 t9 w0 @$ n8 a
</description>
; S# b+ \% |( ~& {4 E3 C5 |, t- n</property>
3 O- Y* s3 ?1 I v<property>& n0 C+ K, y9 a! [( w, A
<name>dfs.client.key.provider.cache.expiry</name>
( Q* Y% v, v8 O! v) R1 K6 r z <value>864000000</value>/ ~, f, V+ q2 D. W
<description>
0 a1 D# p. f/ L) p, L DFS client security key cache expiration in milliseconds.
7 e' V I+ ?: k; \9 y. I </description>
8 {$ O# l+ F% Y</property>& T) _/ ^3 Z# `: M% y4 i U8 f6 Z
<property>1 Y8 ~% Z+ V' H3 M- Q
<name>dfs.client.max.block.acquire.failures</name> U' d2 j/ M. h5 J! F2 y
<value>3</value>
1 G4 Z/ L3 H. T1 z6 O/ b5 @3 I <description>; H& H5 y. G, r8 p9 _* v3 t
Maximum failures allowed when trying to get block information from a specific datanode.) {1 E, }9 \% h/ E) J6 z @
</description>1 l8 [4 ^& P& [* O
</property>1 k; u' ^- G3 b) c1 Q
<property>) @" C/ f. j) l! `/ o6 o. O
<name>dfs.client.read.prefetch.size</name>0 ^7 M- p# C" S
<value></value>+ ?! N& y7 O* Z1 G5 V% u! f/ d
<description>
, h7 q( e# @; F* ^2 o% h( _! B2 ~ The number of bytes for the DFSClient will fetch from the Namenode# ^: v \1 x% v6 ^
during a read operation. Defaults to 10 * ${dfs.blocksize}.
* r. }& s' k# _) C9 V </description>
* [' V* R/ S6 F$ p0 N$ R% i</property>; p4 c) J | Y
<property>
9 S4 _7 W' X* z! z; J( {# B <name>dfs.client.read.short.circuit.replica.stale.threshold.ms</name>- w2 Z" u: ?( c( N5 o1 `! l
<value>1800000</value>5 z# ^7 c5 r2 J9 f; t/ b
<description>
5 _3 x, e, w7 y# K$ o& R Threshold in milliseconds for read entries during short-circuit local reads.1 v: g( b8 R+ y: W) `$ z, B) B
</description>6 V& W8 H) W$ e" i
</property>
( f1 m( \ t, c$ B<property>/ \: P' i. S, L' G* w3 `
<name>dfs.client.read.shortcircuit.buffer.size</name>
/ p0 L% g3 Z9 f0 ^ <value>1048576</value>
! C3 b5 `( f/ k, Y4 N' t <description>
4 A% |$ H, a& ^! V& B) C2 X! v+ t" p$ o Buffer size in bytes for short-circuit local reads.8 A- f- ?; S% o! Z/ M% D
</description>
* m t, [7 c0 s, N. J</property>; J) v8 F5 g) @$ \, K. z, c2 W5 C
<property>) j& \/ S+ }. V: Z
<name>dfs.client.read.striped.threadpool.size</name>
& X. c8 b: K( D% n+ g, k2 X <value>18</value>' J& G3 `1 v* R( J8 o% {7 q
<description>
, O: r, t6 I$ ` The maximum number of threads used for parallel reading6 w: K4 m2 k, \6 V& p
in striped layout.7 d$ h2 X' z/ g+ x
</description>- J& s, n, {% o, d' \5 Z n
</property>
! W4 ?* x# ]) P. D# \* A& D<property>
" L4 O, J7 R [9 R8 _ <name>dfs.client.replica.accessor.builder.classes</name> U; x2 a: o, Q. u$ A6 F h" }' w
<value></value>4 g7 ^9 e; r8 w7 N# U4 d+ f8 y
<description>
& ^* m0 ^' g* G; I+ h, n Comma-separated classes for building ReplicaAccessor. If the classes3 z7 z; R0 O9 @$ t; d
are specified, client will use external BlockReader that uses the
, h( v) J$ b2 Y0 X ReplicaAccessor built by the builder.0 `5 B& l9 Z, A! f6 A
</description>2 I% P7 t& x' F7 z& X X
</property>
# W4 T" D( _* K; V$ n- D+ W) |<property>5 h; B- [9 x: n* w4 X1 [0 I
<name>dfs.client.retry.interval-ms.get-last-block-length</name>
N) l. n& Y6 z$ ~ <value>4000</value>; B# g/ N0 h, ^$ ]
<description>
4 k( M% e1 N7 } Retry interval in milliseconds to wait between retries in getting
1 F8 v: Q7 d/ e! S- Z block lengths from the datanodes.$ y9 d5 z8 [7 a4 N8 S+ I
</description>4 ^. W3 q, r0 P9 ?, M, c9 p
</property>' C* p C4 O2 Y) g, x
<property>; a# ? d) @1 F6 O( m
<name>dfs.client.retry.max.attempts</name>
. G7 C7 `, q) c5 V <value>10</value>
% s1 @ H: N. Z2 \; A! _4 } <description>& Y; ]& ~, A8 @/ z% t
Max retry attempts for DFSClient talking to namenodes.
8 w* G" |8 q& ? </description>' A- t1 k. X' }: j l
</property>. ~, N7 x7 `) p. X5 f# M
<property>2 x( H) Q. F0 ? M- \! X
<name>dfs.client.retry.policy.enabled</name>
3 @/ ~( }8 X# ]! E; N/ m0 A <value>false</value>
* s8 n) p8 h; N9 {5 K6 N <description>/ G2 Z! q8 Z3 F" g9 @) z9 b5 e0 f- B
If true, turns on DFSClient retry policy.
9 \/ @! X- L/ e8 \" X8 a9 c </description>
; Y+ _" ]2 L7 ~( @" S' R0 C$ X% u</property>
; n# g, r3 k, g. p& y# m<property>% p! I: _- w+ S" F+ a
<name>dfs.client.retry.policy.spec</name>
" s2 s! V4 X' }7 K/ X8 m <value>10000,6,60000,10</value>
3 {& Q; `, a; D! | <description>+ b3 A+ O6 O& H( T! S3 _8 N( A
Set to pairs of timeouts and retries for DFSClient.
, p9 i7 d+ r* ]/ V; U2 x6 J& g$ \ </description>
% D8 U) M* M0 t# Z0 f2 l0 l1 W( c</property>6 r5 U/ {' \' S; ^9 E4 Y; a0 X2 W7 c( @$ b
<property>0 f4 z* N% F, {3 S6 K
<name>dfs.client.retry.times.get-last-block-length</name>& b9 x6 j/ {. T9 l7 y/ o7 F( K
<value>3</value>9 X5 K. Z5 d" J. m5 ?/ B
<description>+ d6 |4 a- W% |5 W8 Q+ k/ S8 q1 B
Number of retries for calls to fetchLocatedBlocksAndGetLastBlockLength().
8 V1 I- t4 o8 D, o1 N f8 E; O </description>
- h' L9 A7 |" A2 l8 G3 e; E/ D* |</property># d6 V5 n y& V( j- ~" u! t
<property>8 m; o: F& ~: n4 K+ s0 l T K; I# z
<name>dfs.client.retry.window.base</name>
% d) n1 w0 F7 E/ f/ _2 c1 z: d F+ q <value>3000</value>! B6 }: R0 W8 w- V
<description>
" y" }3 F2 M8 x8 E: c! U4 A5 n Base time window in ms for DFSClient retries. For each retry attempt,1 p' V" l5 X1 `/ t
this value is extended linearly (e.g. 3000 ms for first attempt and
. G: T: T0 F9 N' C* G first retry, 6000 ms for second retry, 9000 ms for third retry, etc.).
9 i" z- D, l% v5 V* c! a </description>
' n& z c, U/ b+ Y3 h% o</property>
) y8 Y4 w0 Q/ K$ j' E+ i<property>
R8 M( D% Y3 y4 u <name>dfs.client.socket-timeout</name>
- r- i- ?& Q, n j- \ <value>60000</value>
: {. w( ?1 g) R9 N+ o- K) k% j <description> V7 D5 U% |+ e2 T) X" }
Default timeout value in milliseconds for all sockets.
, J4 Y0 u: F- g' @1 g2 G7 S </description>
/ L% ^% e. s6 j' Q- z2 N$ ~9 t</property>
3 F% X! v- X. _! e: Y4 P<property> Z' g9 V$ y: S
<name>dfs.client.socketcache.capacity</name>
. Z* `5 }" b+ p2 C3 H* w <value>16</value>
" N+ B% M$ @# h2 Y9 @ <description>1 f" O/ \0 i5 {
Socket cache capacity (in entries) for short-circuit reads.% f( C" e) L P/ q
</description>
9 k' b- J9 y6 t2 ]/ Z/ B. S, }</property>+ M+ _7 t4 I% ~" l
<property>9 ~, R; k4 T- N1 p1 ?
<name>dfs.client.socketcache.expiryMsec</name>2 Q3 L& c- i- Y6 z5 W4 r
<value>3000</value>
2 I2 c8 P6 ~1 D% L- { e4 m <description>
6 Z6 b5 \! H( s1 r( V9 R Socket cache expiration for short-circuit reads in msec.
5 ^8 T, M2 a) l </description> Y1 g) O ^; Y3 j! \
</property>
+ ]: D# k; @/ T& v) n+ h3 g2 n<property>
2 b8 V5 _2 v% y) V/ a9 n: d6 j <name>dfs.client.test.drop.namenode.response.number</name>
0 ]% F& k7 h; A1 R1 l <value>0</value>: D6 O4 d0 e: G: X4 @
<description>
3 W" [4 T9 s. S4 W c The number of Namenode responses dropped by DFSClient for each RPC call. Used
& y9 T( X; N n for testing the NN retry cache.
# e& t* @2 m0 a3 W3 j </description>5 J+ R; l0 F* |' @4 M9 Q" }) e
</property>5 t' C) `1 ^( P! I! j5 ^5 O5 {
<property>
! c. [1 b0 ?5 B) o3 s* s$ \/ y <name>dfs.client.hedged.read.threadpool.size</name>5 |. B1 Y9 y* _3 U; u
<value>0</value>3 {1 e! [+ ^4 G: u
<description>1 z" r- D. E1 {
Support 'hedged' reads in DFSClient. To enable this feature, set the parameter1 k" Q/ Q# j; f, Z
to a positive number. The threadpool size is how many threads to dedicate; q. [% ]) ^& Z1 e
to the running of these 'hedged', concurrent reads in your client.5 Q5 L1 V3 B4 q" w; u$ I" U
</description>1 z! q4 e" p. O) R9 G
</property>
4 H6 f7 j7 s7 \2 A) B<property>1 g' S) g4 v6 O5 W: }! g6 n
<name>dfs.client.hedged.read.threshold.millis</name>$ x5 y3 s8 C6 }1 s0 |) Q
<value>500</value>5 i" G' _& g) `; Y' \- Q- `$ f
<description>+ x, M7 |' D) Z- n, x
Configure 'hedged' reads in DFSClient. This is the number of milliseconds
; ~! }5 d" D3 _, x+ s to wait before starting up a 'hedged' read.
6 R9 `" m, G. I. }. | </description>
2 x1 o) ~/ M3 B' H- p& \6 f</property>% n) I: X7 c- V" R
<property>
5 d8 `4 o0 L2 t1 ^ <name>dfs.client.write.byte-array-manager.count-limit</name> s3 c/ g: ~. m5 p3 S# m; c
<value>2048</value>6 L- S: z5 h. _) ^" O' n+ |
<description>0 U f5 d2 s7 {* i% s0 m
The maximum number of arrays allowed for each array length.( c/ z: e* d$ x- a: w9 y) U! f" v
</description>2 N, i; q7 D* [% Q- r3 a
</property>
* R, y8 w" B& h" d<property>/ x% F) a# W% Z4 @
<name>dfs.client.write.byte-array-manager.count-reset-time-period-ms</name>4 }' O1 W i) o. h% c
<value>10000</value>
6 l7 G/ J6 M. d6 z% ~+ b0 u( | <description>
9 n* H3 z5 e# Y The time period in milliseconds that the allocation count for each array length is: T6 ?$ k+ h: l" m& {" d
reset to zero if there is no increment.
6 L, `% {4 x$ I# I( P </description>$ d7 D* G0 Y. ]6 K" j1 H
</property>
1 ^) G4 J2 \: O8 v<property>( @2 p$ X7 u% m: h1 h0 t
<name>dfs.client.write.byte-array-manager.count-threshold</name>
" A: F* O& f5 t5 {2 V% s <value>128</value>
( K6 n u! D$ A ?: N2 H1 f0 {5 u <description>4 `+ _: g# T. u. n
The count threshold for each array length so that a manager is created only after the
$ `( H/ Y( x* c l4 E' X% K) d$ j allocation count exceeds the threshold. In other words, the particular array length
9 O2 Y2 e, k9 S' @( O, u is not managed until the allocation count exceeds the threshold.: l: K1 m+ o! I$ `! S; g! R5 e0 \
</description>
$ p) E' A u/ K H1 ~9 R</property>
0 G: ^# _4 @7 Y& o0 H. N<property>
7 l0 z! M, e7 l9 R! ^ <name>dfs.client.write.byte-array-manager.enabled</name>
$ `/ Q! t. F% C, w2 M9 v9 ~3 P <value>false</value>
3 C4 a5 p2 }5 t- o% q+ c5 Y <description>
* u$ l1 p3 Y3 B* [5 y If true, enables byte array manager used by DFSOutputStream.5 z2 v" N, y/ r f' | ?/ P" T% U" c
</description>0 q' F5 L8 b+ L
</property> k9 r8 T. Y& p
<property>( C3 J+ V! V2 T& c' o
<name>dfs.client.write.max-packets-in-flight</name>, m/ Y; @$ u8 @) ?
<value>80</value>
" k* G& ], ~2 k- [ <description>
) i3 X* `* o+ k6 [" S& I5 K The maximum number of DFSPackets allowed in flight.
- E1 o, w. N0 ^ </description>+ g) o( c$ D4 k4 {
</property>9 u' P0 J7 i, J
<property>
4 F# v: _8 m0 H' z <name>dfs.content-summary.limit</name>
6 N0 c& V- u* k- l1 d <value>5000</value> w& {; T: H2 N) M! }, I: Q1 O
<description>8 C& o5 h- x: Z+ f8 C! }
The maximum content summary counts allowed in one locking period. 0 or a negative number
1 D6 t+ Z6 v, X% ~+ V+ L means no limit (i.e. no yielding).
* I( d1 T6 V' [" _ </description>
% m6 X* m) d' r</property>
" g6 N; E: m2 m<property>
/ o% V! E9 K& z <name>dfs.content-summary.sleep-microsec</name>5 ]+ ?4 p7 s# v; F
<value>500</value>2 ]4 Y& S" k9 F4 _1 H K* L6 ]
<description>
* a1 C" m8 @4 i2 g$ u* P The length of time in microseconds to put the thread to sleep, between reaquiring the locks
' V; {+ B- C* _5 R% m s in content summary computation.; `1 ?' G1 Q) \0 J2 O& x
</description>
4 U; r7 i) T+ I</property>0 M3 z7 Z: R( n' T2 S
<property>
, |. e' Y" g4 D. {! r( Y <name>dfs.data.transfer.client.tcpnodelay</name>! J, K h# r/ E( A- t3 y0 H
<value>true</value>
0 R% k- L: ]- l9 D* C <description>
J# o# L+ b- C5 T' @1 A% ~7 ^ If true, set TCP_NODELAY to sockets for transferring data from DFS client.
, v* H" _8 J9 j& v8 ]+ D# W </description>/ D# E4 S* P g/ @7 K$ ?
</property>$ a; w; h6 F% f8 |
<property>
8 M% Z4 A' x: S' O5 N( E8 v <name>dfs.data.transfer.server.tcpnodelay</name>/ ^1 B( {! S9 Y8 ~& [+ j2 E
<value>true</value>
4 `( x. ` I9 p <description>
6 a5 O6 }% R; L& {9 a, c If true, set TCP_NODELAY to sockets for transferring data between Datanodes.
4 |* F+ Q. j0 k2 r" l </description>
( x- h, o. X7 L ^. X</property>; k6 _/ g) m& u- G3 m/ F
<property>
8 B1 t# r; G! p! V% t# \ <name>dfs.datanode.balance.max.concurrent.moves</name>
$ M$ x4 x* Y- g* E5 { <value>50</value>
0 X6 L5 H+ u; v. ^2 E <description>! Q! G3 ^9 y" B f
Maximum number of threads for Datanode balancer pending moves. This
+ f- k* n t+ ?8 C' D4 o value is reconfigurable via the "dfsadmin -reconfig" command.7 p- l' s. H0 R% N- v( F P
</description>1 D- Y" N Q, G& k* j
</property>& l6 ]; i8 B9 `; ~
<property>
4 S: K5 a/ m& p- E }9 O. R <name>dfs.datanode.fsdataset.factory</name> a' Y0 w2 w' h6 v3 E# ^
<value></value>5 c* v0 C) ^3 M3 u( j3 w! I0 d9 v& {
<description>
' ]$ ~6 y5 L/ e9 W! t5 P* z/ P The class name for the underlying storage that stores replicas for a, a. X! g% r' W/ }# t9 q! V
Datanode. Defaults to1 f( `& j* i! K( C1 f* p
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.
8 ]4 t0 b% }8 A! G, _! T0 X </description>3 q! x3 `8 u5 q2 l
</property>4 k. d2 T+ B t5 U N9 s4 p) ^
<property>
, ]& x+ G( T9 t5 r. [0 t <name>dfs.datanode.fsdataset.volume.choosing.policy</name>
; u) l/ D$ r- k. u# e* ~+ y! s F <value></value>9 l& }1 D/ f& T! L
<description>
! t0 i7 m# X/ `& ] The class name of the policy for choosing volumes in the list of8 W9 o4 V& L! f0 f& M
directories. Defaults to) {9 X4 c1 g) D8 Y4 [% X ]
org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.1 ?3 v3 s8 n5 G5 r
If you would like to take into account available disk space, set the5 z# F8 ?* J' u2 J
value to
0 L9 d7 Y2 [& G% E/ e "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".' R K8 [& @) M2 N) N
</description>
`! U8 G0 @+ @/ }7 u/ z, L5 e</property>+ A3 j% d) G$ ~9 {* [
<property>% D1 k y( m b% `$ e2 q
<name>dfs.datanode.hostname</name>8 c4 O8 y+ U/ D- @: C
<value></value> A& J1 Z- L3 z2 P4 |# [. H4 z
<description>; N& R# Z$ B* [: U1 F# Q
Optional. The hostname for the Datanode containing this
4 K8 a4 i& B( m6 y4 b configuration file. Will be different for each machine.
5 k3 v% [% Q% a( \2 v Defaults to current hostname.
# N! d- C# N6 T9 q0 X </description>- h4 U/ J8 h; u, b
</property>6 h1 M6 W2 M7 z. s8 e. C. C5 t
<property>
- v# O6 E8 S5 r9 }- v8 C, p. F) r% { <name>dfs.datanode.lazywriter.interval.sec</name>; w& a8 S N) X% _0 E# P D
<value>60</value>% I% y+ M! o e: O! n9 w
<description>9 j1 W8 t0 M8 f
Interval in seconds for Datanodes for lazy persist writes.
6 Y; o5 }9 U; Q9 Y. @ </description>: u: N) S' k, C/ f q# o3 L! n
</property>7 N* _( b: p- [8 l4 ^/ S
<property>6 I) j* e, z7 g$ c7 H J
<name>dfs.datanode.network.counts.cache.max.size</name>: \0 N+ x( q% R7 p
<value>2147483647</value>( S3 {4 u* } _' i
<description>
3 b% [ U4 L, a' a The maximum number of entries the datanode per-host network error& t& ^: A2 t8 n8 A2 l6 X. V
count cache may contain." M* A; l( Q, W! [- C* Q5 y
</description>, K5 H% j2 S1 x9 [ ]' M
</property>
2 O- Q/ @9 K$ Q<property>0 L% W% g. p9 C; q4 R+ S0 A
<name>dfs.datanode.oob.timeout-ms</name>
3 c, q7 P8 r) z2 L* R <value>1500,0,0,0</value>
% o' X: m4 V: s+ Q- K <description>
/ g3 q7 O( O1 u8 W, q/ m Timeout value when sending OOB response for each OOB type, which are
; x5 A5 ?) v1 B0 z OOB_RESTART, OOB_RESERVED1, OOB_RESERVED2, and OOB_RESERVED3,7 |6 Q9 {3 ^# `2 x5 L
respectively. Currently, only OOB_RESTART is used.
8 v" m, U- b1 Y+ q/ | </description>
: e8 U* n H) O- L) n6 O</property>/ a( `1 h9 P) D& x9 f: W
<property>
0 U( e/ y4 i3 M <name>dfs.datanode.parallel.volumes.load.threads.num</name>: e, x* h5 N* t# l' R( f5 w
<value></value>+ E, R. K3 N4 m; A% U! o
<description>
. [4 `4 x0 G& r( q" ^+ r' W Maximum number of threads to use for upgrading data directories.: r& A2 T* N, ]* i. e9 \
The default value is the number of storage directories in the& I. _: y. V; x# O1 f' I5 T* G: m
DataNode.
- K' w) s; P; S5 k </description>6 r& {! f" e0 f1 X# A% _
</property>
& J) P B: l2 z2 z0 f. P<property>
6 J$ u* |# X' P/ V7 v5 j- S% r <name>dfs.datanode.ram.disk.replica.tracker</name>
# M/ k% q# `% n) Z# D- @* l <value></value>$ y% O6 F4 ]) K2 y2 A
<description>) V+ ], @: d' v
Name of the class implementing the RamDiskReplicaTracker interface.
$ W0 A' R5 N6 E# l Defaults to0 w# A2 M" o' ]. T
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker.& j+ c! M, J1 J; z6 ~
</description>
U, J' K0 Y- S w</property>% `8 V3 S5 X1 |$ _3 `
<property>
" Z7 M* }3 L- I/ H6 w8 c$ w <name>dfs.datanode.restart.replica.expiration</name>$ o o% U3 a% j4 |
<value>50</value>6 J7 d/ d2 p2 `/ M$ }8 |
<description>+ C# d3 N% Z: i9 @& W k% S
During shutdown for restart, the amount of time in seconds budgeted for
# f% T6 V x$ _2 x! I" j4 M; L) f& m datanode restart.
/ U% r% x- O$ b# t7 B4 P9 ] </description>& O) C8 {! V8 [! C) L
</property>
4 B' P9 V! X4 y5 S- p8 K6 J<property>6 I( `2 k7 K# U- G1 ~ w0 n; F9 \
<name>dfs.datanode.socket.reuse.keepalive</name>
D7 Q" A, {, Q( ~ <value>4000</value>( u3 D, g% l& g* X+ E; E; u
<description>
" |; y, x1 P& w8 m( d1 \* Z4 ~2 T The window of time in ms before the DataXceiver closes a socket for a
+ a1 H) y8 F/ a% n: `7 P2 f single request. If a second request occurs within that window, the
1 x6 t) M: B0 g3 f1 } socket can be reused.
* v% f, V" r* i8 ~$ O/ ~ </description>! V7 h2 A2 x5 v3 |: B/ A3 m, @
</property>( y0 L! |. l& e5 S( q& k
<property>9 A5 d6 S- u. c, Q- g
<name>dfs.datanode.socket.write.timeout</name>
" C5 E/ Z. l6 X5 Q; _5 u <value>480000</value>
' ^% F! I5 W/ q T7 } <description>
8 f% @ t) @% s3 Z* g Timeout in ms for clients socket writes to DataNodes.3 J( L7 x2 E& T
</description>
K) ?' W! @6 Q7 ~2 u</property>" |3 _6 E$ O8 b! O( w# Q
<property>& |6 I+ k6 Z( {8 m# k8 m
<name>dfs.datanode.sync.behind.writes.in.background</name>
/ K* n |# X1 N6 F <value>false</value>
3 Y& d* n6 i! M <description>& D( T# g* A; i. O, N
If set to true, then sync_file_range() system call will occur7 m4 \1 S' g' ` K, Z
asynchronously. This property is only valid when the property0 C7 ]5 E+ f' k$ V2 ? g0 B
dfs.datanode.sync.behind.writes is true.6 x1 S; F2 X& m% v9 }) H8 A
</description>
7 k% r. {& A- u2 `3 S; c</property>/ T# Q0 d- f8 U0 c
<property>2 y) u' x% u# ^& O* Y$ E
<name>dfs.datanode.transferTo.allowed</name> e* M4 ~5 a" g$ j' a" N
<value>true</value>
$ L6 _' h4 H H( k <description>
2 h9 k' |" ^9 \6 x1 k4 T M( g If false, break block transfers on 32-bit machines greater than
2 L" I5 y$ Z3 o( j or equal to 2GB into smaller chunks.5 X9 |! [2 d9 x0 M3 L Y% |( j
</description>0 ]! r& b V" S, y7 ~
</property>
2 |/ z+ k% B' g' o<property>4 U4 T. F1 A- U* `9 i( u
<name>dfs.ha.fencing.methods</name>/ q3 q! s9 Z* X: C5 S
<value></value>1 _! C( [$ t% k `) ]; D- G, p. t6 n
<description>" d4 c4 ], @: p$ r N
A list of scripts or Java classes which will be used to fence, o/ @' I6 u$ A9 X7 K
the Active NameNode during a failover. See the HDFS High- k$ z, }5 K ^" X/ @/ `) U$ j, c
Availability documentation for details on automatic HA; V# w2 y9 [0 x. z- P9 o1 H- O1 x7 ]
configuration.
& c l9 c6 u5 u! P, ? </description>
( N- C- n& d/ `% O* [3 h! U</property>
2 X# e; a+ _3 m0 q$ e% g8 b<property>, S# e0 I7 V1 g' E* D
<name>dfs.ha.standby.checkpoints</name>. @! E+ K0 m" t
<value>true</value>! V7 c4 R; A2 A0 v1 B @( q
<description>
1 h% N4 {) c! a+ \' Y# ?, ? If true, a NameNode in Standby state periodically takes a checkpoint
% [: ^8 {' ]7 ?: E# c2 L3 F/ {2 L of the namespace, saves it to its local storage and then upload to
. |3 E; {- q5 o/ P: M the remote NameNode. l& n5 E8 F3 m6 H! r% ]8 G* J% n
</description>
2 X1 o x, N/ R+ J9 _</property>2 F* `9 H; ]! s" `. A7 _$ x) m+ H0 @1 v
<property>
: {: {- j; e: F/ {; g <name>dfs.ha.zkfc.port</name>
0 t1 U& x- H+ a% m1 E <value>8019</value>
5 _3 x: _5 Z! r" @) {1 t <description>
8 f0 N( C& m( H% D The port number that the zookeeper failover controller RPC
9 v% d. N: | A( K% n server binds to.3 _. o/ P/ U& f
</description>$ k" r+ A; o" b+ D
</property>
. Y% M; L4 v' |) H/ _5 s4 x4 D1 O<property>
- U0 D! h+ d) }! `8 @" s! j <name>dfs.journalnode.edits.dir</name>
. W' K* t2 y8 H) p" _7 O <value>/tmp/hadoop/dfs/journalnode/</value>
6 j6 \. T2 y2 A+ c& S% v <description>9 {9 V8 m+ u! v
The directory where the journal edit files are stored. u7 ~, m- x6 Z# T) o
</description>, r; P8 B: e+ c. g2 p- Q6 W
</property>! \% w, |3 l: g
<property>
' M5 w* y; [% M# g2 u2 K <name>dfs.journalnode.enable.sync</name>% s! o6 m4 ]) X% b) v
<value>true</value>7 g( R) X; j# m$ x! l5 i& v
<description>
; [0 W% X5 t, ]3 o! A If true, the journal nodes wil sync with each other. The journal nodes
# k( [/ Q* \% n- i will periodically gossip with other journal nodes to compare edit log
" n* _& _+ S' O, e* b manifests and if they detect any missing log segment, they will download
8 K+ K Q0 _2 {8 @3 s9 z it from the other journal nodes.8 R# _0 l, |# }5 ~! _# a. T% u
</description>+ q2 z+ _; [ K, C- t/ A
</property>
: p/ M% l% P0 }<property>
7 N) n5 D/ ]( f. V, _, y% w! Q D <name>dfs.journalnode.sync.interval</name>2 {" p6 W1 \) @0 ?! g5 c. |
<value>120000</value># w, _" \+ p }& `% [
<description>5 Q% W M5 B" A i0 z2 ~: ^
Time interval, in milliseconds, between two Journal Node syncs.
" u7 l7 K' v; @/ H This configuration takes effect only if the journalnode sync is enabled7 H- U! O1 ^8 c* s6 ^( ^8 V# h. S& H
by setting the configuration parameter dfs.journalnode.enable.sync to true.; N) ?$ P4 M4 B: B9 T* P' v+ F
</description>
' z( d( d, S, V& P1 Y0 q m</property>
3 P/ s& E$ f; K, _) R<property>
) I- n0 D1 n( R. H <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
+ }5 }' q2 h5 V+ h) a D7 V- ~: l <value></value>" ^ B+ M! o; e$ ^
<description>
, O7 ^3 \" Z- j A1 E Kerberos SPNEGO principal name used by the journal node.1 ^' J! z+ C) B# }; O
</description>* U( S$ }6 J' y0 d p3 v, I5 @
</property>
" @7 h1 ?+ H* ^/ v+ g! Z<property>4 w, b7 t) R7 b4 J9 V
<name>dfs.journalnode.kerberos.principal</name>7 Y$ t: X9 n( U* ]* E$ c
<value></value>
# c- M' E1 X' q1 _: I <description>
" h* D. ~3 U0 Z4 R5 Z Kerberos principal name for the journal node.
. T b7 i' v2 ~ </description>
# H/ v' h9 H% B( U</property>
+ S8 l# C1 a& F* W<property>
& u' H' a1 T3 k# Z <name>dfs.journalnode.keytab.file</name>, ?( f. Q* p9 ~) {3 M/ a& C+ h8 @
<value></value>
0 P/ E. J' A4 j& v& t5 [9 x <description>1 F: T2 c# O0 m! l8 G% }
Kerberos keytab file for the journal node.( ~) a* ]- l1 @" P ^
</description>
5 o" ]; ~( b" z R, J</property>9 U: q0 p9 p4 ?+ }$ v
<property>
! W. U T ?9 i) K% \ <name>dfs.ls.limit</name>) X3 H- w+ ~3 k; ]4 P( C. T2 N
<value>1000</value>
0 C) O" ~8 H. a. ]& D' \ <description>
* u1 P% i% o: l Limit the number of files printed by ls. If less or equal to8 u( n1 n- w6 `4 ?+ A- y- J1 {2 H; _
zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed.
; _, }9 i0 n& t1 n, x& d2 d </description>
: O7 I4 [0 a b; O; r$ g2 p; y</property>
4 T7 `' A% d/ e% q! k# G<property>
- h% @; @ X: q7 D2 \4 r: p <name>dfs.mover.movedWinWidth</name>* V- d) p' i! l# t: w' y. O, t4 o
<value>5400000</value>
7 \1 Y- K' Y* i- f8 N" ~2 k <description>3 ~3 K7 v# I/ x4 a3 z/ Q; B6 G4 N
The minimum time interval, in milliseconds, that a block can be
) [+ V+ Z* [, d moved to another location again.
5 q o, ^$ B2 v8 G b" F2 c5 ` </description>
3 m7 e# V3 E9 g3 R+ I</property>; y) b- T7 Q& y& w/ x
<property>& M) e; U( O5 _
<name>dfs.mover.moverThreads</name>
) H# i7 `7 A- E <value>1000</value>
J# {! w+ i7 T( U$ L4 J <description>: i4 o) m I8 j Q5 e
Configure the balancer's mover thread pool size.* u) W" u- `) H% l/ r
</description>. E) @5 X* Q9 S0 g/ j% I6 w
</property>% c7 J& c4 C' ]& {
<property>
' c [3 t9 q4 X# S/ S <name>dfs.mover.retry.max.attempts</name>
# h+ K; _/ Q8 y <value>10</value>
0 O* R/ @6 M- U/ z& n2 C5 N4 S <description>% o% Z$ D; v; ?# Q: ~8 Q: n
The maximum number of retries before the mover consider the8 ~7 g, y1 u+ V) W
move failed.
: R8 W, q4 E2 Q: T) A$ V8 z </description>5 ^" l. t& e- o, a# }; Q3 z0 X) S
</property>9 ]- W! p5 [: r/ g/ \8 g3 O
<property>
3 j0 M0 _( W+ x! l5 K: c <name>dfs.mover.keytab.enabled</name>/ Z$ f+ X# F. `& `+ G/ E; s
<value>false</value>4 ?6 d/ f8 A) C. ~8 e* V E' X
<description>
! V7 l' a; u- }" W2 y4 r Set to true to enable login using a keytab for Kerberized Hadoop.
2 T0 f$ m m, I0 J( P- s* c" K </description>
7 B4 a( ~: h# t# G</property>% o1 i+ d/ g9 S) O. q, p- L' ^% ^
<property>
* {9 P9 p3 L1 t) Y3 { <name>dfs.mover.address</name>* g) K6 C! p- X( j9 W
<value>0.0.0.0:0</value>
, Q# A! N% M3 F2 }/ O" X" r <description>
8 l3 V- @& |" m The hostname used for a keytab based Kerberos login. Keytab based login
! s# C; Z+ k; u9 L6 ] can be enabled with dfs.mover.keytab.enabled.1 u% X% Z* P* B. Q$ j i
</description>) f: b7 X6 m. c# J8 ~3 U& J
</property>
2 U2 v9 q$ s) e1 r! O<property>( F9 g$ ]: h% V& e
<name>dfs.mover.keytab.file</name>
, G7 `* A) m7 X6 Z/ d' Q3 E3 H <value></value>0 a0 }8 W+ K+ \* U
<description>7 [% p1 W% X: H# X9 P
The keytab file used by the Mover to login as its
2 D5 R6 [8 j9 d3 J0 L$ ]$ u1 H service principal. The principal name is configured with: M7 b& F- s3 e+ ^. i: c# b# @
dfs.mover.kerberos.principal. Keytab based login can be5 f8 z3 X) F, W y, N2 u
enabled with dfs.mover.keytab.enabled.
/ M4 P9 _- P3 M! n7 I8 D </description>
" a# U4 f7 k2 K( P' B$ t2 E</property>; o t5 c! D7 L7 u. U
<property>
5 T/ t+ x9 i, G/ V5 v7 e3 E <name>dfs.mover.kerberos.principal</name>. C2 y) B, u8 U0 [
<value></value>
A/ N7 D% {. M) O- J V+ M <description>' Z2 U# l- E7 `
The Mover principal. This is typically set to# c% |. F% F2 w, F
mover/_HOST@REALM.TLD. The Mover will substitute _HOST with its
- Z( u" z3 _) k( O4 h" s7 M own fully qualified hostname at startup. The _HOST placeholder
% j) c8 r+ R! x# T) E( G% Y allows using the same configuration setting on different servers.1 V d) A& P" W. B& A
Keytab based login can be enabled with dfs.mover.keytab.enabled.
# R: K4 R0 G3 ~ </description>
! n( ~& G1 C% b `6 T3 J/ Q4 {+ b</property>
0 e& W, T: v1 R2 _<property>2 s6 ]. t7 ?& t. i
<name>dfs.mover.max-no-move-interval</name>/ j5 L/ h" U; _3 w3 I
<value>60000</value>
5 g {$ s' y. G9 U! h; ?5 [+ |# W <description>9 g, K) M, L' P! \" ]: L
If this specified amount of time has elapsed and no block has been moved
) l2 w1 b7 }: C8 } out of a source DataNode, on more effort will be made to move blocks out of
: q1 a/ e' h' |; }/ T1 F this DataNode in the current Mover iteration.
# R. B2 Z. a2 J5 H </description>" a% Q" m; x6 j' S% n
</property>$ H1 F) v6 k, @* z/ C; H, f
<property>
9 e' N5 S4 ~ [: A/ J1 W! O/ k <name>dfs.namenode.audit.log.async</name>- G8 v6 P' Z# Y* _1 u' V' I
<value>false</value>
! B$ H& u& `$ d <description>3 b x; z, T6 R! n7 I B; L' f
If true, enables asynchronous audit log.
1 O2 Q% s$ q$ c, @$ y+ x </description>, k% T$ Y( `* ^
</property>8 K$ ]# b- G- B: ^$ i4 i/ A3 a
<property>+ ~; y1 r8 C$ `( `( x2 ~/ S
<name>dfs.namenode.audit.log.token.tracking.id</name> a: t+ u% p1 h) O
<value>false</value>+ g$ y- t5 g! r+ U: c, g
<description>& b: v* b& k" A8 q$ M& G; O
If true, adds a tracking ID for all audit log events." e- H) C: @( N) W7 V, b
</description>
7 `: }. A3 g8 c/ G; i$ B( H</property>
8 j0 h8 z$ F9 ~1 k( b- n<property>
* y! P1 V, Y- |1 X <name>dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction</name>0 ~3 `$ U4 n2 k4 r2 m. i
<value>0.6</value>$ S, \! d: H( R- y1 a6 M a
<description>% i( o: n! Q8 j! g# _
Only used when the dfs.block.replicator.classname is set to. c* ?5 c% i. \. o( M
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.
. d4 H Y. r" w& p0 N Special value between 0 and 1, noninclusive. Increases chance of# p, O1 [; u- n" @
placing blocks on Datanodes with less disk space used.- Z5 [" ^+ R: L
</description>
8 k6 i# s- `: M</property>
0 T6 d+ d( ?& N5 z3 P<property>
8 U# f9 ]" j ^& x% I( g <name>dfs.namenode.backup.dnrpc-address</name>2 s3 P0 t5 k; O
<value></value>
. M# l8 |1 C- ?1 i& N$ K: G( _ <description> P ^) D' e1 f5 ^
Service RPC address for the backup Namenode.
0 c5 s. ]: `* ]' v4 ^+ G </description>
& i4 ~5 T% S- [4 z' z" c</property>
. d" D$ v0 Y1 {: W; E3 p+ q+ c7 [<property>' U D" r% I* M& `6 F
<name>dfs.namenode.delegation.token.always-use</name>( ^8 C$ P; b( J/ n8 X; g
<value>false</value>
2 l' W% z3 O' l+ p# D" L; Y <description>5 m0 T$ r. E } Q! g4 t
For testing. Setting to true always allows the DT secret manager
4 J4 Q; q& r( L/ k; K to be used, even if security is disabled.
$ O* r' M9 w) r) ~ \ </description>
3 u' B/ b+ ~4 s' p</property>
5 l& D; @2 K' h, {$ r& X5 ~1 S<property>! G, E- G+ {8 T( n. Z/ ?
<name>dfs.namenode.edits.asynclogging</name>% A- h" Z; g) Y0 ^( |0 h
<value>true</value>
, T" a1 v! I6 U6 g: Z <description>
5 r: y# ^+ K) T6 r/ \ If set to true, enables asynchronous edit logs in the Namenode. If set% Y9 W& d% K+ @: ^, A# ?3 ^: O
to false, the Namenode uses the traditional synchronous edit logs.
. K4 n" F! L. J! n/ C </description>- x( @3 K! K0 n) I- A9 f& e8 `( Q
</property>- @3 ^- E+ i7 v2 K
<property>
2 S, t. E" S; j4 R" P) ? <name>dfs.namenode.edits.dir.minimum</name>/ u+ P% A: z4 k/ _
<value>1</value>( r' m+ S; Z4 X& p) q! O- ]; t9 ]1 x* ]
<description>
1 ^% c- c* f# d; E8 D6 |9 x0 I dfs.namenode.edits.dir includes both required directories
2 c* N5 s& f' K% s6 `. a (specified by dfs.namenode.edits.dir.required) and optional directories.( T* q. W! X0 u# n( L
The number of usable optional directories must be greater than or equal& S, }- [8 y9 [6 `% v
to this property. If the number of usable optional directories falls
; {; R0 \& E \" {& L below dfs.namenode.edits.dir.minimum, HDFS will issue an error.+ f: o/ v; Y- ], p+ B( ^0 n8 @ _2 K
This property defaults to 1.
- x- G* S5 b5 C; U! S </description>
2 G+ B0 f! @2 N8 k1 p</property>
* [# B" n* Z1 }: z<property>0 M/ \1 D; F5 v6 n- Q# l! z9 i
<name>dfs.namenode.edits.journal-plugin</name>
; a( |4 F" }8 G <value></value>
" Z: d. l( b2 o: g4 K% I0 n" V! G- O! d <description># G- {/ P* h8 d
When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir,
% d! R5 q$ w+ u and it encounters a URI with a schema different to "file" it loads the8 X# M/ z/ u) s
name of the implementing class from
2 ]. z1 \$ _1 o: s ^4 U "dfs.namenode.edits.journal-plugin.[schema]". This class must implement
9 H7 p4 V, P0 `$ _" d0 { JournalManager and have a constructor which takes (Configuration, URI).
! b) h( s" o. W) {. c </description>7 z4 P+ Y" | |. H' `
</property>4 r9 Y5 c8 Y7 \5 A2 _
<property>4 [* U9 j" ~% }. N) z$ t7 r
<name>dfs.namenode.file.close.num-committed-allowed</name>
( a/ a* S! {2 _ <value>0</value>% v- ?/ M3 o4 F- J% j. n4 ~
<description>
3 i) b. _! D1 D: [7 q: s2 c Normally a file can only be closed with all its blocks are committed.# R. Z2 `, M. M
When this value is set to a positive integer N, a file can be closed
; ?, j/ Y5 d; X2 z/ L when N blocks are committed and the rest complete.0 v7 x) {1 l+ z# L) g
</description>0 C0 `* q( h1 m% Y( U, @
</property>
# t/ J' {( `: t" @<property>
$ b3 @( S! e# E0 d6 S <name>dfs.namenode.inode.attributes.provider.class</name>
& l* ]# l! g! b) P( y <value></value>; C4 o# L. Y A9 F0 \
<description>
" P' V N$ a/ u* }2 a Name of class to use for delegating HDFS authorization.3 Q5 U& c5 u' ^* O/ a% `
</description>
) G/ O3 x3 p* C+ K8 o</property>6 e* d- Q2 |7 u" R
<property>
b) A+ D: K- D8 J7 P$ M# K <name>dfs.namenode.inode.attributes.provider.bypass.users</name>" T3 X' R2 c6 l+ c0 b+ K
<value></value>
9 s( l! m$ X( z0 l1 M <description>! t, T% v2 v8 o; Z' m; \
A list of user principals (in secure cluster) or user names (in insecure
- \! G3 R$ A i' N cluster) for whom the external attributes provider will be bypassed for all% B- f' W1 s# W7 Z1 q
operations. This means file attributes stored in HDFS instead of the
8 _. E/ K' _' {& h9 d6 m: e" P external provider will be used for permission checking and be returned when p; V' s% A$ x
requested.* T( S0 `8 I; U4 D: L5 Y+ `
</description>
- i: m7 o" u/ f1 a. W5 f$ A$ E _</property>$ I) N, W0 s$ E. u' M( f
<property>
3 C; i! ]" d/ ^+ f8 E' | <name>dfs.namenode.max-num-blocks-to-log</name>' V9 z9 F4 z7 }( R" W
<value>1000</value>
1 c; b! r) l) b3 h4 `: P/ F <description>9 w, v" G+ q& P' R& ^4 b
Puts a limit on the number of blocks printed to the log by the Namenode
* Y+ B3 `* C4 M after a block report.
5 Q) t" c; K( ~% Y </description>% n0 ^3 M$ d' }4 f
</property>
5 A) F% X3 y, T5 x5 B8 S<property>" W# W5 e; C- v% @. I0 k/ q
<name>dfs.namenode.max.op.size</name>
. V( s2 O* o# w1 a; D3 i% } <value>52428800</value>4 l e2 b/ y7 }
<description>
) w+ j$ ~* R* v' h Maximum opcode size in bytes.4 W, J( G% Q f+ P5 C8 @* S
</description>* j0 Q6 ^: a/ }0 u" f, O1 \8 W
</property>
& R( h1 Y+ L) j( ]4 d: ]/ w O* F<property>
+ }6 w }: x+ C$ @. ] <name>dfs.namenode.missing.checkpoint.periods.before.shutdown</name># Y5 w* Q( X' s+ c) B/ t1 {
<value>3</value>( G, Y; V* |! A {1 b9 i# V2 K8 y
<description>
6 i: z1 k8 D6 t% N6 G The number of checkpoint period windows (as defined by the property6 T& u- t8 E3 A2 x
dfs.namenode.checkpoint.period) allowed by the Namenode to perform+ a% s$ z- o8 P) U% N. B6 {+ {# q
saving the namespace before shutdown.9 ~" k. M$ ~9 f# L3 N9 K
</description>
: u+ B h3 D! e' S( S</property>
! U/ k5 g0 @. a( X<property>% w8 j6 q5 z* l) e2 t3 Z
<name>dfs.namenode.name.cache.threshold</name>+ T) N A H4 m8 M& C4 g+ q- @6 K& J
<value>10</value>
0 A5 }9 n$ g& _6 H- K <description>/ @3 {) j G, l6 H: O5 y
Frequently accessed files that are accessed more times than this1 q2 v; p3 p, C8 O7 n$ Y$ _; f% @
threshold are cached in the FSDirectory nameCache.9 ^; J6 ^- P/ @9 }8 B
</description>
! I2 _5 g* K5 m# Y3 @</property>
9 I. l: N8 P6 i" {! F0 D. Y<property>0 ]8 S7 I# U3 k; A2 f8 P; O
<name>dfs.namenode.replication.max-streams</name>
j: \- }( T1 L' O; s <value>2</value>. D& w, ]- R2 a5 v2 q. n
<description># T A; G4 P) e5 G# B
Hard limit for the number of highest-priority replication streams.
+ w9 X+ D- u, ]! n7 Z8 l$ Y </description>
* V4 @+ F$ s3 p% E2 }) ^' f</property>$ w' _2 J% F$ W
<property>
) r* A! g; ?5 U/ i" \8 W! H7 b. B <name>dfs.namenode.replication.max-streams-hard-limit</name>2 k9 w4 g- \, _4 H( y
<value>4</value>
. @4 \6 {4 c8 p; F9 W, C+ J) M2 e0 z <description>1 | J. d8 y6 J, X1 w
Hard limit for all replication streams.& x; {* N8 b9 d# c
</description>
/ [( B/ D9 u, {</property>
' ~3 d1 Z" p) g<property>
, R8 X$ B x/ Z. n* C- b) X2 q <name>dfs.namenode.reconstruction.pending.timeout-sec</name>4 n1 B& V8 _0 N! Q7 v& g
<value>300</value>
$ X/ }! V' }& A) p' ~ <description>
% H- o g" |8 i- ?0 Q$ r1 M Timeout in seconds for block reconstruction. If this value is 0 or less,
6 u0 V0 h6 D* F* N0 r# b then it will default to 5 minutes.8 Y, q9 J" Y" G: ?# _* ?
</description>: k1 G m8 P) ]8 F/ j: E7 `
</property>, k1 |: ^9 q2 i; A% r: Q' w2 B
<property>
/ n+ ?) Q6 u! h$ b/ C& z0 m <name>dfs.namenode.stale.datanode.minimum.interval</name>( P3 C/ g# f3 K) q; X0 u5 q5 k
<value>3</value>0 G! j" ?3 T0 A7 A
<description># T% k N5 M+ S) c/ z! b9 j
Minimum number of missed heartbeats intervals for a datanode to
4 _( p3 J0 a# }/ X. T be marked stale by the Namenode. The actual interval is calculated as
) u/ z- c( L5 b4 W (dfs.namenode.stale.datanode.minimum.interval * dfs.heartbeat.interval)
2 R4 X6 b- G8 K3 A% R- y. I' y in seconds. If this value is greater than the property3 `1 S+ g8 p$ ]9 l
dfs.namenode.stale.datanode.interval, then the calculated value above9 \( y: W3 ^+ C6 }% c- x
is used.
4 s" x1 Q) D& A# f+ q0 a) C </description>' K) V) `7 x5 v* @. o1 F
</property>
- @; P: ^- c2 l- `& u" x+ j- j4 v<property>7 v$ @8 g; U6 {$ e
<name>dfs.namenode.storageinfo.defragment.timeout.ms</name>
. s z i5 I0 d Y <value>4</value>- k* X J; e+ A. j g
<description>
9 ~. x: s! c$ c" v3 b/ ] Timeout value in ms for the StorageInfo compaction run.
: v# t) |7 W. i5 k- r, p. t2 _ </description>
$ N4 D5 l+ N, S/ B% y8 [</property>" B$ Y4 S- [4 P$ Z2 @" V
<property>
/ H# Z* I2 b. d. e! Y2 i <name>dfs.namenode.storageinfo.defragment.interval.ms</name>
# |$ h& j. Q: G' W' p! c <value>600000</value>; m* Y# b0 [$ Y& C& y- Z- B
<description># t2 N1 |- J: x3 j
The thread for checking the StorageInfo for defragmentation will/ o% ^! l5 [& U& Z4 {, p( C& ^% f
run periodically. The time between runs is determined by this! x' G3 V4 Q _7 H4 _+ g4 ~# u
property.. q8 ]* [7 R+ S0 Y X" `# S
</description>5 r' K1 @ [0 e; n% O
</property># x+ q' Y9 | @4 @7 L, J: s
<property>
) X' V/ `/ \8 T8 S <name>dfs.namenode.storageinfo.defragment.ratio</name>$ e8 a9 y5 T ]# L( m7 L" q
<value>0.75</value>
2 V. H- o/ ~+ v# Y <description> F( d3 ~/ K+ n+ @' C; V
The defragmentation threshold for the StorageInfo.
; r$ ~2 |) b: d; } </description>$ }$ N* B( p w1 A
</property>
7 G- I* E' E8 a" x4 r- f<property>+ M1 g* U/ v. l$ A4 w2 I& @3 R/ v
<name>dfs.namenode.snapshot.capture.openfiles</name>- F4 Y- T! s, b! ~/ V
<value>false</value>
1 B$ Y. `, @' n3 a/ W8 v" T; T <description>
+ j/ [$ @4 ~* \: ?) W3 }. | If true, snapshots taken will have an immutable shared copy of
1 ^% e! Y: t; `7 Y3 A/ Z! Z5 f the open files that have valid leases. Even after the open files% }" g0 g, U7 x7 ]( O
grow or shrink in size, snapshot will always have the previous
1 N/ N3 H# U% \- Q9 p3 N point-in-time version of the open files, just like all other
; Q( Z& G1 T" q( R+ j/ Z3 B closed files. Default is false.
; d9 x. @; T% `9 R Note: The file length captured for open files in snapshot is1 J4 q/ \+ Y$ m8 V% X' L( d0 m y* a
whats recorded in NameNode at the time of snapshot and it may
4 W% g/ e: n _4 A1 j& h2 L+ i' |& h be shorter than what the client has written till then. In order
3 G V8 {- e2 X: C. v to capture the latest length, the client can call hflush/hsync
; K, J+ n& g2 T! ? with the flag SyncFlag.UPDATE_LENGTH on the open files handles.
- _5 |2 C/ ]2 j, j( y </description>
/ q6 s1 v& |) [</property>
1 y8 d1 }0 e" z. F+ i+ b, ?<property>
5 j* f N7 W, q4 { <name>dfs.namenode.snapshot.skip.capture.accesstime-only-change</name>
2 ?. C1 _% g; N( H u <value>false</value>4 u' _7 w' y. `# g$ [! G
<description>( ~# i# N2 D, U# U( F) t. d
If accessTime of a file/directory changed but there is no other$ i$ _$ y8 Z3 d2 m
modification made to the file/directory, the changed accesstime will/ N m* T+ X7 J( I
not be captured in next snapshot. However, if there is other modification; S% y3 ~. f+ z9 z1 T
made to the file/directory, the latest access time will be captured
$ V' g' ~" I- R% x" ?, q8 D together with the modification in next snapshot.
- R" Q! K! J3 b3 @5 ^5 s3 ~: Y! s </description>
) g' X; F( A; T# Q& r</property>* a( ~( g7 W3 C; y
<property>
" m2 N2 Q G* { <name>dfs.namenode.snapshotdiff.allow.snap-root-descendant</name>
6 |& I+ Y: N }8 B; c- w% k <value>true</value>" h1 V6 v$ d# b* D4 \8 ^
<description>& ?7 @& P$ h, r4 g
If enabled, snapshotDiff command can be run for any descendant directory
" v+ ]1 t: u: i ]! M! u under a snapshot root directory and the diff calculation will be scoped
5 D3 O/ q3 b' T to the given descendant directory. Otherwise, snapshot diff command can* _/ T7 h# ~! N5 b3 r* v7 l e
only be run for a snapshot root directory.
. }- B |2 {! w) g6 q) W$ r4 Z( @ </description>" M: b( [* W: d
</property>
% j; i9 p$ g# W5 ^/ V<property>6 |/ J+ b6 u5 u8 L6 J
<name>dfs.namenode.snapshotdiff.listing.limit</name>- ^. {* ^7 M( q3 h
<value>1000</value>
+ W9 B! I/ v- ]& }* c) N0 A Q <description>
' M4 ^7 n! U f' o1 F4 D Limit the number of entries generated by getSnapshotDiffReportListing within
9 i9 L; m9 i" `8 u one rpc call to the namenode.If less or equal to zero, at most3 B' c0 f# S i2 B/ d
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT (= 1000) will be sent
& M5 j4 ?) _5 L$ x5 H: P/ B across to the client within one rpc call.9 L4 B% }" R: [8 N# r/ I" Z
</description>4 Y/ X# t7 x, A' H/ u3 v
</property>
! n4 A! x) B9 j! l3 e2 ?* |& [<property>, G3 k* {4 b3 k
<name>dfs.namenode.snapshot.max.limit</name>! C2 ^3 h0 M1 Y+ C: S7 W4 l
<value>65536</value>8 v5 Y0 v# i0 }7 W5 k5 i' |. R+ v
<description>
& ]& M8 z& q, T4 x Limits the maximum number of snapshots allowed per snapshottable
% ~4 N6 {1 h9 S; Y- Q" s& i directory.If the configuration is not set, the default limit
, F% ]6 v }& z8 e* Z9 M1 f" C for maximum no of snapshots allowed is 65536.
% M5 n% o" ?$ H* ~ </description>$ n& `' Y5 S( Z# m( Z
</property>
, C9 I1 q" i$ a: {# i<property># l7 ?8 k8 p# X: F: a0 A8 \
<name>dfs.namenode.snapshot.skiplist.max.levels</name>9 o" ~/ x! S6 k& x8 x `; g# m
<value>0</value>7 U5 P/ {' u1 Z$ A$ `+ [
<description>
8 U P6 ]* X! B. n; r Maximum no of the skip levels to be maintained in the skip list for$ l& b$ O4 H [( C4 `& K
storing directory snapshot diffs. By default, it is set to 0 and a linear) q F) _6 j( ~3 m$ l
list will be used to store the directory snapshot diffs.
6 O- ]" i2 K5 n) @: ^ </description>
6 ]$ S) O$ Z4 S6 \! Z</property>- {8 w7 z4 V: e& y- y6 P
<property>
- m) W4 j; K& o( Y: k <name>dfs.namenode.snapshot.skiplist.interval</name>
: N! b N- I; _5 H7 r <value>10</value>
" }2 Q. {1 L w+ F3 ~( d6 t <description>' v! M5 G$ |) ^$ }8 H5 x4 {) i
The interval after which the skip levels will be formed in the skip list w3 j5 M& W" r" C* M
for storing directory snapshot diffs. By default, value is set to 10.! v3 K* E+ g4 B: Y" b$ n
</description>' [0 D4 ]* Z+ X1 }
</property>
! s( S" O2 f" h- J4 e<property># G: |- \+ z( Q# ~; U1 n: Q
<name>dfs.pipeline.ecn</name>
+ Y0 i2 c( C8 B2 [( w% b6 D! h5 F* O <value>false</value>
9 z4 L* m% ^* ~; V5 A <description>
2 q$ _( w* w% _, p7 ~' ]6 ?: B If true, allows ECN (explicit congestion notification) from the% c4 W( ~/ `; T- h
Datanode.0 Z% ]4 r, G6 C7 {! K9 v# Q
</description>
8 Y; x& i ^; p2 B</property>
4 r5 M. f/ m1 p, M<property>
; H, ^" N' B/ M/ e7 T9 e <name>dfs.qjournal.accept-recovery.timeout.ms</name>9 F2 ~+ e# G9 C3 K
<value>120000</value>
2 T! d3 r: h: j' ` <description>3 b8 f2 p j! e* z# D. L. T
Quorum timeout in milliseconds during accept phase of z+ q& z4 n, w. I9 R
recovery/synchronization for a specific segment.
( y5 Z; m2 I# x$ x7 E7 w% _8 q </description>
" S7 V* _5 O9 i4 t) b- Y</property>3 n% N. U' E3 ~8 H$ r
<property>; X2 X; k* t7 d' }0 l2 K( f6 i
<name>dfs.qjournal.finalize-segment.timeout.ms</name>+ L- b2 U/ v8 ~. C
<value>120000</value>
+ f) U G. b9 [ @ <description>
@7 n6 ~8 \% C4 S3 L% E+ ] Quorum timeout in milliseconds during finalizing for a specific5 r" w m+ h* Q5 \% O( g0 ?+ ~
segment.
z2 f: _# W! ^1 w/ G( ` </description>
2 ?2 s% F0 T2 s+ {, k</property>2 \/ y0 z9 d4 f6 ^" g- o2 m+ F/ Y
<property>
: _+ e9 t* M) W$ v: u <name>dfs.qjournal.get-journal-state.timeout.ms</name>) }8 ]6 |; T. s6 c( b. O
<value>120000</value>
9 V! `3 _% C1 j0 l: s <description>% z8 Q* J2 f4 f- z2 z' K/ P8 V' p
Timeout in milliseconds when calling getJournalState().
% `/ ?# R$ S4 z9 ?4 w JournalNodes.
" u0 U4 [4 H% d7 v, f </description>
3 W( @5 L; V! O! ~7 W" r# l</property>
& g. n) b+ B. x) K, d5 r& i<property>
$ o E+ v: C- G6 |9 e( X f: u5 i <name>dfs.qjournal.new-epoch.timeout.ms</name>& P; r$ P& |/ z1 w# K E8 ?
<value>120000</value>
# K0 M7 q9 z+ O+ x3 v- N <description>( L) R1 X" Z/ h, c& a7 s6 l
Timeout in milliseconds when getting an epoch number for write
" G7 W" A( j* o: h access to JournalNodes.5 i; z. G+ \, J; m
</description>
: o1 m" \- R( o- `</property>8 C% v1 U) z/ T- ^( C
<property>; W* a, Z7 t$ t
<name>dfs.qjournal.prepare-recovery.timeout.ms</name>
6 N; p, y. L- p <value>120000</value>
! V" H- ^* S! w/ ~ P$ o9 H2 a U$ R <description>) }! ~$ T, Q8 ~4 w2 h
Quorum timeout in milliseconds during preparation phase of
" p% _. F' n& l* B! o+ F recovery/synchronization for a specific segment./ l* t& Q) V8 G; e/ E
</description>& ?+ _; P w, b7 b/ F
</property>% x* J4 D3 z+ c6 t: A
<property>
: d5 g/ C/ ~8 }2 w <name>dfs.qjournal.queued-edits.limit.mb</name>8 h4 \1 k# _" V% Z4 H& n2 _6 v7 G
<value>10</value>5 _) D2 {8 D) s9 D) r, u1 z' g
<description>
/ x9 m4 s; N- t+ V$ | Queue size in MB for quorum journal edits.
2 K8 C* s5 u2 i. b7 v' U1 z1 g! O4 f </description>
: E+ n6 k2 W% @, f</property>/ B6 O% [$ h. p; ^- ~
<property>
8 b Y7 \+ s- K <name>dfs.qjournal.select-input-streams.timeout.ms</name>
/ o+ h( `7 A# P2 k* h <value>20000</value>9 M$ d L& L# s8 f! P8 m9 g/ h
<description>1 y4 J5 o0 B$ l3 |" T! J9 R
Timeout in milliseconds for accepting streams from JournalManagers.
9 \ W, h* x# \$ y; m$ N0 l </description>
+ R# B, ~" L: K4 ^$ t. ]$ \( c- L</property>
( h2 t+ E! f8 }' g; }<property>
9 ^8 y# g, r; f. K. l/ {1 c <name>dfs.qjournal.start-segment.timeout.ms</name>
9 J" Y: T! Q7 P2 J <value>20000</value>( h# R5 @$ T" _# k8 \; K
<description>
0 X1 j! p2 S9 Y4 N7 N$ V Quorum timeout in milliseconds for starting a log segment.# x" p; \6 A3 a# c+ w& y
</description>* d5 t- F! G! n6 L5 Z. ~5 c" f/ X
</property># y: A& B$ M+ D/ ?# p1 J
<property>
8 u0 s1 p0 {" Q* W& } S$ \ <name>dfs.qjournal.write-txns.timeout.ms</name>$ z% Z1 P5 I3 k+ s
<value>20000</value>9 j- S+ |+ V6 `8 A/ c0 t
<description> R. W( V* [: I2 a" b U+ _% K, V$ F
Write timeout in milliseconds when writing to a quorum of remote, q1 W9 V( w9 m3 ]4 P
journals.
$ S6 o# _! w8 E& m# w p1 _ </description>
5 }8 U- ]$ [! {</property>6 O ]+ ?" j6 S) Z
<property># h, E" z* j6 O; }0 y, u) F( p
<name>dfs.quota.by.storage.type.enabled</name>. d. i2 \" e( u( u' T M
<value>true</value>
6 O/ N1 U; c: s# o! ^* e <description>2 A% s! t# p; {+ F) _; j; b$ i1 |% a5 C
If true, enables quotas based on storage type.
7 n2 @' j: I/ z </description>& D0 c2 I0 R, C, n, z; T4 u r
</property>
, v/ N$ b m( w<property>
" Q8 a+ M2 z) s" ^( R, N3 e <name>dfs.secondary.namenode.kerberos.principal</name>. M" d+ \$ ]1 q& Q$ d8 O; v( E4 Y
<value></value>
9 A0 n& S s& A% Z <description>8 `" D: J0 g- p
Kerberos principal name for the Secondary NameNode.1 S5 H/ e; y2 v
</description>
8 `) t1 }7 |; Y( s</property>7 U; Y5 M; w8 |4 k5 }
<property>& c) [' X) |+ _5 \
<name>dfs.secondary.namenode.keytab.file</name>+ L! ?& E+ e2 d1 X* r4 P( ?
<value></value>
! O# @8 w$ f3 w( r4 l <description>1 F6 `0 \* o6 |5 S, F
Kerberos keytab file for the Secondary NameNode.
' W4 G# B9 ]9 _" U1 s. a </description>2 f! L) ~6 ^/ |7 p# u1 s! c, @
</property>
+ s, `7 f' v" A: o9 U<property>/ g! B/ M+ j" t+ }3 ^; e
<name>dfs.web.authentication.filter</name>- J4 Q# w2 Q0 H9 h! J9 t4 H8 J
<value>org.apache.hadoop.hdfs.web.AuthFilter</value>9 e5 \ f: C; n3 l3 N
<description>
5 k7 b+ D$ O" t$ {. E9 _9 S Authentication filter class used for WebHDFS.
9 P @3 ^3 \- t& M7 Z; U8 B </description>9 f& n& Y: p& S3 X8 K
</property>$ D3 y3 f9 [# f4 V) B2 M0 l. f% q6 m
<property>: U; l4 g, p1 ~% u# o p7 h) j
<name>dfs.web.authentication.simple.anonymous.allowed</name>
9 K! Y( f$ f' \5 O7 ~& j, V& k <value></value>
1 R+ E9 s7 w6 C+ h, y; b <description>
F% e& ^4 c/ Q* h. J$ Z5 m If true, allow anonymous user to access WebHDFS. Set to
0 a6 x7 v' R5 {+ U- }! l; ^; R3 D false to disable anonymous authentication.
: K- T; L% L2 ?2 U </description>: y4 \6 B! T% l0 E9 L6 N. @
</property>, k" ^: W% _8 b0 V
<property>3 C4 y) w4 K% E& y# c$ _6 i* E4 u
<name>dfs.web.ugi</name>" s5 Q$ m, m$ z1 M5 l
<value></value>
e& z0 j) F* c# Z5 K <description>
) Y7 k+ s2 j3 h+ J dfs.web.ugi is deprecated. Use hadoop.http.staticuser.user instead. Y- S3 B) L$ }5 X6 c
</description>4 N6 W, [. `0 y8 ?4 }
</property>
* M. o: j9 D, x2 ?9 U* q<property>
$ o! h8 p+ a* G" M s# ] <name>dfs.webhdfs.netty.high.watermark</name>9 @$ L2 V% e: x- j
<value>65535</value>
5 @+ D+ W; F4 c, ^3 R <description>1 ?6 m- G: w7 ^9 ~( A3 d
High watermark configuration to Netty for Datanode WebHdfs.- X: O9 A& I0 M8 O
</description>$ q0 F! \: t# a& `5 f
</property>! F+ v) e0 Q' F: ~+ ?! c- d" T3 {
<property>
0 V1 [ b! C) y2 u <name>dfs.webhdfs.netty.low.watermark</name>, l$ z: Y' S- z* S7 ?
<value>32768</value>9 x' i6 @9 r( V/ P# K k
<description>) B0 n9 o0 Y; l* g8 r! j/ f' i0 R
Low watermark configuration to Netty for Datanode WebHdfs.7 n5 |/ @* a7 H' L* i- Z/ o* `
</description>
6 t4 [& i) e/ R# r* c0 P3 o</property>& j. Z. {# f# R" ?
<property>
) `6 o( r* m% F7 q7 k) L& ?: N <name>dfs.webhdfs.oauth2.access.token.provider</name>
) [- B# V& T, l2 z9 W5 |7 V3 ] <value></value>
; h4 K' b; n! M4 [$ `4 j2 t <description>
& w* @8 w( G4 M, j z& N+ K4 E Access token provider class for WebHDFS using OAuth2.3 }. S2 R1 a. B& b; {. b
Defaults to org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider.
' @% b. t5 i* w$ g1 Z </description>
+ m, f% P2 s/ ?3 _2 i2 d3 X+ _</property>
/ a4 n7 o! y `/ A0 ~5 i1 ]" G<property>
) U9 H* z4 t I4 w <name>dfs.webhdfs.oauth2.client.id</name>+ J2 I' e' s2 n5 v+ K
<value></value>
+ @! {/ z& z1 B+ r <description>6 ^4 i7 K" ~& a- k3 ]' {( p( a
Client id used to obtain access token with either credential or
/ M6 V; u- J N+ U, J7 Y2 o6 R; f refresh token.
+ i% t" F4 c- L1 P! e </description>
# r* ]& |; s' W Y</property>
8 G$ N; W- m( o. q' d( O<property>
+ A2 w3 V( N* N6 ^' o. t <name>dfs.webhdfs.oauth2.enabled</name>
. L! s% U, n0 [9 g( g$ I7 _ <value>false</value>
u1 d5 }% l. z2 ?+ J" E7 J. K <description>3 Y' v2 [0 Y# T# O# ?
If true, enables OAuth2 in WebHDFS
- d3 u) m: _( X& ]3 B3 o </description>* N, }; Y# {( \8 m4 {
</property>+ c1 Q- N; ^0 A3 _4 ?
<property>
3 S9 F4 l- D' I0 k5 i$ R: z <name>dfs.webhdfs.oauth2.refresh.url</name>
, H# ], {! m8 X F* o: d5 Z' q <value></value>" G3 h, Z% r$ k: \
<description> o! F: j; j; T% ?2 m* I8 w
URL against which to post for obtaining bearer token with
# c6 G: v. T/ K7 B& O$ _ either credential or refresh token.$ h% J" w) w* H3 \) Z+ {
</description>% F# n' D3 [2 w- U8 z
</property>4 `# o9 D, Y' B
<property> E; _1 R! [- k% \
<name>ssl.server.keystore.keypassword</name>/ X% }- i( t* F7 @. j
<value></value>5 ]/ B2 s& z8 [$ U
<description>
$ v7 H) t8 J4 C( m Keystore key password for HTTPS SSL configuration( }2 j9 b3 m- P# S
</description>3 T. f8 Z% O3 N' q# ]
</property>0 ?( L- l' i' d: |3 S1 A z
<property>5 T: i9 C- |. p6 A8 k
<name>ssl.server.keystore.location</name>) c# k# n# i f/ y
<value></value>8 n4 t- Z ~6 A/ z# c
<description>9 o9 C. E3 @/ w, B" M
Keystore location for HTTPS SSL configuration$ G2 S. c/ D, g2 I" n
</description> {3 {& d! x( x( s$ a; y: A! v" x, e
</property>
7 Q# }1 b5 T0 b5 I; L& ?& {0 q+ j<property>% ^, d8 E9 G* Z! \
<name>ssl.server.keystore.password</name>
( T/ P* N. c& s) p5 f- A <value></value>2 M$ c/ I4 B. N- ~% {- e' \
<description>
9 ]( h! E% s- ?8 f2 t7 P Keystore password for HTTPS SSL configuration
]( p+ {. O' D4 R$ y </description>4 v0 ?5 U! t$ Z; |- n# F
</property>7 B$ U6 p$ I5 q/ C2 p: I8 C
<property>2 f' o- ^7 n% z9 n
<name>ssl.server.truststore.location</name>
9 r! n1 }6 i7 `) H* v9 Z) j, f# v <value></value> W2 H- ~' L6 S5 Y* g
<description>
[7 I8 f# G$ u8 x7 O4 G Truststore location for HTTPS SSL configuration
% D: C! y/ N! s </description>
. s+ P- L y0 B</property>" v* w. k5 O7 X+ D% V$ a
<property>
9 r( _" g; K4 k6 N <name>ssl.server.truststore.password</name>
2 [1 r$ n/ j1 L <value></value>( f1 k2 M8 S3 g
<description>! c7 K9 b4 V" Z! u$ t
Truststore password for HTTPS SSL configuration0 e- p. _: y2 U* S3 O: L
</description>8 Q+ y9 ^2 [( g; R/ G7 t8 d
</property>! Q4 R, m6 I5 v% P [
<!--Disk baalncer properties-->/ f+ F2 Q# u4 [9 | C$ M! t; l& S
<property>
$ T9 ?0 W. d8 ^, b a7 d b <name>dfs.disk.balancer.max.disk.throughputInMBperSec</name>* m% S% y. ]* K- Y8 ?( W2 J
<value>10</value>
, R( u a- d) Q: P- E: i. X <description>Maximum disk bandwidth used by diskbalancer' U, m! {) o5 X, }
during read from a source disk. The unit is MB/sec.3 w- i5 D' H* |
</description>
6 f0 p; Z0 q% O& X7 j </property>
$ g+ A8 G3 j- }6 h8 j1 n <property>( G1 v% F+ e: Z! ?
<name>dfs.disk.balancer.block.tolerance.percent</name>
3 O/ k+ Q4 S# S <value>10</value>
' i. ^' V) m) n& h <description>% l/ Y6 T% }% n, O
When a disk balancer copy operation is proceeding, the datanode is still8 l9 e5 ?2 R" y, l3 W, t! j
active. So it might not be possible to move the exactly specified, E2 r& t; _& W, c0 v
amount of data. So tolerance allows us to define a percentage which8 Z9 g+ `& `; ^5 `6 M3 X
defines a good enough move.
" |: x" V& Q2 `* p8 `* S/ g+ {7 K </description>
* R) w# H: C7 ?9 d+ u </property>
$ [' R% R1 s2 b4 X [- ` <property>6 F/ o5 h: l6 p1 h2 G; Y% w/ |7 r6 ?
<name>dfs.disk.balancer.max.disk.errors</name>4 a' E2 ~& ?! G; B5 p
<value>5</value>2 _& K8 e2 w! Y+ ~8 t. U; J' S
<description>
" ^" `# ^% t* T During a block move from a source to destination disk, we might4 Y' T/ n. K% H& n
encounter various errors. This defines how many errors we can tolerate
" k, s$ |& i4 }$ u+ \1 e, @' b before we declare a move between 2 disks (or a step) has failed.: u, [7 S: O* \/ ?7 C0 }$ _
</description>
" e: t" X, N' t a </property>' Z! i9 [- l* ?: Q* d; i- x# p v
<property>
5 c6 v% J# R, y, `' Y9 W) ` <name>dfs.disk.balancer.plan.valid.interval</name>3 i; n' J7 ?1 {7 p
<value>1d</value>0 ~- ^. ?! |7 S5 ?0 z
<description>9 h: ^4 f! P0 H" N$ F) @) U3 p" o0 E
Maximum amount of time disk balancer plan is valid. This setting
1 d; t0 _( Y3 }2 R" y- N% x& r4 o% g supports multiple time unit suffixes as described in
6 i! M7 R* \( k' s dfs.heartbeat.interval. If no suffix is specified then milliseconds- }' ^3 |+ N- o% n$ t% Z
is assumed.
+ ]$ v& s* Q, Y) Y5 ?4 K# H& K* I </description>
6 ^4 F% w( C" X: a8 s- Z </property>$ j5 ~9 q+ M/ r2 T
<property>
. ]+ \- g' o$ m3 H2 k* u9 n <name>dfs.disk.balancer.enabled</name>
: F: n n- ^2 m" b4 x# F9 d7 X <value>true</value>
1 x- r$ g1 m7 ^* y3 Q8 J/ h/ f <description>
, b5 S1 _! L( \5 i6 {; B, | This enables the diskbalancer feature on a cluster. By default, disk
- a4 K8 X( _: ?1 F' N% L( M balancer is enabled.
- p: Q' h6 ? L3 X. d4 c, R' C; N </description>
( S* z# I6 r& Z- y" D6 s R7 d& I% E </property>& u$ d: p! ^1 F
<property>: F4 a* g) O) F B$ w5 L, d% ~
<name>dfs.disk.balancer.plan.threshold.percent</name>6 [1 ?6 ^8 T4 e8 p8 J0 w, a s
<value>10</value>
/ _! ?; @- \* o6 B/ h8 U) W <description>
$ H! H# t% ]/ [" D The percentage threshold value for volume Data Density in a plan.
6 X9 E) w$ H& } `: p" `4 g! E If the absolute value of volume Data Density which is out of. H4 p( M. k; i/ C3 t7 g
threshold value in a node, it means that the volumes corresponding to
6 _+ f: E8 @% f0 A! x/ g3 K5 P the disks should do the balancing in the plan. The default value is 10.
$ x3 l6 \; O, P& k4 t </description>9 `: }* q1 F0 x6 l( M: j2 @
</property>
2 K9 y# j2 Y- R/ z& ~ <property>
& k Y: K) F M* Z( Q# { <name>dfs.namenode.provided.enabled</name>4 s/ J% s K) I) o
<value>false</value>
: o) b6 T' _; U* ], T9 L <description>( K8 F" V" h L2 \8 B' i9 d
Enables the Namenode to handle provided storages.9 Z; E% a- U% V) C8 v Y# e, U. ?
</description>
, w+ r) i1 {0 B1 B: X. Y9 j </property>
% T" F. M* ^7 j" c5 ] <property>+ Q! ?$ k6 m" X f3 n. ?
<name>dfs.provided.storage.id</name>2 G5 p' B3 m/ ?/ |( c" ]
<value>DS-PROVIDED</value>/ q5 n6 H% j: H" [: \
<description>1 a3 ~* _! k; e9 N& m
The storage ID used for provided stores.
( G7 e K8 H* b1 L </description>
3 f& V4 W) O; z& B/ g% ? </property>3 f( R3 I: l; F( p U) [. N
<property>2 c! S( T4 }# [
<name>dfs.provided.aliasmap.class</name>: f+ N- e) z+ C. V
<value>org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap</value>+ k( l6 L9 l1 d+ [" H# l) J
<description>
" B9 }5 ? C) Z# f9 b$ W The class that is used to specify the input format of the blocks on" S7 |1 v) c+ k" Z: y
provided storages. The default is( r% V3 q- g: k) N
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap which uses
" W6 W _8 {4 X2 G. P file regions to describe blocks. The file regions are specified as a
6 _! d. }4 H4 a delimited text file. Each file region is a 6-tuple containing the
4 R( ?' M3 M3 ?. e7 [& m block id, remote file path, offset into file, length of block, the
( D8 F4 F6 h- ^' G" T block pool id containing the block, and the generation stamp of the/ N' i9 p: M. p' E0 u5 O
block.+ J8 i: ], D) P- j% u0 I6 @' u
</description>
. m% V2 P0 Z" H4 k5 l# g5 _ </property>
& l8 s7 b) |) u3 o. c5 k8 R' V; H" U, Q <property>
& f2 C" N; _: K1 l <name>dfs.provided.aliasmap.inmemory.batch-size</name>
9 Q0 I; Y4 x' E" B% u+ p9 A' e" I; l1 e <value>500</value>: A' A4 j1 M7 Y
<description>
: i% E5 y* n: ^' t8 M! d4 v The batch size when iterating over the database backing the aliasmap, p2 g0 `0 R. A
</description>
6 O' [0 t9 H e* t& b </property>
, s2 _2 Z% k) P3 v! {' e <property>
4 q7 A4 Q- R4 h, a <name>dfs.provided.aliasmap.inmemory.dnrpc-address</name>' ?% ^" M8 W6 Q
<value>0.0.0.0:50200</value>
* b0 \: x* F! |2 w V- r1 q <description>
' n* f" ?* y; ]8 c# C The address where the aliasmap server will be running
! ^! U/ }, }9 w' ^ </description>: Y6 Z! O, Q1 n4 z1 S4 `+ \/ Q+ e5 Q
</property>" m1 G( ?& I. D4 n
<property>
/ g$ r+ v; z7 n+ A m& t5 F <name>dfs.provided.aliasmap.inmemory.leveldb.dir</name>
7 u [0 z' T" `8 g y: |" ?/ b <value>/tmp</value>
h: i1 _1 i, ]3 C- s <description>
; D. {' E. `( \. Q3 X The directory where the leveldb files will be kept
& D6 q* Z3 e5 I: S* g! H </description>
/ p0 [6 i7 c I8 D- ~" x </property>& \8 w/ @) l; d7 L
<property>1 P2 S c% W# W* _ ?9 n
<name>dfs.provided.aliasmap.inmemory.enabled</name>+ s2 j) p3 F! E
<value>false</value>
3 H- b! G, P9 c1 o) {: z7 M <description>' M& K& m& x, F/ I. K. z
Don't use the aliasmap by default. Some tests will fail
6 T3 l7 g! c& O O/ u v because they try to start the namenode twice with the
! l9 J9 z0 h+ R# D# Z$ c$ {! u2 B same parameters if you turn it on.
2 M9 \. _( ]5 a& Q2 `; W2 X: I0 S% C </description>
: [! c s+ \& h8 L; Q </property>. W6 M9 ^" o& h. C9 y2 O
<property>0 {( u6 D" h) B+ g/ S/ _) j" R
<name>dfs.provided.aliasmap.text.delimiter</name>
) m; z2 O8 N9 g1 x7 q <value>,</value>7 n3 x4 V+ Y& f( b- B7 c
<description>
6 I. e6 Q% u8 A' H The delimiter used when the provided block map is specified as
( O3 w, t1 F$ L ~& v a text file.6 P) K9 G& e, j r
</description>
7 k* c% w# l. j </property>2 A$ o/ ^3 c3 P2 r' i
<property>
* ]9 X( b; W/ N. G. F" Q6 A <name>dfs.provided.aliasmap.text.read.file</name>
1 W* z L$ t& g <value></value>
2 h4 n5 ^( _1 J' Q <description>9 k* @0 X* m! Y# u; T7 {1 C; R6 T5 |
The path specifying the provided block map as a text file, specified as
: ~7 }7 Q6 h3 T* `8 |9 |9 ]2 K- m a URI.8 F, S8 E8 i# |
</description>
% w& \% u9 X# e* Z- ?9 t </property>/ s7 ]9 _- u3 c- K4 v& b4 t
<property>* Q, [* D) d1 O9 v! ]
<name>dfs.provided.aliasmap.text.codec</name>) {% m# X) p$ i% F3 t9 h; o9 X& M
<value></value>) }0 y% a/ j, R2 C+ X
<description># \& ~2 x/ ]+ N7 y& i& Q2 u$ h+ T7 h
The codec used to de-compress the provided block map.: |2 d0 v7 H$ S
</description>" A0 x& S E# K: ^
</property>
" S1 v& O# Q/ o; P <property>0 `2 A% q( f9 q/ Y% Y+ H1 v
<name>dfs.provided.aliasmap.text.write.dir</name>- L/ K! C% J' t$ a/ `( Q
<value></value>% j. ]- M( r" c+ q
<description>
+ E9 D. b5 m% b& I$ x The path to which the provided block map should be written as a text* t- u, H, z# T8 K. [
file, specified as a URI.
$ D/ l1 w0 Y b </description>
1 i; G. u& j" N, ^$ i3 ~& [3 } </property>
9 `7 j. \) ]0 m8 o5 O$ P <property>
- A7 A5 `8 ~ h' J <name>dfs.provided.aliasmap.leveldb.path</name>2 j3 u* s; t7 W5 m/ a h) k5 l( r
<value></value>4 A. L) Z) C+ o2 i1 d
<description>
+ `! d' n: e0 p The read/write path for the leveldb-based alias map4 E5 u4 h1 X) x3 I+ E# z
(org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap).$ C! W, Z1 E% u Y5 m
The path has to be explicitly configured when this alias map is used.4 d. J2 K: `3 `! C: L
</description>9 b4 @6 G) H! t9 C* ~
</property>2 S- O! S+ Q6 n' E" G# u
<property>! G9 K4 G- [3 e1 n' v2 y
<name>dfs.provided.aliasmap.load.retries</name>
$ U; i8 T7 s; i0 C; l0 G2 g; Z <value>0</value>, o1 N/ }# G' k
<description> _6 a$ G: r0 R; Z4 l* ?; L6 f- z
The number of retries on the Datanode to load the provided aliasmap; f# p5 p/ Z2 h- ^; D
defaults to 0.
. j! W5 t% c% ?/ a3 n </description>
. C7 c) |( H$ `9 H( w </property>
7 i" k- Z8 _9 G3 j% b1 ^" f <property>
5 r) K4 \0 ]. E9 x9 m! w& Q <name>dfs.lock.suppress.warning.interval</name> L0 M# o' \+ ]2 k6 D( b% K
<value>10s</value>8 e; y3 C' t& T$ I, F9 S/ J @* m; u
<description>Instrumentation reporting long critical sections will suppress
8 y; |* Z6 G5 V9 f+ z# Y$ x consecutive warnings within this interval.</description> ^3 h! W4 X; C5 P- K x, ]
</property>
* ?- a! ^! `6 a2 U4 O7 k <property>
5 i9 ~1 q. o# i( t9 m( s <name>httpfs.buffer.size</name>+ d: n2 a" c' Y+ ~* `5 v' j/ s
<value>4096</value>" G2 q( G; Y" {- m
<description>
m: m; I& m$ g/ A/ j6 ~ The size buffer to be used when creating or opening httpfs filesystem IO stream.
$ d$ W9 I9 ^/ U" }( @8 }' m </description>
) e7 M2 X& [, x </property>1 v3 A r; S0 C; K0 j, q
<property>9 A F- K9 U* ^2 ]
<name>dfs.webhdfs.use.ipc.callq</name>
z" a( I2 U( \7 n$ Z1 x: n% F <value>true</value>% Y6 }$ g, [9 [( I9 Q! A) K% b2 `0 ~
<description>Enables routing of webhdfs calls through rpc% ]7 O8 G9 M$ f9 }6 D
call queue</description>9 s& R3 F5 V5 t* u
</property>
4 _" M/ D, b) Y* j i <property>
1 t A, _+ m2 U* D' [ i0 c <name>dfs.datanode.disk.check.min.gap</name>. U# f* z6 b2 Z3 H
<value>15m</value>
6 s& I# ^8 z; i: s, _( F <description>
/ X4 d5 Q1 o( M2 S, @1 e7 ] The minimum gap between two successive checks of the same DataNode
, N5 E% l/ y, q5 h, { volume. This setting supports multiple time unit suffixes as described
4 k) V% Z! ~6 r# f' K- u0 N in dfs.heartbeat.interval. If no suffix is specified then milliseconds. y; U( c8 s/ O5 G
is assumed.1 B& v, v: b; n% G/ f4 L$ i
</description>' P2 n; ?7 J f2 A3 x
</property>9 o+ R3 i! _9 u5 ]8 O
<property>/ j+ n: q+ g/ J) A
<name>dfs.datanode.disk.check.timeout</name>$ H) r+ g' H5 M
<value>10m</value>. ?5 b0 p" Z% y3 x
<description>7 K3 |4 ^ X- G# J$ V5 m% i
Maximum allowed time for a disk check to complete during DataNode% |- d4 S7 M B/ {% @5 }
startup. If the check does not complete within this time interval+ b0 s; z7 w$ S4 k+ l3 V
then the disk is declared as failed. This setting supports
9 M, S! `0 D+ X/ ]7 m- R( ]/ H multiple time unit suffixes as described in dfs.heartbeat.interval.
6 ]& P0 ?' D8 Y If no suffix is specified then milliseconds is assumed. w% H: @) O F4 |: h8 k+ z
</description>
" r" A |! Z5 j4 T </property>
8 O, t; p2 j5 k <property>
0 A7 [- L/ z0 p <name>dfs.use.dfs.network.topology</name>" p6 }$ y# g3 J: X% D. r
<value>true</value>! r1 p: c" u" a" s
<description>; r h! f `1 \% k/ Q) y
Enables DFSNetworkTopology to choose nodes for placing replicas.& R6 t1 b% J8 d/ |; {- _9 Q
When enabled, NetworkTopology will be instantiated as class defined in( ~7 M, E. y; S' Y7 v6 J& Z$ N
property dfs.net.topology.impl, otherwise NetworkTopology will be, X. E6 `6 X1 G
instantiated as class defined in property net.topology.impl.
H% O' S5 b% D/ j </description>
( v5 d# F L. s/ q. c, H* ?$ N </property>; l$ h8 z0 B# s" I. V( A
<property>
. I, o/ w# O, w) N+ o/ A <name>dfs.net.topology.impl</name>
5 @ s3 u( z- R( \4 i <value>org.apache.hadoop.hdfs.net.DFSNetworkTopology</value>1 s$ w5 C: n/ C* [
<description>
( \6 ~6 h( o, p/ D/ A- v% X U The implementation class of NetworkTopology used in HDFS. By default,# \% S. `0 t1 Y' Z) P, x! i' [
the class org.apache.hadoop.hdfs.net.DFSNetworkTopology is specified and2 h9 U; L: ]9 P) e( o0 x' A
used in block placement.+ i* b/ r4 q) u) R
This property only works when dfs.use.dfs.network.topology is true.
! S, ^" q% {. m( _2 G2 P+ c </description>
1 D+ ~( |: g% j$ j, g </property>; `; c- c0 l+ `. q+ W* m1 k
<property>) ]/ x: m- C; [; B4 I
<name>dfs.qjm.operations.timeout</name>
4 t% J0 Q7 j k <value>60s</value>6 b6 B3 b7 ~" A! k) w" a% [2 F
<description>
u' d3 S" T* p$ R/ R- t* o Common key to set timeout for related operations in
" n7 z2 ]6 ~) N4 n" B QuorumJournalManager. This setting supports multiple time unit suffixes" A% U3 }0 w' Q, S! W) Q9 _ T
as described in dfs.heartbeat.interval.
7 y1 m |9 j. E& N If no suffix is specified then milliseconds is assumed.
; H2 E1 p& ]3 E </description>
( `2 [1 k2 z0 C7 w </property>" ~# T& z r* g) E: Y
<property>; c/ O7 N' _/ M s7 f' x# @
<name>dfs.reformat.disabled</name>! _% n& u- L6 C+ s
<value>false</value>
/ Q" L. J8 n! _9 L2 K <description>
0 C$ S( r' Y7 U4 V. I5 _4 E Disable reformat of NameNode. If it's value is set to "true"! ^2 v7 Q. z) H4 e* s2 W- B! z
and metadata directories already exist then attempt to format NameNode" b+ h0 b% I8 @$ L1 c+ y9 e+ a
will throw NameNodeFormatException.- E. N& `0 q. X' Q" ]. w
</description>3 c, E& V& A/ v$ q6 {6 B" P
</property>
3 [2 U( l1 k3 b! Q/ p</configuration>
- ]0 t9 E1 |% m6 P! a: b3.mapred-default.xml% J0 s* a& D3 b- @. J
- Z. s2 I+ d) M3 O/ F<?xml version="1.0"?>! [1 S# W- X# `. Q9 ]6 Y( r
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>" L4 J5 C" w$ W j
<!--: s2 ^! S* ~3 y# a$ m: F* P
Licensed to the Apache Software Foundation (ASF) under one or more
3 h# I/ f% x( p contributor license agreements. See the NOTICE file distributed with
6 b0 Q5 r3 ?7 Y: V3 H6 _' s this work for additional information regarding copyright ownership.
8 f) B. X( j) d9 J, |% X+ ~( C The ASF licenses this file to You under the Apache License, Version 2.0+ p" x5 Z3 V$ x* e7 I+ e
(the "License"); you may not use this file except in compliance with
: W! d. h' P3 [ the License. You may obtain a copy of the License at3 z- H& ^8 h3 i8 t, S
http://www.apache.org/licenses/LICENSE-2.0( T5 ~4 g3 [0 u% r4 L8 P
Unless required by applicable law or agreed to in writing, software
& E$ K5 L. V1 s7 {: A: ]& B7 A3 l! p: B distributed under the License is distributed on an "AS IS" BASIS,
2 _1 w3 o$ q$ m# f& a7 i, k WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
" M9 T' N# _) X: E) z' S5 F See the License for the specific language governing permissions and
. t" I: |( n! B# p limitations under the License.
) W- b+ c8 U' M7 {# ]-->
h I: b9 c% K& U* X$ [2 Z<!-- Do not modify this file directly. Instead, copy entries that you -->' {. x/ [. s! o6 m3 B, B
<!-- wish to modify from this file into mapred-site.xml and change them -->
; S- V2 b4 i* V8 q" i$ g4 A0 h" X<!-- there. If mapred-site.xml does not already exist, create it. -->
% n4 [# h8 C' G8 G/ |6 i/ Z<configuration>* F2 D% o$ @3 N
<property>
( H' F! P; |5 U; j {- T <name>mapreduce.job.hdfs-servers</name>2 ]8 z( c5 p0 }" D* U
<value>${fs.defaultFS}</value>7 m9 J7 m0 l3 W/ @- Z' K
</property>$ H* }4 G* H0 V" i7 g- C
<property>/ N$ E" g9 @7 o4 H6 g M, _
<name>mapreduce.job.committer.setup.cleanup.needed</name>9 ~; m6 C: [* R2 L1 e/ |8 \
<value>true</value>9 k& l. N _. s) V# p. \
<description> true, if job needs job-setup and job-cleanup.% H" T6 Y' j. @5 P6 p( a
false, otherwise. p, z+ ~7 N* I7 k
</description>! x# S$ W. C4 M/ M
</property>
/ \) ^6 }, o" ^; i) Z2 A" Q1 }<!-- i/o properties -->
) a# I8 h& r( R6 I- y+ r4 g<property>
( D- P7 J9 M7 O$ m" S <name>mapreduce.task.io.sort.factor</name>
5 d, v: ]5 B/ p. o: F <value>10</value>- @* h3 |8 y9 x" K- k3 j& v9 m
<description>The number of streams to merge at once while sorting( X4 i3 E& a* v& b
files. This determines the number of open file handles.</description>; k# w8 T) w& L) Z
</property>7 v2 f* }5 y0 E* `
<property>
& V& W+ _4 n% Q; ~% m0 `7 Y" T <name>mapreduce.task.io.sort.mb</name>' v. e5 Y% S! m
<value>100</value>- L% [. e( f! M Y, E
<description>The total amount of buffer memory to use while sorting! V% {1 F I4 o
files, in megabytes. By default, gives each merge stream 1MB, which
5 I/ Y7 M: V7 R2 S! x4 }0 l should minimize seeks.</description>; C* U* d8 H; e
</property>% W" q3 ?* R# F) `) W. q) s; u. f: i
<property>) C( Y: S9 {2 t; U5 _; n7 t
<name>mapreduce.map.sort.spill.percent</name> ?" R6 `7 F" v
<value>0.80</value>+ ?) D; ]8 ^% V% W2 [' O
<description>The soft limit in the serialization buffer. Once reached, a
! `3 q+ M" I; N! @& r thread will begin to spill the contents to disk in the background. Note that/ a+ q, V9 I5 \# z6 `- B4 Q
collection will not block if this threshold is exceeded while a spill is
, m% z V6 g3 t! I already in progress, so spills may be larger than this threshold when it is h) s1 ^% D9 l) K2 y8 E- N) K
set to less than .5</description>
% Q+ M+ ?# m& T5 ?9 A: M3 i& b</property>
1 B2 I1 V4 D5 I( I# G<property>
5 O; Y; e$ O* n$ U5 u- ^. K <name>mapreduce.job.local-fs.single-disk-limit.bytes</name>+ K% y; Q) K- [7 [) M7 `% v
<value>-1</value># p4 G9 v$ u+ D! S, ^' ^$ E
<description>Enable an in task monitor thread to watch for single disk
- A8 ]0 r/ f2 _9 m5 }0 y consumption by jobs. By setting this to x nr of bytes, the task will fast
. U+ w, U# p1 p9 F) }3 ^ t fail in case it is reached. This is a per disk configuration.</description>
8 V0 }* ? @' n' V6 b, c# |</property>% f8 C; U0 m: E# Q
<property>
9 N' f6 D# f8 C7 T: P% @( l; f( g <name>mapreduce.job.local-fs.single-disk-limit.check.interval-ms</name>
" M8 |4 Q5 g* X <value>5000</value>
& e% A8 ]6 z" a9 W <description>Interval of disk limit check to run in ms.</description>
( A/ l2 a/ p( N) `# C# K</property>% V4 q( W' l( t# P, m# @
<property>: z4 I$ G, i$ K9 S4 b4 ]
<name>mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed</name># I/ l% H' z$ B) }+ X5 g
<value>true</value>
; ^8 Q) b1 a4 r0 @, N <description>If mapreduce.job.local-fs.single-disk-limit.bytes is triggered+ \9 P5 T+ |5 l2 o6 q7 K2 B c
should the task be killed or logged. If false the intent to kill the task# h0 w/ ^' v6 d0 s
is only logged in the container logs.</description>
- r3 ]6 O, b5 I</property>
3 `2 ~* n& ^+ M* ]& u& J+ {4 [<property>
U5 D4 U+ w* h+ n/ b1 t8 H <name>mapreduce.job.maps</name>
7 \+ l- T1 p o& ` <value>2</value>
, T6 a# m' J) G8 Q( Q9 @& \9 a9 F3 \ P <description>The default number of map tasks per job.
* A) r) \- }8 ?& B4 y l2 n: I Ignored when mapreduce.framework.name is "local"./ R" o6 Y* i m( ~, ^
</description>$ d; b* y! J0 q" S5 y9 O
</property> t' \- W. T/ e+ ~
<property>/ Z, h* @; j2 k# Y+ `
<name>mapreduce.job.reduces</name> K/ r% U8 J% W- l. v$ S/ H
<value>1</value>
; R5 r* ?/ S( s# y) Y2 b" l3 v <description>The default number of reduce tasks per job. Typically set to 99%8 ], k3 I8 D* f9 R
of the cluster's reduce capacity, so that if a node fails the reduces can3 s9 ?2 u2 j$ e. V- p3 S
still be executed in a single wave.
. M4 G4 r* s6 N- b t Ignored when mapreduce.framework.name is "local".
% o* t, a2 P6 v0 w& _ </description>
1 C, M: L% _% X& H A u* I3 E- A7 @</property>" _: O' I+ y' R2 I4 `; ~- B9 x
<property>
8 J! e" L7 g0 e* s$ l1 h5 N) N; l <name>mapreduce.job.running.map.limit</name>0 S. d4 n+ y' V [2 r" t9 m
<value>0</value>
0 D9 |- A L: E! ^( c <description>The maximum number of simultaneous map tasks per job.: W- @% k/ Z( x, X4 |
There is no limit if this value is 0 or negative.* g! ^ _2 ?# o# K7 [+ e, H
</description>
9 [: m: e0 P: c) y/ {6 ^</property>
4 I; e& P, I) U S<property>
$ k4 n8 E$ T8 ^& \( S8 ~ <name>mapreduce.job.running.reduce.limit</name>
4 o' Q" b' r7 v3 j0 E) o. f <value>0</value>
2 _: d: `; l# x3 b0 X) Q <description>The maximum number of simultaneous reduce tasks per job.5 \" _& Z- J1 g v6 Z
There is no limit if this value is 0 or negative.
1 v" U/ `9 T: J </description>
3 w$ G# [5 \5 \$ ?( ~2 _</property>! M3 x8 \ ^/ u7 g& _% i5 S0 d
<property>
1 g5 g. E$ U" k4 R G <name>mapreduce.job.max.map</name>4 b" R2 i( l* s: L. y
<value>-1</value>
1 c2 z8 p% j- O <description>Limit on the number of map tasks allowed per job.
% k7 E" j8 e# ~, M! g3 G* W3 x There is no limit if this value is negative.! F2 v- h( ^9 ~% F
</description>
# a: P/ I D, }, g+ s</property>& \; Y2 s! s% `+ k2 {2 x1 S5 G
<property>
% Z o3 \( C, H2 S- [ <name>mapreduce.job.reducer.preempt.delay.sec</name>1 @: O: _9 R* }3 R* f- Y2 n1 t
<value>0</value>; x" A+ s/ j4 O$ p
<description>The threshold (in seconds) after which an unsatisfied+ j8 F. V1 j: U5 m
mapper request triggers reducer preemption when there is no anticipated; j6 n0 `* K/ @( i
headroom. If set to 0 or a negative value, the reducer is preempted as9 i5 g* { _7 D, B
soon as lack of headroom is detected. Default is 0.
; \! R4 G1 {& p8 S2 y </description>0 J3 |$ l* e: A* [
</property>& ?: Z1 W+ u3 h7 k/ q
<property>/ D8 q' l( o* m- y) O
<name>mapreduce.job.reducer.unconditional-preempt.delay.sec</name>
( o @ r7 n2 ?: x! J5 `. C <value>300</value>) l+ Q$ V) C9 O+ S: d9 T* z
<description>The threshold (in seconds) after which an unsatisfied9 v$ c6 {: ~% X' C# V
mapper request triggers a forced reducer preemption irrespective of the
9 W8 Y ^8 D: K% M$ x! S anticipated headroom. By default, it is set to 5 mins. Setting it to 0
' f4 D2 J4 @9 @6 K/ G" y leads to immediate reducer preemption. Setting to -1 disables this4 X1 X- ~/ ]) n: e D
preemption altogether. l; r7 {" `' X. M7 ?) W8 l6 f
</description>& {& C7 X0 o0 z* ?
</property>
9 C+ w2 }' P1 o3 l <property>
2 _9 ?3 G. T Q <name>mapreduce.job.max.split.locations</name>; V. M! O( d. M y; V
<value>10</value>
9 G7 j' I+ z8 q' y( Q. c5 ? <description>The max number of block locations to store for each split for( v/ y- u+ x* L2 V, [) y
locality calculation.
3 q6 r# B8 H' O( M; H </description>
% o7 |6 ?4 N% _+ f" ~/ D5 U" z</property>8 n% |2 |. b, _
<property>2 }/ P1 k$ L; f4 u# t5 y1 n& V
<name>mapreduce.job.split.metainfo.maxsize</name>4 }6 F; R1 P) [2 q, m6 U
<value>10000000</value>0 _' S! O: V/ I' X
<description>The maximum permissible size of the split metainfo file.
4 X+ x3 \$ P/ q5 M) ?/ j) T& D2 Y The MapReduce ApplicationMaster won't attempt to read submitted split metainfo2 Y) }6 U& v/ d7 Z& R% g
files bigger than this configured value.
* Z, G# x9 m# `% o) x6 R6 E5 N No limits if set to -1.5 t1 t( C( ?# ]% N
</description>
( p( w6 R0 r5 { p: `1 t' Q</property>' N/ S y( @ t" h! t' j! N
<property>' L' N8 u- H0 v7 C% f
<name>mapreduce.map.maxattempts</name>3 k( b1 @7 s& u* }, s2 I" E
<value>4</value>, @7 L1 q* s d; I5 O8 W9 k; L6 |
<description>Expert: The maximum number of attempts per map task.
! F5 n* y4 d3 x& e' z In other words, framework will try to execute a map task these many number. c$ z+ V3 I. ^; H4 \9 k2 Q: u- v& T
of times before giving up on it.+ L3 f* ]; Z/ N* w
</description>
9 X2 h, V C7 H- y- u6 P2 z</property>
D5 P& n/ F; E) ]+ _$ j: s<property>
# A* p# X- y: ^0 F <name>mapreduce.reduce.maxattempts</name>
, H0 e1 k5 R+ w4 [ <value>4</value>
( a+ @. [+ v( a" e <description>Expert: The maximum number of attempts per reduce task.
- r: M M5 f" R, q( } In other words, framework will try to execute a reduce task these many number/ z% H! p4 k1 h
of times before giving up on it.
; G# F' ` M$ B' |8 U, ] </description>) O7 `7 P9 m& K$ k# _
</property>
( d# U. g @7 r5 D<property>3 D7 `7 C: C/ j) S' s- v# n
<name>mapreduce.reduce.shuffle.fetch.retry.enabled</name>
7 H$ [( D( H+ | <value>${yarn.nodemanager.recovery.enabled}</value>3 ^. k p2 k% ^
<description>Set to enable fetch retry during host restart.</description>- x. h# A* J2 S, L
</property>! X/ Z, k$ d; G N
<property>
/ l \4 T. f! r. i( g <name>mapreduce.reduce.shuffle.fetch.retry.interval-ms</name>$ O2 K& a6 C3 A; z* J J2 d( _
<value>1000</value>5 ^" r! D# `5 P. m0 M. G; W
<description>Time of interval that fetcher retry to fetch again when some8 ]# p5 p& L" R. Y- L n
non-fatal failure happens because of some events like NM restart.
, q9 i* {2 }* ?' r& S: Z </description>
* {0 \4 P$ U2 S/ `7 V( z</property>; T3 F! |5 p; p/ j! X0 a1 s
<property>
( q. K: B9 u# B2 A <name>mapreduce.reduce.shuffle.fetch.retry.timeout-ms</name>3 l3 h9 F0 E( Y! k- j
<value>30000</value>7 T. e: F+ S/ V! V3 s1 q; |2 u
<description>Timeout value for fetcher to retry to fetch again when some; k) c- d- d$ ~% b8 x0 _6 Y$ o
non-fatal failure happens because of some events like NM restart.</description>
% A$ x9 J" ^1 i% K</property>
1 Q/ s' s7 A! n5 p1 p0 q<property>0 n# X C. y T8 ~. J5 L
<name>mapreduce.reduce.shuffle.retry-delay.max.ms</name>
6 P3 q' _, ^. L <value>60000</value>* y! x! |# g2 t# K* o- Z5 j0 X1 C
<description>The maximum number of ms the reducer will delay before retrying$ r( L8 s7 F2 d! V
to download map data.
x, K/ Q6 x! V8 S4 r+ Y </description>
. O5 R; d" `4 g) @/ ?# K* s2 S</property>
3 ?/ o& ~! L% E* |" M<property>( G% L4 J6 l# g( {5 B/ g2 C6 I
<name>mapreduce.reduce.shuffle.parallelcopies</name>
& u$ L) r0 C. c6 A9 | <value>5</value>- F; U+ ~- i: Y
<description>The default number of parallel transfers run by reduce& o8 D+ a: w. y4 p# M& m0 i. `
during the copy(shuffle) phase.7 j; }& s8 e- J
</description>4 f9 Y- W6 c0 ?* E6 ?
</property>
% t( A- V R1 h' t& c<property>
6 a3 r: v: U% [: I: d0 | <name>mapreduce.reduce.shuffle.connect.timeout</name>
0 C$ C8 d- Q8 K' [. \$ Z <value>180000</value>
6 S2 K1 @* f+ B9 m1 l( d) J5 Y) N <description>Expert: The maximum amount of time (in milli seconds) reduce
* j7 P/ R% w1 e r e5 N8 U! e task spends in trying to connect to a remote node for getting map output.
3 G5 O, D. K( ~$ j" H w </description>- s( T8 Q2 x. q7 p9 L& W7 j! `
</property>
8 {. i# \' Z3 q2 w, P<property># y$ X# k& M! F! Z
<name>mapreduce.reduce.shuffle.read.timeout</name>
) u' z! I7 L7 `' _, }* d <value>180000</value>
2 t$ B: \! l A; j. B <description>Expert: The maximum amount of time (in milli seconds) reduce
3 S2 G* {8 V4 M2 y1 j+ I task waits for map output data to be available for reading after obtaining4 d( S/ i6 c' \) I
connection.! B0 f. P# d* Q3 A: {* p" ]
</description>( m3 f r% u1 ^, z- D1 V$ R+ a- I
</property>
5 }, q$ X, j/ J1 R& k<property>7 ~$ B6 Y# F* j4 N
<name>mapreduce.shuffle.listen.queue.size</name>8 |) z* P! p% G! G
<value>128</value>6 [4 z9 \* Z. P8 `3 h& m6 [
<description>The length of the shuffle server listen queue.</description>7 x% _( D" R! C( H
</property>
+ E2 Z/ r/ {$ {' K0 B<property># W. |$ S+ Z' V6 l
<name>mapreduce.shuffle.connection-keep-alive.enable</name>
4 |( J, l( q2 M: H3 J( H4 ~ <value>false</value>3 u& v1 @. J" o5 m3 o
<description>set to true to support keep-alive connections.</description>
; k5 F0 P6 j$ p" x2 e</property>
# y7 B! j/ X* N, l& Z! i. j<property>9 M2 c* K1 v& u! g6 F# H
<name>mapreduce.shuffle.connection-keep-alive.timeout</name>+ b4 y }' {& M( I- P8 c
<value>5</value>
* J, P) r0 R+ l9 h <description>The number of seconds a shuffle client attempts to retain
/ e. ]* R' ]% C0 s, _7 d http connection. Refer "Keep-Alive: timeout=" header in
* X7 ^! h; s5 Q0 L* h0 x Http specification/ |- \2 i7 G9 z4 M3 B
</description>
: j3 @5 f4 p# p+ W5 U+ m. m% h: C</property>4 |3 }. ^7 H" O7 \( v9 `; |
<property>) H7 z+ k1 z- j4 M" i
<name>mapreduce.task.timeout</name>
! p4 f3 t" X+ |$ M <value>600000</value>0 w# q6 \( j7 L3 d9 v* U
<description>The number of milliseconds before a task will be
. W* m7 D* T) D I( i terminated if it neither reads an input, writes an output, nor' R& a5 x. g2 @% A
updates its status string. A value of 0 disables the timeout.7 _' A8 J( ~ y4 N% B
</description>, k7 `. D) l6 G% h9 c
</property>
0 m: e' w) ~6 h/ ?1 m+ b- l3 r x<property>/ F% g3 d% ^' V, Y& `' Q, d
<name>mapreduce.map.memory.mb</name>7 q' G* m+ e5 a6 y; i7 ~
<value>-1</value>% E- A1 ]7 B7 q/ ~0 _9 y
<description>The amount of memory to request from the scheduler for each
7 j: i* h% H/ M, E9 L( e map task. If this is not specified or is non-positive, it is inferred from6 v1 }2 M ]: m* o9 q Y2 W( L
mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
! m U6 O( y2 M/ B6 A1 u6 J If java-opts are also not specified, we set it to 1024.
0 X' Z& e# B d1 n9 N" Z </description>' l1 c) c; Z6 ?1 w$ B& _* w
</property>. C, A! Y8 X3 e, F, [5 `
<property>: a2 @. h9 T0 B9 n# W q1 Y M
<name>mapreduce.map.cpu.vcores</name>: V8 U* S+ a( r$ f4 W
<value>1</value>
; u5 D+ r) I1 Z9 k7 m8 Y9 u <description>The number of virtual cores to request from the scheduler for
- _% L4 P# q' s- s2 K- m each map task.
$ S' U* j, p' Z( | </description>
9 ~* y9 u9 X; G3 \6 X$ X+ a</property>
5 U- }( v( C$ l<property>
# d6 l5 j% C. w( Q <name>mapreduce.reduce.memory.mb</name>) y. r: [# |6 f1 v
<value>-1</value>
; k8 ]2 l1 K' Z2 V( e. b <description>The amount of memory to request from the scheduler for each
1 U5 A7 h; y | R reduce task. If this is not specified or is non-positive, it is inferred
/ G) a, R2 r" F7 P/ i- X from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
r- R9 s' {: ~: a If java-opts are also not specified, we set it to 1024.8 a3 k5 {4 T+ D( J
</description>$ @2 ?3 K' o* p0 y$ E5 h3 v M
</property>! v3 X% j: y+ e/ |6 l0 v
<property>9 R( c3 o& F- H: V
<name>mapreduce.reduce.cpu.vcores</name>) T! S+ {4 R; s2 d
<value>1</value>
+ j3 Q* a' S6 k& L. z8 F+ P9 v <description>The number of virtual cores to request from the scheduler for
0 P' K/ k- x" v* Q each reduce task.8 ^: f- A- O9 T0 D8 X
</description>
8 e. O+ c. w& U2 P. Y6 q</property>
5 V" Z8 t7 L7 L<property>( v* G- n3 m" z6 v' _/ U& v
<name>mapred.child.java.opts</name>
2 ^) `3 K/ D$ P8 w! i <value></value>
4 k1 o- k* z8 t6 w <description>Java opts for the task processes.
_9 e" g% `* L& U& J. v z The following symbol, if present, will be interpolated: @taskid@ is replaced
4 `8 k- K. i- A5 n by current TaskID. Any other occurrences of '@' will go unchanged.
$ t5 c+ c& n6 b For example, to enable verbose gc logging to a file named for the taskid in
# s4 D/ u! O) L5 t. Y% e /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:& d& I" O$ E1 r' q4 O2 e
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc+ Q8 p3 ~9 F' `
Usage of -Djava.library.path can cause programs to no longer function if
H2 _ r6 j. I$ ~3 ^$ X hadoop native libraries are used. These values should instead be set as part
8 i B; y( I! @# \; [ of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
' V; Z" l! j/ B+ K; R mapreduce.reduce.env config settings.
$ R- d; B7 P6 g& C( r2 x If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and" q7 u0 i ^9 F" g; Z, G' H
mapreduce.job.heap.memory-mb.ratio.) I# b$ e" C, S: _: @
</description>
( _, e- o, c! l( s8 u L' e* g</property>+ o5 y8 U% U P7 i3 H+ c
<!-- This is commented out so that it won't override mapred.child.java.opts.
2 @) W3 Z' _1 l$ p* b( u<property>
; `: n3 M5 t4 Y+ Z6 Q- m" l <name>mapreduce.map.java.opts</name>: T) u" ^8 n: Q. C6 y8 ]* `
<value></value>% U3 X# H0 w: Z* B- D
<description>Java opts only for the child processes that are maps. If set,
, j N6 }+ T! X# H this will be used instead of mapred.child.java.opts. If -Xmx is not set,' ~3 }% L* s7 d, P
it is inferred from mapreduce.map.memory.mb and
* F, ]% L# H' O( u5 j mapreduce.job.heap.memory-mb.ratio.
, o- U F) B1 K G </description>
- p2 Z8 }, G# u& E$ w7 ~</property>, w% \" e- ~& x
-->; P9 Y& J! W* @: Y- d
<!-- This is commented out so that it won't override mapred.child.java.opts.
* w. y& {+ T/ C, Q1 A<property>
0 D% u1 J! N# w' G* y) c/ t <name>mapreduce.reduce.java.opts</name>
! N8 v0 L+ i' t <value></value>
# {& `0 B5 j- M2 _3 O+ r <description>Java opts only for the child processes that are reduces. If set,
0 V; h5 A* U" w6 e* D6 `) J this will be used instead of mapred.child.java.opts. If -Xmx is not set,2 p( q! y$ i8 @( k! ^$ u
it is inferred from mapreduce.reduce.memory.mb and
4 r+ M1 q Z( }, T$ ^# p mapreduce.job.heap.memory-mb.ratio.
1 b2 {) g0 Z8 P5 J3 m </description>
" X/ M: b T( j& i</property>
G( R1 D2 I& Q1 T-->
: U# _9 z* A+ p6 T% m$ l$ _<property>
' X: o6 P9 m8 y# S <name>mapred.child.env</name>6 d7 P. H% N3 b7 l' |7 {/ D
<value></value>" b l o6 J( @' m0 Q
<description>User added environment variables for the task processes.6 u$ i) b! h# C6 s" f5 d. p
Example :. H: U+ U3 d; Z- T
1) A=foo This will set the env variable A to foo
9 H, |& }) s/ {# J- ~# k 2) B=$B:c This is inherit nodemanager's B env variable on Unix.2 h% W' t' x5 f* m! i' R$ ~$ V
3) B=%B%;c This is inherit nodemanager's B env variable on Windows.. a3 J6 t% F* E* j S2 r
</description>1 u, W* y1 F( M: d* a i
</property>
3 e; w3 L, S' c+ [* @* l) |, Q<!-- This is commented out so that it won't override mapred.child.env.! @* v/ S4 p9 b- G+ a& P
<property>
) I2 F4 T! a. s& p3 d# K <name>mapreduce.map.env</name>
" k( r3 V: {" M7 e) r/ z <value></value>% {8 R7 y/ e$ P
<description>User added environment variables for the map task processes./ t3 `: M$ X" @/ u
</description>+ p, O" u( C; T: ]9 h' d
</property>% d3 ]; z: i5 Y0 L$ W
-->; I+ w$ I8 |3 {' W/ z' S
<!-- This is commented out so that it won't override mapred.child.env.
- y8 v: C2 u$ Y9 l! \4 \- i<property>
0 \' q5 U1 D2 A3 S5 n <name>mapreduce.reduce.env</name>
8 H6 K* u* s# @( b/ a+ D2 I/ [9 l <value></value>
/ ?3 o) D+ }9 e% I) S2 C <description>User added environment variables for the reduce task processes.
: T0 w* {4 n; g9 ~+ ?+ J. K" o' } </description>
1 ^2 m: i/ t7 [. |</property>! B' G+ O! y+ x& ?& ]9 l( S
-->0 w9 t1 M) [# V( p& P
<property>
i* \0 r% e. K' q- o x* \ N <name>mapreduce.admin.user.env</name>
# I3 k1 {2 F2 _; e6 C <value></value>( q" ~! Z- }2 Y! N3 z
<description>
; ~: G$ I( t+ `5 m$ A: u8 p7 g: q Expert: Additional execution environment entries for
4 n9 e$ d6 g1 @4 X" a3 r/ Y. | map and reduce task processes. This is not an additive property.
, h7 F8 ^; x; u) ]( Z0 \9 z* M You must preserve the original value if you want your map and, n0 g+ G `5 n! v# S, J: {' F
reduce tasks to have access to native libraries (compression, etc).
" }8 m0 ~' ]# r. J6 T When this value is empty, the command to set execution+ V; K) I( `! i( B
envrionment will be OS dependent:
/ E9 G! P* v" t& p For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
; G" t6 A# }' ]) H+ O: E For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.
* ~; g# p% g w' T* v </description>
7 s# s- V/ k+ w9 P" c, Q4 o</property>
# A+ |) f; b1 V$ i. C<property># V0 D, ? U/ Q5 k& P. {& E3 [
<name>yarn.app.mapreduce.am.log.level</name>
# J9 a* a( J; g. U8 a3 K2 P' s <value>INFO</value>
9 L( a, x( S: _6 o% _1 K, C' f <description>The logging level for the MR ApplicationMaster. The allowed3 {/ ~- a9 q( C6 R5 |: x0 J3 z% Q
levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
2 c3 k* M7 E: r$ v8 R The setting here could be overriden if "mapreduce.job.log4j-properties-file"4 Q; |/ \( s2 S4 }0 ^0 B
is set.
# d( j) C+ n6 v </description>
) G( m7 r# ^" ]6 ~( C# `' C</property>
8 r: F) z# B! j. ]6 r<property>
- l+ s6 ^% Y$ M4 ?( { <name>mapreduce.map.log.level</name>2 t" ]6 f' I# {8 {# l/ K
<value>INFO</value>! w* w( u. h& D: J
<description>The logging level for the map task. The allowed levels are:
! G& h8 x; M M/ h& w1 x OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
, d# Q6 }. S s$ l$ U The setting here could be overridden if "mapreduce.job.log4j-properties-file"
3 d6 S% V4 ]$ x' p' Y is set.
6 `: x# C' n$ p, R' S' I4 v6 B </description>5 {, Y1 h, A' A9 e% A% `
</property>1 q8 c- m' M. U" c4 A
<property>3 v9 y+ c& Q* I/ X, x
<name>mapreduce.reduce.log.level</name>
: m2 L) U0 u% p) V5 L% w; I <value>INFO</value># Z3 ?; F% A4 X1 R* w
<description>The logging level for the reduce task. The allowed levels are:* c7 O$ g; M) r$ A. b5 ]1 Q, Q
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.; D- K7 c4 Y; I9 s' ?. a
The setting here could be overridden if "mapreduce.job.log4j-properties-file"! }7 V+ m" D5 g& T
is set.
) a) I/ i. j7 s </description>
! A8 L( e8 D7 R9 |% S: b</property>
1 B/ S, c4 y1 d/ k \) s" `<property>1 t8 ]- ?, Q# Y- M& Z
<name>mapreduce.reduce.merge.inmem.threshold</name>) S5 U( Y9 C, @2 l: f# ^
<value>1000</value>- _, r& e1 N3 c4 C! W+ b( e
<description>The threshold, in terms of the number of files
* f1 a' i+ j6 D: ]. d for the in-memory merge process. When we accumulate threshold number of files! g# c! i) ^( B+ k; g! z9 J, B
we initiate the in-memory merge and spill to disk. A value of 0 or less than% |' f! C- E3 |! y5 a+ |
0 indicates we want to DON'T have any threshold and instead depend only on
! \2 L4 S5 Z/ u7 | the ramfs's memory consumption to trigger the merge.
2 l0 R6 R# R/ |% j9 X </description>
' o! |1 S3 q" J</property>
- J! U8 ?" J, q/ j, I<property>
. i; W; s9 Y; Q0 S) R# | <name>mapreduce.reduce.shuffle.merge.percent</name>' f. @8 X6 H( m: `' g0 s& K
<value>0.66</value>
4 p- W, W. O4 x8 B& U <description>The usage threshold at which an in-memory merge will be
+ n" }( Q/ F- S- @4 Q1 }- C* x initiated, expressed as a percentage of the total memory allocated to
! O# k$ A' E- @$ @ storing in-memory map outputs, as defined by
5 B1 y" b! g6 g1 I) S9 Q- ~ mapreduce.reduce.shuffle.input.buffer.percent.! `# m! K8 l+ p, y+ @7 M
</description>+ v" l4 z; } _# j8 q
</property>+ z7 }8 c! K; u& K0 W
<property>6 f& L& v- U- M0 X
<name>mapreduce.reduce.shuffle.input.buffer.percent</name>
' Q R; P7 z/ i" B- } <value>0.70</value>- H: z. T/ \0 @) H
<description>The percentage of memory to be allocated from the maximum heap/ y# } |* {0 J5 `
size to storing map outputs during the shuffle., G3 ]& S$ w9 j: Z& Z
</description>
1 D. E5 ?* ?* i2 Z9 Y</property>
) t2 u! s7 L2 E9 C! |8 D* \7 S<property>& j N* q6 X- d; b7 @7 ~9 t$ m4 d0 X
<name>mapreduce.reduce.input.buffer.percent</name>5 S; W' Z7 ^! q7 N+ l Z1 `7 K
<value>0.0</value>' D+ s* L' f6 R$ z1 R
<description>The percentage of memory- relative to the maximum heap size- to) n/ ~+ ^% H i6 O( H
retain map outputs during the reduce. When the shuffle is concluded, any" Z: e& Y& {( o( J o7 B
remaining map outputs in memory must consume less than this threshold before
# h9 W7 W( `# _% b0 b# t4 L# |3 P4 w& u+ c the reduce can begin.. \* M9 z" N: _& F2 ]
</description>
# q$ [4 z4 K z7 d2 F# Z</property>
* { }2 L0 E0 N2 u2 E+ R<property>, H7 f% X: v, r- b
<name>mapreduce.reduce.shuffle.memory.limit.percent</name>
" P& y* \" w3 ^ }* \8 e <value>0.25</value>: M# e3 y8 h; E0 @/ u8 ~
<description>Expert: Maximum percentage of the in-memory limit that a
7 l p" d2 P0 t, N) ^7 D; ?# Y single shuffle can consume. Range of valid values is [0.0, 1.0]. If the value4 O, X& N0 x5 ~+ |
is 0.0 map outputs are shuffled directly to disk.</description>' Q" h7 k2 O4 f7 r) q$ r
</property>
1 J w4 B# d1 B3 d/ g8 w<property>, Y2 D5 P0 c# c
<name>mapreduce.shuffle.ssl.enabled</name>
4 l+ a1 Z6 z7 h/ j# u/ s& R <value>false</value>8 q5 D& w) C3 E: n
<description>7 _& z" D, A- \/ w$ N- q% ~: M
Whether to use SSL for for the Shuffle HTTP endpoints. S5 h9 d# x1 c5 H
</description>
8 E) x! M( J5 l' R</property>- y8 R: B+ L1 g; b0 o+ z
<property>" Y Z3 o% g9 \7 j4 X: w4 O" J
<name>mapreduce.shuffle.ssl.file.buffer.size</name>
6 g5 S4 P Q4 ~( a! o# i9 R: w4 v <value>65536</value>+ M4 e* M* X7 q/ a
<description>Buffer size for reading spills from file when using SSL.
" J3 H& t8 H4 N, I </description>, X; x+ p6 P9 @* t0 o6 G
</property># J, O/ e9 Z% i, t
<property>( A2 D4 H% e* K$ A/ a
<name>mapreduce.shuffle.max.connections</name>
$ e# o8 h4 X- O% t <value>0</value>
) [( \2 w3 v% f+ M" g0 l( A3 u <description>Max allowed connections for the shuffle. Set to 0 (zero)- N) j: m. t5 e; Y- v. A/ c& P
to indicate no limit on the number of connections. r( C+ l4 | A; G6 V
</description>
% } t* [+ _ a3 ]</property>
0 `( G/ S2 C! E7 G5 p<property>
5 G5 f4 M% d; S% C <name>mapreduce.shuffle.max.threads</name>6 L6 J! N3 @& U% s
<value>0</value>
5 b" S3 f J& J6 g5 ]/ L2 W6 v' ? <description>Max allowed threads for serving shuffle connections. Set to zero
- F" f1 D9 u' S; h' J3 M to indicate the default of 2 times the number of available' f8 x+ D' w/ V# [) L2 I
processors (as reported by Runtime.availableProcessors()). Netty is used to
% u% q3 V) z8 g serve requests, so a thread is not needed for each connection.
. a; l/ X7 u0 n% X5 L </description>6 w0 G1 B5 J6 @6 N# g
</property>: N5 _6 w, ]9 {" Y
<property>
7 j# Y7 `' p( P" s7 G7 I) F <name>mapreduce.shuffle.transferTo.allowed</name>
- O5 j q5 w; k0 D3 ? <value></value>" G& j7 W' B7 o4 U# a0 Y: H) j( T
<description>This option can enable/disable using nio transferTo method in- S$ U2 Y; O# s/ A# F
the shuffle phase. NIO transferTo does not perform well on windows in the
" H. p; `) |+ k/ ?( N2 X shuffle phase. Thus, with this configuration property it is possible to
. A7 O5 ]* V2 f$ J+ J& ~' g disable it, in which case custom transfer method will be used. Recommended
3 [) Q! b4 B1 _- i+ w- Y& C value is false when running Hadoop on Windows. For Linux, it is recommended3 M2 k4 Z+ e0 J( s8 K
to set it to true. If nothing is set then the default value is false for4 q+ r7 y5 X( Z- p. f
Windows, and true for Linux.
: m( k: S+ W5 I J' I2 { </description>
$ D" c2 T3 d9 ]9 o0 X2 z5 R4 {$ B3 G2 w</property>3 V' N5 \1 ^+ _- O
<property>8 j! j! Q* \# D; p5 p6 B
<name>mapreduce.shuffle.transfer.buffer.size</name>. r/ h% K+ q9 n1 i, N
<value>131072</value>
% a/ H' d8 D7 L% n, ~; a( P" |% S( A( T <description>This property is used only if: D% `# E0 M$ L
mapreduce.shuffle.transferTo.allowed is set to false. In that case,
/ k# ^9 t# B5 q+ W! x/ N this property defines the size of the buffer used in the buffer copy code* M. p7 Z- l+ Q: T; I
for the shuffle phase. The size of this buffer determines the size of the IO9 W' `/ W0 R( d5 F- ]' Q" z
requests.
; _6 P+ ], Y0 w. C3 Y. p1 D </description>
. k4 N) Z# e* Z1 o</property>
R1 D. H6 M- v4 |% K( n) W<property>, u/ C% ?# }& I6 c
<name>mapreduce.reduce.markreset.buffer.percent</name>" e6 R* {! R# I' m3 @- }
<value>0.0</value>
# D# Z5 e/ a2 ^0 t0 L8 I <description>The percentage of memory -relative to the maximum heap size- to
8 o' P* V. B. T+ s+ v* p b: M be used for caching values when using the mark-reset functionality.
: z1 A1 v6 m0 Q7 L6 A, O </description> Z, B& u5 I0 o5 ]
</property>
& P P& R3 H6 U5 v4 T% N3 E9 Y<property>" r5 R9 m) s2 _& T& R6 b
<name>mapreduce.map.speculative</name>
: `6 }: t4 Z- p: F <value>true</value>
: M" }3 d3 t* q4 X( V& c <description>If true, then multiple instances of some map tasks0 S. s# r- N! a, q4 E8 J* u
may be executed in parallel.</description>' \# Y% @2 H. c% ~
</property>
( v: N& k, {- ^ D* r<property>' |; Z" e( W0 j" w2 o0 b% `
<name>mapreduce.reduce.speculative</name>( @* @9 G( k S4 X, b
<value>true</value>
& P3 P6 u, R) y3 v- ~0 v <description>If true, then multiple instances of some reduce tasks8 _3 W# v T' r. D. T/ L+ N- L
may be executed in parallel.</description>
0 V5 r4 x6 @' d</property>$ v& b% v" J$ c6 N' s" i
<property>& F! A1 U' D& F& ~6 H! d
<name>mapreduce.job.speculative.speculative-cap-running-tasks</name>
: n; w$ b1 }8 M/ ~7 W1 ~; @ <value>0.1</value>9 {6 Y3 R ^' ^
<description>The max percent (0-1) of running tasks that7 N+ j3 _! m% o( r3 a
can be speculatively re-executed at any time.</description>3 g8 n. o2 ~+ n/ `
</property>0 x. ?3 U! a* L S0 e) Q: P9 q
<property>
3 m6 G+ I3 h$ S$ `5 I- V. w <name>mapreduce.job.speculative.speculative-cap-total-tasks</name>
- `8 K; \" }; O9 Q% U <value>0.01</value>0 Y+ w7 {2 `! X& h0 F7 p
<description>The max percent (0-1) of all tasks that
- }, p5 k, \9 Z4 N can be speculatively re-executed at any time.</description>2 F5 a: w$ t' E1 V, M
</property>
1 @. u( s; L* `2 w<property>
- s' o; N* M2 W% e3 {2 d6 @+ P, w <name>mapreduce.job.speculative.minimum-allowed-tasks</name>
" q+ G5 _) n& J+ [ <value>10</value>
# ` w8 ~( O3 a4 c- i5 i) e( a3 X r <description>The minimum allowed tasks that
8 Z0 R% T" g2 p3 q can be speculatively re-executed at any time.</description>
. ?! A& J/ B! t8 T* D+ z1 N' `0 x</property>
0 X. Q- G c7 D) z8 [& c<property>; U/ P3 B; x$ d# r
<name>mapreduce.job.speculative.retry-after-no-speculate</name>0 s) ~' n$ j9 ^
<value>1000</value>
( o# k% L; h, k4 I& ^( p <description>The waiting time(ms) to do next round of speculation/ m% o: E0 @7 x" Q n
if there is no task speculated in this round.</description>
$ k$ ~8 o4 M0 e% ?+ X</property>
. @0 z- ?! _3 c<property>0 P t% H: d9 u5 A
<name>mapreduce.job.speculative.retry-after-speculate</name>
! q' E/ W- M4 L$ S B <value>15000</value>
7 D1 t7 Y5 Q6 I8 o6 j( G j <description>The waiting time(ms) to do next round of speculation
1 {6 b! }' g1 ] if there are tasks speculated in this round.</description>8 ?$ ^: Q1 x( I- q" T4 x: T1 C) o
</property>$ Z& Z; r7 u4 P2 b9 ]
<property> g3 c: z) n8 s. \
<name>mapreduce.job.map.output.collector.class</name>
& |: W! w+ J: w L% k% m8 J) p. I <value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value>
1 @ z% F7 c- k <description>
, E; r' R* i4 k$ X2 x The MapOutputCollector implementation(s) to use. This may be a comma-separated
S* T5 }- T+ W' H' k9 c list of class names, in which case the map task will try to initialize each& Y$ h8 }" j" k& i. j E
of the collectors in turn. The first to successfully initialize will be used.
6 I8 N* y J8 `( [ </description>
* K& v! f2 L3 ]$ R0 Z6 ]4 y</property>; D" Y- x9 A+ E/ i. {
<property>
& l8 ~* V# f+ h6 c <name>mapreduce.job.speculative.slowtaskthreshold</name>8 r+ {2 m! f4 O" n' q6 g$ j" ?' O- b
<value>1.0</value>
( V$ N0 A `3 ^6 `* I! S' L <description>The number of standard deviations by which a task's6 k) m; U9 j+ V; `
ave progress-rates must be lower than the average of all running tasks'
7 _& A, a' K; f for the task to be considered too slow., e5 p" t$ D# n9 F
</description>
: K) l- l, ]. G</property>
* E4 K" E1 w1 {6 `3 i$ N* B# @<property>
+ `( f z" |# k* m* t# X* c& A2 m' M <name>mapreduce.job.ubertask.enable</name>& ]4 @( M2 e9 I. @% u
<value>false</value>
; W5 [* b5 P0 J( @- H# u. X! s <description>Whether to enable the small-jobs "ubertask" optimization,9 a: u+ Z! U) |' p$ _; R. z* t
which runs "sufficiently small" jobs sequentially within a single JVM.
9 w7 }/ H$ V0 w* k "Small" is defined by the following maxmaps, maxreduces, and maxbytes' |' B1 Q2 U. |& c
settings. Note that configurations for application masters also affect
* c5 e) }6 _) K, O1 E the "Small" definition - yarn.app.mapreduce.am.resource.mb must be
' }0 q0 _3 E( P" ^+ D% g/ G larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb,
- d$ M5 m8 [* h1 Q and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than
* f) O* G' W9 O! w. V both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable/ q( o7 ~5 }) R3 O6 U9 t
ubertask. Users may override this value.
3 [4 U5 e% [9 ` </description> H9 j- o7 I; h. ?
</property>1 v; o, J `" t) B6 T
<property>$ T) V' D: g& U, i5 Y9 L
<name>mapreduce.job.ubertask.maxmaps</name>, V, _$ K4 b8 q6 B* p) J- A
<value>9</value>
' _( {8 i; d" Z <description>Threshold for number of maps, beyond which job is considered& J! F0 T. ~( E: d5 n
too big for the ubertasking optimization. Users may override this value,5 p: ]; y* m' _0 ? g4 l; k
but only downward." ^; q2 X0 a+ b5 ~1 R. A3 o( S s: ]7 m
</description>5 ~8 @6 Y u( W( V
</property>
# i | C4 ?+ C2 } s<property>
1 b* L; @& e& O <name>mapreduce.job.ubertask.maxreduces</name>
. I, R; v! D& f <value>1</value>
! B0 g; x: V4 o# i, z <description>Threshold for number of reduces, beyond which job is considered
$ Y4 E( e0 d) I3 u3 U; i8 _ too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT
% [' r9 \ Z( R/ F& T( Y MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max,4 v* d7 ~" E1 _1 ~# B, I
however.) Users may override this value, but only downward.2 I6 _, J$ \+ @6 [
</description>
: W. o7 B) n% c6 v: A8 o</property>
0 T- z5 n, z2 v. a3 o* a. E! T; C<property>9 d1 r& q) E, F9 u7 n9 o& A
<name>mapreduce.job.ubertask.maxbytes</name>
5 d5 Y6 B1 V$ Z5 t <value></value>
! O- d9 u+ `& T <description>Threshold for number of input bytes, beyond which job is7 h- @) x1 Y9 [3 b: h ^0 q b
considered too big for the ubertasking optimization. If no value is
) K' r0 J# W0 V specified, dfs.block.size is used as a default. Be sure to specify a. q0 O* J0 _- V+ M, I
default value in mapred-site.xml if the underlying filesystem is not HDFS.
* X! y$ Q8 [2 ]/ j+ i; _ Users may override this value, but only downward.
4 }7 k7 Q. P* f- h& D N </description>4 ~# z7 f" a9 _5 y" w4 l* T, h
</property>8 f2 E& d# k# ^
<property>/ {0 A. g! @& |$ W- D4 ~
<name>mapreduce.job.emit-timeline-data</name>
, Z) R) g \! B <value>false</value>
! x$ W, I7 P# E) b& T <description>Specifies if the Application Master should emit timeline data
! f9 R5 W% {2 v: r7 @ to the timeline server. Individual jobs can override this value.
, G! b; `9 W0 z4 j8 O1 T </description>
. W: T, L! T! P9 [+ T9 a9 d</property>+ V6 E! V. |% o9 _
<property>; e* w3 \9 X% \
<name>mapreduce.job.sharedcache.mode</name>. V0 X4 n- D- s9 w9 g% v. F: Q
<value>disabled</value>2 |6 g1 C) y3 n/ U. l, J2 ]
<description>3 H2 \- g3 z5 _% K8 g; d+ O0 \9 H
A comma delimited list of resource categories to submit to the shared cache.+ r/ a3 W: a. z) a% q
The valid categories are: jobjar, libjars, files, archives.
- X0 j& r d6 s4 C3 W, C s1 e If "disabled" is specified then the job submission code will not use7 `6 I0 Z2 k& V2 U+ L
the shared cache.& j" U4 _. Y* N' J6 c2 h, _
</description>5 S/ x1 s) _# C% _2 a2 I
</property> _) o3 ~/ _4 F+ c j
<property>0 q, s0 l ~6 \2 P7 w( U
<name>mapreduce.input.fileinputformat.split.minsize</name>' x: H$ x7 T4 F; m
<value>0</value>
' F! r2 M! w+ O4 Y <description>The minimum size chunk that map input should be split7 j( ] V0 R- P$ a& o; l% M8 a
into. Note that some file formats may have minimum split sizes that
C: c* T5 V& T6 K6 I take priority over this setting.</description>
. F% U1 R# z, ?5 s j</property>+ ]4 V, U3 _8 {
<property>, t d2 }8 i& r+ t/ J0 F
<name>mapreduce.input.fileinputformat.list-status.num-threads</name>
# h4 O1 b/ ]. T2 Y" b# k9 @& h8 { <value>1</value>1 h; O$ @9 h; D8 K1 Y
<description>The number of threads to use to list and fetch block locations6 x( U! Z, d- i
for the specified input paths. Note: multiple threads should not be used1 o1 Y2 q* ^" z; f
if a custom non thread-safe path filter is used.) w- g$ C: K. V
</description>
~* e6 q) [7 L" y# u</property>
3 `7 {% [. n4 b5 _# y<property>
' J1 i& l+ H2 e' g3 m/ l <name>mapreduce.input.lineinputformat.linespermap</name>
* z K- j6 |1 o7 a( I <value>1</value>9 Y0 L' T% F% R# p7 f- V% u, _- d
<description>When using NLineInputFormat, the number of lines of input data. f* o$ Z6 R; Y& T6 n$ W& G
to include in each split.</description>8 X" f2 I7 X% R6 b& b1 b# u0 f. F
</property>
% r* w. \6 _4 x6 Q& _( U$ g! V<property>$ `0 P2 K K! [0 b7 ?+ E
<name>mapreduce.client.submit.file.replication</name>) m8 q V( l4 |
<value>10</value>9 \) s H d# c+ _9 `" a1 A! L; y* e
<description>The replication level for submitted job files. This
m- |2 K- H9 F) ~, ^. c' W) K should be around the square root of the number of nodes.
X: V/ { a; C) u7 e( L; } </description>
- ~3 H+ H1 z7 i5 W) f( j% S2 t, D</property>
* ?1 Z# Q. O5 S& o; w$ I<property>
+ P% W' A# h) V- F7 n! N <name>mapreduce.task.files.preserve.failedtasks</name>
" {* ~/ A: c1 e1 L5 t <value>false</value>& _3 w4 x2 O u- _; D' R6 F
<description>Should the files for failed tasks be kept. This should only be
% w9 b1 J+ n6 z: x used on jobs that are failing, because the storage is never" r- y+ j* S( T# ?
reclaimed. It also prevents the map outputs from being erased1 m3 D$ Z) W$ `( c. {, g$ U
from the reduce directory as they are consumed.</description>* g) S& }0 y4 {3 ?6 M9 B. N
</property>7 L5 q8 ~( x0 K9 u& f# X
<!--, P7 Y: \0 a/ M6 H
<property>
1 R* U( G% N# z- i& @/ A1 T <name>mapreduce.task.files.preserve.filepattern</name>
& [) m5 `, w! E. V$ F <value>.*_m_123456_0</value>
% x) G1 z* j2 @9 w3 K6 ~ <description>Keep all files from tasks whose task names match the given$ p2 d' w7 W T1 U& _ [" o
regular expression. Defaults to none.</description>
9 z- D* H6 T9 i6 \6 n </property>
1 x! k# r7 |$ g0 t! F; B, x-->
1 w; A4 [- e# g<property>& P4 V' b# m5 q4 ~' C4 f
<name>mapreduce.output.fileoutputformat.compress</name>& R/ v# b: o; J6 e
<value>false</value>1 n W& C. ~" l# d; s3 U( Q/ D
<description>Should the job outputs be compressed?
' p# r5 [; }0 x& A$ z- G5 k </description>/ f/ i: F: |6 s5 N" Y- R
</property>
H0 F. V8 L, n$ n5 z' L<property>- J: \! o% b$ m7 z* U# Z% x
<name>mapreduce.output.fileoutputformat.compress.type</name>
3 D( g0 o8 J. B) a <value>RECORD</value>
' k# p( \0 ~' f4 m# R. i <description>If the job outputs are to compressed as SequenceFiles, how should) V3 I& g1 A4 d9 L L% U
they be compressed? Should be one of NONE, RECORD or BLOCK.3 ]& r# l/ r2 l) K- f' U; z
</description>
1 Y, q# W2 p5 Q; o# {6 f$ L0 l</property>, E5 C1 y: `( a6 ?
<property>. H- o' Q. ?- T9 O4 o5 d; Y+ \
<name>mapreduce.output.fileoutputformat.compress.codec</name>
5 k+ d2 V0 s2 g/ { <value>org.apache.hadoop.io.compress.DefaultCodec</value>
6 O1 e* e) }5 L6 [; M7 K4 l <description>If the job outputs are compressed, how should they be compressed?% p' v; r% D+ }' T$ T
</description>5 X5 @/ d9 M) j8 e) `& R
</property>
, u5 {7 ~! F9 h<property>
_" [* v1 \+ c4 j+ U8 h/ { <name>mapreduce.map.output.compress</name>. N8 @5 D) l: I1 B) y
<value>false</value>
7 c2 W/ W- d/ F5 w6 C <description>Should the outputs of the maps be compressed before being& K& s; L& _7 y5 Y: f9 [
sent across the network. Uses SequenceFile compression.
; I' Q1 s" O; I/ c: o </description>5 V0 C) e- W3 u* x6 D G3 v
</property>
/ n( E/ J1 B6 h<property>
, d) `% V+ X; ?; F* e' e <name>mapreduce.map.output.compress.codec</name>
; ]2 x- H7 Y% K* b2 S8 {' ` <value>org.apache.hadoop.io.compress.DefaultCodec</value>1 ]4 n- Q2 v: }2 q6 P0 `
<description>If the map outputs are compressed, how should they be6 X& Y% P6 `1 Y6 p
compressed?
$ n. p. |9 C! x% y </description>
/ y( ^# e% e: G4 _. D4 z* D) }</property>7 \; n2 }/ P. _
<property>$ G3 q& c, @+ O5 l! z. {- v
<name>map.sort.class</name>: b6 K" ]! d6 Z& E5 a2 h& X
<value>org.apache.hadoop.util.QuickSort</value>
9 r+ a+ v3 ?7 ^4 m2 z9 o <description>The default sort class for sorting keys., f7 K% K# B2 {7 e) B' t
</description>' n5 w& f* a3 s2 S# ^
</property>
' b9 f$ ?0 B0 |! R0 {5 C<property>( a1 u; R. s; T
<name>mapreduce.task.userlog.limit.kb</name> G+ ?' }! \, b z
<value>0</value>
0 K! `" M$ L5 o# ? <description>The maximum size of user-logs of each task in KB. 0 disables the cap.% g. j x7 |; S+ P+ q
</description>" c# B/ V" D; X( s
</property>
/ Q$ H3 j$ z! v: k3 L) n<property>
( Z8 @ w+ b* {5 W3 @; [ <name>yarn.app.mapreduce.am.container.log.limit.kb</name>
. K' X: i/ `# p4 J <value>0</value>
" ^8 }. @/ @1 ~: R R, r2 `# S <description>The maximum size of the MRAppMaster attempt container logs in KB.
0 o; K$ Y: r0 ~; w2 C: J6 F 0 disables the cap.
4 e6 ?" n* w7 a. V7 V </description>* L( ^8 A3 Q% V+ f# [! ?
</property>, H2 g7 g: h+ L' B2 L) M# S i
<property>' {9 G: x( d* ~+ O0 _! V1 N5 E9 M
<name>yarn.app.mapreduce.task.container.log.backups</name># N- n4 a$ y$ y9 K' c
<value>0</value>
" T" X' K9 i& Y- p( Z, e <description>Number of backup files for task logs when using. @* l, v. s4 Q6 b: ^- A
ContainerRollingLogAppender (CRLA). See
8 |# o, w6 u; i+ \# b' n% j org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,7 x; l6 i" U7 C/ N! \' k* `4 q
ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA
+ _6 g$ `/ K# b is enabled for tasks when both mapreduce.task.userlog.limit.kb and& W1 L0 e6 b6 ^: o, W' v: X
yarn.app.mapreduce.task.container.log.backups are greater than zero.
6 U6 I4 ]& `. G </description>- G8 ^2 n7 U) L+ ?
</property>
1 a5 @- G# y, c4 ^/ L% ]<property>
( y5 a* |- ~' k! S <name>yarn.app.mapreduce.am.container.log.backups</name>6 e, m& | d5 H0 b7 Y( S. x
<value>0</value>
" \8 ^4 T* D( Q( K; d& f' c# r4 a <description>Number of backup files for the ApplicationMaster logs when using
* a! R) w4 |& y; F7 v" E6 L7 @ ContainerRollingLogAppender (CRLA). See: @, V1 h J6 Z% W* b5 V5 N
org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,7 D: O; d3 L5 G, v6 @2 o
ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA& z3 k7 E4 v; T5 E' z# {
is enabled for the ApplicationMaster when both9 n- R$ L0 s/ i, E0 I/ [. t' z1 F( W
yarn.app.mapreduce.am.container.log.limit.kb and$ I3 H0 @1 i* r% K M4 k
yarn.app.mapreduce.am.container.log.backups are greater than zero.
5 n0 |) _; u& @ </description>
5 d% e% A' C, l' \; A6 K' R</property>6 ~5 J2 Q7 p& Q& P
<property> b, q$ A: V2 l1 e" R2 y/ s
<name>yarn.app.mapreduce.shuffle.log.separate</name>! N# c* ^( y; w' C
<value>true</value>" _4 u) @! r3 r, B( Z2 e
<description>If enabled ('true') logging generated by the client-side shuffle
( J8 S" C' o% r! j1 B$ w classes in a reducer will be written in a dedicated log file
: ~8 F: k; T+ g( o0 u o1 q 'syslog.shuffle' instead of 'syslog'.
E& O% C8 F6 r- w! j2 V: | </description>1 _: C8 `1 H7 j( F( N5 G
</property>
* ^, _* u& c, s W<property>0 e5 E5 X% k7 Y# M: E7 B- s* P
<name>yarn.app.mapreduce.shuffle.log.limit.kb</name>
% V& P2 b. q3 a4 f* A2 x ? <value>0</value>
, u$ G6 V- Q g" B/ B/ W <description>Maximum size of the syslog.shuffle file in kilobytes' s3 ^: ^9 y6 C; X" D8 y( I8 v
(0 for no limit).4 _- p# a9 S' G: J- ]
</description>
7 Z+ s5 f' W+ K. S# W$ l+ R</property>
6 V' R, {0 m6 ~7 i3 {& A e: b# k<property>1 V6 \" }* P+ ~ _; `" w1 E0 k9 G
<name>yarn.app.mapreduce.shuffle.log.backups</name>
9 }7 r7 o9 ~/ E; g) e5 V <value>0</value>* z1 G8 A0 `9 m4 o7 O
<description>If yarn.app.mapreduce.shuffle.log.limit.kb and" F5 s& j% c( k; p' @ v& n
yarn.app.mapreduce.shuffle.log.backups are greater than zero
+ t2 X1 i" K2 s then a ContainerRollngLogAppender is used instead of ContainerLogAppender* ?5 s) { p$ O; `, V D8 u
for syslog.shuffle. See
# R* f( q6 ?1 h7 K( D5 ~ org.apache.log4j.RollingFileAppender.maxBackupIndex$ e% A. t! E& p
</description>' ^5 h0 ?& b. B% q* V% F
</property>0 {/ e) ^1 A" v0 L' S% L' z& `
<property>( l. N) l# K' g6 ]
<name>mapreduce.job.maxtaskfailures.per.tracker</name>+ b% y, ?' b4 q% P# F9 e- l; |
<value>3</value>
$ i$ N3 D! D% ?3 r: S8 C <description>The number of task-failures on a node manager of a given job; k1 u8 X5 }( @
after which new tasks of that job aren't assigned to it. It
: a' ]: {0 S4 J; w- j MUST be less than mapreduce.map.maxattempts and
+ w( O1 j$ s# L; \0 O mapreduce.reduce.maxattempts otherwise the failed task will
' E& K8 q4 O6 S never be tried on a different node.& X3 g6 l( u W9 J
</description>
$ f }2 _8 l4 V+ F4 j</property>
2 ]# W3 o& k p( s<property>
$ W4 D* n# m" h5 e( H <name>mapreduce.client.output.filter</name>0 }5 T* d; k- c
<value>FAILED</value>
- M$ {5 M. {, h4 z( X <description>The filter for controlling the output of the task's userlogs sent
4 n9 [/ l& z3 h to the console of the JobClient.
, `$ D: G) l: G6 e The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and: _. B& f* c6 }( h& r1 \
ALL.3 ~: h# U$ `* E `8 p9 n' D+ s% z
</description>5 `3 W+ t4 E* s
</property>
4 a$ Q6 b3 Y6 {. A( v/ n <property>4 ~. U$ F6 r! t! H
<name>mapreduce.client.completion.pollinterval</name>3 S3 j: x6 B& L
<value>5000</value>! i, h5 {4 K" r i% T2 S8 ^
<description>The interval (in milliseconds) between which the JobClient7 |/ L2 f4 |, F0 N3 H1 u3 F. t
polls the MapReduce ApplicationMaster for updates about job status. You may want to, y+ {! d O3 _/ H
set this to a lower value to make tests run faster on a single node system. Adjusting1 q8 I5 A8 O) K3 M5 s# J, T( f. u8 s2 S
this value in production may lead to unwanted client-server traffic.1 E" C2 z+ L7 F
</description>
. u! Z+ v6 x$ V/ G* g' j8 \ </property>5 H* ?; T4 ]# \& A9 ]6 Q
<property>3 i8 y2 a' y# j) M
<name>mapreduce.client.progressmonitor.pollinterval</name>
+ I+ X6 R6 x! B' {/ x& g <value>1000</value> |- h: [* C7 u
<description>The interval (in milliseconds) between which the JobClient1 A h$ L8 ?1 U# |
reports status to the console and checks for job completion. You may want to set this
# L% _3 }# _- d to a lower value to make tests run faster on a single node system. Adjusting! |1 h+ {# O( S4 k. f ^8 i
this value in production may lead to unwanted client-server traffic.
* }8 p9 ]% F |; Y </description>0 _# a1 `+ Y7 g% Y# F, S
</property>
: q6 e4 b/ T6 ?( S. S <property>, |3 V1 ]- U1 f. a4 B
<name>mapreduce.client.libjars.wildcard</name>. P* R0 B6 P/ Q
<value>true</value>
1 ]7 j4 [; V" a" s0 z$ O) O; p <description>9 z9 k$ H* H! S) c! V' [% T
Whether the libjars cache files should be localized using3 e d5 S/ W3 s! A9 x) P4 b
a wildcarded directory instead of naming each archive independently.
Q' n6 S# y! |- c3 H Using wildcards reduces the space needed for storing the job
- ~5 A& {3 X; i7 p. P information in the case of a highly available resource manager' ~$ k; j, o4 ]6 D
configuration.9 H) Y# U: G- F4 v) g. ^
This propery should only be set to false for specific
/ U. W& J0 f4 ^" T jobs which are highly sensitive to the details of the archive
& c+ }" L8 A; {9 R4 W& q$ b+ ~ localization. Having this property set to true will cause the archives
- E8 Z/ T. [5 O: R" ` to all be localized to the same local cache location. If false, each
3 ^! I) d% Q5 o D$ e0 K, }+ O archive will be localized to its own local cache location. In both
: \4 i# N( k1 e* w2 N% \/ [ cases a symbolic link will be created to every archive from the job's
2 x; u0 g9 j" u* E/ f4 S4 n working directory.
) f, k$ @) H/ \% z6 ~) i7 z% m </description>4 f* X: P" \( j
</property>
- z9 l$ ~4 N6 M9 l6 M <property>, ~ g: e! }% ~+ i
<name>mapreduce.task.profile</name>- c# L8 o3 U2 H6 |, H) `0 R
<value>false</value>+ I7 b+ s( {+ u- e
<description>To set whether the system should collect profiler
* e5 H. K) K' ?6 C" z8 e+ V information for some of the tasks in this job? The information is stored+ G: U! Q ^% o) Y8 a
in the user log directory. The value is "true" if task profiling
) } F6 E m1 y3 `, C8 {4 U is enabled.</description>2 B; `6 B! B& V2 C- J) s5 y: d7 j
</property>: |7 o0 y4 ]; M% a- G, `9 D
<property>
* w. M$ E! `2 w8 v& {: T' P <name>mapreduce.task.profile.maps</name>% [. W1 f1 B+ i
<value>0-2</value>
9 k, K) b1 L) f# c s, L6 [ <description> To set the ranges of map tasks to profile.. n) s0 X. I7 T* U. q" K _
mapreduce.task.profile has to be set to true for the value to be accounted.+ @- E1 A9 t; [" f, w6 ?
</description>
0 M- @4 |% A) p) l </property>
. n. r/ B% \5 s4 q7 T <property>* }1 j4 @, C) g/ j
<name>mapreduce.task.profile.reduces</name> i- L6 @1 a! |
<value>0-2</value>
. K% t) K2 x- |) k <description> To set the ranges of reduce tasks to profile.# J( e; e3 E8 m% d
mapreduce.task.profile has to be set to true for the value to be accounted.
* b, F* @3 m+ L9 [ </description>6 E1 p, D3 R' M9 ~- T; a7 A
</property>. ?1 M% ]! S% Z5 F0 V
<property>" I0 \; |5 j2 k1 N1 @4 g- t* I! h1 I% \0 B
<name>mapreduce.task.profile.params</name>
2 ]' s& [/ b# I: ]! K; e <value>-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s</value>
- C( s) h! N4 c! \4 Z <description>JVM profiler parameters used to profile map and reduce task
! \9 Z, E+ a1 e3 e6 O& Q2 E attempts. This string may contain a single format specifier %s that will
- L2 q: `" s( `6 z4 l: A2 G be replaced by the path to profile.out in the task attempt log directory., t! e2 `# `( n
To specify different profiling options for map tasks and reduce tasks,. ?- e9 g; E3 Y9 p5 i. ~9 e
more specific parameters mapreduce.task.profile.map.params and
, y% ^3 b2 c1 l: e; | mapreduce.task.profile.reduce.params should be used.</description>
6 u! i; y4 D$ D: z4 E </property>% R' b. j5 Y1 u$ E- d& R
<property>! d0 U6 ~" D9 O u( t
<name>mapreduce.task.profile.map.params</name>
5 L. g+ e% @0 Q' n6 p <value>${mapreduce.task.profile.params}</value>! z/ l% K, d! _1 n: D
<description>Map-task-specific JVM profiler parameters. See" Q! B) R# P6 O* H% q
mapreduce.task.profile.params</description>
# L6 q! F9 E7 w3 n2 z3 ]1 ~ </property>
2 _2 I& w& U/ w9 {( @! s% D <property>
# [' q/ X" l8 |9 @- {# p <name>mapreduce.task.profile.reduce.params</name>0 p" i3 e! x( i" V
<value>${mapreduce.task.profile.params}</value>
' P' Z( s1 a$ E5 X <description>Reduce-task-specific JVM profiler parameters. See$ t# w0 m) X+ I+ q6 {. f7 K, K
mapreduce.task.profile.params</description>
3 W- ~; ]! Q* I: T$ s( e! q9 w </property>
0 O. K8 D' I+ D! Y% } <property>
1 Y; I* I) B" V2 t& i' K" F <name>mapreduce.task.skip.start.attempts</name>3 j0 E, [. m2 C- y4 F. f
<value>2</value>! F; }! p9 a9 F. A# _0 C& h! g1 |, E& h
<description> The number of Task attempts AFTER which skip mode
1 B. A8 I0 J7 E/ Y! [+ E will be kicked off. When skip mode is kicked off, the
0 l" N9 i4 |6 `3 ]6 q9 ~! I tasks reports the range of records which it will process- b: p0 H+ Z5 z4 n: J( i$ y
next, to the MR ApplicationMaster. So that on failures, the MR AM( q7 h3 f" ]( u
knows which ones are possibly the bad records. On further executions,
4 G1 c) B- a' g8 L6 S those are skipped.+ o/ v8 f( ]( I- x- y: W6 r. I
</description>; |9 ?2 o2 a" x
</property>5 w: Q `1 u* E5 \3 D9 d
<property>
* b! _0 G# N/ g: e <name>mapreduce.job.skip.outdir</name>
# [) \& q3 k" w- E" G: a2 D <value></value>+ N& c2 h7 j/ [; a5 [
<description> If no value is specified here, the skipped records are
+ w9 i' [ S; Y; E( Y4 a- J/ C written to the output directory at _logs/skip.) K3 ]( `( b% J; U6 S! _
User can stop writing skipped records by giving the value "none".
' L! w/ c2 Z! i' v </description>- S1 J X: `8 d5 Y$ H: X7 [$ R4 ]( p
</property>
) O* Y- P+ i9 G; G% o9 P <property>
" D: B3 H n7 o& L/ } <name>mapreduce.map.skip.maxrecords</name>
0 {& [ W1 e% E+ B3 k8 ~2 I8 C <value>0</value>
: ]8 W3 n c& q* z: [ <description> The number of acceptable skip records surrounding the bad1 L, ^8 K. P# @* \% { D# r
record PER bad record in mapper. The number includes the bad record as well.
3 [& m5 n _9 s+ M/ g To turn the feature of detection/skipping of bad records off, set the
# m4 Y4 g& _/ p. I5 i# } value to 0.
( t4 i9 ?4 y1 R `1 Y! _ The framework tries to narrow down the skipped range by retrying
9 @5 z- [4 r" ?; U until this threshold is met OR all attempts get exhausted for this task.0 `* F) S4 G8 O# q) A) p! t; d
Set the value to Long.MAX_VALUE to indicate that framework need not try to2 r, K l9 L$ [$ G
narrow down. Whatever records(depends on application) get skipped are
|* D8 Q% t9 D5 q acceptable.
- T+ [( Y" J$ Q </description>3 q+ v* E) T1 s5 v S8 Z }6 A
</property>5 X6 b8 L+ @4 ?) V5 E
<property>0 E9 }# Q: ~+ V ~
<name>mapreduce.map.skip.proc-count.auto-incr</name>$ M3 N9 G' J/ X/ T
<value>true</value>
* [1 g- Y k: O- ^ <description>The flag which if set to true,
; U$ v& d1 \: j$ A. o SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by8 X( G' `& G1 H- n) G2 O
MapRunner after invoking the map function. This value must be set9 K( X% l* T/ P( ~% b
to false for applications which process the records asynchronously6 c5 H: a4 g- @! g6 Y, ]
or buffer the input records. For example streaming. In such cases' H7 r' Z+ W+ r
applications should increment this counter on their own.1 S4 @, y/ p" `4 J
</description>
1 R1 {* j g, f) X </property>! Y2 I6 V- p8 m8 X
<property>
( _8 f6 z4 u/ u( ]" @ <name>mapreduce.reduce.skip.maxgroups</name>- n2 s5 C5 S/ A% o$ e, f
<value>0</value>7 U% K+ o6 j" W3 G" Z$ R
<description> The number of acceptable skip groups surrounding the bad7 |) Z: ~2 f' W% `1 E, U6 [
group PER bad group in reducer. The number includes the bad group as well.
7 F# D! J; e0 `3 t" A# E To turn the feature of detection/skipping of bad groups off, set the( b$ I! r+ I* h" d7 s. g
value to 0.
U& _ {) Z8 K D% |, t @ The framework tries to narrow down the skipped range by retrying
3 Q' b' o( i' E3 X until this threshold is met OR all attempts get exhausted for this task.
3 Q! C3 n# @' }2 u u& k7 h Set the value to Long.MAX_VALUE to indicate that framework need not try to
2 W7 G6 X B& t$ w* ^ narrow down. Whatever groups(depends on application) get skipped are) E8 y4 L/ [8 `7 A- j8 w$ b! }4 l
acceptable.
% a4 o. u' t0 P. I" O8 ^ </description>
- K" n4 u# P2 t$ E* F4 ]: B </property>3 @4 a1 c, |3 [' ?% u
<property>( O0 T1 U; V* ?
<name>mapreduce.reduce.skip.proc-count.auto-incr</name>: W, Q: {) u8 G( o
<value>true</value>
2 f( y! V8 I2 D+ b) ] <description>The flag which if set to true.
6 O- `" y8 z- [+ @* P SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework
7 x6 H$ M( g+ T4 J after invoking the reduce function. This value must be set to false for: E1 J7 R: {8 f) _
applications which process the records asynchronously or buffer the input( ?$ P% I" i Z# c% l
records. For example streaming. In such cases applications should increment
( b% {; V; S! {+ W* w; P m this counter on their own.! m+ p, B4 C) W% S% V5 h. D; l
</description>
. _+ F! P! X" M3 H' R% M# A; Y </property>) d- K3 F9 l) s2 V8 G& M- D9 L8 m
<property>
* r! q. M4 y. c! K( s <name>mapreduce.ifile.readahead</name>
8 M; _# s2 [! \. {& Y <value>true</value>+ i% ?6 j! i* _+ J6 K+ J
<description>Configuration key to enable/disable IFile readahead.
2 R- l' L0 d' @/ G6 w0 k </description>' E. m2 p m/ @' s, J
</property>
8 c" H0 ?# E* r9 U4 \6 B3 E6 W <property>
3 d' j6 d, H# ^' a5 \, v% c; ?5 [ <name>mapreduce.ifile.readahead.bytes</name>( L9 B4 t; v* l; o
<value>4194304</value>
8 p2 d9 E) V# H+ Y; }% y# ? <description>Configuration key to set the IFile readahead length in bytes./ e) A6 H3 @. Y7 l; Y
</description>
6 o0 I# O7 C5 p, [/ g) k </property>4 m0 r1 P* b; b: F0 Y% X
<property>
$ `/ l! q+ X- n2 U' @ <name>mapreduce.job.queuename</name>3 W. J0 u9 p& s+ s' `
<value>default</value>9 L# H! W8 p. h& b$ @
<description> Queue to which a job is submitted. This must match one of the( l7 {& |/ m3 a
queues defined in mapred-queues.xml for the system. Also, the ACL setup
- x6 y% {2 I, M7 ^ for the queue must allow the current user to submit a job to the queue.
) V. E1 K; H/ J: D/ n+ L Before specifying a queue, ensure that the system is configured with7 W6 S/ s& B/ R3 T8 A$ c; X
the queue, and access is allowed for submitting jobs to the queue.0 u9 m0 @) i' p8 @. ?
</description>
; K3 G K- O5 I( n4 h</property>: V6 V0 H7 `( X! n
<property>
, e. H% {. U5 L x' C' ^1 y <name>mapreduce.job.tags</name>) m: u. K- s3 h0 S' ?' V
<value></value>
& M# F, m% y! \- L S. I <description> Tags for the job that will be passed to YARN at submission
. E. ^; f, w7 v time. Queries to YARN for applications can filter on these tags./ T: D; R1 f4 U s! p# O
If these tags are intended to be used with The YARN Timeline Service v.2,$ c! R; F1 w! t; N' ~9 I
prefix them with the appropriate tag names for flow name, flow version and
# Y# e: S) @5 e% H1 z flow run id. Example:
* g* d! W P, C' o! Y* ] timeline_flow_name_tag:foo,
8 F+ d5 }6 Q- I) W timeline_flow_version_tag:3df8b0d6100530080d2e0decf9e528e57c42a90a,! ^( I$ A$ D: A+ b6 [5 c
timeline_flow_run_id_tag:14652463485994 s. B# } c; L9 d& O
</description>
. V- q9 F( Z0 r/ K </property>
+ J8 P. e8 j1 t<property>
3 W% h$ p, W+ q4 e <name>mapreduce.cluster.local.dir</name>/ I7 c9 T& P, [) f) a
<value>${hadoop.tmp.dir}/mapred/local</value> e' t* F F4 l
<description>
6 a9 \% S2 ?" Y* G4 c) N, Y; N0 C The local directory where MapReduce stores intermediate
0 h; b4 l F5 \9 I4 m) P Y( ` data files. May be a comma-separated list of! M) C0 J" u& R4 V: T
directories on different devices in order to spread disk i/o.
+ m1 M J6 C& c( Y+ W& v, g- _% q Directories that do not exist are ignored.
; J2 H+ f& G8 b, Z! i5 V </description>8 \/ X* C) O9 u4 Z( q. i' W8 `
</property>
) G+ Y0 i/ T1 S t1 X# Q<property>1 V/ R* L' l! A. c8 A; C" X, j& a, J
<name>mapreduce.cluster.acls.enabled</name>
8 c+ v9 p8 {3 L* B; r/ }/ C& l <value>false</value>
: T4 C5 Z' H0 J$ n* ` <description> Specifies whether ACLs should be checked
; G. m* K7 S8 O( X/ o" q for authorization of users for doing various queue and job level operations.$ P# O% S7 k, |* s) w7 U
ACLs are disabled by default. If enabled, access control checks are made by; f( B6 U% }" u* G5 g" r4 W, l
MapReduce ApplicationMaster when requests are made by users for queue. M7 I. ^( U2 `- l2 v
operations like submit job to a queue and kill a job in the queue and job
& E; ?5 w) J9 `8 o9 _, p5 S" H operations like viewing the job-details (See mapreduce.job.acl-view-job)/ m$ G& L+ W J( s
or for modifying the job (See mapreduce.job.acl-modify-job) using
3 F8 g/ ^" [& f$ f" g( W7 @0 h Map/Reduce APIs, RPCs or via the console and web user interfaces.! w; p0 l8 O: o' y8 O/ l
For enabling this flag, set to true in mapred-site.xml file of all
2 l( p2 O8 M J' M8 R. I MapReduce clients (MR job submitting nodes).
+ o; x% R- \8 l) R& h: n </description>
* i0 }" n, E% ^4 w</property>* b/ T0 X* ~; ^* J" l, `! D- ~
<property>) _8 Q* r4 F7 n5 |- s. |
<name>mapreduce.job.acl-modify-job</name>
; S" H( V/ y0 y <value> </value>
1 `" S5 j/ x8 F4 o2 P: o$ p <description> Job specific access-control list for 'modifying' the job. It
0 Z7 u% _9 p; Q/ x( y is only used if authorization is enabled in Map/Reduce by setting the
x4 p3 {4 B' P- m$ U* T configuration property mapreduce.cluster.acls.enabled to true.; V/ }( E4 H0 K2 B. t
This specifies the list of users and/or groups who can do modification$ ~- z% A) @# d5 n6 f
operations on the job. For specifying a list of users and groups the* T. a! r$ B% K7 L. u8 H& b X
format to use is "user1,user2 group1,group". If set to '*', it allows all
: j, Z) [/ B5 A+ U: k- z users/groups to modify this job. If set to ' '(i.e. space), it allows
7 Q2 e/ |( H* p( T none. This configuration is used to guard all the modifications with respect
& D% {5 Y/ Y9 u: } to this job and takes care of all the following operations:
! \0 ~( k! y/ h# G o killing this job! Z- S: D/ c, s5 `
o killing a task of this job, failing a task of this job3 |# z6 E2 O+ N+ G; e$ G$ S( ]& m
o setting the priority of this job' O* P2 B* N+ S' U3 a5 F5 `
Each of these operations are also protected by the per-queue level ACL. _. P5 y6 z2 |, S/ _6 k
"acl-administer-jobs" configured via mapred-queues.xml. So a caller should
- a' G. b" \! P* P" p8 K2 o have the authorization to satisfy either the queue-level ACL or the; H8 A' D0 j, q
job-level ACL.9 ?/ C0 q% Z) s1 m4 W, }
Irrespective of this ACL configuration, (a) job-owner, (b) the user who( B" L- Q) |7 C8 K# R, `* K8 f
started the cluster, (c) members of an admin configured supergroup
; c+ [! @; g$ F/ X configured via mapreduce.cluster.permissions.supergroup and (d) queue- p( J6 ~9 d' w5 s" F6 a6 M
administrators of the queue to which this job was submitted to configured
# I- K! ~$ T& p& D) S1 O via acl-administer-jobs for the specific queue in mapred-queues.xml can
& Z, F% o6 D% q do all the modification operations on a job.2 Z. i' o+ Y* j0 c3 J
By default, nobody else besides job-owner, the user who started the cluster,
N# U7 K+ ]; [1 }* C members of supergroup and queue administrators can perform modification
6 Z: H6 r" a" V: E1 B) u1 i operations on a job.
6 M. W% {2 K0 o% r" O: D J1 }% P </description>8 g V" ]3 e6 v( W! c* _
</property>
0 y- F( t3 k- `/ M5 Q0 y<property>3 j- b1 b P6 C: L6 _% W) {
<name>mapreduce.job.acl-view-job</name>
j& p. r i$ c$ W. u7 Y <value> </value>
4 `% X! ~: X1 I+ z <description> Job specific access-control list for 'viewing' the job. It is* c% k# x; K# P, A. ]+ V
only used if authorization is enabled in Map/Reduce by setting the
0 L* q5 M1 U5 m" F5 c configuration property mapreduce.cluster.acls.enabled to true.
* [- a3 i4 ]6 U! o _4 u This specifies the list of users and/or groups who can view private details
9 e5 T5 {, i+ I about the job. For specifying a list of users and groups the
: m. ?8 l2 N; j3 l/ N format to use is "user1,user2 group1,group". If set to '*', it allows all
9 ]8 I# R3 N: Z- p2 G users/groups to modify this job. If set to ' '(i.e. space), it allows
1 i$ @/ D1 Q6 v none. This configuration is used to guard some of the job-views and at
% r& L. _ N& H6 H. s1 N0 d C- [8 r present only protects APIs that can return possibly sensitive information9 j* x. K3 f6 P; i' _$ f
of the job-owner like
1 ]9 j/ f N$ Y6 }3 o/ m1 K7 P o job-level counters0 S) Z- X& E9 j: B7 \
o task-level counters
1 M* t" L7 c- u$ g' p o tasks' diagnostic information
" Q& Y! Z+ p0 E- u0 o7 c2 c; S o task-logs displayed on the HistoryServer's web-UI and
$ S9 e T; s) p6 ~9 Y o job.xml showed by the HistoryServer's web-UI" h5 K% Z& f1 |, R. w: N
Every other piece of information of jobs is still accessible by any other
g) x! X& [/ g1 ?) X user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc.7 `* G& r6 s1 C
Irrespective of this ACL configuration, (a) job-owner, (b) the user who
% e2 N' ]5 r7 Z, |; Z0 x" H# @ started the cluster, (c) members of an admin configured supergroup* ^# r k2 Z6 M7 L9 ^3 j* S# `# [2 j
configured via mapreduce.cluster.permissions.supergroup and (d) queue
4 f0 A8 Y3 N" M" ~' W administrators of the queue to which this job was submitted to configured
% w9 P0 e- M0 O, o( ^$ E3 J via acl-administer-jobs for the specific queue in mapred-queues.xml can. @# Q; ?* j: q6 U
do all the view operations on a job." I9 x. m8 I$ C( B% A. i
By default, nobody else besides job-owner, the user who started the
: f }: d0 o# p& e) l cluster, memebers of supergroup and queue administrators can perform {' R* u5 s8 H! U
view operations on a job.8 }) J! N" F+ Y+ g& B; m
</description>- o3 |3 ~( `/ s9 Q- ]' T2 ^0 e$ C
</property>8 i- F, C; Z+ H
<property>$ c6 P5 I3 m0 w+ a# {. V+ \' B( J
<name>mapreduce.job.finish-when-all-reducers-done</name>
% f; R" M% k. V. a1 g7 E8 q) r: U <value>true</value>7 b' E! Y* I) J) u* l6 Y1 w
<description>Specifies whether the job should complete once all reducers
$ \- a1 N! X! |6 ]! D0 F7 _1 A have finished, regardless of whether there are still running mappers.
0 T1 H% D6 y; P) t) \ </description>
0 w+ K/ {& R9 N. k( G' V</property>0 z6 [% E2 e( x4 K8 U7 n
<property>
: `' M( g# D3 h7 Q <name>mapreduce.job.token.tracking.ids.enabled</name>2 W+ D0 S* r* P, f
<value>false</value>7 d, @4 G6 Z s9 U( E7 R! H
<description>Whether to write tracking ids of tokens to. W$ `$ d# B4 J% G
job-conf. When true, the configuration property, S( d) m3 H, _
"mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of
2 F- m* a) Q0 @8 m1 l the job</description>2 p S5 ?2 r" }6 L
</property>
, u; N' J0 w* C) F; c<property>
" |+ q* p1 A. P6 Y( r5 d <name>mapreduce.job.token.tracking.ids</name>
6 e4 W" v8 \$ @# x# R3 a# n9 V9 m <value></value>% @. l5 s. [1 o
<description>When mapreduce.job.token.tracking.ids.enabled is
) G, F, k+ g- x: M set to true, this is set by the framework to the
7 I: M: J, L, s/ s+ m token-tracking-ids used by the job.</description>
" o* J4 W# G, Q3 `3 p! r</property>; h/ I/ F* K K% X& [& V
<property>
. v b( A P! g( G" b <name>mapreduce.task.merge.progress.records</name>8 ?- J7 E& M) X) R8 B
<value>10000</value>$ X0 H- c2 d3 d
<description> The number of records to process during merge before* v0 U2 T! O. w
sending a progress notification to the MR ApplicationMaster.
, E5 H6 l7 R0 s3 @0 w- }& c6 j; \ </description>- t2 d% U3 r# \5 n' C4 G
</property>$ z9 E$ u! Q$ C9 Q. P
<property>$ V# _. O) \- Y/ v+ e
<name>mapreduce.task.combine.progress.records</name>
$ @4 O! g' y# P# P <value>10000</value>% ^6 I) [5 e; S! Z/ u& k1 P; Y
<description> The number of records to process during combine output collection
7 @- X! M; b4 R! x& X before sending a progress notification.( _$ j0 n: l/ G
</description>
6 F0 P2 P/ a: R* z</property>4 k: f, S) M7 \7 i. n" G
<property>2 B4 w$ n, ?, t: [$ H& n% k
<name>mapreduce.job.reduce.slowstart.completedmaps</name>
0 N: F3 |! ?: h4 g <value>0.05</value>4 b2 R% O+ w8 n3 v0 F# o
<description>Fraction of the number of maps in the job which should be! e5 A& Y8 v) a2 V2 J. ? L0 U
complete before reduces are scheduled for the job.
# _$ P% P3 ~6 ^- t- {9 n </description>
9 L; Y" \( U8 |5 j9 b1 E2 [3 `6 m, Y</property>
/ z) A! l4 N# B* N# r<property>" ^! f, O" j' z" L, w
<name>mapreduce.job.complete.cancel.delegation.tokens</name>. y1 `+ X4 B$ f' h% m
<value>true</value>
% O! K; r3 t# | <description> if false - do not unregister/cancel delegation tokens from1 g- ~: w! u( ?$ G
renewal, because same tokens may be used by spawned jobs
( ?' H) h& s+ @. H2 J5 y </description>
~2 g2 m6 D0 r& p% d' ~</property>' I+ z) P) m, ?) p& r
<property>
* p. J M2 @! y <name>mapreduce.shuffle.port</name>
: {8 w1 y4 w' Y! E" W <value>13562</value>
9 I/ h3 u3 b' |, Y1 @9 p- O, ~2 e <description>Default port that the ShuffleHandler will run on. ShuffleHandler
8 I4 U* I9 M+ L7 _" k$ B, b# V9 v is a service run at the NodeManager to facilitate transfers of intermediate
: e/ V0 Z% N8 X+ ]) g S Map outputs to requesting Reducers.
; I0 z! n2 N1 i3 A5 q- w! s4 | </description>
3 r. e3 r6 m+ _5 `! F; v( y</property>2 i; `' P; p8 \! A: H
<property>
1 r' v4 M- `+ s* {$ k <name>mapreduce.job.reduce.shuffle.consumer.plugin.class</name>
9 i" J( t: }, g9 c1 D0 Y <value>org.apache.hadoop.mapreduce.task.reduce.Shuffle</value>
4 X& g. B8 U# `. w. P- w <description>( N+ m; _/ ^% H$ F! ^
Name of the class whose instance will be used
" _% r7 d! k' R' [- M2 a, A/ \; [- l to send shuffle requests by reducetasks of this job.
& T/ l# \$ E/ I( @. ?. j The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.6 a0 g" P3 U# X% n2 j7 e5 H( x
</description>! `+ {" m9 ~5 p D3 l
</property>
# \$ h- g; H2 q' n4 u( E<!-- MR YARN Application properties -->7 R6 t! m" N$ f6 b v
<property>4 a: R. c# @2 `; O% t+ Y/ h
<name>mapreduce.job.node-label-expression</name>- ]) Y* ~2 f& k! v1 }
<description>All the containers of the Map Reduce job will be run with this' Q7 Y2 }' N! p4 C y
node label expression. If the node-label-expression for job is not set, then
4 i- t. U1 Q. \ it will use queue's default-node-label-expression for all job's containers.
- v4 K: f4 I' l. n7 r' Z </description>8 o; b( X. M+ x3 S" L
</property>. ?& j6 H4 a& d( e! c+ L- P& Y
<property>0 X9 O0 b1 X: I8 J: W
<name>mapreduce.job.am.node-label-expression</name>
0 [" f: y7 h8 J7 | <description>This is node-label configuration for Map Reduce Application Master
3 r2 o0 X, [9 R, h$ h+ } container. If not configured it will make use of* L4 e6 d3 o; ^, R, E) z0 T
mapreduce.job.node-label-expression and if job's node-label expression is not3 K- l! M# f9 F6 Q5 c
configured then it will use queue's default-node-label-expression.
: G; R( @; N& x2 r% E2 ] </description>
+ h- ~7 i) T0 c8 c4 I</property>2 M1 B9 x' X' |1 h2 b- ^
<property>
% k9 h/ ^9 \* D! p4 V <name>mapreduce.map.node-label-expression</name>
3 S* `2 J1 O" Q) P8 X <description>This is node-label configuration for Map task containers. If not7 @7 h2 s1 A; |& w0 i* B5 {
configured it will use mapreduce.job.node-label-expression and if job's& J8 y, i8 \$ x5 S
node-label expression is not configured then it will use queue's1 b8 @4 x5 g3 o! \$ r. y6 X
default-node-label-expression.; ~6 c& N% Q( Q" C' |: J
</description>
- z+ d1 e& { O' w</property>
8 Q, b0 V& p8 L# o3 \! B- K<property>* ~: y j+ Q/ ^/ x4 K% r$ a" r4 F" C
<name>mapreduce.reduce.node-label-expression</name>. b/ e- ~3 ]$ M P+ n% q: e0 Y
<description>This is node-label configuration for Reduce task containers. If
& l7 |+ @1 T' R' q' p' A$ }% { not configured it will use mapreduce.job.node-label-expression and if job's: h' }% D& k! x+ F' Z& ^# @( g
node-label expression is not configured then it will use queue's
$ R) k4 J* _) { default-node-label-expression.
" R5 Y. ?" s( h </description>
9 }* \5 J) ^# t/ c& j</property>
( ~0 A3 @; e4 U- ]- l" K<property>
6 d- Y( \# }9 q: B <name>mapreduce.job.counters.limit</name>
8 c- [& q3 V* S& Z5 R" Z <value>120</value>
* k( q; i8 U4 s6 s <description>Limit on the number of user counters allowed per job.8 {% b/ p5 J" k% K+ L% ~. A
</description>2 L. K9 u3 Z7 G: z8 |; Y$ ]
</property>
* J% l4 t8 }3 F: c) `, _' m7 }<property>
* N; \- }# E% p <name>mapreduce.framework.name</name>
3 C5 P: `8 x- O$ _" Q5 B& q' E <value>local</value>! a. i6 f* Y" n* V/ G
<description>The runtime framework for executing MapReduce jobs.6 ^8 o& \7 e4 m5 E; u6 j# I' [
Can be one of local, classic or yarn.
/ i4 B/ ^; P' E8 R! R$ P </description>
3 O ?- y* U8 t. A W</property>
! ^9 T7 B! ?% e5 s- i<property>
- o* M% {& s4 z* s <name>yarn.app.mapreduce.am.staging-dir</name>, R T& T5 y7 }
<value>/tmp/hadoop-yarn/staging</value>
: k& v/ J0 ^, B3 n9 ~ <description>The staging dir used while submitting jobs.
4 v+ s! G& _" \0 f9 X* m# ^5 O </description>
' x2 J. \) b, o- P</property>
# z: d' T7 j1 K+ d/ Y& e2 C<property>. Y& o) Z, C/ D7 _6 `3 W% X
<name>yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled</name>: h N1 Z# H" @8 F1 O, G
<value>false</value>
% O2 X) x) t; W( \% T! s- G( H <description>Whether Erasure Coding should be enabled for
0 R/ m& t. q6 q* Z" _5 e0 P files that are copied to the MR staging area. This is a job-level; g$ A2 o# v* [6 ^% M3 l1 g+ `& S6 |
setting.
0 q+ X4 g8 d4 j( e </description>
9 U- v0 K2 |& Y1 y</property>3 |1 o" c) ~$ V$ W2 w& S
<property>
) `! D0 M( Y* V$ I: C <name>mapreduce.am.max-attempts</name>" U" J9 A/ ^! D! r$ [% z8 w
<value>2</value>
! G8 [5 R$ B* l s& g( N+ k2 T+ H <description>The maximum number of application attempts. It is a
* g- W) M' K! N$ d application-specific setting. It should not be larger than the global number
( Y/ a2 }& E8 I' m) E set by resourcemanager. Otherwise, it will be override. The default number is1 m( Y0 O( \0 v: x
set to 2, to allow at least one retry for AM.</description>
6 E4 v9 u# y: M7 d3 @7 o' r</property>
# p8 Z i: M# a- C<!-- Job Notification Configuration -->
' f' x* [3 Q H<property>
- L! |9 x, @- ]: l6 n; R9 K <name>mapreduce.job.end-notification.url</name>
" P' k+ x9 w: t& _% S0 g7 h( \4 p( W3 ^ <!--<value>http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus</value>-->
" x' A6 |6 z4 q6 G <description>Indicates url which will be called on completion of job to inform
% l- r. D3 z0 [/ Y end status of job.
3 g; J9 R! c& E8 q: B1 J User can give at most 2 variables with URI : $jobId and $jobStatus.
], b0 _. e% t3 s( d0 ~) J If they are present in URI, then they will be replaced by their: T! |4 M; q3 e" H0 }
respective values.
! s9 x2 Y' _% F* } b6 b. h</description>
8 n- w- {3 G8 B8 e, x2 _- _. }! b</property>% e3 V) L8 J2 W6 D% L8 H2 n
<property>
" H+ X: v) x9 V3 d2 q6 [3 {4 c <name>mapreduce.job.end-notification.retry.attempts</name>
+ J9 a4 | U) T: o/ h. u <value>0</value>
+ Z1 X$ i) v& f# X/ C <description>The number of times the submitter of the job wants to retry job. d, r- ^8 b" R; f# x. \
end notification if it fails. This is capped by& b% n( Q/ }$ Y
mapreduce.job.end-notification.max.attempts</description>- N$ l* }2 {2 @1 `! `
</property>
) x$ u2 F; l2 x; o2 S# |# J+ [<property>
0 b0 L; t) f w* w( w <name>mapreduce.job.end-notification.retry.interval</name>
, b, i8 E) c' q4 q) X1 J <value>1000</value>
- ^- u& L3 \" T7 \5 N3 \ <description>The number of milliseconds the submitter of the job wants to
) |& O0 @. C: k& x8 O p wait before job end notification is retried if it fails. This is capped by
( z/ }, }# }8 Q/ E; n mapreduce.job.end-notification.max.retry.interval</description>1 A3 b9 k) O& c) @0 X- ^/ q# ? G. M
</property>0 b& Y/ H' m5 q1 w$ w
<property> ^& h q7 y1 R+ H4 z: t4 v) ?( m9 M
<name>mapreduce.job.end-notification.max.attempts</name>: ` i& J; E1 k
<value>5</value># s+ E% d1 n- y/ g" h
<final>true</final>
a* s( @. l$ Q n$ W <description>The maximum number of times a URL will be read for providing job
7 o0 @. K2 T- g0 X end notification. Cluster administrators can set this to limit how long) \+ w$ K1 O B8 H
after end of a job, the Application Master waits before exiting. Must be! c5 r% N) t J" y1 B) c
marked as final to prevent users from overriding this.
8 j0 M( m2 ~6 i, a; `9 o9 ?7 ~' k </description>
% R0 s/ M9 l8 P4 G/ q+ v! j; L</property>' P1 U) [) f/ L2 J+ a
<property>
, u) s2 c5 B+ l, [( X5 f <name>mapreduce.job.log4j-properties-file</name>2 i1 _" t8 S$ ]3 J2 w' G# b* E) ]
<value></value>
/ I, A" A- O$ h* P$ j <description>Used to override the default settings of log4j in container-log4j.properties
& B$ I Z m% d, r for NodeManager. Like container-log4j.properties, it requires certain
8 D, N0 M( C( w) I5 s framework appenders properly defined in this overriden file. The file on the
% Z( e% f' I) F: f! i" ^ path will be added to distributed cache and classpath. If no-scheme is given/ s! Z- c2 K9 R. B3 a, ?3 o
in the path, it defaults to point to a log4j file on the local FS.
, G& f6 v$ b1 j" Y) j& L% C% i </description>
1 D" ` r) Q( Y( v" ^ </property>
$ }- g* S, b5 P5 @<property>8 p% Z4 g3 j2 w
<name>mapreduce.job.end-notification.max.retry.interval</name> N' d; ^: [" @1 J- X! ]3 w% Y, F3 l
<value>5000</value>6 R, }+ Z+ e. K# \6 @/ T; {
<final>true</final>! O: o9 ^/ e: N. a1 }
<description>The maximum amount of time (in milliseconds) to wait before$ @/ c: Z; b1 r2 \% N8 b8 u1 w
retrying job end notification. Cluster administrators can set this to2 C7 _6 v0 L* V: b( s+ _
limit how long the Application Master waits before exiting. Must be marked. Y _! F7 Y0 e7 ?; N& C
as final to prevent users from overriding this.</description> Q. n5 S* V$ t% c& T) G! M
</property>
% L+ i. x) k1 k. V3 k$ C1 w<property>7 a5 @$ `4 T2 w2 g- S |
<name>yarn.app.mapreduce.am.env</name>
) E8 ~9 C+ d1 X$ p2 z% s1 ~ <value></value>! e& }- T- K. u1 c! k7 J
<description>User added environment variables for the MR App Master( i0 x, i6 q# V: S9 E
processes. Example :: p+ o- k! m6 Z/ h
1) A=foo This will set the env variable A to foo
. V3 @- T4 k% D; J 2) B=$B:c This is inherit tasktracker's B env variable.& t9 g2 b1 j/ p& n
</description>! z5 B m7 ?- b3 f# H1 T+ E, N3 u
</property> D2 }# Y6 b" j7 y5 w# E6 `- K
<property>$ z* P( d4 z, Y& B. ^* h
<name>yarn.app.mapreduce.am.admin.user.env</name>
* q1 g# M5 e1 Y4 N, @8 F& [ <value></value>
* r9 f. m* g) }: \3 m$ K$ b4 x <description> Environment variables for the MR App Master
3 [) b- j4 y- e1 k processes for admin purposes. These values are set first and can be+ D7 N: U7 t/ u5 _" X/ ~
overridden by the user env (yarn.app.mapreduce.am.env) Example :
0 \8 a# [' W& Z* o; _! ]3 l% o 1) A=foo This will set the env variable A to foo
4 D: ?& [" p' L6 |! O 2) B=$B:c This is inherit app master's B env variable.7 g, i. b. o+ ^8 I$ ]/ W0 k6 ?
</description>
v9 k% y- a! J, O</property>$ }- z6 z# [( }$ p) E" ^3 }
<property>
6 R! D( b) } ? <name>yarn.app.mapreduce.am.command-opts</name>
* l5 g' p. F, c& d' f% P, z, a, | <value>-Xmx1024m</value>
0 A" j" e6 Y9 o @% H# r9 R1 s, x <description>Java opts for the MR App Master processes.
% m0 {1 q* w7 Z0 l The following symbol, if present, will be interpolated: @taskid@ is replaced* E; T# w: Q; H1 q3 Z
by current TaskID. Any other occurrences of '@' will go unchanged. Y% K' f( f# m ^
For example, to enable verbose gc logging to a file named for the taskid in$ ^( D/ E$ _# b
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:+ A. e: k! B! k
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc8 m5 P' n8 g8 k6 S3 V# }
Usage of -Djava.library.path can cause programs to no longer function if
& D5 P R, U3 k Y6 T hadoop native libraries are used. These values should instead be set as part' i3 z$ c! ?" X# w8 ]
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and; {8 Z2 d& u' n- v0 J/ N% G
mapreduce.reduce.env config settings.
' N' E. p3 d$ ]' d# r7 [ </description>
' X0 e* ]- L3 j</property>
( n6 V- L) y! H9 t0 S6 {<property>" G3 I5 h- s5 A3 y
<name>yarn.app.mapreduce.am.admin-command-opts</name>
- n/ j$ f; t; @' L! Z <value></value>
5 o* y. K. I' b( W3 x <description>Java opts for the MR App Master processes for admin purposes.8 ~* m# y6 x* u! Z
It will appears before the opts set by yarn.app.mapreduce.am.command-opts and
0 L. x# a# {+ i2 }' r thus its options can be overridden user.
# [5 P& M& K, Q* B, ] Usage of -Djava.library.path can cause programs to no longer function if* a; d9 P# |) I( s/ u# G0 A
hadoop native libraries are used. These values should instead be set as part E6 _% ]$ k+ p! L7 F# G
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and' u" A' K# u7 D( m; D+ ]: x# P' H7 g
mapreduce.reduce.env config settings.
0 w& W0 o3 P2 X; F+ S </description>
/ _: S" v3 a0 u7 q) f6 l' _; N! r</property>8 W) b. q' m8 O$ T5 L y
<property>
7 |8 F9 G i0 Y$ V: A: _ <name>yarn.app.mapreduce.am.job.task.listener.thread-count</name>
- o% u& ~( Z' Z# [- h; y7 ^ <value>30</value>! a4 v1 y: d# `
<description>The number of threads used to handle RPC calls in the/ m: C" w' R }
MR AppMaster from remote tasks</description>
) }. f+ j) Z4 {5 Q% h) g& ?</property>
0 [2 N& L9 H, U# e<property>( L8 a& Q1 o6 a/ P1 E8 ?
<name>yarn.app.mapreduce.am.job.client.port-range</name>6 G: n7 r l. _/ O$ `
<value></value>3 Y/ q9 C4 t s) w; U# B
<description>Range of ports that the MapReduce AM can use when binding.' {$ l3 }. P. |" G6 N2 V7 b$ Z$ g
Leave blank if you want all possible ports.
5 j( F7 A; p3 P1 G) f/ c) [3 T) g1 p& C For example 50000-50050,50100-50200</description>
5 V1 }- E! a+ }</property>
& Y( }# H/ X* Q- Q# V( Q<property>
, a3 J6 T2 K, n4 n* T' ~ <name>yarn.app.mapreduce.am.webapp.port-range</name>* w- G* J* [" {( {& U' h3 \
<value></value>7 G6 [1 q9 o# z$ m4 A
<description>Range of ports that the MapReduce AM can use for its webapp when binding." E4 X' ]& [% E% t: p4 C3 e( `
Leave blank if you want all possible ports." h* |- V# S+ B( {- l4 h, l( Z
For example 50000-50050,50100-50200</description>6 u5 \7 `/ M7 I* G. k k
</property>
! T, n. L, k8 T3 K<property>: p1 z% n+ N* w( |* l
<name>yarn.app.mapreduce.am.job.committer.cancel-timeout</name>
' V+ Z8 }. I: c) |/ k <value>60000</value>4 r3 g" v( P3 o
<description>The amount of time in milliseconds to wait for the output8 u0 F+ t! R( A% [. C8 x% a
committer to cancel an operation if the job is killed</description>
9 p$ z% l* g7 `7 f7 j5 m2 k, r</property>
: ~- O/ S! k# h0 ~<property>
/ W# z( U, K& D' l Q# T <name>yarn.app.mapreduce.am.job.committer.commit-window</name>) t( R; y& U' r, r: V8 @6 b# B
<value>10000</value>
) n# r( X0 u7 E <description>Defines a time window in milliseconds for output commit/ G) K6 i/ k( T! q: Z) O# }
operations. If contact with the RM has occurred within this window then
. {: \6 }- G5 E6 F; V; y commits are allowed, otherwise the AM will not allow output commits until7 ?+ @. B/ q- ?( ~
contact with the RM has been re-established.</description>
O _% G3 a% h, I</property>1 X" V% |: H: P' A4 z* I
<property>' p) f, q6 \" q
<name>mapreduce.fileoutputcommitter.algorithm.version</name>! x& A6 P8 ]# e# l5 g. L8 g* m5 i2 I
<value>2</value>
7 x9 t! }( |% l% Z' N7 e <description>The file output committer algorithm version
% N ^* c) H& j! H valid algorithm version number: 1 or 2
7 k# l& c9 u. |1 q, y default to 2, which is the original algorithm
8 u5 A1 x( ]0 s9 C4 f In algorithm version 1,7 o; e/ _4 {. z' l6 V
1. commitTask will rename directory& W- U3 `) q$ M" J' X
$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/( }# T; d: ?9 v0 v% V2 T7 j4 ]
to
& J4 {% P F, c2 V) s1 f$ H $joboutput/_temporary/$appAttemptID/$taskID/% Z N( \9 f+ p9 {5 Y# _
2. recoverTask will also do a rename. N4 u3 y" }0 `' V7 M
$joboutput/_temporary/$appAttemptID/$taskID// U- f- W" O# m4 E6 \
to
( j1 P& U: [1 I7 o $joboutput/_temporary/($appAttemptID + 1)/$taskID/0 `% w- Q# s @. P G
3. commitJob will merge every task output file in7 V4 U9 G X2 P- r, r
$joboutput/_temporary/$appAttemptID/$taskID/: Z, C0 b/ O+ m6 U3 x
to z* ?5 |% R9 w6 w* _# x) \' Q( I' ~
$joboutput/, then it will delete $joboutput/_temporary/
* V' D7 R! m4 h and write $joboutput/_SUCCESS' ]( s( n4 o* f" A; @8 W0 y
It has a performance regression, which is discussed in MAPREDUCE-4815.
5 q; l j. z3 \' _ If a job generates many files to commit then the commitJob
8 _" V" A' Y. H% ~0 h method call at the end of the job can take minutes.2 I a' y2 \. W
the commit is single-threaded and waits until all
$ j$ m% P1 K' N tasks have completed before commencing.5 K1 k. u# z6 L) }
algorithm version 2 will change the behavior of commitTask,9 U+ s2 m8 P" Y& _
recoverTask, and commitJob.# L; l! @6 }0 r
1. commitTask will rename all files in% X d; n y( J: x, Q
$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/; I1 W9 a4 b5 F3 D/ Y& i
to $joboutput/$ n' w# C) b4 t4 d% W: e ^
2. recoverTask actually doesn't require to do anything, but for
# i c. P' l" w5 f upgrade from version 1 to version 2 case, it will check if there
1 ^# d: V3 e. J8 Q3 r6 U are any files in' Q( h# h, y( ]6 L! {) ]4 n0 y# U
$joboutput/_temporary/($appAttemptID - 1)/$taskID/1 F7 V* y, n/ [" F
and rename them to $joboutput/. c1 N3 D, B, v# Q% [
3. commitJob can simply delete $joboutput/_temporary and write
' [% u6 B6 ~, b! F $joboutput/_SUCCESS1 e! L; m- N$ Q* u
This algorithm will reduce the output commit time for
$ E7 ^7 `3 `, m& x/ C) d large jobs by having the tasks commit directly to the final
) q$ U- v6 B& p0 b. K% s output directory as they were completing and commitJob had! r+ d% v+ r4 T, W1 Z
very little to do., H" _) e3 F! m1 X
</description> L7 P3 h! d& a, l# o& I9 K
</property>
^# u8 h6 c( Q; m* h3 G X<property>( b ?5 }5 p1 h- r/ \6 q
<name>mapreduce.fileoutputcommitter.task.cleanup.enabled</name>, m; _8 _% F8 G% l4 D1 L
<value>false</value>
9 P( ~ k2 ]' ^ <description>Whether tasks should delete their task temporary directories. This is purely an
2 j$ i. j5 ?2 O' t optimization for filesystems without O(1) recursive delete, as commitJob will recursively delete
& ~4 y- y m; Q, ?& I6 B/ E- Q the entire job temporary directory. HDFS has O(1) recursive delete, so this parameter is left0 [+ h. z% o9 j
false by default. Users of object stores, for example, may want to set this to true., `. M8 X% W+ ]$ }) `
Note: this is only used if mapreduce.fileoutputcommitter.algorithm.version=2</description>
& C! P6 V' |4 B, a</property>! Q$ w; I9 T6 P! y
<property>2 B& }( R/ T, D/ ~2 t9 [; R' S1 E
<name>yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms</name>& C8 ~/ |+ ~1 R
<value>1000</value>
! f4 `# r! ~$ @ r! b3 V+ B+ U <description>The interval in ms at which the MR AppMaster should send
3 m8 w: X C8 \9 u# ?: j3 Q! ~/ i heartbeats to the ResourceManager</description>
& u& m4 K& t, P* |0 z2 d! t</property>
0 F' S1 ?4 _% x8 |- n+ |3 z<property>5 W1 n- F& `, x" ~ ]
<name>yarn.app.mapreduce.client-am.ipc.max-retries</name>0 ~+ d/ M& V# j5 t F0 E3 c# X
<value>3</value>
! P J9 ~, c# b <description>The number of client retries to the AM - before reconnecting
" n& z+ u( u' J to the RM to fetch Application Status.</description>
5 D& G, e$ t, d0 u s- D</property>
& n9 H( r$ C3 v$ M$ L2 t<property>
' Y7 c" _- x8 ?& B5 I0 c <name>yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts</name>
$ ?: q. x. E& m: q. }+ C <value>3</value>
3 Y- u& ^# j" N; e <description>The number of client retries on socket timeouts to the AM - before
" A% {( Q, Z d reconnecting to the RM to fetch Application Status.</description>
' t y# p4 z: A; T, L& R</property>
1 k5 g4 C+ [9 A- g+ I<property>
0 _: F9 Q" V( S* q& R <name>yarn.app.mapreduce.client.max-retries</name>6 Y3 C. ^. b' p# C
<value>3</value>. z" V; n* G1 ^4 G
<description>The number of client retries to the RM/HS before
2 N! W% |; w J; x" x0 O8 E3 k) b throwing exception. This is a layer above the ipc.</description>. Y. O1 A% e/ B# ~9 U
</property>1 H; D8 V1 K a: m4 @
<property>; Z3 S' n1 W0 N
<name>yarn.app.mapreduce.am.resource.mb</name>
3 p- \0 e" o: \. J* B <value>1536</value>
, [2 }8 T* t3 E% e- g; @0 C <description>The amount of memory the MR AppMaster needs.</description>8 F4 g1 q8 }" B# i. @
</property> W; s; `, c1 G" `+ A6 R5 ?& m" h
<property>0 ^6 K" D( K. N4 ?+ s5 N( _# |
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
5 h8 o% `; J4 p2 b0 c <value>1</value>
1 M( A) P0 P- m7 e( [, N+ t <description>3 [" @# ]2 H- I' ^* U* @( r
The number of virtual CPU cores the MR AppMaster needs.$ K4 d L8 |1 w! l' n* m$ N# a, _" U
</description>
4 o8 z& }5 x3 V! b' i$ |9 J7 @$ X</property>9 P% k1 S& v! U& |% }
<property>
" n% e& O4 w) ?; T) O x <name>yarn.app.mapreduce.am.hard-kill-timeout-ms</name>, e k* ~) C: m+ P9 q; ]
<value>10000</value>! O; U' t" N/ g8 q
<description>
- R9 G3 [0 l; R. j- [4 x% { Number of milliseconds to wait before the job client kills the application.* z$ F, m, d9 @# E& \7 H0 N+ H
</description>3 H7 d% `- U/ p
</property>
% ?. V- [! K% `! F+ _& M7 l<property>
* a" U& w# \# c0 [ <name>yarn.app.mapreduce.client.job.max-retries</name>* a% v8 D( y! d
<value>3</value>
( N) p, v- o1 k% p3 ] <description>The number of retries the client will make for getJob and6 ~8 Z8 K. |: S3 S3 p
dependent calls.' e, f# T, U! t- v0 s5 y
This is needed for non-HDFS DFS where additional, high level1 d, [5 ?6 S6 @: Q% h# x7 Y
retries are required to avoid spurious failures during the getJob call.3 @2 S: U( I+ p" _
30 is a good value for WASB</description>- M' p) t1 @# {9 c4 z+ ?
</property>
# G- c \3 h5 } E: o<property>: @( k" Y6 t3 q7 U
<name>yarn.app.mapreduce.client.job.retry-interval</name>8 H% ]! D+ B7 n( P
<value>2000</value>6 o9 D' S1 }) W- X( _( D% V5 B6 i
<description>The delay between getJob retries in ms for retries configured( S3 @7 V H5 w9 n0 P* _
with yarn.app.mapreduce.client.job.max-retries.</description>3 |( l' L9 Y& ?# a5 `" l4 K
</property>5 r! n( Z% P$ @; h
<property>
& K5 t# z1 B8 Y7 P* s! L <description>CLASSPATH for MR applications. A comma-separated list
! A: C' t* }" m1 y3 H- x of CLASSPATH entries. If mapreduce.application.framework is set then this
5 p ]. b1 n0 l/ N must specify the appropriate classpath for that archive, and the name of
5 J2 ]' L* K4 G% A; g' ^$ m) J the archive must be present in the classpath. D* y; R3 I; i+ a. H
If mapreduce.app-submission.cross-platform is false, platform-specific
4 Z% D# B0 R! F$ ~& E environment vairable expansion syntax would be used to construct the default
% L& o0 W* B8 R) {& U CLASSPATH entries.# g8 A& [% D* N9 u1 \; j
For Linux:$ Q; |* [! N% e, |4 U
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,: b7 z/ _4 A5 l# D r7 O3 \' |3 J
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*.# A+ W, N. z- w5 H; q" c
For Windows:
0 ` t! M5 |0 G0 f" i %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,4 E& G% l3 g9 s& k! j
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*.
( ?# N' ]/ Y. D2 {% f8 L! x6 U If mapreduce.app-submission.cross-platform is true, platform-agnostic default% ?1 T! T4 K0 k+ x# _
CLASSPATH for MR applications would be used:3 W2 G9 q0 z1 N5 R/ ^" w! M( e
{' ^# w9 C: R/ v; ?
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,0 E7 ?+ z! o0 x, f8 p- G: W
{+ n6 n u2 J; a8 s M
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*. h1 w, z( J2 N" N. a @
Parameter expansion marker will be replaced by NodeManager on container
4 X) t! t' ]6 Q launch based on the underlying OS accordingly.
- p& Q8 G5 m. h: V+ |# s# F U </description>
* u8 y. r& n5 e' k) m) B3 P <name>mapreduce.application.classpath</name>" o& m9 p* y" C# i4 x) D
<value></value>4 ^+ U% ?& ~; x& T7 Y7 l) X
</property>8 X4 Z: A7 Z$ L( v' w( C' p6 h
<property>
9 d( _! ?; [7 ?. t <description>If enabled, user can submit an application cross-platform
+ y& A! X9 @8 @0 W5 i2 e! X9 z i.e. submit an application from a Windows client to a Linux/Unix server or* a7 w$ Z+ k: }' X- Y9 J/ K
vice versa.4 ]( x$ ^( z8 G1 S
</description>+ u7 Y1 P" [2 Q! u
<name>mapreduce.app-submission.cross-platform</name>
1 `6 O7 C+ O% h& c+ `. \5 M <value>false</value>
, l, H2 K7 B; s! T5 X</property>
( _; `: D$ s! R& g/ p8 y<property>
; S' l5 l& d+ O W <description>Path to the MapReduce framework archive. If set, the framework) L, N$ T$ H& v- L& q( e# A
archive will automatically be distributed along with the job, and this
" I9 k1 H4 O3 U% P path would normally reside in a public location in an HDFS filesystem. As$ e! @( X3 j: T1 s: z9 f4 V
with distributed cache files, this can be a URL with a fragment specifying6 K% g( A( i; b8 W* J9 r- m7 g
the alias to use for the archive name. For example,) u3 M. A+ t6 V: I
hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would
2 d& ~/ W" _ {) \; x, Y m alias the localized archive as "mrframework".
) N4 V: Q8 x; l- v( w$ P% c2 k Note that mapreduce.application.classpath must include the appropriate
) c, R* `: c# R) \% N2 e classpath for the specified framework. The base name of the archive, or
, R* @9 W: k) x8 ^, n2 G( H a# c* O alias of the archive if an alias is used, must appear in the specified% q: d5 n" T# U- I% o
classpath.
8 h% S' I" i9 `3 x5 b+ `- W </description>
8 V W5 W1 s, X <name>mapreduce.application.framework.path</name>
" {; t& R& B( D/ u <value></value>, Y g1 Q1 c6 ~. l7 k) _* N* e
</property>
% X5 J. {$ L$ `1 M2 A2 \: P7 c<property>( f/ ~4 h% f; Q' V8 `0 X3 _% [* D8 @
<name>mapreduce.job.classloader</name>
- T% ]4 N* l- j$ P7 ~ <value>false</value>* [$ j, O6 {9 s7 z
<description>Whether to use a separate (isolated) classloader for: O1 s1 j8 G" p- ]$ L- k: @) Q
user classes in the task JVM.</description>! r) {8 i3 l1 b* ?: b/ C p0 {' _
</property>! [0 B0 {9 R) @+ p
<property>
# {+ A4 d4 K% p ~5 V1 p" J <name>mapreduce.job.classloader.system.classes</name>
5 w# {: _9 m# i! c# M/ T! w. @. x <value></value>7 T- I, W0 \2 M. f
<description>Used to override the default definition of the system classes for
2 a6 q4 D9 @1 G, ` the job classloader. The system classes are a comma-separated list of
/ Z& _- f9 k0 X& K4 ]! @ patterns that indicate whether to load a class from the system classpath,
% F3 V! Z: E# ^/ B' r" ~ instead from the user-supplied JARs, when mapreduce.job.classloader is7 z( K. K# f$ }' T# c
enabled.- H# ]( A* o( X5 H0 N
A positive pattern is defined as:
$ g/ Z, W, u& w" ?8 j) D! ?% I 1. A single class name 'C' that matches 'C' and transitively all nested* k& s+ X/ P3 F/ d0 h$ e/ M9 Y% ~
classes 'C$*' defined in C;
0 [! N2 l& q# f1 O8 ]/ Q+ P& B 2. A package name ending with a '.' (e.g., "com.example.") that matches3 A+ f. e" f+ I P, g; ^
all classes from that package.
4 n+ f& B- W5 O7 O A negative pattern is defined by a '-' in front of a positive pattern
5 O- O$ }. x# ^- s (e.g., "-com.example.").
" a5 ^$ Z/ c( U" z$ ^ A class is considered a system class if and only if it matches one of the
4 k* F5 ?" A, f7 i9 e positive patterns and none of the negative ones. More formally:
# {+ u: {1 K( j+ K) L8 p A class is a member of the inclusion set I if it matches one of the positive
+ I* S9 d& }0 x3 h& X patterns. A class is a member of the exclusion set E if it matches one of
1 }% [( L" k/ D$ i the negative patterns. The set of system classes S = I \ E.
, m) ~1 b. ~8 U3 r: M </description>" t8 L6 k/ g Y2 i6 H1 S2 Q
</property>5 {% r" _& O9 V0 {" }8 j& Y1 e
<property>
( A) Q( W' s- F0 B( X <name>mapreduce.jvm.system-properties-to-log</name>7 R1 q8 j3 Y* }
<value>os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name</value>
; S% m# j' @9 o+ T <description>Comma-delimited list of system properties to log on mapreduce JVM start</description>
0 X% f0 s7 t* K% ^</property>
& }9 q2 b; ?7 ]0 l: N% h6 K<!-- jobhistory properties -->) N, N% L" `; A) d
<property>6 o; c* a& }3 T8 h+ P
<name>mapreduce.jobhistory.address</name>! R, \9 g- g. {0 X+ a( `$ Y3 W
<value>0.0.0.0:10020</value>* A/ q- S* W( m# E. \; L
<description>MapReduce JobHistory Server IPC host:port</description>: Q" ^/ {2 O6 Y& g: v" `
</property>
9 @& h$ ~& y4 m( A S" g<property>
1 a* c' _/ x% u) c% O <name>mapreduce.jobhistory.webapp.address</name>3 Y) q3 y3 @& @
<value>0.0.0.0:19888</value>8 o% H7 V0 ]& |+ q* c
<description>MapReduce JobHistory Server Web UI host:port</description>0 ~ D# ]. g* t& `4 s( ~
</property>
' T" z2 Q9 e8 c6 I. f+ v- D2 K( l<property>
, D1 L0 t2 a9 X) M7 y# ^1 j <name>mapreduce.jobhistory.webapp.https.address</name>( E8 h' y0 e6 r8 q" ^7 ]( k0 t
<value>0.0.0.0:19890</value>: V* u7 x8 j: h1 C! r {
<description>7 j' S, c% v K; F7 V5 f- x
The https address the MapReduce JobHistory Server WebApp is on.
) ~7 _2 i, y- q </description>1 D% x0 C$ u6 c+ _+ e9 F$ K9 a
</property>
I: p( p" _$ N3 q+ N* J* o, f+ k<property># H6 ^4 L" k- q* T1 V
<name>mapreduce.jobhistory.keytab</name>
3 C& x/ H3 h" W8 F5 `1 a! m& x <description>! }/ U# t. \& R
Location of the kerberos keytab file for the MapReduce
+ U# q1 V, m: }5 W JobHistory Server.
% h' A5 Y3 j0 p </description>
# D8 }4 {. a. j0 e2 k0 l- n <value>/etc/security/keytab/jhs.service.keytab</value>% t7 G, A9 `" u# X+ G
</property>
( @, v% V9 `1 H<property>4 L6 x4 T# Q# I+ d$ A" ^
<name>mapreduce.jobhistory.principal</name>" _ W) a, E; U2 x( d K
<description>, Z. m) U- e# y; Z
Kerberos principal name for the MapReduce JobHistory Server.
8 v+ S8 I. \4 N </description>- T% S, ]& i" E2 \
<value>jhs/_HOST@REALM.TLD</value>" J" e4 \- d! d! F: T8 D. o! @0 |
</property>* A& M- x8 b* M4 p" E
<property># u: S+ B& R- G) h( {0 `& I
<name>mapreduce.jobhistory.intermediate-done-dir</name>
5 L& }6 d0 t* g, t+ P <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>6 l9 i F7 U2 Z' j1 o6 e+ Z
<description></description>
5 ?3 x3 x; ]5 I1 x</property>
! c5 h) h1 g1 i `% x<property>$ T/ F3 o7 \1 g# j: u7 _. T+ u/ X
<name>mapreduce.jobhistory.intermediate-user-done-dir.permissions</name>9 S' x E$ B8 T' s: E+ H$ {
<value>770</value>
5 S) B; {( a7 ^: Q1 s1 b- a <description>The permissions of the user directories in$ r$ f( Z4 b& @; v- Q Y
${mapreduce.jobhistory.intermediate-done-dir}. The user and the group' l8 y `% n- `. g
permission must be 7, this is enforced.
2 n* @ B( `3 ~; V9 H </description>
9 M$ T# n' ~5 H</property>" \/ M) o K3 N! V$ @! ?% \; ]/ h
<property>
' S% ^5 H+ R m <name>mapreduce.jobhistory.always-scan-user-dir</name>. X! D3 D2 V/ }5 ~; b7 m9 v
<value>false</value>7 Q6 d4 m! `+ t& D1 M* g, w
<description>Some Cloud FileSystems do not currently update the; E" y' p3 J1 D' P4 `
modification time of directories. To support these filesystems, this
8 R1 ?& d3 v* ~ configuration value should be set to 'true'.
% ?7 D- s0 x" @6 o% L </description>
: `: L( }; L. p' |1 E+ v" Q</property>
; Q( a0 Z( d" y4 m<property>7 l2 w8 ^9 a5 ^* Z+ D
<name>mapreduce.jobhistory.done-dir</name>
. O; Y7 {/ h+ g: r ~ <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
3 z$ \# O x: ] <description></description>: h+ {) U( u9 e0 h9 s# `
</property>! L! D% y/ I: `+ @; X
<property>
3 Q! ]0 R: a' R) ^! `% r <name>mapreduce.jobhistory.cleaner.enable</name>
6 u0 I6 I. }% K5 V <value>true</value>
' `. l$ t( T: I; u" G' T0 L5 L7 _ <description></description>6 `( Y7 C+ K4 G" x
</property>
1 ?" r. C9 ~* u# F; p<property>, G5 i9 G7 W5 o* ^- F0 t
<name>mapreduce.jobhistory.cleaner.interval-ms</name>
9 P# Q3 S: O/ B% Z" K5 T' n) g <value>86400000</value>
; e5 W- a' \) ^ <description> How often the job history cleaner checks for files to delete,
0 G* ]2 f7 y4 X+ [2 \3 u; R: f in milliseconds. Defaults to 86400000 (one day). Files are only deleted if
9 i. {! v* g- K+ c* ~# U9 |3 L& U they are older than mapreduce.jobhistory.max-age-ms., Z( z6 N0 m7 w0 @
</description>
/ `8 I6 `' y9 E8 l</property>
7 k7 u: s' J: I5 g: m<property>0 T$ D4 d2 m6 P/ A* V) M
<name>mapreduce.jobhistory.max-age-ms</name>* M4 i. _. X$ Y5 w0 j; B
<value>604800000</value>' y7 M1 b* `# c; c, d
<description> Job history files older than this many milliseconds will
5 M! _4 \9 x" J be deleted when the history cleaner runs. Defaults to 604800000 (1 week).7 O, N7 P: v% Q
</description>8 z3 f" J' [! G
</property>; q+ u8 b8 N4 B6 N2 R7 X# Q2 F; {
<property>2 L7 C Q! L7 K0 r1 i
<name>mapreduce.jobhistory.client.thread-count</name>8 E1 _/ w f9 J# y: O9 a S
<value>10</value>
4 ^# i ^, q) D% a" y5 b <description>The number of threads to handle client API requests</description>9 Z7 K% g% z( ~+ g
</property># W& R/ t( o+ G# a9 t7 W: M' ?
<property>
8 U: S; n# n3 d7 F <name>mapreduce.jobhistory.datestring.cache.size</name>
! f) s. i" _$ d6 {# m' u$ d0 d <value>200000</value>4 @0 t" x: o, J$ r3 }' }
<description>Size of the date string cache. Effects the number of directories
. M& v. ^1 X& B! M" M! G5 A- q which will be scanned to find a job.</description>& P$ t$ l C, E$ A9 P1 q( ~
</property>' t2 @5 h9 ?3 R9 \
<property>6 }" z. `2 Z x* a4 i' d+ ~
<name>mapreduce.jobhistory.joblist.cache.size</name>4 a1 u# [+ ]6 B6 Z2 b! V" `
<value>20000</value>6 e! \2 A- S) b9 j9 U$ a0 y
<description>Size of the job list cache</description>2 A8 K8 F" S, ^& _! H& I* ]
</property>
9 @. m/ w6 ~4 V" K<property># f5 T$ x) W3 f& Y, N3 j/ f
<name>mapreduce.jobhistory.loadedjobs.cache.size</name>
$ V5 B3 j1 C9 F* ~0 q9 i, A, ] <value>5</value>
$ \1 t: W/ M; u0 c2 D9 d: U: y8 U <description>Size of the loaded job cache. This property is ignored if% c. P; |2 o f8 t( x. Z$ L( h
the property mapreduce.jobhistory.loadedtasks.cache.size is set to a# n4 i }, v# T/ U: j7 y
positive value.. D" F8 ~; ^8 C# X' Z5 |, [
</description># }1 a& I1 w" q1 _7 ~& [4 _
</property>
* |" r+ A- T4 P' q2 G<property>; l# D# t* r! n* B3 f! @
<name>mapreduce.jobhistory.loadedtasks.cache.size</name>. p; U7 d; G1 T
<value></value>5 x" O1 }' Y1 D1 f# g5 Y: H- W% o
<description>Change the job history cache limit to be set in terms
! B4 @$ a8 r/ B: Q of total task count. If the total number of tasks loaded exceeds
! e1 x7 v, ^) M( D this value, then the job cache will be shrunk down until it is
% t- F9 \ _! c) M. Q under this limit (minimum 1 job in cache). If this value is empty
% n6 _# s9 C& B; e or nonpositive then the cache reverts to using the property
1 z6 N$ Z/ c+ B e, K+ N! _6 c) Y mapreduce.jobhistory.loadedjobs.cache.size as a job cache size.
7 I# r) z- u, _1 E; `$ V7 H Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size4 P. y8 x+ H; c2 b. Z! ~
property:# g8 h$ M ~. N& U2 h
1) For every 100k of cache size, set the heap size of the Job History8 L& d- u6 G: J9 V1 o- f" H% s! R4 n; x
Server to 1.2GB. For example,
/ G) c* E+ t6 T' p6 R: q1 G mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.' y' q/ n, w1 ?6 t$ u$ z" u
2) Make sure that the cache size is larger than the number of tasks, v' H0 c1 y. H( d
required for the largest job run on the cluster. It might be a good: ~. f! h. ?( r7 |) d0 V, h! T
idea to set the value slightly higher (say, 20%) in order to allow
1 |& E% V& A8 S0 m$ d" h for job size growth.( Q s2 t6 j7 M6 R
</description>
3 K' ^% a" C, m0 G& r2 x0 z( h</property>
* r! m* s% P/ H0 g0 e<property>+ T/ t7 v9 E: [! ~1 f
<name>mapreduce.jobhistory.move.interval-ms</name>
6 I- o6 L- _9 e" X9 E <value>180000</value>2 b( w: W! ` i
<description>Scan for history files to more from intermediate done dir to done2 k: B- A0 Q; e( B; F. D% ], C
dir at this frequency." z" b w" o4 ~( c$ W, C6 r' k
</description>. l) k ~' E2 S B# i
</property>
' h, ^0 ~: ?/ B) |<property>8 R' P: D* E1 i2 w5 e, z5 N6 D/ {( g
<name>mapreduce.jobhistory.move.thread-count</name>
9 l& `/ E5 p D# R. l <value>3</value>& w; g4 H0 [+ l) b) G( j
<description>The number of threads used to move files.</description>4 q. v& w/ ~: `2 v" {6 y1 K
</property>" T- W4 _- x/ |- V9 L" M" L
<property>
: H2 h G- |7 @6 a' a1 U/ d, P <name>mapreduce.jobhistory.store.class</name>
- e/ a' \5 I) m* d3 V& U8 J& e <value></value>
! @- a% G- M, z! F <description>The HistoryStorage class to use to cache history data.</description>$ b1 Z, N4 h- `8 S
</property>
0 Q7 W8 z6 j) g: G& @. l# s<property>; s3 B! G& v& ?8 ^* }1 ]3 M
<name>mapreduce.jobhistory.minicluster.fixed.ports</name>7 F/ X1 H6 w* |* o* l: h
<value>false</value>
# a2 G( J* W, K5 a! Q <description>Whether to use fixed ports with the minicluster</description>, o9 C" g2 e1 C8 V7 i
</property>
+ z% N; I! i$ a( C4 D+ p<property>
# c- H4 C! }7 [8 b* J <name>mapreduce.jobhistory.admin.address</name>
& [5 \- s8 B& |+ F0 I3 k9 _ <value>0.0.0.0:10033</value>
! s( c& D. w& ^/ v: b9 Z <description>The address of the History server admin interface.</description>
# C# c/ {9 s) u7 ~+ @# I</property>
, L% C/ T) Y6 i$ \' w1 H. M<property>
- x- S' ~+ v& | <name>mapreduce.jobhistory.admin.acl</name>
8 X/ r, Z0 P/ V& Z <value>*</value>( m# D# }5 Q4 X, ]+ [ Q* X, P/ I
<description>ACL of who can be admin of the History server.</description>% H3 U& s- B; |5 u# f- F
</property>' m* x4 h3 N; y- z1 a
<property>
% ^ G- ^2 |1 Q+ f <name>mapreduce.jobhistory.recovery.enable</name>
* ]% c- u2 d) [) s <value>false</value>$ B: Q3 P6 {5 ?* [3 Z% s
<description>Enable the history server to store server state and recover" i* t! q" t1 \. A1 q& z' _
server state upon startup. If enabled then
8 f$ n9 l5 a* S mapreduce.jobhistory.recovery.store.class must be specified.</description>
! G' K( L% r6 |0 k7 c5 j% g</property>
7 y- A4 e; p* ~. ?<property>6 }0 L6 y/ o+ d- g$ h
<name>mapreduce.jobhistory.recovery.store.class</name>
5 C k1 |9 [( r0 E <value>org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService</value>
! O0 @' o% `5 D! u* m$ y% u( o# T <description>The HistoryServerStateStoreService class to store history server
1 Y" N, y! q% Z7 }$ ~ state for recovery.</description>
: \ |6 [* W3 s+ o( J' c/ ?$ k</property>
: U/ L! l9 p- p! @7 U. r<property>$ B5 K3 X" b, v* @9 ?$ ?* u
<name>mapreduce.jobhistory.recovery.store.fs.uri</name>" ?# g2 C) l2 K- i: D( @
<value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>3 J4 A* h/ D2 H( F1 c
<!--value>hdfs://localhost:9000/mapred/history/recoverystore</value-->% v7 Y& O6 a! V' M8 R; l q1 }, R4 P
<description>The URI where history server state will be stored if, P+ {( | h8 h" m4 H7 L
HistoryServerFileSystemStateStoreService is configured as the recovery
+ {' V f. r* e$ A storage class.</description>) f/ ~; M0 p; y5 o
</property>
3 D- z- \, K2 o6 d( d<property>: F6 Y) r- V% _( I3 M, r
<name>mapreduce.jobhistory.recovery.store.leveldb.path</name>% M: K+ Q, D7 [- D" C
<value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>
+ F- g6 ?$ r8 v& y <description>The URI where history server state will be stored if! M# z4 f8 K4 a/ W
HistoryServerLeveldbSystemStateStoreService is configured as the recovery
8 I* M. O3 W( L) p( H storage class.</description>$ V& m3 h' a9 x: Q5 Q
</property>
5 y. D0 I" H5 s* u. u<property>7 F- i# {4 m- j- r
<name>mapreduce.jobhistory.http.policy</name>
2 G% j$ q7 p J# x- `/ o( e5 R <value>HTTP_ONLY</value>
, o S4 s* i D1 @: q# X" ~ <description>
8 `2 d2 L" Y; [. m; j# r( K3 N This configures the HTTP endpoint for JobHistoryServer web UI.' D0 J% ~$ C/ L# l6 m% w% w! k
The following values are supported:
{3 v# w( h$ X4 C2 P5 ^ - HTTP_ONLY : Service is provided only on http
8 }; ]2 M7 f# b: N1 W" z- v, M - HTTPS_ONLY : Service is provided only on https
) G) M( N% l2 ?. V& l% h( b5 v </description>
" ?1 N3 s: c$ J</property> `' f* P3 y4 q2 r0 b+ m
<property>
3 X/ @5 T0 g3 ?0 S1 o0 H <name>mapreduce.jobhistory.jobname.limit</name>
3 X) |0 e) i0 y) n2 q7 L' p <value>50</value>
( v' [( U, k* Y8 \ <description>
' |1 P5 e- b: w$ x Number of characters allowed for job name in Job History Server web page./ u, Z k `9 n4 t/ I" a2 [
</description>9 U4 W/ C+ p8 E. S( l: P9 b2 P
</property>
$ J' a& I2 b. _" [1 {4 v<property># V/ {/ |+ |9 j( ]+ E. Y: B5 B1 T
<description>- h' |. c( l" u* _* O i
File format the AM will use when generating the .jhist file. Valid
. x$ G5 G+ A( e [! _5 K values are "json" for text output and "binary" for faster parsing.
$ J& d! X0 ~" u' Y </description>
$ y( n! R. V# m/ [. ` <name>mapreduce.jobhistory.jhist.format</name>) e; o5 D- Q- C# v' A
<value>binary</value>
% d' z: Z @) v( p4 p9 L0 h+ y</property># d; n+ {7 m7 n! }5 l
<property>
9 Q1 w# a/ M' ?) C6 o <name>mapreduce.job.heap.memory-mb.ratio</name>
, ~' Y4 `- |7 |1 |! J <value>0.8</value>
8 o' Q" I( d1 I4 Y2 H! [ <description>The ratio of heap-size to container-size. If no -Xmx is
3 M y1 r2 ~3 C' S- w) W1 t specified, it is calculated as' k8 ?; O! f5 a a* v
(mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio).5 h7 a# c3 y! E- D( m C! t
If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is( P4 o5 q6 z1 l
calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
0 W c4 q5 a" P5 w" E2 Z& ?9 Q </description>
7 x. F D1 x& p" y L</property>
- D2 c/ j/ @8 ^" `# q<property># J6 m2 b/ J& S Q3 A+ H! B) X
<name>yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size</name>
6 ^( n6 E% k3 W <value>10</value>
9 B z( h2 ]7 ?9 G( N& G! C% { <description>The initial size of thread pool to launch containers in the
" Q$ J, Q( D$ V j" Z app master.# H+ n2 U& {# n H/ ?. f
</description> X m( M' X% _0 J* M. Q
</property>3 Z3 G* F, R; G2 _! d" v$ [
<property> T5 f9 A. E. H4 K4 g. o: O8 @
<name>mapreduce.task.exit.timeout</name>+ z$ z4 H+ G. M$ U) {: f# ?
<value>60000</value>
4 ^" Z( O1 f. N1 v& W6 V <description>The number of milliseconds before a task will be
' |- w3 \6 a* h terminated if it stays in finishing state for too long.
( m' z6 N# Y+ g- z1 }1 v After a task attempt completes from TaskUmbilicalProtocol's point of view,
, w# a" Y. h- V) W7 v5 j8 c it will be transitioned to finishing state. That will give a chance for the
% H Z: F0 b5 e& s+ a$ [ task to exit by itself.
7 X! r# t: s' n" b. g </description>
4 f1 u# C) W4 `- D4 i4 f z! ~' A</property>
( J' t! n# Y6 ?9 X8 P<property>' n/ p6 A" V! P, p8 h6 j5 d& e6 m
<name>mapreduce.task.exit.timeout.check-interval-ms</name>
$ e3 j$ \/ W' G( \1 [ <value>20000</value>6 r3 h* V+ [$ U$ \9 |) \) U
<description>The interval in milliseconds between which the MR framework
2 w v3 G, q, A5 w9 a- B* B+ o9 T j checks if task attempts stay in finishing state for too long.
8 A) v( P! Z, Q% y9 \4 N$ e% R$ P </description>
* b( s; ]7 E$ Q</property>/ f6 t& v, q4 s6 J
<property>5 ?4 L3 ^8 V0 J! E
<name>mapreduce.job.encrypted-intermediate-data</name>
# w' Q b$ O7 A+ M <value>false</value>2 B6 L- e$ h) ^4 m, @" Q
<description>Encrypt intermediate MapReduce spill files or not
; C4 b) [2 u' ^) Y default is false</description>
" ~2 e& f9 k6 K</property>& x3 }4 L0 r/ h" b- v+ o
<property>
3 P2 ` M+ Q% c <name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
k1 h! W/ _- K <value>128</value>: @" A E! D" |+ f1 |
<description>Mapreduce encrypt data key size default is 128</description>
7 b+ z N8 H" T</property>
2 L2 O* i5 A& a" n<property># }) {* S; x6 I" W7 _; I* Y
<name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
* K# q2 ^ c9 m <value>128</value>
/ Z) X7 j9 p8 z5 e <description>Buffer size for intermediate encrypt data in kb4 i' [$ Z- k8 w6 g' b
default is 128</description>
4 t) N6 S2 Q. G4 {8 P</property>
. A3 V* s, W8 d# I& c8 x) y<property>" F( u. U2 T! ?- i: H
<name>mapreduce.task.local-fs.write-limit.bytes</name>0 x! M1 k) R. \9 j3 y
<value>-1</value>
: u) Q. |% Y% H' j" z1 U; x- g3 q <description>Limit on the byte written to the local file system by each task.1 {; g" h# F7 Y" |7 U2 l
This limit only applies to writes that go through the Hadoop filesystem APIs3 `; W) c) e2 }9 A
within the task process (i.e.: writes that will update the local filesystem's
q" g I* l3 N Q: j% x BYTES_WRITTEN counter). It does not cover other writes such as logging,
7 T5 i; M3 @% }1 I sideband writes from subprocesses (e.g.: streaming jobs), etc.
0 \: k. {! b" ^% [: v$ ?1 \9 d Negative values disable the limit.
$ V$ J" O/ h0 e" t6 ~9 Q default is -1</description>
$ b( {2 A9 ^" Q( s- V- E</property>
& x# q. f+ y% Z7 r+ ~* f2 q7 j$ A- G) U. N<property>' ], B! `, d' A1 y0 N* o
<description>9 P" |! d* \( k* C1 C
Enable the CSRF filter for the job history web app1 X0 }! R; `9 E
</description>
2 p8 R! p" [* b* i7 J7 \" W% O <name>mapreduce.jobhistory.webapp.rest-csrf.enabled</name>
; X6 S4 `! g2 D; l6 }4 C <value>false</value>
- q# }# X7 ]8 V* I0 u" a9 k</property>1 N! D$ H9 ^1 e: \& }9 a- V
<property>
w9 F; h" `- _- w3 T <description>
8 ]$ `5 p5 V1 a6 s2 C- [+ l+ T& P+ Q Optional parameter that indicates the custom header name to use for CSRF
" n- U) Z; @9 g7 C) B' X: N. \ protection.7 g1 n( q/ y0 O/ K L
</description>
4 f) M5 N5 H0 {+ C) N2 k) Y$ h <name>mapreduce.jobhistory.webapp.rest-csrf.custom-header</name>2 B+ X+ ^& e% c U8 C2 K8 R# a
<value>X-XSRF-Header</value>9 ^/ b/ q9 H& L8 [ C
</property>1 U) j# R( y) |- X7 |) m* Q: f
<property>' f1 I5 y D' L$ f5 {, y* F
<description>8 M' c% p$ X- d% X
Optional parameter that indicates the list of HTTP methods that do not1 n; P; x) U* W
require CSRF protection
+ m1 k* _: Y+ f1 U) Y2 [' u8 B </description>
6 Z& x& j6 b( U: A# H <name>mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore</name>
- M1 l* V4 {6 y ?3 `! A( x <value>GET,OPTIONS,HEAD</value>( Q) C0 E1 @" R* P/ x
</property>
- R9 U( [: z* c7 ^; S4 o) d<property>
. D2 o8 p# |4 M- P) j4 r <name>mapreduce.job.cache.limit.max-resources</name>8 @ i8 I: w4 x4 p, W
<value>0</value>
" F7 V% \2 z8 ]) _4 I <description>The maximum number of resources a map reduce job is allowed to
$ s* X% W. h3 ~" W7 m/ W submit for localization via files, libjars, archives, and jobjar command
1 q6 R2 g; T3 l& E line arguments and through the distributed cache. If set to 0 the limit is
4 W* A3 ]; v- [, j/ y: _7 L ignored.% U! B$ X9 K* M4 ?
</description>
" ?& J0 a7 Z* u</property>! _ X' w' c6 J. J! n
<property>
$ [/ ?9 U( J# H! h+ C% A <name>mapreduce.job.cache.limit.max-resources-mb</name>; [- O& T2 J4 T# s( c" c: W& W
<value>0</value>) G' d" d# Z* o5 G0 r2 C9 X
<description>The maximum size (in MB) a map reduce job is allowed to submit
# h! W0 l# @- s# t; k9 ]0 o% @7 e/ b for localization via files, libjars, archives, and jobjar command line
* C- `+ J8 G$ x7 l: L arguments and through the distributed cache. If set to 0 the limit is
0 E2 C" c- a$ O& |# P( Z+ W4 A ignored.
+ A, O- b& s O" j6 {( D </description>6 U2 S4 f- p1 G
</property>* r4 |2 U. i1 {: T! a* x2 M$ Q: \
<property>) a! h2 m, c' S/ [/ H/ I- F3 F
<name>mapreduce.job.cache.limit.max-single-resource-mb</name>6 T- b. e' {1 _% p
<value>0</value>
! G6 M) i: U; \: `, P h9 ? |& ` <description>The maximum size (in MB) of a single resource a map reduce job
6 k* p# K, a' ]$ F1 H5 T! v is allow to submit for localization via files, libjars, archives, and
" o* t3 y1 G. s u& f jobjar command line arguments and through the distributed cache. If set to
1 _5 [6 }4 l5 T# f 0 the limit is ignored.
! k( B! N% k0 U9 j1 B) |+ G- q. { </description>
: Z* Q! I! @4 V$ J: D6 z</property>
$ F7 ~; R1 F) }$ q' f! C4 k% k<property>
/ X' E- V. q/ F3 E <description>/ M3 E) n2 {% F( L2 a) Z6 W2 N
Value of the xframe-options
}8 _& p9 R6 g& ]. Y5 r- E </description>/ R8 h3 A, t/ `7 Z$ f6 ]) f/ a
<name>mapreduce.jobhistory.webapp.xfs-filter.xframe-options</name>
3 E3 }, I* E4 s% k! {8 p" Y7 f! x. U <value>SAMEORIGIN</value>
) x/ V' w. a& Z' v# i6 y</property>. _- m& `& K" y6 S L2 U1 j$ D+ G
<property>6 K+ g+ J1 \: h4 h* p5 ]8 x
<description>. [. g- s2 F8 G, ~6 I1 A' Z2 \* N
The maximum number of tasks that a job can have so that the Job History0 [1 ~+ i7 K# i/ X, a0 r
Server will fully parse its associated job history file and load it into2 J$ p* [" W2 @+ w4 t7 z+ q2 c
memory. A value of -1 (default) will allow all jobs to be loaded.
7 ?0 ~5 X, T9 \, q </description>7 Q. Q' w& N+ D1 B
<name>mapreduce.jobhistory.loadedjob.tasks.max</name>
* Z, b1 d# R* F- i: X# w <value>-1</value>( V# n9 f9 F1 [- x
</property>
2 [) X: |0 K6 d/ k4 D8 U<property>
9 j9 X( e" v( x, G <description>: G- {3 B/ {( h7 L2 g1 R0 h
The list of job configuration properties whose value will be redacted.! S- n g' P. g. f. y7 {7 P6 M/ b1 f
</description>1 Z/ a% p4 m! i1 M6 G5 a" N. l
<name>mapreduce.job.redacted-properties</name>
- d! K1 C: b/ F' v. E: G <value></value>
: M* @" x$ b# o M/ G1 n5 C8 F( T</property>$ C5 p# \3 p+ L% _; y8 S- F
<property>- p: r2 g6 N' o' d
<description>- Y( \8 u5 w, E8 U
This configuration is a regex expression. The list of configurations that: \+ r$ b6 D$ Q h# Q7 h3 K
match the regex expression will be sent to RM. RM will use these" v" C: b) [: C/ M( e
configurations for renewing tokens.
9 @' c [! n& O: g8 i$ g5 }; g This configuration is added for below scenario: User needs to run distcp5 W( Q% s( b" ^6 V0 w8 s2 N
jobs across two clusters, but the RM does not have necessary hdfs: }4 N$ z2 n+ G5 p/ |
configurations to connect to the remote hdfs cluster. Hence, user relies on" r7 N& l" @3 e/ _4 u+ f; z
this config to send the configurations to RM and RM uses these0 z/ ?5 {- h' J* a# q
configurations to renew tokens.+ u, c8 P9 w6 m( r% ~5 l$ X2 Z$ x/ x
For example the following regex expression indicates the minimum required
! A5 t) e0 ?; {) N& I configs for RM to connect to a remote hdfs cluster:% L Q' v9 a$ ]2 s
dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$|dfs.namenode.kerberos.principal+ B3 Z3 R! Z5 b- W, Z- m0 j+ E
</description>
+ \7 f; X8 L. ^ <name>mapreduce.job.send-token-conf</name>
8 `" @0 g1 T; E% S& I' C; N! z* z( C' b <value></value>
5 Y% l2 W! k+ _, w3 D# k# o</property>
6 ~. b! H2 H) z! C' @<property>
% a, H# r* z: n& ^ <description>) _3 w' k/ `, @# U
The name of an output committer factory for MRv2 FileOutputFormat to use
) m3 S# ~ _( V3 N/ U for committing work. If set, overrides any per-filesystem committer
$ L- b0 W3 n3 X defined for the destination filesystem.
' h# J, p4 _( w& E) @ </description>0 D( E# A3 e8 U* G+ w- b
<name>mapreduce.outputcommitter.factory.class</name>
' R/ p: w3 b+ z4 b# |2 w& R <value></value>- x$ _# `" q1 i2 [7 G9 s
</property>
, H$ h. Z1 Y; w2 i0 K& F<property>
& x( B7 x7 q# E8 ^" @ f7 Z <name>mapreduce.outputcommitter.factory.scheme.s3a</name>
( X1 |1 r0 k. r* i <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value>
2 r0 E: M4 _* u <description>: M9 O- }7 u; ^* W. [ j
The committer factory to use when writing data to S3A filesystems.+ x$ p" w! u- J, d9 X2 w
If mapreduce.outputcommitter.factory.class is set, it will D C5 |/ e6 G- F
override this property.: Q7 M' F8 b! B
</description>- |0 @8 x- w( [8 o
</property>
6 ^ U3 { r& Y: s( k, n5 N7 ~</configuration>: w; ?) h! Z' `6 R. G7 \4 l( u) Q8 H
4.yarn-default.xml
) _; E. a4 Z7 _
0 H# ^/ Y2 f5 ~, G+ x<?xml version="1.0"?>
6 f( X& [; p# x. W5 H+ I! p+ D7 B<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- v' d- R' l6 b. D. B$ w M' W% y<!--# W. V# o* M5 D, K
Licensed to the Apache Software Foundation (ASF) under one or more6 G: A3 q; H3 p/ j: c
contributor license agreements. See the NOTICE file distributed with0 V. ~% q9 @. f2 b
this work for additional information regarding copyright ownership.' f- n; n0 u, i& B/ K4 S
The ASF licenses this file to You under the Apache License, Version 2.0. c0 v; w! A; {( s. t# K
(the "License"); you may not use this file except in compliance with
1 M' I: l# m( h N+ x the License. You may obtain a copy of the License at7 J: _! `, Q& i" A; I( W0 _. I
http://www.apache.org/licenses/LICENSE-2.0* S* C# T7 [7 u1 K7 w$ y
Unless required by applicable law or agreed to in writing, software
/ o0 x S& w) u' ?" u, s distributed under the License is distributed on an "AS IS" BASIS,. V+ Z$ I# A/ p* P" N& X' v2 ?% a
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.) {% E7 r, X: x8 q O3 e, p
See the License for the specific language governing permissions and
; k) l+ |* O' Y/ x! x( Y" y g limitations under the License.
4 y4 N) u0 ], t; i-->
' q m! A# _& }4 d# J+ S1 A<!-- Do not modify this file directly. Instead, copy entries that you --># V3 X0 _6 ^! |1 g
<!-- wish to modify from this file into yarn-site.xml and change them -->
; B9 _; ~. `9 M& S# L% }<!-- there. If yarn-site.xml does not already exist, create it. -->6 k8 \( u4 Q- w: z Y
<configuration>
% Y o( ~9 P0 ]% v <!-- IPC Configuration -->2 A: Y) ~* S7 l) t' k" n
<property>) X8 H+ r1 g* N$ R) A
<description>Factory to create client IPC classes.</description>
u" @$ _! b( ]( x6 J6 Z; M <name>yarn.ipc.client.factory.class</name>1 L2 X( x# B( `1 s
</property>
! q! B- t3 l9 i, B% W1 ? <property>* A0 u4 U; @. j/ N& a# U$ c$ ^3 S
<description>Factory to create server IPC classes.</description>
" ?! t7 r% C" @) i, [1 B' Z- l <name>yarn.ipc.server.factory.class</name>& u% @+ h1 S. g! h* h4 q. S
</property>+ C; ~2 ~; p X8 P2 D) U. |
<property>
9 K) y$ i/ v0 x) a; s <description>Factory to create serializeable records.</description>' Z* [& `) v' F4 F- k+ R4 F: R F# V
<name>yarn.ipc.record.factory.class</name># @# K6 l! G4 u1 Q. k q
</property>
+ n% ]& O/ v' G$ i& f7 C+ D <property>
2 V6 y8 O: W, j <description>RPC class implementation</description>
3 E; j$ O5 a y$ o2 t <name>yarn.ipc.rpc.class</name>
4 o$ j) W& q% c3 H; g <value>org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC</value>) I! F) I2 j& U- f) t6 k% N7 c! X0 q
</property>; o+ V. Q3 X7 i2 W- p s
<!-- Resource Manager Configuration -->9 u7 T' I8 }8 l) O/ h Q% U6 q
<property>8 C* b% `4 |6 Q% R
<description>The hostname of the RM.</description>
) J$ A) @- H- f% r9 ] <name>yarn.resourcemanager.hostname</name>
1 j2 S- R& k# T8 k <value>0.0.0.0</value>
0 @' ]2 Z. U% i </property>
, J$ W9 E. f; \; A$ s3 H <property>3 t8 o5 ^0 S! |, w: B- \# U/ ?. y3 Y
<description>The address of the applications manager interface in the RM.</description>
$ Y) b; c$ _5 h: q s <name>yarn.resourcemanager.address</name>
# U; T. r1 R" F2 C4 b; Q <value>${yarn.resourcemanager.hostname}:8032</value>5 S c$ ]6 C# V9 O% b1 j" }1 Y2 } v
</property>. l1 l1 ?+ @. a: x% k: S
<property>; I4 J- P# e- \: m3 t
<description> ?: J* R" c/ q' ?" q% [, D: k: Y
The actual address the server will bind to. If this optional address is
6 Z2 L/ D: f$ c set, the RPC and webapp servers will bind to this address and the port specified in( e9 I: k* r2 l1 A
yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This
; Y/ \% u3 v" ~+ e& Y is most useful for making RM listen to all interfaces by setting to 0.0.0.0.6 h/ v. V5 L8 f# W/ _/ {- l
</description>5 e6 g) l, \% g5 [ Y2 R# I" m
<name>yarn.resourcemanager.bind-host</name># l& d5 B/ I& M. G$ _
<value></value>6 H+ e8 [# E7 e/ Y: g+ f
</property>1 c! O4 k2 u/ H- p# E7 W9 x+ A
<property>) G! M H- N4 Q% f: ^ @, ?- E
<description>
( S C9 w' j w* C" k! e If set to true, then ALL container updates will be automatically sent to1 a3 |4 e9 U& y* Y7 o+ _
the NM in the next heartbeat</description>
6 i6 y! W8 n, L1 E+ Y <name>yarn.resourcemanager.auto-update.containers</name>
$ |$ r# h2 s6 Q" T4 n1 h! Z <value>false</value>' Z3 o% b! U, I* V! i
</property>3 o1 c, s$ @; M( H6 b) F
<property>
- E, ?& n e0 V& v( S8 E2 ?- A <description>The number of threads used to handle applications manager requests.</description>5 g' g! f" P/ `0 h4 w. r4 |6 l
<name>yarn.resourcemanager.client.thread-count</name>
1 ? E$ T% Z! ]5 K% X5 D9 o <value>50</value>
9 g6 J7 M& _2 U' R; m! t9 `& _+ ] </property>
! V- {' U0 w) T7 X! N <property>
+ B9 n, X; J* J <description>Number of threads used to launch/cleanup AM.</description>
! n+ a& C7 x, ~0 o <name>yarn.resourcemanager.amlauncher.thread-count</name>0 K& P4 e% u' C- ]) q7 z8 z1 [
<value>50</value>
! R) O3 [# w) o" I. r( K4 x( y </property>
+ T( j( Y. a3 [) U$ ] <property>
. P0 S# D2 Q: j+ s: r" K- v! Z <description>Retry times to connect with NM.</description>5 A1 a1 M; z3 i* v" _5 B+ }
<name>yarn.resourcemanager.nodemanager-connect-retries</name>
* N D: I% q) D& }0 w <value>10</value>
- u/ o5 i' r0 a% K4 S& y* r6 A& I </property>
9 o( x+ @/ h5 ^$ d G- W! }; T <property>' y- o9 i3 X( G& u' N3 A9 J
<description>Timeout in milliseconds when YARN dispatcher tries to drain the* e3 @8 ~4 I4 R" O
events. Typically, this happens when service is stopping. e.g. RM drains
1 h1 m3 }' s4 B* S the ATS events dispatcher when stopping./ E' H& v( O' c
</description>
% U; a; e' z! ]2 L! ~ <name>yarn.dispatcher.drain-events.timeout</name>
5 B: ]4 q( u1 Y <value>300000</value>) | T. b( S. Y
</property>
" x: a. F) k& |3 i3 f8 z <property>- ?, ~, B0 |$ M% ^2 I9 _
<description>The expiry interval for application master reporting.</description>
9 L2 N3 x8 e! p <name>yarn.am.liveness-monitor.expiry-interval-ms</name>( i5 ~3 {4 C/ F8 a' @5 R
<value>600000</value>( O% n' m/ g7 ~7 y1 H( L+ u' \
</property>
/ E8 P! V3 O/ g. F, C8 m# G6 ]- \ f <property>
' M1 T# @0 g1 X# L, O: H <description>The Kerberos principal for the resource manager.</description>; n# ?8 a( n& v
<name>yarn.resourcemanager.principal</name>
' b. ?7 m1 a# H' d. n </property>9 @3 k8 n9 ?9 c* _9 l
<property>
6 o1 Z4 O" _5 `2 P <description>The address of the scheduler interface.</description> S( S# Y. @' d: e6 q. ]1 \
<name>yarn.resourcemanager.scheduler.address</name>+ S/ J5 x# l, E1 ~+ }
<value>${yarn.resourcemanager.hostname}:8030</value>6 {7 z/ I9 g; |4 o* D5 H! D
</property>2 I% `5 u- ^+ q, x1 U% \
<property>8 Q+ g8 `" m. ]8 G+ g4 A+ v
<description>Number of threads to handle scheduler interface.</description>
$ P# q. o) c- }4 G" v, j <name>yarn.resourcemanager.scheduler.client.thread-count</name>
4 h* \, U5 ]$ ^5 d <value>50</value>
) R8 D9 F# p z/ j$ ~ </property> L1 B0 r* O! d! h+ X& M
<property>
/ a( n2 e2 ~2 Y; ?4 v' ?4 }3 Z <description>3 S5 d# i# K& c
Specify which handler will be used to process PlacementConstraints.% H! @3 O- U' Q4 P- ?
Acceptable values are: `placement-processor`, `scheduler` and `disabled`.# r# ^8 P- F; Q; _( d+ |
For a detailed explanation of these values, please refer to documentation.
" J9 i- k2 [* M7 }; l7 r/ V+ ]4 F </description>6 a+ j' v' G: ]- s8 [
<name>yarn.resourcemanager.placement-constraints.handler</name>
3 l/ x) c) ]. y( a <value>disabled</value>
( z3 y7 `6 H8 }9 h7 g: u- ?" H+ S </property>
8 K$ t& D' [ [8 ^$ q1 O, t2 E6 a <property>
1 \4 z' y6 E* N <description>Number of times to retry placing of rejected SchedulingRequests</description>- v1 Y% }' t, Q( I4 C
<name>yarn.resourcemanager.placement-constraints.retry-attempts</name>( Z) E6 w* y8 w- O& x
<value>3</value>" Z1 P3 l& }; {9 g
</property>% @3 h" o M7 b! s S
<property>
7 U7 w" `! [8 Q <description>Constraint Placement Algorithm to be used.</description>
9 R+ B6 e. H7 i <name>yarn.resourcemanager.placement-constraints.algorithm.class</name>
, Y/ E" b% O* { <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm</value>
+ O4 t& ?' D" Z2 ? </property>
' K, q2 X0 a2 K8 V; s- j W1 X; R <property>
& y% c9 I/ J- G$ _ <description>Placement Algorithm Requests Iterator to be used.</description>4 x: M* H9 N' s4 R% ]0 P2 w+ v
<name>yarn.resourcemanager.placement-constraints.algorithm.iterator</name>
6 _2 R' n4 F; p$ G( _ <value>SERIAL</value>9 k0 [* U" r5 D* q/ C9 Q
</property>
" O x0 G+ D5 q z7 O& b <property>! ]2 p0 i4 T4 Q7 |/ B H: }- R
<description>Threadpool size for the Algorithm used for placement constraint processing.</description>7 j. v H, G% L0 j2 ^4 g
<name>yarn.resourcemanager.placement-constraints.algorithm.pool-size</name>
+ v6 e% K6 F1 N+ V( B) Z7 S <value>1</value>
: X' ?. F; w' j' O8 J2 o9 a2 x# O </property>% F$ R- s: A) i% C0 X* o6 t7 {
<property>" r) X. |& b; u% f
<description>Threadpool size for the Scheduler invocation phase of placement constraint processing.</description>/ ~' s7 ?7 t$ s0 G4 |* l/ i) @
<name>yarn.resourcemanager.placement-constraints.scheduler.pool-size</name>
) P# }! ~8 m% X1 z3 ]8 I0 w <value>1</value>+ t- `2 a* j; U4 n2 Z5 v
</property>- r9 b" Z7 D. t% H) C- x; }. _7 _
<property>, ?1 w( C1 O S
<description>; y& M" u$ w# e4 a
Comma separated class names of ApplicationMasterServiceProcessor3 q7 h i; U! m2 z
implementations. The processors will be applied in the order: }+ r( [3 V8 L& J. R: @
they are specified.
) h5 l0 D. I# I7 |0 M8 e8 a# B </description>1 p" Q' F8 x0 F- B
<name>yarn.resourcemanager.application-master-service.processors</name>
- q ?$ _6 C) C) g <value></value>
" y+ q- ^1 C& }: h </property>
. s( }7 P7 W2 N <property>
: m2 P' I0 v) r, I" `0 C+ b/ t <description># q9 W$ H- {2 U% C- f1 e& m
This configures the HTTP endpoint for YARN Daemons.The following5 m( H# K, } d' n9 T% i
values are supported:8 z6 s5 R6 A8 u1 M& X& i7 E: M5 f4 x$ f$ Q
- HTTP_ONLY : Service is provided only on http
! K9 |8 G# b& n" L& M - HTTPS_ONLY : Service is provided only on https
0 H) M* s3 o5 I R- B9 ? </description>7 ~) r6 s# S2 O2 ^# w
<name>yarn.http.policy</name>
7 I0 k2 Z3 _5 u9 c( Q9 R <value>HTTP_ONLY</value>
( p/ S5 V) i; a. b8 T </property>5 [2 i, y- t V# y0 z% J6 E% c+ b
<property>
F9 `: X7 u- f: { <description>
$ x, t% ? H- ^ The http address of the RM web application.
% f6 w+ ~; s: G+ ^ If only a host is provided as the value,( t9 Z/ [1 R1 E4 O( B/ Y
the webapp will be served on a random port.% m3 r% _& Z: X* M& f7 |
</description>
6 f1 h# j* K- S2 E- Z G% q$ @ <name>yarn.resourcemanager.webapp.address</name>8 b q* _- W- ]( L+ w) v
<value>${yarn.resourcemanager.hostname}:8088</value>6 A. t- w& S% Q! \- U F
</property>
; S6 k& c( V% z1 X7 r' W <property>( w7 X) i9 H* x
<description>
" P8 O3 d. R* Y' a6 j The https address of the RM web application.
4 k) M# M5 c) `2 l- L If only a host is provided as the value,
( z8 N: \7 w( W8 `% g the webapp will be served on a random port.4 T& @/ H |/ g1 P
</description>
! G( T5 {! k& x! d <name>yarn.resourcemanager.webapp.https.address</name># _# t2 B; b# M, y* n: m& F
<value>${yarn.resourcemanager.hostname}:8090</value>1 u' f& e" Q9 N
</property>5 z6 ~3 Z( ]( o' u7 r2 Q5 \
<property>
4 n$ n+ d; e* z) ~, ^' j <description>
/ A3 x8 b" l3 B8 A$ }8 I4 m The Kerberos keytab file to be used for spnego filter for the RM web
2 _) U. H+ R/ E) F. f$ C interface.
8 W; u/ _8 w9 l6 `: s8 ? </description>
. k7 v" W8 X3 d( v( S! a. i# ^ <name>yarn.resourcemanager.webapp.spnego-keytab-file</name>
6 U; G1 z; m% g7 n <value></value>
% Y9 n0 }, w7 S5 c4 H3 Q </property>9 U5 I8 b1 ]! E8 H
<property>) L' J: K T7 g3 r' y8 m
<description>( f- f: {4 e, W
The Kerberos principal to be used for spnego filter for the RM web
% y1 s8 a0 o% I1 n1 S0 B8 Q interface.
6 `' O% f$ A$ ~3 w4 ~, ` </description>5 `% c8 J5 M+ r0 l
<name>yarn.resourcemanager.webapp.spnego-principal</name>+ l& r' n$ s. S. u% O. @: D0 {+ d
<value></value>
& n) m2 ?- U7 X% Y! _2 b9 J: U </property>" @- s. k- w, d' k& d
<property>- U/ H5 r# l8 e1 ~6 B. D- Z( z
<description>
- n9 _: Q8 h! M% ]& ]2 ?: F$ S Add button to kill application in the RM Application view.9 z4 b) c# r5 d( B6 |; g
</description>5 E3 K- J- {4 ?8 L7 t4 Z4 i
<name>yarn.resourcemanager.webapp.ui-actions.enabled</name>
$ F( a4 L6 K, b S# p <value>true</value>
. i( e' W$ J! E. O; H </property>
& Z! b! p6 R5 e0 j <property>, Z/ G3 T: z. v& `
<description>To enable RM web ui2 application.</description>
: n& [6 @" {! w* ~$ @6 z <name>yarn.webapp.ui2.enable</name>) E+ R6 z; Y2 s! f
<value>false</value>5 d; ?8 P9 s! i4 \6 O: W8 G& e
</property>
7 G1 k4 r/ }/ o( d$ [ <property>
( l' W% E2 i! D3 n* ]& E0 \. y <description>
' Q1 I! H# [% ?# t# o" m5 K2 q Explicitly provide WAR file path for ui2 if needed.* ?2 R' Y9 `( @
</description>
! n: t/ u; k5 Q Q <name>yarn.webapp.ui2.war-file-path</name>
; D% R7 N# w! Y1 `( x: N- o <value></value>" H* p4 u( e, u' O( V/ ?
</property>4 H; V, p* i6 ~) P. p3 w
<property>
- Z3 w" l! o: }) N$ g <description>5 v) k1 c/ ~0 }8 ], L
Enable services rest api on ResourceManager.
/ r) U9 k9 s* P; t, v </description>1 c5 v- v+ T" k# L: l0 W
<name>yarn.webapp.api-service.enable</name>
; W8 Z( M Y3 z0 {+ i1 f, G, E <value>false</value> @' ]9 z& U* F/ T# u
</property>
2 O# |$ \6 M. i1 S6 [ <property>
( X+ d* O9 v) x" z- p <name>yarn.resourcemanager.resource-tracker.address</name>
' T5 ]6 H0 r9 K7 J <value>${yarn.resourcemanager.hostname}:8031</value>
; l' _1 j; [' L. ]' j& l5 D7 e </property>
/ u. ^. M3 O. P# s <property>
0 s' \ y4 H: a% V, l <description>Are acls enabled.</description>
+ M/ w5 U. j. q <name>yarn.acl.enable</name>
$ u+ L* w% I- H2 n <value>false</value>4 x! \3 O. N6 {' c, h* O @0 n
</property>
( [6 M* p4 q* }5 r) x <property>
: D t2 [5 x4 ~0 q <description>Are reservation acls enabled.</description>3 L% }: O2 ?9 b/ d& _% K5 @% B
<name>yarn.acl.reservation-enable</name>: _2 A3 B+ z; V1 R; ]2 }$ u9 d
<value>false</value>: e: C5 d$ s* _6 U' T
</property>6 p0 y) }9 k6 v. e" {( q0 @% c
<property>, ]. ~' m, _8 X, I/ B$ _% ?
<description>ACL of who can be admin of the YARN cluster.</description>
8 M, h% _/ _ Q& [! m- K$ k <name>yarn.admin.acl</name>1 e* I2 r6 I. n9 y4 m# e
<value>*</value>
; w" q0 V, w6 s' ?- g </property>
/ `% _% V" _5 z& q- \! E3 r2 C <property>
4 P) {5 T# W) ~6 w3 a. M <description>The address of the RM admin interface.</description>
1 g) L6 k: C& d. ?. u- `% _ <name>yarn.resourcemanager.admin.address</name>
& ^ `4 g e( U! M* Y) _ <value>${yarn.resourcemanager.hostname}:8033</value>
5 s' O3 m0 ?+ N$ y* n9 ]) u </property>7 ~" P1 C) r2 B+ k
<property># ? q# e a0 r+ T% B, D, \
<description>Number of threads used to handle RM admin interface.</description>* T; u) U) I( f r5 V( K
<name>yarn.resourcemanager.admin.client.thread-count</name>. e+ o6 ]$ y8 X% M
<value>1</value>) C2 |$ {/ G" r! n5 E; C( d
</property>9 N. F4 B4 T( ~( y+ w4 A5 x
<property>
# ~: i" n2 a" r: B+ c; t# U' K <description>Maximum time to wait to establish connection to5 T t' Z. v8 }) L
ResourceManager.</description>
3 v4 U% X' l+ A0 c+ s <name>yarn.resourcemanager.connect.max-wait.ms</name>
1 }$ i! a4 B$ E! \% G% @& m% C <value>900000</value>: y& s# g `0 a4 P4 l
</property># {) d/ j8 q, E
<property>
5 X' f+ ~, T6 F7 I% s. a- K <description>How often to try connecting to the
$ A# t# @0 h+ M ResourceManager.</description>
/ T7 L2 h. n0 o" h" _ <name>yarn.resourcemanager.connect.retry-interval.ms</name>3 ?( ?. O/ q' I3 H; A( X/ H: u7 s8 ~% h
<value>30000</value>) L. t0 @; e3 h* e: X
</property>
: t' x# `9 T& i- @ \4 t <property>! r2 M) R: ~* d9 w3 F
<description>The maximum number of application attempts. It's a global
l; Q M& s+ L setting for all application masters. Each application master can specify' a" o" h* B2 i# O6 C, ?3 M9 m
its individual maximum number of application attempts via the API, but the! g, }0 H# P8 |: t$ i6 } m
individual number cannot be more than the global upper bound. If it is,
' n0 ?% M) }+ B0 r the resourcemanager will override it. The default number is set to 2, to3 |! y1 h; o- B8 y% N
allow at least one retry for AM.</description>
5 Q8 l- C( u5 P/ O) J0 a! { <name>yarn.resourcemanager.am.max-attempts</name>
7 N$ |9 _: b9 c) S- I <value>2</value>
5 H4 R. y# m9 a- O( K$ y+ O </property>
/ Z1 p+ U3 W$ D, |! v* H <property>
% U$ U& w6 y0 K: u2 b4 V. F& W; c <description>How often to check that containers are still alive. </description>$ G' V7 ?7 K2 s2 [9 o! ?
<name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
; Q5 G( | M& O <value>600000</value>
# }7 {2 I6 {3 o% P/ A3 W8 E </property>! p1 j9 [8 Y7 C3 g4 Q; v
<property>
8 o' V% a$ G) j3 L <description>The keytab for the resource manager.</description>
2 S! N$ a' |( n <name>yarn.resourcemanager.keytab</name>- q0 I/ U, w& l5 T' \. @ s
<value>/etc/krb5.keytab</value>
4 z" g2 {) T6 B) t) |4 Q </property>$ e; ?9 \8 |& [# D
<property>4 r# }, C9 x9 b0 L
<description>Flag to enable override of the default kerberos authentication
3 U: b' ~+ F Y$ p2 ~; b filter with the RM authentication filter to allow authentication using# P5 j# d* g! d& t* ]; D8 E$ @1 s h
delegation tokens(fallback to kerberos if the tokens are missing). Only- d9 B' o' X/ S2 C. \7 F& k
applicable when the http authentication type is kerberos.</description>
" K, I* r5 q$ H( r/ F <name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
2 Q; ^! Z2 C" j: ` <value>true</value>
) V( }0 y V* W: u) \! X </property>. @6 \! L1 P/ U& s! x3 J
<property>
1 b0 U9 T$ m. |$ E: p# R2 q0 b ? <description>Flag to enable cross-origin (CORS) support in the RM. This flag. m9 O$ o) t" j% J: w3 Y, n
requires the CORS filter initializer to be added to the filter initializers9 W" L! D- q4 |+ @) t/ j
list in core-site.xml.</description>
, \0 y p, c+ j" m0 d <name>yarn.resourcemanager.webapp.cross-origin.enabled</name>" r, z" Y3 E5 H
<value>false</value> }6 w1 Z' {* C( s
</property>
5 v& k& L- _* E9 \6 r2 \ <property>
0 H) o) d7 ]# Y2 X6 J4 ~ {1 ` <description>How long to wait until a node manager is considered dead.</description>
3 i* J9 e- o( K+ V <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
4 Z6 C' [2 o+ E5 f; E( ` <value>600000</value>
0 i k1 Y& z; S$ ~" E( [6 @ </property>
4 k7 R. T1 i$ H1 b2 ? <property>
9 _$ A- v( t+ i! }2 V- a- r+ l <description>Path to file with nodes to include.</description>
- Z7 `; G1 F( o+ q% f3 k3 ?& H% v9 p <name>yarn.resourcemanager.nodes.include-path</name>
( u) v9 W3 ]% }; E' E1 I, q <value></value>- u; _( e% G' a% _* h' }! {
</property>+ F1 y9 v" B( Z/ G4 G
<property># @3 b7 y }+ r7 S$ i5 y
<description>Path to file with nodes to exclude.</description>1 S$ j5 x6 q' O2 V
<name>yarn.resourcemanager.nodes.exclude-path</name>
' G( S* E0 G2 d L) K5 Z5 @! l W4 m <value></value>
; s @" a6 }& d5 m! c$ I: Z4 X; E </property>: \. u9 h- S1 w& I- y) P: K
<property>4 A) u+ g# v0 L
<description>The expiry interval for node IP caching. -1 disables the caching</description>
* w1 L+ R+ X( U! o, R <name>yarn.resourcemanager.node-ip-cache.expiry-interval-secs</name>3 c7 j4 U+ U$ S2 d+ n2 c6 s
<value>-1</value>- n7 B" ?) c7 Y+ i
</property>- ]+ D% |% r' e7 _
<property>
+ c4 w+ V+ ~* v! t3 T <description>Number of threads to handle resource tracker calls.</description>( |' f( G+ V8 C) {0 Q/ E, S
<name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
: t7 E7 G/ E* E; r <value>50</value>
/ j# P Q- ^- `: [0 r1 Z* r </property>, Q7 O* z' |, l1 v7 W* Z: l
<property>. Q' ]. |8 @. `: L% m. x. `6 `
<description>The class to use as the resource scheduler.</description>
+ M+ e# T2 N' @: ^' a <name>yarn.resourcemanager.scheduler.class</name>+ a; m c R6 j& R- h
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
i( U$ l% j8 d) H- k </property>
: J" \9 r5 G+ e# _: m; y <property>3 g" C* S- _+ ] B& T
<description>The minimum allocation for every container request at the RM# G! T2 Y' Y/ N3 B. n. ^8 r
in MBs. Memory requests lower than this will be set to the value of this" J! C9 _! D4 I r' L' Y
property. Additionally, a node manager that is configured to have less memory+ X( V* G! v! G5 j9 o
than this value will be shut down by the resource manager.</description># K* o. z+ C/ M) @3 T+ s" }) G$ z: Y
<name>yarn.scheduler.minimum-allocation-mb</name>2 c8 l; M1 q+ |" a* q
<value>1024</value>6 T: ^% |0 K2 j3 i; `8 a$ @
</property>- }+ p8 ~+ L! ^6 \' u
<property>2 w* V+ m. `* s( v4 ?: K! j2 U
<description>The maximum allocation for every container request at the RM( s4 ?& v+ D! o
in MBs. Memory requests higher than this will throw an. z. |3 H7 H: V3 |
InvalidResourceRequestException.</description>
" ~8 [: S( a0 P* b6 ^% | <name>yarn.scheduler.maximum-allocation-mb</name>2 B7 j8 w; l7 P- U6 d
<value>8192</value>5 e4 g+ B: A+ L
</property>
& p p2 o: |% ^' A2 u! x; i8 [ <property>
6 d. D' `$ ?7 B& e1 W# n* { <description>The minimum allocation for every container request at the RM
7 V6 N( l- W* B* _& x in terms of virtual CPU cores. Requests lower than this will be set to the
+ F6 p/ j5 ]. r4 y$ X value of this property. Additionally, a node manager that is configured to
8 s$ A g2 o8 Z4 y: j have fewer virtual cores than this value will be shut down by the resource
' M5 }8 f. i( x& _0 L+ o- l manager.</description>: J D& g+ T9 E+ V1 s
<name>yarn.scheduler.minimum-allocation-vcores</name>; n3 h( k; f% W: I
<value>1</value>
! f" T8 y7 w* }7 @4 L$ T. A </property>
+ S7 p- @& M3 U3 e+ e/ t <property>
$ b8 S$ A0 s# b7 v+ o <description>The maximum allocation for every container request at the RM1 c' v" u( K7 i7 [% S/ E- i
in terms of virtual CPU cores. Requests higher than this will throw an. V; u" z0 k' d& L1 ~
InvalidResourceRequestException.</description>, ~4 Q0 [$ t' I
<name>yarn.scheduler.maximum-allocation-vcores</name>
: T- a% L4 q$ A8 s <value>4</value>
! I! O/ i- d$ Z3 R' c </property>& t: I# |& l" ~4 n9 h+ p
<property>* e8 Q9 i! H( {: `8 K7 A
<description>0 O7 N$ y, R6 B/ d* n
Used by node labels. If set to true, the port should be included in the: v+ `9 Q- ^( o6 }/ l! E
node name. Only usable if your scheduler supports node labels.
8 R! m- U5 L1 M! `8 W) c </description>" b! k! V6 J/ r, S; D
<name>yarn.scheduler.include-port-in-node-name</name>; V- L( z4 D8 S
<value>false</value>5 _' ?/ e" X; u3 t7 z) k
</property>2 [* E3 o I& W1 m) _+ X! g+ p6 F
<property>
' V9 H' A3 G8 J) ^, w# c2 Y <description>Enable RM to recover state after starting. If true, then
: [" ^0 I) n4 D2 a @7 a1 j yarn.resourcemanager.store.class must be specified. </description>$ C& A. i6 D3 O1 l% W4 `; i4 w
<name>yarn.resourcemanager.recovery.enabled</name>
4 ~) a8 ]$ ^8 [8 x; Q <value>false</value>$ Y$ m1 }% v# J& m
</property>/ ?# Y' p2 G, o2 {% K: n
<property>4 P% e: S+ F& W1 h1 a
<description>Should RM fail fast if it encounters any errors. By defalt, it5 ~1 Y4 _ {( c) D
points to ${yarn.fail-fast}. Errors include:" c7 z Q2 Y& |; F" r' J
1) exceptions when state-store write/read operations fails.; [ ]; _0 d1 N' a/ c _8 n2 }8 k
</description>
?8 k& {% w) C <name>yarn.resourcemanager.fail-fast</name>. { ]1 O) N1 P3 Y
<value>${yarn.fail-fast}</value>4 }/ o) b4 o- c5 Y7 D# O
</property>
9 A% k7 l( J/ |3 D8 v1 x0 a. h <property>/ a$ F5 \% e2 H2 d( f3 Q
<description>Should YARN fail fast if it encounters any errors.- { \5 A% m" [- S, W9 x
This is a global config for all other components including RM,NM etc.
" x. k1 a" Y0 @5 { If no value is set for component-specific config (e.g yarn.resourcemanager.fail-fast),
9 {6 Z2 ]! _+ _* B+ u this value will be the default.7 e* e ~2 l' d6 v" k$ ?
</description>: o; g k/ J& \4 ]$ V, ?; A
<name>yarn.fail-fast</name>
[) v4 l3 L8 a. l [7 L+ h <value>false</value>
$ q- A# |/ i* S$ @6 p" v& o </property>
6 k( h" E# n) a! ` <property>
3 [) s2 ?. R; u( ^& Z2 u( r& O <description>Enable RM work preserving recovery. This configuration is private+ C& B8 W% K) n4 `" w. U
to YARN for experimenting the feature.. C2 s; ]/ @ O
</description>
0 j, ?/ i& Y9 M9 A/ ? h <name>yarn.resourcemanager.work-preserving-recovery.enabled</name>+ A, y! u; `& N) F1 y
<value>true</value>
% Y- w& F* L! H7 G </property>: [ T3 r* x" p9 g9 J
<property>
- V( i a! _# i% h0 [/ N- |/ A <description>Set the amount of time RM waits before allocating new5 o! p. N) a, H9 ~; I
containers on work-preserving-recovery. Such wait period gives RM a chance
* L* o% s9 ~2 [% }3 u" _ to settle down resyncing with NMs in the cluster on recovery, before assigning+ ?% M" e! g, o/ R
new containers to applications.. n2 w5 p* {! B* U$ K( u3 ?
</description>& g n0 f% r5 H ]3 q4 V# s0 a
<name>yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms</name>
, h0 @- q. |/ A9 t- q3 { <value>10000</value>8 R! S0 K' ^ e4 J5 K R9 X& G. J
</property>
8 F! N$ P/ |; Y <property>+ |+ @) q, W: Z; `2 t$ X
<description>The class to use as the persistent store.
& |" O* e! ]- R5 q; g0 s If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore4 F$ q' A7 ]* r- S
is used, the store is implicitly fenced; meaning a single ResourceManager* t2 H7 t7 ]- Z4 P
is able to use the store at any point in time. More details on this2 {3 ^8 h- M9 O I/ q. {8 {( H
implicit fencing, along with setting up appropriate ACLs is discussed. Q* G8 B _( q: V [
under yarn.resourcemanager.zk-state-store.root-node.acl.
) g, ?7 d5 a4 [$ @ </description>
: a6 G% W. s3 V- \4 z <name>yarn.resourcemanager.store.class</name>2 E: B* b; p9 F: N6 \1 Z2 e
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
2 e9 k1 B" J4 Y+ E: \( z </property>' a$ X; [# j8 f! c
<property>% P; ]# ]9 r+ Y0 z
<description>When automatic failover is enabled, number of zookeeper, K8 ]9 o* R7 v% I0 X" |5 P
operation retry times in ActiveStandbyElector</description>* x. d( X* L/ o7 H/ v: t
<name>yarn.resourcemanager.ha.failover-controller.active-standby-elector.zk.retries</name>
, T1 }1 e- V! I( Y0 l: s <!--<value>3</value>-->
' c) c n( a8 k4 g5 B3 }& F </property>
( \0 p: L) g( X8 N <property>5 x9 o/ g$ u9 c7 L
<description>The maximum number of completed applications RM state
: h& R3 A; U- `8 C store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}.: V% Q$ t" x7 c
By default, it equals to ${yarn.resourcemanager.max-completed-applications}., J% C: e' ], f
This ensures that the applications kept in the state store are consistent with
" W" j7 @: G4 w the applications remembered in RM memory.
+ r3 Y& u" q% B! h Any values larger than ${yarn.resourcemanager.max-completed-applications} will
; c! |6 g0 v( k+ H1 ~' Z: | be reset to ${yarn.resourcemanager.max-completed-applications}.3 V# @0 e( x8 k/ V9 H7 e+ Z
Note that this value impacts the RM recovery performance. Typically,; t1 `! y3 n7 \ H: l3 d; ?
a smaller value indicates better performance on RM recovery.
0 H u4 i' u5 a </description>
. z. z1 b; A: z1 f* ]) c& E9 ^ <name>yarn.resourcemanager.state-store.max-completed-applications</name>0 F8 Q, r" s% `3 ? M# P
<value>${yarn.resourcemanager.max-completed-applications}</value>
3 o. q1 G5 T7 e; ~8 b9 m </property>
. q( M% ?- _* J5 f0 J <property>
9 Y/ U! L; d. ^1 I" p* S+ P <description>Full path of the ZooKeeper znode where RM state will be
3 U9 k( m# S0 i4 c! W stored. This must be supplied when using" o0 N$ s; a' V4 n# G. `
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
, [; q2 o0 a8 o" q, D& P; K7 Q8 ? as the value for yarn.resourcemanager.store.class</description>; v% L+ c7 q& L
<name>yarn.resourcemanager.zk-state-store.parent-path</name>
- r, A. |) T% j- h [ <value>/rmstore</value>4 T( d* H/ O0 n6 ^3 V* r' X' u$ A
</property>
6 P0 \1 L q7 | <property>
/ v. B% c& e5 M0 D* c <description># D( s/ N* D2 p* ~
ACLs to be used for the root znode when using ZKRMStateStore in an HA' c, q- G. K0 [" s
scenario for fencing.
# r7 \; U4 f) S, p4 ]$ w ZKRMStateStore supports implicit fencing to allow a single
3 i5 i" S6 u8 u ResourceManager write-access to the store. For fencing, the: [. c$ n1 I" {9 A# @! {$ Q* _
ResourceManagers in the cluster share read-write-admin privileges on the
( y H" X$ T8 ^& h root node, but the Active ResourceManager claims exclusive create-delete
8 u5 K/ K5 O- d; ^0 A0 B. D' w permissions.
' D$ v7 S+ i P By default, when this property is not set, we use the ACLs from5 V! O- X& Y3 B1 G& E3 u
yarn.resourcemanager.zk-acl for shared admin access and
! q" J. E `, b9 n3 U: X( F rm-address:random-number for username-based exclusive create-delete' h% }& U$ ^ R) x4 q
access.& s8 _# H; s' G
This property allows users to set ACLs of their choice instead of using4 Q# k' N5 |, s+ I8 T
the default mechanism. For fencing to work, the ACLs should be
& V2 ?& b9 t1 b1 j* f& x carefully set differently on each ResourceManger such that all the
0 k' g* @6 ^( g: X% [. p$ S ResourceManagers have shared admin access and the Active ResourceManger. U. ?: H6 H r- W0 m
takes over (exclusively) the create-delete access.
" i C% x4 l5 j* }9 G" W V h </description>
) s2 |/ n' H0 _# B8 U( e, d- u5 @ <name>yarn.resourcemanager.zk-state-store.root-node.acl</name>
1 Z" y6 \/ v$ D: m( k, e! m! ~ </property>! u" s% W3 t X2 K/ @( A
<property>
" T# `* u2 B- Y, j <description>URI pointing to the location of the FileSystem path where
; i! `9 A4 l& J0 Z( K/ _! ~ RM state will be stored. This must be supplied when using
8 W6 @) h! P# ~2 Q: Y2 {8 g org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
9 o, M5 A* X9 V; K as the value for yarn.resourcemanager.store.class</description>. i6 o7 |/ h! o3 l% m+ }
<name>yarn.resourcemanager.fs.state-store.uri</name>7 X* [' e& }' z" `
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>. `* b% _1 Z( J+ p9 @! [
<!--value>hdfs://localhost:9000/rmstore</value-->
, Q4 v9 F* w( | { </property>
7 u+ w3 k7 n0 T( q( S! S. Z <property>4 c, D6 J5 A, \4 Q) ]- v( _
<description>the number of retries to recover from IOException in
9 i; g+ ]/ Y- R) ^$ u FileSystemRMStateStore.( z2 l5 K& ] M0 ?% m
</description>
2 y% s0 D8 `9 h) d7 @4 S: Y6 o7 J. P <name>yarn.resourcemanager.fs.state-store.num-retries</name>
* k( d k" ]: v <value>0</value>
! p2 q2 \) I2 l* P </property>+ @8 R8 I4 d& B/ ~; [
<property>
" r3 k! n) g0 D0 z <description>Retry interval in milliseconds in FileSystemRMStateStore.
9 M- X& ]" T4 k; ~! ^' ? </description>
; D5 [7 ^) w6 ?; c& W5 v. T/ N& N6 v <name>yarn.resourcemanager.fs.state-store.retry-interval-ms</name>$ @3 K& p: e: c( M o' u: |. [
<value>1000</value>4 R' T/ F; O( M$ a" x- D) X
</property>1 {* ~3 |# h6 `, @% _
<property>. O- ]) r* o! W0 b/ ?! _( ]
<description>Local path where the RM state will be stored when using
6 Z E- w# s! ~- | Q# l0 [& z1 ~ org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore
! W( n; _6 c o$ D5 z; C* j as the value for yarn.resourcemanager.store.class</description>1 i3 t: o3 _& n5 r, M# r
<name>yarn.resourcemanager.leveldb-state-store.path</name>7 t* ]" x, P7 {! G
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>0 U, E+ h& T' b$ A
</property>7 Q/ Y% R6 @' p/ C6 m
<property>3 E& f4 k. Q' Z4 s6 k
<description>The time in seconds between full compactions of the leveldb' C8 u x/ s/ \- t
database. Setting the interval to zero disables the full compaction4 S( M/ {3 t+ |# @* q
cycles.</description>' N9 a0 B) _) J5 D( w' i
<name>yarn.resourcemanager.leveldb-state-store.compaction-interval-secs</name>, B9 u/ s8 Q5 i/ \
<value>3600</value>
. Q$ ]7 D2 d. J6 @ </property>
0 w: _, |8 [4 X1 h <property>
4 m+ X$ F' s+ _6 Z$ t+ \# e& K! J/ f2 L <description>Enable RM high-availability. When enabled,
% V9 ?5 I. i N/ P, M8 s5 T' O/ ]- g (1) The RM starts in the Standby mode by default, and transitions to
4 Y% A2 U3 I* P: \ the Active mode when prompted to.
9 ?3 k) ^, G; F" y* _# [( @ (2) The nodes in the RM ensemble are listed in
7 X1 u9 N) g! C3 H yarn.resourcemanager.ha.rm-ids1 Z* D' h* B( h! }' O2 X8 t$ m
(3) The id of each RM either comes from yarn.resourcemanager.ha.id
8 x5 G" r5 ^" J0 Z# N U if yarn.resourcemanager.ha.id is explicitly specified or can be' L9 _: S+ D- p- `( q1 w( u' k% f y
figured out by matching yarn.resourcemanager.address.{id} with local address- P0 f( c: G, D% |/ p$ t/ X
(4) The actual physical addresses come from the configs of the pattern% x; ]# o: f- }. g! _" h) B
- {rpc-config}.{id}</description>4 e, \: l0 L- b
<name>yarn.resourcemanager.ha.enabled</name>8 l, }! `! k% h. `8 _
<value>false</value>% T# N; u# q# a3 x g# D2 X! G
</property>: D; Z5 _- C" |" x
<property>
* _# v' a2 P$ ^! `7 K; l <description>Enable automatic failover.1 T# N9 c9 G& _5 k
By default, it is enabled only when HA is enabled</description>
# h0 t: u8 x6 A8 l <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
* m: d. u: r+ O. Q0 v3 m7 g8 l1 Y* I <value>true</value>
3 Z$ z8 g8 ]& L5 Y8 H </property>( [+ q/ @6 V* d. u/ S7 E. @
<property>4 e* b1 H' C% i6 R6 Z+ |) G# E
<description>Enable embedded automatic failover.( V2 j. C/ r5 i; D; M
By default, it is enabled only when HA is enabled.
# {5 h5 q1 s2 b6 F* q The embedded elector relies on the RM state store to handle fencing,- o& [" ?" p, S( `! X
and is primarily intended to be used in conjunction with ZKRMStateStore.! ^, F1 h7 J, C) U9 N- z5 ?
</description>
$ H9 G- E- A6 R/ G+ W9 ~ R <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>: r; _5 E. ]2 _
<value>true</value>
6 A' J; u" F' A$ a- ]) U+ n </property>
& T( }! e% s k% z! b4 B6 X7 f" Q <property>
$ M! g# ?$ @( ^: B/ p- k <description>The base znode path to use for storing leader information,# H3 N/ V# w! x
when using ZooKeeper based leader election.</description>" V6 w) O) I+ \8 |, g: Y
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
+ G! U' O, e" b L. t <value>/yarn-leader-election</value>
/ }" I3 I/ z4 R% Q5 a9 i! n </property> R/ w% |; Y. r3 p9 S
<property>$ _5 W" M: }; x5 b
<description>Index at which last section of application id (with each section
' A! S' P9 _7 C& W+ ]7 b8 R' x separated by _ in application id) will be split so that application znode
# @) A1 i: n* P# ?5 [ stored in zookeeper RM state store will be stored as two different znodes3 L$ E, C8 D7 p1 A2 D
(parent-child). Split is done from the end.
8 v1 L4 Q- A1 G, `; u; k- c! S For instance, with no split, appid znode will be of the form0 z, W( W8 M# T' Q1 d2 @' M- U
application_1352994193343_0001. If the value of this config is 1, the. a5 U$ F: `- R- s% L( @ V; E) c
appid znode will be broken into two parts application_1352994193343_0006 |0 y+ v# \; H
and 1 respectively with former being the parent node.# V+ Y# Q- P7 Z6 b, ~- U
application_1352994193343_0002 will then be stored as 2 under the parent: s$ _: h) l' P. I3 I, C
node application_1352994193343_000. This config can take values from 0 to 4.
) q) O5 v6 b4 }) y' @ 0 means there will be no split. If configuration value is outside this/ \3 P) c l* ?- S$ [( o: b, z
range, it will be treated as config value of 0(i.e. no split). A value1 L( z; _$ T$ m/ L, H8 O+ @# R- {
larger than 0 (up to 4) should be configured if you are storing a large number
: D" k" X+ i" S! W; @ of apps in ZK based RM state store and state store operations are failing due to
! k8 t- c& [8 L LenError in Zookeeper.</description>
5 y7 D! U( P1 z0 Z5 B8 @5 h9 m <name>yarn.resourcemanager.zk-appid-node.split-index</name>
& @. g* V: H2 J( t) f, a7 } <value>0</value>
; F6 M; b) Y0 e% k- W. c7 f2 t$ q </property>
e' s1 g" d2 Q( Z6 i* X( B5 |, ] <property>1 I* V6 u* L& A j
<description>Index at which the RM Delegation Token ids will be split so$ S1 Z- P; y; I, `$ e
that the delegation token znodes stored in the zookeeper RM state store' l: B' [' a+ n% `
will be stored as two different znodes (parent-child). The split is done
4 z" I% K/ R" p& s: B; }, S from the end. For instance, with no split, a delegation token znode will
2 L9 P8 k: ~: y4 H1 j5 T2 c* u be of the form RMDelegationToken_123456789. If the value of this config is+ i6 }6 b, R4 ]
1, the delegation token znode will be broken into two parts:9 j N7 ^" U, |! G: g& P/ F( c7 @
RMDelegationToken_12345678 and 9 respectively with former being the parent
. O1 W/ Z. W5 _, |/ [9 P node. This config can take values from 0 to 4. 0 means there will be no
* E8 P8 X, c( ~ split. If the value is outside this range, it will be treated as 0 (i.e./ v1 t! i) ^. \2 b9 u& i" b) |
no split). A value larger than 0 (up to 4) should be configured if you are
5 g# K3 j8 T+ A0 C running a large number of applications, with long-lived delegation tokens
" J$ t: ~8 C/ ~) v9 D" f5 Y and state store operations (e.g. failover) are failing due to LenError in
! p9 T7 A% b" w Zookeeper.</description>
& l, i0 f7 r6 a/ u <name>yarn.resourcemanager.zk-delegation-token-node.split-index</name>
$ v: B% H; {' r <value>0</value>8 B) w# }6 y+ H2 N, f2 n$ _! Z
</property>
$ X0 r j5 ^! t. ]- T3 z/ p <property>
0 ], z4 P2 |( ^2 m7 c: j; A* ] <description>Specifies the maximum size of the data that can be stored! ~) P0 t9 i3 H7 O0 G6 E
in a znode. Value should be same or less than jute.maxbuffer configured7 s8 J# m( T+ k$ ~
in zookeeper. Default value configured is 1MB.</description>! H1 P0 y t+ Q3 [" p
<name>yarn.resourcemanager.zk-max-znode-size.bytes</name>
1 Z" I8 G7 I: M- Q8 r <value>1048576</value>
( I( d( j, F6 C6 c0 C" v) x </property>
: R' ]5 h) V$ r/ C+ P/ X i <property> I1 `0 O8 r- {+ b
<description>Name of the cluster. In a HA setting,0 A! ~. V# p: Z6 b" Y
this is used to ensure the RM participates in leader
' @8 b, r! |% x, T, p) K% _6 n election for this cluster and ensures it does not affect4 _) T! U, {& S* G
other clusters</description>. p6 r' @( B, b/ o
<name>yarn.resourcemanager.cluster-id</name>9 g, j [5 k4 h# Y$ F
<!--value>yarn-cluster</value-->
7 h5 \" L7 E% o( O) M9 v1 O </property>
" j c1 ~, p' f/ d- e# w+ x <property>
3 \5 N/ J. V" z <description>The list of RM nodes in the cluster when HA is
, {! Y0 c- o) G7 h0 o# q/ j2 D% @9 x! r enabled. See description of yarn.resourcemanager.ha
|5 Z& F7 W7 m0 E! H; J& P% | .enabled for full details on how this is used.</description>
8 ^+ {# U* i- _- Q <name>yarn.resourcemanager.ha.rm-ids</name>; g. ^3 o% \( p7 ~. `
<!--value>rm1,rm2</value-->
; M4 W5 P' P6 D0 N' e% s </property>
" V" ^' w. \1 B; q <property>
; {5 b' D: y: I7 F <description>The id (string) of the current RM. When HA is enabled, this
; ` j" q. n. u% Z5 G is an optional config. The id of current RM can be set by explicitly4 v5 d: ?* c9 M1 D
specifying yarn.resourcemanager.ha.id or figured out by matching% c y8 q' H7 e: ^# d; d& d
yarn.resourcemanager.address.{id} with local address
1 P- ]$ }$ ]& u* q See description of yarn.resourcemanager.ha.enabled( \- `8 }. d7 \6 _) f0 [4 b
for full details on how this is used.</description>- k: o5 S8 Q% L5 ?5 d
<name>yarn.resourcemanager.ha.id</name>
+ l/ q, k% P4 ]7 P* R+ [) \ <!--value>rm1</value-->* v( O. s: t9 d3 {
</property>
5 }+ c/ z/ Q$ k* }/ G <property>
. R+ K- \# _& f2 z" \0 ?9 B <description>When HA is enabled, the class to be used by Clients, AMs and
/ q0 q2 ]- `. ~$ B( z4 n' Y4 S# U NMs to failover to the Active RM. It should extend
8 o- F8 s) T3 u9 l% J; M org.apache.hadoop.yarn.client.RMFailoverProxyProvider</description>) {8 \" V+ E0 i% {4 k
<name>yarn.client.failover-proxy-provider</name>4 o. U' \4 z, O4 L
<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
+ B, i! h$ F) B# Y, ~& C" T </property>
9 `$ D' _$ a' [4 y* ^: Q5 D <property>" K: h6 Q2 G7 j7 d/ T, n4 e, o
<description>When HA is enabled, the max number of times4 A! `, e& A J1 ]" a) m
FailoverProxyProvider should attempt failover. When set,% X- E" S! q+ e. w
this overrides the yarn.resourcemanager.connect.max-wait.ms. When
" [6 z+ T# L, f6 e4 J not set, this is inferred from. W% s) R$ @0 k: W
yarn.resourcemanager.connect.max-wait.ms.</description>: z, H' Z" h0 a. _- w2 ?8 s
<name>yarn.client.failover-max-attempts</name>
+ Q6 C- M7 S3 N e$ @* v& D <!--value>15</value-->) s* J7 [5 z/ b( S" G& @
</property>
3 B6 K: B7 @, F2 ~1 J9 y6 n( v4 z' n) b- H <property>% L' N# I5 y9 b' h
<description>When HA is enabled, the sleep base (in milliseconds) to be
( z& I, p+ W l# ?- { used for calculating the exponential delay between failovers. When set,
4 k+ E6 k; I8 d5 Z this overrides the yarn.resourcemanager.connect.* settings. When
( L, |9 k+ D! \ not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.6 Y- Z0 S. q1 W$ V! o9 S
</description>$ a- L1 l3 \ I8 x4 z
<name>yarn.client.failover-sleep-base-ms</name>: y+ [0 H: K, ~* A* P
<!--value>500</value--># O# v/ }& j! P# I4 h' _
</property>1 V& ~. P+ |$ v; P& O/ O% i# o
<property>4 ?2 _+ Q% |9 S& G
<description>When HA is enabled, the maximum sleep time (in milliseconds)
2 d4 z5 I% D1 n3 W n between failovers. When set, this overrides the0 \ b2 l8 o: t
yarn.resourcemanager.connect.* settings. When not set,
/ G9 ^) O1 V" Z yarn.resourcemanager.connect.retry-interval.ms is used instead.</description>8 V: U( q9 y( v8 K" {
<name>yarn.client.failover-sleep-max-ms</name>
! {9 |' s+ G7 B% A0 T <!--value>15000</value-->
) d3 Y; ~. }9 @& K( x8 @5 w% Z- O </property>
$ m+ Z9 h4 b B* g <property>$ ~ l+ ~* l7 @+ U
<description>When HA is enabled, the number of retries per
( k; S5 D) E5 P5 ?1 e( ~/ m0 ~ attempt to connect to a ResourceManager. In other words,
- n& ~) q; Y' s5 R/ } it is the ipc.client.connect.max.retries to be used during1 H# g" u0 K4 |7 A
failover attempts</description>$ c! b+ D% d9 w5 [, j/ L
<name>yarn.client.failover-retries</name>
?/ `! M( }) @& w0 k! ]- E <value>0</value>
# m6 K- d8 s3 B. E </property>
( K$ o, C3 {- Z* j3 H <property>
+ L2 {9 O; m6 k( v) `8 |( L( Q2 @ <description>When HA is enabled, the number of retries per
1 i6 N! \; E+ R% Q- S attempt to connect to a ResourceManager on socket timeouts. In other
8 h8 Z, ]$ b, F5 r. ~$ C- \7 O words, it is the ipc.client.connect.max.retries.on.timeouts to be used) g: |) u* R- E: q% ], j
during failover attempts</description>& J( X2 x0 f+ W h. l% b9 ^$ S+ K
<name>yarn.client.failover-retries-on-socket-timeouts</name>/ A+ `& E, H8 q0 k9 ?" Q
<value>0</value>
5 G; ~: A; g7 Y# o8 y </property>
" r( a9 f% r7 u. T. M! L0 } <property>) z& R/ A c' E) R1 ], E/ I" @
<description>The maximum number of completed applications RM keeps. </description>. a, l6 c$ A* e/ n8 _& ]$ u' a o
<name>yarn.resourcemanager.max-completed-applications</name>: r$ c: H) J* y: A, T" |- X
<value>1000</value>/ j. z$ V. y" c* J( W
</property> S2 }2 x! M- O/ Q1 o+ s3 ~
<property>6 C4 S: G4 O! a5 L( A& g. K
<description>Interval at which the delayed token removal thread runs</description>( ?2 K# j/ L2 {% h; S' Y# v. l
<name>yarn.resourcemanager.delayed.delegation-token.removal-interval-ms</name>3 ~; `$ S* v8 @$ t4 X$ I! _6 a3 p
<value>30000</value>5 a" M. u$ K* ]# J9 a0 Z
</property>
0 v: k T3 R8 O. M, k! Y( K <property>7 u* h9 C/ ?) o: F h
<description>Maximum size in bytes for configurations that can be provided
9 e* [% X% L5 ]* P6 @ by application to RM for delegation token renewal.% }8 d* i* g6 q1 h8 t( u% ~2 q4 [
By experiment, it's roughly 128 bytes per key-value pair. `8 D5 v. e( F6 z" ?( J
The default value 12800 allows roughly 100 configs, may be less.
% M+ t4 V [6 L% r) S9 U </description>2 ]0 V( A7 K9 L; z
<name>yarn.resourcemanager.delegation-token.max-conf-size-bytes</name>
& a0 }# r0 j. u, z <value>12800</value>
. _' _: B: n/ i3 h& ?; F </property>
+ T' `7 Q" u$ e( ]. k <property>% F8 N3 X+ X+ c6 ?4 r# _! T
<description>If true, ResourceManager will have proxy-user privileges.3 s4 Y6 G9 s2 t- R
Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to
6 Y1 K! k- J5 P! J c do localization and log-aggregation on behalf of the user. If this is set to true,
% e9 k a; { q ResourceManager is able to request new hdfs delegation tokens on behalf of
; d( s1 q h; E7 n, _" b% I the user. This is needed by long-running-service, because the hdfs tokens
! l% s/ {3 Y% q7 D, q" ] will eventually expire and YARN requires new valid tokens to do localization
1 J2 G4 q% l+ H2 J* M4 ` and log-aggregation. Note that to enable this use case, the corresponding8 }+ r. P- f9 R1 Z* J
HDFS NameNode has to configure ResourceManager as the proxy-user so that
6 b- {& O* u% u6 Y. R ResourceManager can itself ask for new tokens on behalf of the user when `3 K$ b: |, }
tokens are past their max-life-time.</description>* S, W/ d7 Y! ]: R
<name>yarn.resourcemanager.proxy-user-privileges.enabled</name>( J- k; ~# {# w" E
<value>false</value>- q: Z' e! M! r3 M* O$ W
</property>
/ e4 |1 }) e: f# h& X3 D <property>5 Y) Y7 x/ c# o. k5 R5 E! m
<description>Interval for the roll over for the master key used to generate
! x+ `4 v6 N/ M8 W- {1 G; r application tokens
) y: }% V& R* q% Z8 e2 h7 j </description>- g0 z* @! g+ D0 ~, b3 i; P
<name>yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs</name>
6 b) ^6 H+ G" l2 E, B0 o <value>86400</value>
9 d' c4 I+ |) k1 c& d* }9 ?" Y </property>/ R1 C* t1 k9 p
<property>
3 ?3 V, u+ p z& s8 i <description>Interval for the roll over for the master key used to generate
0 c1 e+ m2 k. J% W) ^, U container tokens. It is expected to be much greater than
+ |7 }) l; G. |0 g! k yarn.nm.liveness-monitor.expiry-interval-ms and7 Z! ]- d( n7 G0 E7 S
yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the
0 O( L' A) h' m* I behavior is undefined.
& ?- m' J6 u" [7 q9 d. { </description>
" ^9 [) p1 ?8 N <name>yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs</name>8 i. _, ^3 l; I7 T0 W; d7 G% P. Y
<value>86400</value>
% Q% n! X; J8 l </property>
( C* _: b6 T/ b) W' ] <property>0 J) |* n, ]; f9 D" E4 x( {6 a
<description>The heart-beat interval in milliseconds for every NodeManager in the cluster.</description>7 `/ _, @' m* }5 |# B# G# l
<name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>" k M" ?: u* |+ i
<value>1000</value>
" r: E! {" R0 c: b: m) U </property>
; i1 Y3 M- P* s8 k' R9 M2 G6 o @ <property>
& F+ b! u: h0 u3 D, U) x <description>The minimum allowed version of a connecting nodemanager. The valid values are$ t* U, \3 b7 ^- s+ I, M; k! B
NONE (no version checking), EqualToRM (the nodemanager's version is equal to$ v% g _3 A8 G) C) A( ^
or greater than the RM version), or a Version String.</description>
. a: H, N- a- @2 _ <name>yarn.resourcemanager.nodemanager.minimum.version</name>5 l3 ]; U: `% D8 ?9 @6 C
<value>NONE</value>" i& i8 V- D! k5 x
</property>" u2 F; i- L( f9 W! s& g
<property>
+ S# p, \" ~$ {0 r1 e <description>Enable a set of periodic monitors (specified in
: ~* H) }: T4 s' H8 A/ f5 T yarn.resourcemanager.scheduler.monitor.policies) that affect the
' c- z7 ^: t0 a9 l K% ` scheduler.</description>" A( W0 x( k/ v' @# B6 B
<name>yarn.resourcemanager.scheduler.monitor.enable</name>
9 u! ^+ U5 U n, \, Y0 I9 R9 _ <value>false</value>
7 w% k. v6 J# z- E# ~) ~; P) z! [ </property>* h5 r; a+ D" k4 N1 X
<property>6 @2 Y" D$ f. Q+ o
<description>The list of SchedulingEditPolicy classes that interact with8 U: {: u$ e& V8 F# P
the scheduler. A particular module may be incompatible with the
" D9 r- _' N6 S% i* b scheduler, other policies, or a configuration of either.</description>0 g; e; P. C3 G c8 e
<name>yarn.resourcemanager.scheduler.monitor.policies</name>
! d4 |1 S. g1 Z4 L" N' x <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value>
, \! L* B3 J& p6 H5 W1 Y </property>3 T8 p) }2 b3 J7 M# |& ?. n/ P
<property>' A* P* h; U) b# v
<description>The class to use as the configuration provider.2 f; _2 u& G7 e' u7 s5 S. n% M
If org.apache.hadoop.yarn.LocalConfigurationProvider is used,
* p4 I) }+ X" r5 `7 }" | the local configuration will be loaded.
$ w }0 e4 V1 P4 y3 Z If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used,
) n( w* }0 o/ W+ V8 p* \# r the configuration which will be loaded should be uploaded to remote File system first.' Y" B1 B# P9 S6 {1 `3 ~; F4 ?! p
</description>3 n- @- E" D4 P8 G% d: ]
<name>yarn.resourcemanager.configuration.provider-class</name>
! u! W- s* q( T$ N/ e5 ^; k' K9 g <value>org.apache.hadoop.yarn.LocalConfigurationProvider</value>& |3 w" o }/ E* ^
<!-- <value>org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider</value> -->6 J9 m2 i/ P. ^" L1 `: d$ s
</property>& x. q( e6 N0 x: ]
<property>
1 E" t( U7 y! x. F. @8 f+ c4 Y <description>
$ B1 M+ \* ^" P! H The value specifies the file system (e.g. HDFS) path where ResourceManager
! K8 g! V: L# d! [: l loads configuration if yarn.resourcemanager.configuration.provider-class
2 j9 x" L1 c3 @. a' M; t is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider.7 `2 m1 j6 i- N
</description>/ |# y% n. j) Q1 w
<name>yarn.resourcemanager.configuration.file-system-based-store</name>, Y/ a9 U# `. y+ y4 C
<value>/yarn/conf</value>
2 _- d+ Z: N: N1 e. h </property>
) M2 v# L: F9 b: O <property>
+ B0 F- e9 R% Q L4 H <description>The setting that controls whether yarn system metrics is0 X* M) G) A# k1 K! Y* p% _
published to the Timeline server (version one) or not, by RM.
+ A& M9 C) Z; U3 K This configuration is now deprecated in favor of$ w- \# l a1 t
yarn.system-metrics-publisher.enabled.</description>! G3 I$ f5 ~2 { M- l
<name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
6 q G$ {% i( h/ i4 Y" x' @ <value>false</value>* _# _! K+ q! M) j0 z
</property>
1 `! u R; U+ L# D3 ~/ B <property>
; |& T; Q! X( u% Y& C" v <description>The setting that controls whether yarn system metrics is
[) Y. U& i" Q8 P published on the Timeline service or not by RM And NM.</description>2 Y, R+ g+ e% t6 V% X6 O+ ~9 [
<name>yarn.system-metrics-publisher.enabled</name>
, ^* a/ k4 `* u2 E( L <value>false</value>1 K6 Q8 O; p2 K4 \6 Y2 B
</property>
2 I, X. R9 z2 I+ i6 H5 | A <property>: T2 d5 x5 p. h+ ~8 ]
<description>The setting that controls whether yarn container events are
4 |4 u8 @3 a" @$ l: B# f' z published to the timeline service or not by RM. This configuration setting( [3 q+ s# Q+ V7 f0 q
is for ATS V2.</description>/ T+ E3 u# N4 @" k* h1 k
<name>yarn.rm.system-metrics-publisher.emit-container-events</name>
' S, ^ A7 F3 @' K <value>false</value>: {$ N. I6 m4 R, s
</property>5 U. i; P" C4 h3 z! l& [2 o
<property>
0 R0 A& z3 n6 b$ A0 o <description>Number of worker threads that send the yarn system metrics7 m/ E6 r0 H& G5 \, {: ?
data.</description>8 W& c! j3 J: P
<name>yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size</name>
& [- g' ~/ c7 c; D- s <value>10</value>
8 }6 n/ j. Q1 y </property>0 M, j6 j, s& ^; o; m# C: T2 d7 o
<property>
3 i. g" x$ C" } <description>Number of diagnostics/failure messages can be saved in RM for
4 F5 |7 P; Y2 o7 [& `- h- W log aggregation. It also defines the number of diagnostics/failure% k5 e% u( x4 S/ ]4 e
messages can be shown in log aggregation web ui.</description>
; p5 u/ R h7 m* r# T% C" T) O <name>yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory</name>
- ?0 M' f2 @( t9 _ <value>10</value>* o9 h! x6 \8 H; Q
</property>3 a; f& n) {/ X/ C) f0 I1 P! ]
<!-- Node Manager Configs -->' s8 J; y+ t5 m
<property>
- D4 g& s9 k! Z5 t5 G" c1 K4 e <description>
G5 ?7 X7 E( A' z8 g* D% X' c; C RM DelegationTokenRenewer thread count2 j8 |4 g B+ D5 {
</description>% j+ e. E. E' q5 A8 T
<name>yarn.resourcemanager.delegation-token-renewer.thread-count</name>
* f+ ^6 Q, x. D! o# [% k2 w <value>50</value>
+ p* G- _ g: O% a2 ?2 w4 a </property>
5 v, ~$ p% y. ~+ ?2 u <property>
* F8 d b- y( [/ b) h( k <description>; @; O( l0 ?( q U
RM secret key update interval in ms7 a3 Q" b& V0 t
</description>' b* D2 \1 x6 \& O8 v3 }! [% ~: x
<name>yarn.resourcemanager.delegation.key.update-interval</name>
1 }) _8 \5 c/ D <value>86400000</value>2 O/ ?; b* M" B# q
</property>
4 j9 w, j2 L6 c <property>
) S9 y& X6 }$ s' A <description>( S# N: j3 |0 A
RM delegation token maximum lifetime in ms
& v. _; f7 f, S </description>
6 c+ {4 G& m3 \0 c4 w8 ` <name>yarn.resourcemanager.delegation.token.max-lifetime</name>
. t/ G. N+ h+ C# t: z/ x <value>604800000</value>
( C& B* C2 V6 A6 H! g1 ?* C* f </property>
" P4 W( Q4 m; `( {' ~9 c0 Q <property>
" w. V0 Z. |2 i5 U( Y. B& E! h <description>
" z4 A; b* ?, c L RM delegation token update interval in ms* X) N" S# O/ p8 M% \) [ E" H* i
</description>; B; @ C( E* F# c4 r9 e" u
<name>yarn.resourcemanager.delegation.token.renew-interval</name>
, D# m" A) x0 s- Y <value>86400000</value>$ R9 F4 f1 I7 ^6 E
</property>; n" G+ `/ F; \& [
<property>
& E8 Z3 E7 @: C# r/ R) n/ R, O! D <description>" D* x* ]& R/ P5 I7 y
Thread pool size for RMApplicationHistoryWriter.
# @# P$ g% a0 x( a5 L8 H* \8 j </description>- x- G9 r5 k0 M% t$ I: v& [/ x
<name>yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size</name>6 y( x! y8 L U
<value>10</value>2 s: W" I- W( ]8 [- ^! P
</property>
* W m' j0 w, f: D6 M <property>6 E+ |& Q4 `: O$ c& ~6 d0 F. a+ b
<description>( }1 O1 g& O( P
Comma-separated list of values (in minutes) for schedule queue related
) J/ P4 S( \% w4 }3 T( d metrics.
$ u2 k3 M: O+ a# M6 D </description>( t# @$ H# H( I$ U3 `& ?- p3 e
<name>yarn.resourcemanager.metrics.runtime.buckets</name>
) ~- H1 N5 G/ L! c- ]! d8 V. E <value>60,300,1440</value>
9 j9 {. a# t% @6 Z$ n, d4 i5 J </property>5 z; J, p. ^1 B. o& _9 A0 v
<property>0 d- k! L) ~& J3 }2 s, N* W! y
<description>
% p2 \- g+ j5 |1 P! p( t' z- |4 D0 k" u Interval for the roll over for the master key used to generate) ?* G, x$ N: x# l/ C4 U0 n+ q
NodeManager tokens. It is expected to be set to a value much larger
( c$ m5 x4 X3 H/ S+ r# ] u# i than yarn.nm.liveness-monitor.expiry-interval-ms.' b+ X% p z9 ^, o, J3 B% [! g3 p, [
</description>
7 I" e. `& j+ v; M; B2 M: m' g4 Q <name>yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs</name>( X) ]2 a- P6 ` q
<value>86400</value>7 l. o% B1 H4 b6 a
</property>% [8 ]5 M; Y8 |8 ^
<property> u. x2 a1 G* ~* x r
<description>/ X" p! T4 c# c+ b/ Y4 K4 f5 ^
Flag to enable the ResourceManager reservation system.% U! N5 r& |, g4 C
</description>
) l) {: I$ m3 u. ?9 { <name>yarn.resourcemanager.reservation-system.enable</name>" _& x$ S) b) Q/ U& N7 D, L% K/ G
<value>false</value>
2 J( D! F4 d' [. n: M </property>
- {8 Y+ J' c1 d. r <property>
7 d1 e1 }2 T. f) F- v <description>
: e& |$ O$ O7 l+ m7 N/ R The Java class to use as the ResourceManager reservation system.
9 o' W3 b+ ^: [4 m By default, is set to3 F4 g0 e3 @* v g
org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem7 k8 L# V" `2 E
when using CapacityScheduler and is set to4 f$ ?6 F p0 ?
org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem
1 K1 U# E+ p, [: q1 J9 c! j when using FairScheduler.
$ f8 U+ ]) ]1 h. u; F3 f </description>8 U9 h6 v% r6 @
<name>yarn.resourcemanager.reservation-system.class</name>
/ a- b; \& M6 {1 @: w <value></value>
4 ?9 t& J+ s8 g5 t1 `8 U </property>+ H% ^$ M, F+ x0 }( l$ _" o+ e
<property>
6 D {+ h2 c8 t+ v$ I5 ~: t <description>7 n E. N& j( x+ M9 g
The plan follower policy class name to use for the ResourceManager
+ j: s) a6 z/ i8 t reservation system.
# y$ k4 T* D6 ^+ o# S. I. Q By default, is set to
1 v4 }) M8 j$ | b2 n! q7 D& T- r org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower
* P1 Q5 o4 A Q$ a# H/ M is used when using CapacityScheduler, and is set to) A- p) `2 J. Y% o
org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower
; A/ z7 l( G3 w0 c9 P when using FairScheduler.
' q7 b+ T0 [$ ~+ C+ x9 u </description>
9 f" c; L3 X" W <name>yarn.resourcemanager.reservation-system.plan.follower</name>% W9 @) E6 x3 x# g) B' }# _0 c
<value></value>
) o: z! N4 B. L# n( H! T& L </property>1 e3 I+ q% k- R4 l& q( W
<property>
2 \9 a& K% a$ V" l, F! _7 P' g @ <description>
9 S: ?# c; E5 C* L Step size of the reservation system in ms9 `. T3 I3 \+ k( d: l
</description>& I, ~$ B8 t) B' U F
<name>yarn.resourcemanager.reservation-system.planfollower.time-step</name>
8 r _9 [* t4 c9 j <value>1000</value>
/ K) u, c, |0 Z* [$ w6 w/ t </property>
9 K( h% v! X8 i: J <property>
3 @% ~0 _" T4 x' |) T& g$ |0 E7 F <description>
- ?" K" E o2 S1 I. Q4 J The expiry interval for a container
1 R) T* ]$ T4 L! J0 p U </description>3 k7 {9 _2 q# p% I7 H5 s
<name>yarn.resourcemanager.rm.container-allocation.expiry-interval-ms</name>
* e7 Z' \. {- v$ \% k9 r: N7 h <value>600000</value># ?% Y, F+ x& h% G9 L1 r
</property>1 ?8 n% W3 s* c! q& I
<property>& N9 w/ o) p3 M; [
<description>
0 {& I5 Y8 M/ a; N7 k, I: f$ X Flag to enable/disable resource profiles1 Z& g! L b' @
</description>
, m* H' j D5 y) l/ {+ m9 r- w <name>yarn.resourcemanager.resource-profiles.enabled</name>
4 Z( p2 d0 h1 C {- M <value>false</value>0 L! ~7 x/ W% T# y& k% C- i9 W
</property>
+ N' f9 J( Y. N+ \5 u, L' V <property>- C( h* o& k) T* a. @0 {" w' L
<description>
- a+ ~; ?: J2 n$ g6 k0 h; l" r: b If resource profiles is enabled, source file for the profiles
% D) F, ~# F8 ?% c0 A* H5 x </description>
; s9 [! v( X' E <name>yarn.resourcemanager.resource-profiles.source-file</name>
0 T3 [5 e6 y1 m% f: U v1 Y <value>resource-profiles.json</value>$ Q: g0 @' L# v$ X! ?3 N! p
</property>( @" S; N( I7 Q _* Q. B4 [
<!-- Node Manager Configuration -->
/ F8 {2 _0 v# z0 j0 L <property>
! k T7 a" M, I <description>The hostname of the NM.</description>& t0 k; o0 O2 v3 O2 d- R- p V
<name>yarn.nodemanager.hostname</name>; h, G- O W- Q# P9 L
<value>0.0.0.0</value># l* f* A5 n: Z e1 @
</property>
; Y+ K; Y) E( c2 y e <property>
" j9 ?/ `1 g6 _. n <description>The address of the container manager in the NM.</description>
9 v6 i8 T ~- h& e! t, G9 j <name>yarn.nodemanager.address</name>. B! h/ k% X) c, e; W
<value>${yarn.nodemanager.hostname}:0</value>" `) E( P& h% o4 ?
</property>
+ w4 b% u- I8 w7 ` <property>
5 Z" o1 _$ @$ j' A2 } <description>6 I0 q0 _3 f$ M7 X, {/ m1 }0 M8 V
The actual address the server will bind to. If this optional address is
: C/ ]! S" _" y, n' O2 d& ~ set, the RPC and webapp servers will bind to this address and the port specified in
$ W# m( ^2 s$ x" `: \ yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is
# B8 Z8 O( J* B+ s/ o8 ^& T most useful for making NM listen to all interfaces by setting to 0.0.0.0.2 ] v9 T+ X3 a2 x
</description>. l+ }5 E2 S& \2 K- A
<name>yarn.nodemanager.bind-host</name>
[1 b; `. r# ^ <value></value>
1 U) |1 N4 |. D" P </property>( W9 P# Y" A* t, e3 X
<property>8 b: ? t! @' J7 m; D$ X
<description>Environment variables that should be forwarded from the NodeManager's environment to the container's.</description>4 x3 A/ S( x! p6 P
<name>yarn.nodemanager.admin-env</name>
7 k) s6 ?; p. q <value>MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX</value>" v4 G# S$ ^& m: a% g) Z
</property>( [$ [, S& {; }4 D) W. U
<property>8 p& }% D# n4 |5 X0 B
<description>Environment variables that containers may override rather than use NodeManager's default.</description>9 q: }: r5 W/ a' q, l
<name>yarn.nodemanager.env-whitelist</name># `, C6 m' i; @7 o1 R8 F" @0 b
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>* e" X' E* M0 ~, M9 v$ P. {
</property>
& y2 h, _; E. L* d <property>
) q7 W' F% ^. F3 {' | <description>who will execute(launch) the containers.</description>
1 W, \8 w5 U+ m <name>yarn.nodemanager.container-executor.class</name> p. K( q, U: M( E# m
<value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value>4 g4 v# j- @9 x. |7 a/ X5 m
</property>
1 T, v6 E. x/ W, ^9 H- q4 t3 o5 x [ <property>
* A1 {9 ]' F/ @0 ^4 N3 J6 a. o% e; {$ ` <description>Comma separated List of container state transition listeners.</description>
+ s6 P0 L2 r; [, Y( o. B <name>yarn.nodemanager.container-state-transition-listener.classes</name>, n4 {- E9 a. S( Q3 ^3 A7 A
<value></value>
% [- h. a% G3 d2 b& d% E7 t! J ]; l </property>
% f! [ j& R0 T0 E+ C& c0 F <property>! z2 h7 M$ B/ x
<description>Number of threads container manager uses.</description>
9 g4 q; y3 y. }1 u% I" a2 t8 j+ y <name>yarn.nodemanager.container-manager.thread-count</name>
1 }/ n& y2 q! y3 E# m4 g/ t* E <value>20</value>
' h, g2 L$ a" U9 I </property>6 `& l8 I) g9 z' D+ D: z T
<property>, P8 n3 v( S* T! m# x* k% u6 @
<description>Number of threads collector service uses.</description>
" s {$ q' ~* D. B$ m8 H <name>yarn.nodemanager.collector-service.thread-count</name>: ], P2 n$ S& @ w z8 h/ r
<value>5</value>
5 n2 `! }; D" t8 j </property>8 d- J7 d0 @, ^
<property>/ Y3 G4 f& j! E2 I w! T
<description>Number of threads used in cleanup.</description>
1 B V. ^& I& [! B" n5 I <name>yarn.nodemanager.delete.thread-count</name>
1 k% Z! h2 W, W9 _+ d <value>4</value>
: e# f6 I& z9 I* X </property>, x7 B; _* H- I. Q* ?6 N* B, L7 o
<property>4 @' O$ `9 n ]- r n
<description>Max number of OPPORTUNISTIC containers to queue at the
b! S' z' U- w. W. b, H+ z nodemanager.</description>
. K# G! H3 F2 A <name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>
& S8 D" s' f* T$ N" E <value>0</value>
) v/ `6 k& ?5 @ </property>/ w: Q6 r( V# A) V q! r3 R
<property>
1 d1 s# D$ f1 ^" T' O& } <description>
_: n$ T$ F7 C! i Number of seconds after an application finishes before the nodemanager's
3 @1 |2 t) P5 J: u+ W& Q DeletionService will delete the application's localized file directory7 v2 m2 S8 D% E% ?; [
and log directory.
/ [) z/ m7 o! O7 `6 J To diagnose YARN application problems, set this property's value large$ `/ b( X: @6 V: f" j& l, i
enough (for example, to 600 = 10 minutes) to permit examination of these' L$ \0 E* A9 o
directories. After changing the property's value, you must restart the
2 D1 M" j/ @) `% S nodemanager in order for it to have an effect.+ G# j1 `2 U) m8 a$ F
The roots of YARN applications' work directories is configurable with
/ e& k4 s! g( o the yarn.nodemanager.local-dirs property (see below), and the roots
1 |- l# g' a( ?+ n2 ^ of the YARN applications' log directories is configurable with the
$ E9 p# a; b2 C' w yarn.nodemanager.log-dirs property (see also below).
; o' L. a) H, i! |; _ y* G. S" u </description>
/ I, b2 V* J$ Q6 m+ d: C; Z <name>yarn.nodemanager.delete.debug-delay-sec</name>" B1 Q9 D$ T( {5 Y6 h8 ^% |
<value>0</value>0 b: L$ A2 J. D1 U
</property>
1 K/ U4 F- Z; {/ V <property>
2 j7 Y- O2 C, a7 }7 | <description>Keytab for NM.</description># E( ?, T9 i O( u9 K6 R; [
<name>yarn.nodemanager.keytab</name>
% i' n. P6 O2 \7 t1 o9 k <value>/etc/krb5.keytab</value>
6 O5 y" x& i5 g7 H9 e+ H& I" N0 M </property>
& M* \1 G) W) u# R8 K) C: T8 V$ ] <property>
; Y$ Z0 E6 O e3 r; ^ <description>List of directories to store localized files in. An
& u: \5 {) \/ ?+ o# ]- _ application's localized file directory will be found in:* j/ I" k P& h
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
" M8 _5 f5 l2 b2 e. r! ?' a Individual containers' work directories, called container_${contid}, will8 h; A S% U7 E
be subdirectories of this.; b' @+ P% Y/ ]; G
</description>
- g* ?% O4 u8 m5 l7 w <name>yarn.nodemanager.local-dirs</name>
8 r9 C& k T% j0 i k+ b& p <value>${hadoop.tmp.dir}/nm-local-dir</value>
5 u. Z% Y: G: ^ </property>; K% b( c4 k' m$ M$ J
<property>/ j% c. h" t6 S7 x
<description>It limits the maximum number of files which will be localized1 ]' g) j" F' S5 J' P6 I, V
in a single local directory. If the limit is reached then sub-directories
" s/ g3 P. t0 X Z+ S6 Y, d& y- _" G will be created and new files will be localized in them. If it is set to
- U3 g% A6 k) M: `- S' S* c) r8 w a value less than or equal to 36 [which are sub-directories (0-9 and then
) E% g8 y# V3 [1 U6 T d! ` ?' u2 {# M a-z)] then NodeManager will fail to start. For example; [for public
$ w: J. B0 z, y) V% T cache] if this is configured with a value of 40 ( 4 files +
0 y/ z. V0 \7 f! e, j! c2 U- ^7 w 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will& f% F7 m7 A# \- a P
allow 4 files to be created directly inside "/tmp/local-dir1/filecache".: x" M( f' G5 t
For files that are localized further it will create a sub-directory "0"
" Y% a: f" p0 N: O" N* i8 \# ~ inside "/tmp/local-dir1/filecache" and will localize files inside it& N4 u, C/ x9 z% c1 O2 I
until it becomes full. If a file is removed from a sub-directory that
4 ]. j: i" {. D1 y3 U# @0 B0 e) f is marked full, then that sub-directory will be used back again to) f0 D$ L- z$ S1 }; B0 J
localize files.. s- _' R/ I+ I6 w1 Y9 H& t
</description>$ H8 P$ C8 j9 t( Z! J0 u0 P
<name>yarn.nodemanager.local-cache.max-files-per-directory</name>
9 W& l1 R* c- P6 [, m. k <value>8192</value>2 q+ Z+ F2 k. l: W/ ?+ T }
</property>; n, g B6 j0 Z/ v; j
<property># e) i1 b/ N+ H
<description>Address where the localizer IPC is.</description>: H( a. t3 G0 {( f7 Z8 K. H5 Z
<name>yarn.nodemanager.localizer.address</name>
0 z% `: v. Q X0 ?1 m" U <value>${yarn.nodemanager.hostname}:8040</value>
" {3 F2 \* v% `+ S- A& e </property>% M V1 k; M, N; Y
<property>
8 j8 d3 P3 E- c# D6 L <description>Address where the collector service IPC is.</description>
; y% o! T9 n: i0 c: _ <name>yarn.nodemanager.collector-service.address</name>
' K0 \0 `4 _) C0 H; d+ T <value>${yarn.nodemanager.hostname}:8048</value>
$ \$ D: r% E; _! G; Q5 h0 S </property>
" b5 J8 [% f2 b <property>" ^0 d3 B {2 `
<description>Interval in between cache cleanups.</description>2 V7 w7 ]" f1 M+ ^! L
<name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
6 |2 H* @5 y3 G2 k$ X$ N <value>600000</value>, ` |1 m6 Z" A3 Q
</property>
5 r6 V# K6 O6 W. j: A <property>. j* D( [& ]- B# v( q
<description>Target size of localizer cache in MB, per nodemanager. It is
1 M% y9 ^! q5 l3 s/ M& A* y a target retention size that only includes resources with PUBLIC and
t! @! `, z" I4 ]/ Z; K PRIVATE visibility and excludes resources with APPLICATION visibility
* x6 N3 \! |6 v </description>% ?+ y# T+ @3 t# N
<name>yarn.nodemanager.localizer.cache.target-size-mb</name>$ L% B( i* r1 S) O {* _! {
<value>10240</value>/ f+ f) l% n4 D
</property>
+ }% B' j' c9 }7 R0 f% } <property>
6 G/ A7 J# U. n* B7 L <description>Number of threads to handle localization requests.</description>
6 y1 w5 u( x! Q' u( i# i# u <name>yarn.nodemanager.localizer.client.thread-count</name>/ n" A1 H9 O. v* }4 d& e5 X
<value>5</value>
3 W1 @8 Y4 v8 l4 g </property>
% x, {- D/ E& y; v ~% ?; l1 e7 f4 F9 x: _ <property> x5 C% [' H# y. T
<description>Number of threads to use for localization fetching.</description>$ {- A0 H( ^7 N% F, U1 `( e
<name>yarn.nodemanager.localizer.fetch.thread-count</name>* x( o* H! l3 G
<value>4</value>& ]% a0 b; {; m8 J7 \
</property>
# c* v$ H% `9 f0 e9 s; y( w1 n <property>
7 V6 I* u; }7 }" p <description>
# w6 o4 ?6 J! u2 c9 g* o1 @) W </description>
& `+ `1 |6 r' H <name>yarn.nodemanager.container-localizer.java.opts</name>; j( ~) J4 C0 ]9 r/ M4 m! T6 s' \
<value>-Xmx256m</value>- @, ?4 z& f$ e k4 v7 l
</property>
w$ K+ Z3 w& P* ^% W* R; d <property>
" z3 Q6 u) @: e1 s$ D <description>- h( g. u6 h1 U3 \4 E* Y
The log level for container localizer while it is an independent process.
/ s6 C1 h) f! \& ]9 l* V </description>
% B7 P4 T6 I5 Y <name>yarn.nodemanager.container-localizer.log.level</name>9 t' j3 [- m0 Q1 Z( C
<value>INFO</value>
4 U4 J+ G0 ~2 S3 T# e </property>5 X R# Y% }1 P; S- ~# V
<property>
3 k& d3 x1 b7 l! f$ j <description> T' t9 Z/ a; y5 q% I+ O
Where to store container logs. An application's localized log directory
( @1 j, x% N7 P; M will be found in ${yarn.nodemanager.log-dirs}/application_${appid}.& N6 o3 F2 N3 N& l/ J9 J W% F* q
Individual containers' log directories will be below this, in directories
% M% T. {# x; u. Y* }' W named container_{$contid}. Each container directory will contain the files) t7 P3 k2 p9 A e- t0 E8 U9 G
stderr, stdin, and syslog generated by that container.* x' n3 v0 X" z' M/ e8 r2 K4 F
</description>/ h E1 E$ b. n5 W& ^
<name>yarn.nodemanager.log-dirs</name> r9 q6 f; U1 l
<value>${yarn.log.dir}/userlogs</value>( g; w1 B' Z1 m+ F2 T! R3 O
</property>
! B8 n$ }! G7 V; B* ` <property>% D0 [: l! G O% p0 L& X
<description>! R1 G" e! Z8 _+ y
The permissions settings used for the creation of container
+ c: L. a6 @6 o( k7 J: K/ d8 r directories when using DefaultContainerExecutor. This follows5 _& L% T. y3 J: ~* F
standard user/group/all permissions format.5 J7 A, `! K; Y: }0 l
</description>1 f& V6 q! ?* N4 F% ^
<name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>: k% a, n% a) B8 v
<value>710</value>
5 F# Q q8 U4 K/ `$ a </property>% w" I+ p' F5 ]" s+ j1 f+ c
<property>9 _9 N# ^/ s: m2 |4 n6 k6 m/ C
<description>Whether to enable log aggregation. Log aggregation collects
2 Q; k0 `% I/ J' S, d- G each container's logs and moves these logs onto a file-system, for e.g.
) c) X2 @! s, y6 }" Y; K9 T9 C3 k HDFS, after the application completes. Users can configure the
' O0 `* ], n) w& V+ l "yarn.nodemanager.remote-app-log-dir" and
9 f2 w4 G2 u. _3 z; R& F' B "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine4 `" G, }6 @3 v
where these logs are moved to. Users can access the logs via the
$ s( F/ ]$ m% Y7 \9 j Application Timeline Server.( {; w& n" k# b4 F/ y1 c+ A
</description>
7 J+ Q0 o O9 n( o" m! Q5 o <name>yarn.log-aggregation-enable</name>
/ j* M' `) |+ G$ @$ O) @* I <value>false</value>
/ l& [5 p& N8 b. s </property>; B( o0 F1 d2 C( \8 h4 L$ p
<property>
6 W5 M- `! j' _4 Z <description>How long to keep aggregation logs before deleting them. -1 disables.
* u/ E( g, k8 v3 t& G Be careful set this too small and you will spam the name node.</description>
# G% ? O1 I* o; s+ T <name>yarn.log-aggregation.retain-seconds</name>
5 l" j+ F; m$ Y/ h; j- B <value>-1</value> z: Z( o' N' Y! b
</property> : ]6 E9 r) D0 z$ `" c- \! F% f8 F: v
<property>
' K% U# h" H! z2 ?& l1 A <description>How long to wait between aggregated log retention checks.
$ i' b7 e* n9 \( m7 @7 t If set to 0 or a negative value then the value is computed as one-tenth6 d+ A6 {( E2 w1 V) `$ h' n
of the aggregated log retention time. Be careful set this too small and$ a5 k1 b* U0 z. n' [$ z
you will spam the name node.</description>
+ Z& K) b$ E+ U/ x% r: h+ s v <name>yarn.log-aggregation.retain-check-interval-seconds</name>+ `( P- n8 t$ G0 W
<value>-1</value>1 }' Q9 d7 N: C+ m" V
</property>
/ f) n% }" N( m* V1 M <property>" Q) D( C: ^- z( g
<description>Specify which log file controllers we will support. The first
% M7 Z o# d$ [: P( E file controller we add will be used to write the aggregated logs.
/ O6 A( O, \9 B% I' ?. i This comma separated configuration will work with the configuration:. n6 o, W/ k# j- d1 [
yarn.log-aggregation.file-controller.%s.class which defines the supported3 e3 _8 c0 m2 \* M# ]* \9 ?
file controller's class. By default, the TFile controller would be used.
' e2 e- o# [) I% q( p9 ]: F The user could override this configuration by adding more file controllers.
4 S) j! k/ {! Q3 I+ s3 Y To support back-ward compatibility, make sure that we always
: E. u+ J5 \+ |/ L3 s3 }- Q add TFile file controller.</description>
# x I* x" V4 K- Y <name>yarn.log-aggregation.file-formats</name>
' w+ {4 |4 S8 D& k3 S1 n <value>TFile</value>
& G$ c9 Z( y( P a! |3 I </property>
6 [8 U, { L1 }1 {- S <property>
2 b8 Q- J$ J" n3 M y+ L8 J <description>Class that supports TFile read and write operations.</description>
1 B( q# d- e7 [6 p) {$ y <name>yarn.log-aggregation.file-controller.TFile.class</name>
. Z; N% r% L7 m <value>org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController</value>3 n% e. _% I4 l0 h4 V0 O; C C2 ~
</property>
1 P {+ X: w3 ~7 a( Q9 W <property>! z. w/ a- z- j. O- ~+ Y
<description>
- b ^# u2 c: ~8 b# i How long for ResourceManager to wait for NodeManager to report its5 e- e; I2 r% \, `% \
log aggregation status. If waiting time of which the log aggregation
0 W) ^* M% _$ S: @1 ]- Q status is reported from NodeManager exceeds the configured value, RM
/ D* [+ \$ W% k* r# e9 s$ \ will report log aggregation status for this NodeManager as TIME_OUT.% v& `. `7 s4 {& {
This configuration will be used in NodeManager as well to decide0 Z8 P+ Z/ b" `/ W! u; K
whether and when to delete the cached log aggregation status.
I! s' f, x3 ?6 u; Z. S- H </description>/ I. n' ^% q: S" F( t& L' i
<name>yarn.log-aggregation-status.time-out.ms</name>) }0 \& ?7 v, Z9 p
<value>600000</value># H( D3 V4 t7 s) K" [/ I, N+ L
</property>
2 E) T! ]! q3 I9 |" }; s% { <property>
3 i2 Z& M) s9 q* [7 K" d <description>Time in seconds to retain user logs. Only applicable if- R8 Q5 P5 M' H6 G
log aggregation is disabled" y0 u0 f* s0 g) a! ^6 V* A2 s
</description>+ w9 f* \+ d" Q p: C
<name>yarn.nodemanager.log.retain-seconds</name>4 M9 N$ y" F9 h4 M% X1 H% D! s! a
<value>10800</value>
9 s" d1 r( x& d9 W+ U </property>
* i$ w; g+ t( @ <property>" ]7 l: z6 `' ~" J
<description>Where to aggregate logs to.</description>
5 p: }& V% B" j+ E <name>yarn.nodemanager.remote-app-log-dir</name>
+ s9 S5 o; |/ r; `1 Y* ^ <value>/tmp/logs</value>
+ H& I3 f4 J4 x+ q7 G </property>
: ]& ~" Q$ y1 n q3 t V1 B/ F <property>8 m9 p0 \* w( i: R
<description>The remote log dir will be created at 6 o0 V' }& F( _- Z1 @
{yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
5 ]2 r0 {0 _( g </description>2 C, b$ J1 q& s2 V w! L
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
' u4 e0 X3 e" X7 o E- _* S <value>logs</value>
* v* N' W7 f0 o* W, T) E% | </property>6 t: H. u6 v- J9 p4 s
<property>
. q+ }9 X3 U2 Z. t <description>Generate additional logs about container launches.7 } v% }5 N) c+ Z
Currently, this creates a copy of the launch script and lists the
: o, y& Y9 h) V& y directory contents of the container work dir. When listing directory2 [" T3 ]' a5 \; ]) G
contents, we follow symlinks to a max-depth of 5(including symlinks8 \, N' ^2 b# u+ [& m A M- L$ q
which point to outside the container work dir) which may lead to a5 K* @8 N$ P: k2 j P7 s. @
slowness in launching containers.5 ]8 @ q( D, w' u
</description>
- o( J) \+ L' _/ s <name>yarn.nodemanager.log-container-debug-info.enabled</name>, v, ^9 L$ o6 T7 \& V( b c( m4 ~
<value>true</value>
; e9 y: L; B8 \, z. I. v1 p </property>
b3 {: P+ A/ ]# |6 i <property># v8 J. ]5 V% ~
<description>Amount of physical memory, in MB, that can be allocated
* E7 Y7 |" _* i) s for containers. If set to -1 and
4 y3 Z; y/ d2 i yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
4 K: K) ?" n- p8 C7 D automatically calculated(in case of Windows and Linux).
8 K9 d6 B4 x5 Q! }- a In other cases, the default is 8192MB.
. g8 k9 O- [) a8 b) Q5 a </description>1 x$ `7 A! M, X* j" O$ a
<name>yarn.nodemanager.resource.memory-mb</name>
# [7 b; k/ A/ {6 _1 I: S" Z) d <value>-1</value>; y. j8 C5 o; B5 ]
</property>- Z& W, P! Z9 C# Y8 f0 \; G
<property>
+ s' h4 \2 x; [/ g5 V <description>Amount of physical memory, in MB, that is reserved% n2 J' {$ v& V! b$ j% V
for non-YARN processes. This configuration is only used if: ~' ^% n- |, b
yarn.nodemanager.resource.detect-hardware-capabilities is set
+ o, k$ G9 q: f* A1 R: }0 G to true and yarn.nodemanager.resource.memory-mb is -1. If set
' _; y( X3 H. b/ d to -1, this amount is calculated as
8 ~, g% n* b8 I: d 20% of (system memory - 2*HADOOP_HEAPSIZE)
* @# j! d1 ?4 ^6 S6 g- w" G9 S </description>/ E. n% q2 ]: N$ S6 {
<name>yarn.nodemanager.resource.system-reserved-memory-mb</name>
$ H& p; d4 f5 K1 a* e( ?4 c3 r <value>-1</value>
0 ?" k4 B; t/ ~- ~6 {" v </property>0 ?$ _- f! E: X1 v: {9 l
<property>7 E! ?. a4 U* |- Y, g8 b7 R* N* }
<description>Whether YARN CGroups memory tracking is enabled.</description>
O) c4 E" X1 w: L <name>yarn.nodemanager.resource.memory.enabled</name>2 F Z4 A: d% d( _
<value>false</value>
2 H0 s3 h5 l4 R2 ?; M7 `0 b" k </property>% j; `) _2 U; ^0 @4 G+ ~: ?& b: t9 L
<property>9 w0 b* @- P6 ^( ? W9 Q
<description>Whether YARN CGroups strict memory enforcement is enabled.; Q/ _( |$ T1 f/ h) K% i h# X
</description>
/ }& G9 N2 G* i$ t; P/ X K, S <name>yarn.nodemanager.resource.memory.enforced</name>
/ A) F/ n; p& X5 N3 h* R <value>true</value>, ~* A1 _: w0 Z7 A, u4 C g
</property>
/ H$ M. N) _7 i! f' L <property>) @( n% t% m2 d+ X9 s0 Y7 H- K4 H
<description>If memory limit is enforced, this the percentage of soft limit
( Q& ]. [5 o$ S( P2 {) M- Y! { compared to the memory assigned to the container. If there is memory9 y: v$ @- l+ ?0 j- X. t- x
pressure container memory usage will be pushed back to its soft limit
) \5 z$ {7 c+ K. R# @+ p by swapping out memory.
' ?: X- [6 \$ W$ j, T </description>
) {' T6 G% t! p1 S- k <name>yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage</name>$ V' S5 q1 b- f. Q1 V- U$ `* X
<value>90.0</value>6 X! V; I, N% a' ^, v9 E
</property>8 B! d/ k! O$ t- T+ F" W
<property>
1 o1 Y. j/ D$ F+ I2 Q <description>Container swappiness is the likelihood a page will be swapped
/ X; l- C- }, G$ ^ out compared to be kept in memory. Value is between 0-100.- x: [& D3 ~' }; u5 b; f! o
</description>+ v4 l% L# f7 n. o& B5 S
<name>yarn.nodemanager.resource.memory.cgroups.swappiness</name>
# e# }" j3 {# h <value>0</value> D4 u/ p" q, h1 t4 H4 \7 P0 d) J
</property>
4 s1 V, ~0 h2 G/ ]' U! X <property>
2 L1 v6 i3 X/ Y- S+ ^6 a$ g <description>Whether physical memory limits will be enforced for
* @8 @- `& ~9 \* |5 `& a! ^: r containers.</description>
# ^" U$ a; R6 a: X4 h' S, Q <name>yarn.nodemanager.pmem-check-enabled</name>1 g2 T F% S% l( G
<value>true</value>
8 f3 y! x7 M3 o& x6 f/ M1 l </property>
0 a" @/ |. H) C4 @* N l% J: i& Q* q <property>
. g0 J( G6 E( w' {1 K% T6 j4 A4 g <description>Whether virtual memory limits will be enforced for: ~8 p! X3 w1 s6 [* @
containers.</description>
+ |( m# H; m+ g+ F <name>yarn.nodemanager.vmem-check-enabled</name>
% a/ V7 H( W5 V( ], c/ ?/ o <value>true</value>- W4 [/ a V% O1 T$ l
</property># T2 U6 b R' z1 H8 s2 ~4 B
<property>% _0 n$ i0 @7 N% Y- k+ U% u
<description>Ratio between virtual memory to physical memory when7 U5 Z9 u6 h; ~9 ?$ f& y
setting memory limits for containers. Container allocations are
# l% k7 I; D) A% {9 m expressed in terms of physical memory, and virtual memory usage H& q% |; n3 `: O# t
is allowed to exceed this allocation by this ratio.# W4 O. q. p7 V) h
</description>3 v. Z8 x: L& k2 t/ V( F3 h
<name>yarn.nodemanager.vmem-pmem-ratio</name># ?1 X1 m/ V5 r7 H. `
<value>2.1</value>
) ~1 z" Q B, ~: V3 H- y# n) b </property>+ V- D! e1 r8 g A+ R/ Z; {
<property>
( m) [. W5 {& }- I1 @( p2 a9 ` <description>Number of vcores that can be allocated
* E8 ^# K+ e- ~- I8 V for containers. This is used by the RM scheduler when allocating$ z/ a) L* G6 d) Q# d: }' k
resources for containers. This is not used to limit the number of
1 [- B+ e4 ?9 @! ? \0 d3 g) {, f CPUs used by YARN containers. If it is set to -1 and8 T; [$ N) a- _& u Y2 |9 [9 Z% k
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is# M* n# g3 P+ c( E+ C
automatically determined from the hardware in case of Windows and Linux.0 {5 S( N: y: x- H- ~& ^
In other cases, number of vcores is 8 by default.</description>/ v D3 f7 O2 L8 X# M8 o
<name>yarn.nodemanager.resource.cpu-vcores</name>& q$ c! A' f3 M' H& G8 ], A. Q
<value>-1</value>
. c$ V0 }# _ e7 B7 c </property>
) X! e3 f' U/ W. z0 d. M3 Q$ v* f <property>
7 C5 M" ?1 X8 i% w1 Y0 p$ Z9 b <description>Flag to determine if logical processors(such as
A2 ]: k5 u# g9 _ hyperthreads) should be counted as cores. Only applicable on Linux
! c( T6 L( u; i+ s when yarn.nodemanager.resource.cpu-vcores is set to -1 and6 E0 [6 E1 ?2 B, S7 |
yarn.nodemanager.resource.detect-hardware-capabilities is true.
: v& d, X% Z! f9 f0 K7 I </description>
9 _6 r7 r0 F5 J O% B) s <name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>' E! `4 t! d1 ^, I. l
<value>false</value>
7 T9 X' x) ~. l7 j( e9 h$ ? </property>; z: P* M# O6 J
<property>
' O7 A4 U9 Y/ [2 q <description>Multiplier to determine how to convert phyiscal cores to: ?# U) M1 z: L$ F1 {2 j1 Q
vcores. This value is used if yarn.nodemanager.resource.cpu-vcores
6 S* c6 k, a3 V9 A3 W is set to -1(which implies auto-calculate vcores) and
& ^3 a" ^5 g0 g+ Z# x* n yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The
7 L+ j' D3 ~$ _/ w number of vcores will be calculated as
2 |7 i* q" g. r! N number of CPUs * multiplier.! y' R0 g& S3 c% x" t
</description>
. A, e7 r, n# k; r7 o4 ?+ N6 x <name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>
2 U+ d! y4 [. T" d1 }: s( Q2 r2 b <value>1.0</value>
2 ~& K# Z& a# Z# n, J, u </property>$ T) l- p# K7 X4 ?- O; g
<property>% Y \4 ^% T. G/ r- k4 C! _
<description>+ t" k% D: F$ `5 }% f1 F
Thread pool size for LogAggregationService in Node Manager.
5 I+ \2 B' o! _% X/ E </description>/ G/ h- q! N% X) k( l, F8 T% Q& }
<name>yarn.nodemanager.logaggregation.threadpool-size-max</name>
% m) W% {! i$ f/ ]! l: U <value>100</value>
2 p# Y9 ^& q! I5 b </property>
. Q8 o7 k4 O8 r+ M* z$ a b <property>1 C* D `6 r5 i
<description>Percentage of CPU that can be allocated
: F' p7 w3 K5 p4 g for containers. This setting allows users to limit the amount of
* S& q) M! y( c CPU that YARN containers use. Currently functional only
6 |8 f* i+ o) j+ i4 B* _/ B9 w7 S5 ]; p on Linux using cgroups. The default is to use 100% of CPU.
1 ?6 r9 I0 e c3 n, Q. `. E/ U </description>% ^; ]% O X* I% g5 K
<name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
" H! C+ I3 d2 @$ D" Q# D <value>100</value>
( h' G; I! @+ I- Q </property>+ a& }, c3 A) ]# B8 A. y5 ?& U% Q
<property>
+ p( z ~ \! u I" M: K& A" [ <description>Enable auto-detection of node capabilities such as. V& @$ u9 y! O
memory and CPU.: B6 d6 N; Z/ x c/ X7 B. j
</description>5 O% x/ t5 z1 v( O1 g
<name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
; A' I, c. V# F <value>false</value>$ [, H7 G# h* Y8 Y6 Q5 L* Z
</property>! ? d l1 u( t( m% b
<property>8 r0 e, V9 N1 p) e
<description>NM Webapp address.</description>
- Y' f# X6 M3 l9 `+ ~ Q <name>yarn.nodemanager.webapp.address</name>8 _" P+ ]4 }+ j5 ]1 w/ _0 Y& a
<value>${yarn.nodemanager.hostname}:8042</value>
7 \$ r1 q6 \# c# @ </property>7 W6 m5 d9 X' ^: g
<property>7 d0 r& B; A3 t m$ D- m. a, L- {' u
<description>
* L0 H: _* ]' x" b7 A The https adddress of the NM web application.
2 k6 g5 S2 g, k- O# P </description>- N- Q2 u, c: Z0 `! |0 ]
<name>yarn.nodemanager.webapp.https.address</name># h. S4 Z( l' {6 H. d: J
<value>0.0.0.0:8044</value>
& w' J/ X: A( A s8 V' K; T </property>1 s) c/ Z. c7 F
<property>
: P3 d' n7 U5 P" v <description>
# k4 [0 W* @3 L+ O The Kerberos keytab file to be used for spnego filter for the NM web
# a/ y2 s6 z& L/ [ interface.
) B6 k0 Y5 }) V% L/ ~ </description>- S9 K; ]6 N1 v$ t
<name>yarn.nodemanager.webapp.spnego-keytab-file</name>
8 E5 Z+ y0 r* `' L! ?$ x' l <value></value>
! m3 t8 w3 K4 I* H6 Y L+ y% u </property>
) s, q$ L7 C2 w <property>' S$ S D5 P" n, \
<description>
( w W( l. E# a; A The Kerberos principal to be used for spnego filter for the NM web
/ g9 W4 X( N. m$ \* ~: M- ^% B interface.
+ h& ^1 ~: f- N. R </description>( O* ~6 r4 _& N
<name>yarn.nodemanager.webapp.spnego-principal</name>1 k3 x y8 l" ?$ W, v$ Q
<value></value>- q" Q, t" @5 `& Z7 T9 z
</property>- n {# f# [( [) J2 D; R1 q% D5 M; }
<property>
1 t5 W% c/ W/ `% g; {: }% V* N <description>How often to monitor the node and the containers.2 H$ i8 V" S4 |. R
If 0 or negative, monitoring is disabled.</description>5 c' `' k# \9 B9 m" @- a/ j- |
<name>yarn.nodemanager.resource-monitor.interval-ms</name>
1 c" |( M/ h P: U <value>3000</value>. K4 W, E& \ Y; P/ O" q
</property>% ~7 J5 g0 q) E' ]" X, }$ b) U
<property>
9 k- h7 G% m5 Q7 g4 p/ G <description>Class that calculates current resource utilization.</description>
+ Y) X# Q+ K+ U: [9 i. A <name>yarn.nodemanager.resource-calculator.class</name>4 w9 j6 I* `# C
</property>
* ^2 k4 S. z/ B4 I) j+ J( p8 k <property> I b5 T- f3 C- j1 c
<description>Enable container monitor</description>
6 C. D( M- W E8 m" P <name>yarn.nodemanager.container-monitor.enabled</name>) i9 D9 |: P# d5 `+ s1 @
<value>true</value>) ]6 R( _! @* ^6 v1 L2 G" g
</property>
, M- N" \ \: B! Z7 J* o <property>
4 K; B9 d$ l2 j' L$ H) f <description>How often to monitor containers. If not set, the value for4 _' e3 D `: t/ }; h& k- M5 L
yarn.nodemanager.resource-monitor.interval-ms will be used.
. S* u8 A, d: o M8 _6 e If 0 or negative, container monitoring is disabled.</description>
$ |. P/ ?/ H, b' F& L& s8 X <name>yarn.nodemanager.container-monitor.interval-ms</name>
6 b5 g$ ^/ v+ }- ] </property>
4 N# Y( K* j. h1 e6 }7 q2 Y% o5 Z <property>
* T' x0 ~; W$ y' y4 d3 u( m( _ <description>Class that calculates containers current resource utilization.
7 e& ~$ Q+ U, B If not set, the value for yarn.nodemanager.resource-calculator.class will3 t; z" [) F$ d3 D" j! {1 c. o! X, ]
be used.</description>
9 G- a6 l! V: h3 G <name>yarn.nodemanager.container-monitor.resource-calculator.class</name>; Q/ P4 P# j9 l4 i( z- x
</property>5 v0 H# S: Q7 W, f. ]9 q
<property>! I f; v& m0 R2 \+ O
<description>Frequency of running node health script.</description>
2 `8 @8 i4 @/ @9 g9 S/ P/ j <name>yarn.nodemanager.health-checker.interval-ms</name>
4 r) k4 q+ n6 S. N& V <value>600000</value>4 d* H, M, I1 p: Q+ w7 a0 Y0 P4 i1 j
</property>- }5 ^: b5 E+ C5 r
<property>
6 j. ~5 W) a \2 {% q# p' e- \; F <description>Script time out period.</description>
/ A- k# U! k0 i/ f <name>yarn.nodemanager.health-checker.script.timeout-ms</name>" N3 X) B( i1 s! {4 |. c
<value>1200000</value>
, W' q2 p+ a) P2 u </property>% ]( }# Y* u- u' H% Y, B/ l( x
<property>
) S H* i4 f! b5 C4 A, E! K <description>The health check script to run.</description>
3 @$ u- e. o+ n; y. V$ O0 L <name>yarn.nodemanager.health-checker.script.path</name>
$ \5 t$ C+ v* S <value></value>
3 {/ U& L b( O6 A3 t2 V7 t0 K: V3 K/ J </property>
+ p4 A1 R4 C9 A& C4 u8 B <property>
+ k0 {' c d$ [" |- K <description>The arguments to pass to the health check script.</description>
/ i( P0 \, y, i" G# d- ? <name>yarn.nodemanager.health-checker.script.opts</name>
5 m: r/ h* s, `& F; c5 ~% W <value></value>
" e. D7 M: y" Z0 I) A </property>- e5 Y0 ]7 X8 G+ Z* W
<property>
K- d1 }, i- u' \! R1 |7 G8 I <description>Frequency of running disk health checker code.</description>
; p1 F! x3 z# g) f1 l8 @6 \/ j: D <name>yarn.nodemanager.disk-health-checker.interval-ms</name>
& X, y- p) A& p7 z9 M: ]. V <value>120000</value>6 i7 o+ S* V$ E7 }( H0 o6 k2 o
</property>
4 Q$ P" O0 l1 B% q$ |9 H <property>
5 Q1 k/ d% _4 e7 S; l G6 C A) W5 H <description>The minimum fraction of number of disks to be healthy for the/ |' g* h+ @7 R: {6 {% {) |1 `
nodemanager to launch new containers. This correspond to both
* U U8 l8 L2 m# z# v yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there
7 B- D4 _/ E7 d6 y2 z are less number of healthy local-dirs (or log-dirs) available, then
8 {1 l# F* Y* R: K5 i new containers will not be launched on this node.</description>" m4 O/ y: X0 @! u
<name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>$ x9 c( A& i4 O. Q% m: D
<value>0.25</value>
- M, }5 \ J( `& N8 ~2 k4 h9 F: b </property>
, ? g; K% j* c2 ^ <property>
% U b' g( y) B1 a& e <description>The maximum percentage of disk space utilization allowed after
8 ]! a: }# A2 {% s which a disk is marked as bad. Values can range from 0.0 to 100.0. * {6 c; O0 K; J Q
If the value is greater than or equal to 100, the nodemanager will check
1 V }5 ~" n: `, B- C for full disk. This applies to yarn.nodemanager.local-dirs and
3 {+ j/ @; v% g2 j8 B& }, @ yarn.nodemanager.log-dirs.</description>* m' A* P6 ^# S z8 P' @0 B
<name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>- @% I& D) O! t5 ?5 L0 n
<value>90.0</value>9 N0 N! b! Z3 a9 T; K5 t9 `. T
</property>" a+ s' r1 ]# z7 N* r
<property>
6 u, q) b5 P5 |% k6 ?1 @# n <description>The low threshold percentage of disk space used when a bad disk is5 L' [3 Y0 M1 ^" q: p* @1 ^
marked as good. Values can range from 0.0 to 100.0. This applies to" K6 f6 K1 n" }3 f+ N2 a8 r
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
$ ^0 Z' n0 H( y1 ]0 ^& L* S Note that if its value is more than yarn.nodemanager.disk-health-checker.
. c8 i. ~8 P, N5 a# Q5 I max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as
2 D: b4 J# d& h- L/ v$ {$ Y/ D yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage.</description>
# @2 {; L' x8 a1 a8 s5 R* Y <name>yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage</name>6 j5 S r* {- ^- {& y' B! w+ s: N& |
<value></value>8 }+ {6 @) |6 Z2 D
</property>
2 k) Z$ Y* W0 h5 }* t5 z t <property>
! O6 x I( @ V# l7 v! H1 N <description>The minimum space that must be available on a disk for
) H0 U% k2 W7 T5 u9 o it to be used. This applies to yarn.nodemanager.local-dirs and5 o6 A' z/ O0 V0 M
yarn.nodemanager.log-dirs.</description>
9 H# M0 R k$ Y1 q7 i <name>yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb</name>
- `0 P+ Q: p% i2 c7 Y <value>0</value>" I4 [% X8 B/ m5 m
</property>( o: {& m- }8 i% X1 |$ q
<property>: L9 B1 k/ L5 x% D
<description>The path to the Linux container executor.</description>. K9 \! x2 D6 b
<name>yarn.nodemanager.linux-container-executor.path</name>, u5 R2 O% I/ z# ^8 ]# }9 J
</property>
1 p. {+ n' M6 h$ Y4 }% j3 @ <property>8 j' S. K* Q6 f# [
<description>The class which should help the LCE handle resources.</description># a9 x E$ Q' [2 c
<name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>0 f# y* }0 L. l7 h5 z# V9 v
<value>org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler</value>0 ~! z2 ~/ ?3 Z0 i8 H \ D2 P% ^
<!-- <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value> -->
6 e$ B6 V) W6 R: d0 w) j$ W </property>
K7 ]" U; R* M8 r1 `7 n <property>
+ X9 r r- ~! E0 N- B, Z( A <description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas).
5 N) m/ Q# x1 A0 W( ~# `! I: N If yarn.nodemanager.linux-container-executor.cgroups.mount is false
" x4 Y: G3 n6 g* R4 _7 e (that is, if cgroups have been pre-configured) and the YARN user has write) E/ T! I" q6 E+ m
access to the parent directory, then the directory will be created.5 Y( N! T7 i5 ^% m. A- _2 s+ `% L
If the directory already exists, the administrator has to give YARN
. T# }* L$ k' @' R0 S write permissions to it recursively.! j n N/ ]! C0 d1 G6 o
This property only applies when the LCE resources handler is set to
% G; w5 ]- L% e" I# I CgroupsLCEResourcesHandler.</description>
7 O) i( i5 K4 P& O; W <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
0 n S3 M) L: d <value>/hadoop-yarn</value>
/ A! w+ F$ s2 E0 W( Q* y </property>+ N, ?" E! A" c+ V) ^
<property>
" C) d; C# y* |6 e <description>Whether the LCE should attempt to mount cgroups if not found.
$ W+ o" H4 B" e% y& _- T This property only applies when the LCE resources handler is set to# ~: X( `: ], |+ M8 q
CgroupsLCEResourcesHandler.
2 @+ Y* Q2 K( w# T </description>
h+ q% H8 U- D! B <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>8 I& `6 V& t" u2 {6 e% v/ w
<value>false</value>" A* L- P7 M* ^; r& t( p
</property>, p1 j2 V8 c1 P7 x6 D; U! L
<property>/ ?: t5 q$ M+ U9 t( w3 c/ o
<description>This property sets the path from which YARN will read the
# L* t# _! r' j6 t' U1 ~ CGroups configuration. YARN has built-in functionality to discover the/ a$ i. `" @; p5 P9 M) n
system CGroup mount paths, so use this property only if YARN's automatic
( x4 g& w3 i1 [3 l mount path discovery does not work.
$ z/ M, @0 {1 F0 x% S% a The path specified by this property must exist before the NodeManager is6 w5 l- z0 L9 L8 S5 @8 S
launched.
' \* q i9 q" `' H8 ?/ n If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,5 m& N) ^2 H* |# S
YARN will first try to mount the CGroups at the specified path before9 j ]9 \6 z4 i0 r8 `9 M3 k- [2 _; g
reading them.. ^" G. Y6 D; W1 Q
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to
2 e( d0 p2 O- V1 F- p6 I false, YARN will read the CGroups at the specified path.
: ]' M+ k# F& F/ e If this property is empty, YARN tries to detect the CGroups location.
8 k8 s6 B' J$ R. `! W8 Z Please refer to NodeManagerCgroups.html in the documentation for further) I- ]& \! V0 e8 @% }
details.& V$ a. m v1 d0 s4 u' p9 t
This property only applies when the LCE resources handler is set to9 T& B" E: Y. x, E
CgroupsLCEResourcesHandler.
# x% O- |; ]/ P% l% d. D/ w </description>( U6 \* }9 V$ A2 Y" O$ S/ k
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
7 p! W }' _9 g9 `2 E </property>
4 l3 h/ `5 d9 Q% L <property>
6 B* H! ?, a0 {3 @/ k# i. E <description>Delay in ms between attempts to remove linux cgroup</description>
4 `1 g( L& s% @8 @) F* C; M <name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name>
( p* q+ f t3 B9 a9 F. q <value>20</value>
# A U, T& G) A! { </property>
, @# v0 u2 R: c) Y8 o <property>4 Y' z5 M {* o; X! j% T2 I; ]
<description>This determines which of the two modes that LCE should use on+ P) [2 e/ c: V) K/ R" B8 R
a non-secure cluster. If this value is set to true, then all containers
$ g& Y5 C+ D9 y will be launched as the user specified in
! v8 o( _% Y) ^3 r yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If) t$ P. ^. U* R5 E3 i5 L l
this value is set to false, then containers will run as the user who N; B$ x: g) y" u( B+ u8 N* c, D! J8 b
submitted the application.</description>% b: {0 q! Y& Z9 A/ Z; u
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
3 P) p% |9 i% d1 g$ ^2 B& A <value>true</value>( Q* L- V$ T: [2 j
</property>
# f+ j; N5 I A `3 ^3 W5 t <property>
; m u' C& P" x) ? <description>The UNIX user that containers will run as when7 h% D9 E( `) o# m* V* p
Linux-container-executor is used in nonsecure mode (a use case for this
' a8 r" [: }2 c) _5 R* d% z3 D is using cgroups) if the, a& G8 q) `8 Q+ a% m, N: N
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is
5 O# q6 }* }# _. z {5 O set to true.</description>! U* y1 f$ Q: A. G0 _* W6 ?
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name>
B5 Q3 {& |! x1 t( U; i <value>nobody</value>
5 W. W, d8 e- ^8 h9 u- _" X' D </property>5 x) }4 n* g4 O3 T$ r5 K4 Y6 m
<property>
8 _& {, v' n, i1 U+ M* y$ Y <description>The allowed pattern for UNIX user names enforced by
5 g! F, m% U- I% [ Linux-container-executor when used in nonsecure mode (use case for this8 B& Y2 x) s7 ]1 {0 O
is using cgroups). The default value is taken from /usr/sbin/adduser</description>4 l4 E2 q% O* d& D$ O4 Q$ W
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern</name>
+ E' {8 L6 m/ \4 B <value>^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$</value>
% q1 a$ h% i# j L. H$ q. \ </property>
S8 K5 ~4 e4 s1 A9 Q8 ^% @ <property>) S* L9 {7 O, L( Q/ a2 r) I
<description>This flag determines whether apps should run with strict resource limits
+ X5 K+ y# I) G! d& M or be allowed to consume spare resources if they need them. For example, turning the& ]$ G) g* ~) o/ f9 i
flag on will restrict apps to use only their share of CPU, even if the node has spare2 O5 u0 k) ^) Q
CPU cycles. The default value is false i.e. use available resources. Please note that
( h/ T1 o c) s; h; z turning this flag on may reduce job throughput on the cluster. This setting does8 P/ `) W1 j, z% A
not apply to other subsystems like memory.</description>4 \+ |" {8 y& w& N% N$ a; U+ d! o
<name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>! y) Y1 ^9 S2 H) E
<value>false</value>. [& T3 y' ]; R; K
</property>3 G+ [8 Y( a9 G. d
<property>% g4 q& r( B# {% n
<description>Comma separated list of runtimes that are allowed when using* {) ?: M5 z! q$ ~4 T- F9 p
LinuxContainerExecutor. The allowed values are default, docker, and6 B2 S5 n, ~5 s1 t7 ^0 k0 k
javasandbox.</description>
8 o3 z" A9 C$ Y% h3 d% @. s <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
$ j+ ?- G: {- ~* w- e <value>default</value>
" P: ~4 _. Q2 q0 y/ q6 `" ~! ]2 N/ Q1 I </property>
3 ~& F( i0 `8 i/ L, l <property>4 B- C7 k, n% G/ ^; j1 M
<description>This configuration setting determines the capabilities
5 K; H" Q3 e M; r assigned to docker containers when they are launched. While these may not3 e$ m! {1 o1 Z* b5 i1 F
be case-sensitive from a docker perspective, it is best to keep these
; M" \! a3 n8 l8 n' d uppercase. To run without any capabilites, set this value to1 D* z6 r! e2 h" M+ `/ t) v
"none" or "NONE"</description>7 ]3 _! i5 a- z: }
<name>yarn.nodemanager.runtime.linux.docker.capabilities</name>5 W8 n7 g' l8 `0 K" @0 {+ U# m
<value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value>
: P4 F" E$ h% S& `2 H </property>
8 T. Q; m& j7 c4 i <property>
: @3 e- H! n& \8 e, Q6 Y6 e <description>This configuration setting determines if8 B8 n! R; k6 u4 n9 ]4 [
privileged docker containers are allowed on this cluster.. o$ K# m& h1 m
Use with extreme care.</description>' g4 E( v! d2 d& Z' N2 O
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name>
: Z( Z3 i7 E$ X: e1 f8 b& c- \ <value>false</value>( b* G; X8 \1 K1 C+ M6 l# o
</property>
( }9 a9 V& v' T F, N- n5 d <property>
0 T& j- t! @# }0 R |( F* c* N. C <description>This configuration setting determines who is allowed to run' u4 C( L4 {6 F, X
privileged docker containers on this cluster. Use with extreme care.3 W" z0 ]- M" _0 t
</description>) N- Z* p6 g6 q7 l6 P1 s$ M8 H
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name>2 ?$ a. p% n$ N- J, e9 v
<value></value>
, v5 F; I( {0 u! m) w2 ^ </property>. a, e* [; l5 x6 Z; x4 k& |
<property>
1 i$ n6 B- P7 D <description>The set of networks allowed when launching containers using the) [6 I) h7 }% f* y, _+ w, A. N6 C9 [2 x
DockerContainerRuntime.</description>9 ~# ?2 v2 `' C4 M+ D; c" J- V
<name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name># J( x, S" S2 L5 W; ^- a R
<value>host,none,bridge</value>8 a( L$ c5 L! r3 I
</property>
" N# L, s( L6 G/ G7 K( w* e <property>: @$ I+ h: j1 W* [ _$ |9 n# Y
<description>The network used when launching containers using the
x3 H1 M+ k1 ^( H& z: X0 R: G DockerContainerRuntime when no network is specified in the request
( r3 S2 ?+ e% k) v; y2 Y . This network must be one of the (configurable) set of allowed container3 V5 D; m& o" F' e" b. M: O8 [
networks.</description>
2 T9 [+ V4 Z$ t* b1 l <name>yarn.nodemanager.runtime.linux.docker.default-container-network</name>3 j( \7 X/ O% {
<value>host</value>
0 l/ U. z9 |( J: W' r% t+ L; l9 X2 K </property>; E, I w/ c3 `0 C; t
<property>; H1 |7 ^$ h- d" F
<description>This configuration setting determines whether the host's PID
3 @* A8 r& U. @2 k namespace is allowed for docker containers on this cluster.
) \) v6 Y0 Z0 D Use with care.</description>
! l# f( |) n# w I! Z+ M <name>yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed</name>
' J; N# w1 K0 v, s3 N6 J8 b <value>false</value>
" U! C+ }6 c- K+ w0 m' G </property>, U& |* W* F% ~, ]+ e
<property>0 W) E$ `% x; I
<description>Property to enable docker user remapping</description>- V" l; ~$ u' v3 Z5 l
<name>yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed</name>' K7 I. _9 h/ c7 Y8 x3 y0 T" @
<value>true</value>
4 w6 f% }2 u9 A </property> B: w* V: o/ f2 U' S$ q
<property> [ }, @7 x' I g7 R
<description>lower limit for acceptable uids of user remapped user</description>" N# S& U O2 _ q5 W' u; s& w% O b
<name>yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold</name>
6 p& _: G* `* q3 ]' e* l# [- h+ q+ H <value>1</value>
1 |; ?2 |' p# ~1 M0 V </property>& e r2 q; n8 n6 O4 M
<property>
- K5 R3 u+ g# r3 C/ c' l <description>lower limit for acceptable gids of user remapped user</description>
$ V" [) s' j. e4 H <name>yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold</name>3 t; Z! w, H' G9 o, g
<value>1</value>9 J/ \0 Q7 R' X$ w: f; l
</property>3 K* |# D7 B6 C0 i) Z3 ~: q+ L# S: K
<property>
O' W0 r' N# F5 ? M6 d8 e <description>Whether or not users are allowed to request that Docker/ V3 Q! e6 }! j7 x2 |6 X
containers honor the debug deletion delay. This is useful for
. ^# |# q# ?4 u; a$ e! A# R troubleshooting Docker container related launch failures.</description>/ W8 K0 l1 V7 J) D
<name>yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed</name>
! q2 @/ i! ?2 z <value>false</value>
/ c( e8 n2 J5 q0 X! @: x </property>+ |2 I3 J- y- X* B
<property>& W5 o, l8 O j
<description>The default list of read-only mounts to be bind-mounted& S! Q$ C) ^0 ^( N/ X# D
into all Docker containers that use DockerContainerRuntime.</description>
3 S4 X1 Y! P. t/ `9 | <name>yarn.nodemanager.runtime.linux.docker.default-ro-mounts</name>+ r( l6 C- |# N# i x4 a. ]! E U) N) f
<value></value>
, d6 H7 X% y, ^. j" g1 p2 D( O/ o </property>
, Z/ s% S# `$ O+ R5 ]# F <property>
/ e) G K, D/ I P/ ?. k p$ \4 R( M3 \ <description>The default list of read-write mounts to be bind-mounted
8 t% h3 ~. t9 G% t2 k4 b" u& A t0 B into all Docker containers that use DockerContainerRuntime.</description>+ J7 @: W+ x+ [ e" d# u; @3 u
<name>yarn.nodemanager.runtime.linux.docker.default-rw-mounts</name>
8 f, i/ v. B7 V: \% _ <value></value>% C$ I% b* J# {! {$ n
</property>" }, j# E& G C4 d5 e2 C: B
<property>/ B4 [( T, `& i% |
<description>The mode in which the Java Container Sandbox should run detailed by
1 y5 e- k9 x8 i the JavaSandboxLinuxContainerRuntime.</description>/ y/ j4 p- p# Q# V+ O1 w$ N, u2 k
<name>yarn.nodemanager.runtime.linux.sandbox-mode</name>; \/ S0 F3 Q4 p0 c, \+ O
<value>disabled</value>
) c# K8 Q' J6 |' c3 j) `3 Z8 I% [ </property>2 ^' n' n/ l3 y- H$ {/ |
<property>
: J7 V# R' n/ Y; T. ?8 m <description>Permissions for application local directories.</description>
! C; r: u5 R# z7 E# X; } <name>yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions</name>4 }3 ], F8 I. ]- t
<value>read</value>
6 S- T1 w* X& N% T2 B- E5 K </property>
. E5 o9 c+ q; g3 W" L. C* [ <property>2 z! |$ o9 k& \0 g# E1 q
<description>Location for non-default java policy file.</description>: q |. k d8 Z6 `! q
<name>yarn.nodemanager.runtime.linux.sandbox-mode.policy</name>4 p+ a1 N9 j, f9 d! V3 V, V" Y
<value></value>
$ i. F# u2 w9 j8 t" C2 K </property>% k5 W# Y: T, a: n
<property>
% G% b6 V" b9 ^0 \9 Z! w9 R2 R# C/ R, V <description>The group which will run by default without the java security/ W; i7 w r" A. I* L1 P
manager.</description> |* d5 |3 N$ m. v7 N6 o
<name>yarn.nodemanager.runtime.linux.sandbox-mode.whitelist-group</name>" X9 G5 e, ?0 N& p( F
<value></value>% e/ f5 E6 w/ y+ U" Q
</property>
- J) w" r3 D. g9 `, Y <property>+ R& \9 b+ E# {5 \
<description>This flag determines whether memory limit will be set for the Windows Job' T$ y9 [0 r) @& t% ?, Q
Object of the containers launched by the default container executor.</description>
]! M6 V' {7 a: z; q- ~2 p1 P <name>yarn.nodemanager.windows-container.memory-limit.enabled</name>
* v5 S1 W6 D; z1 d& O& Z* x <value>false</value>. S: Y- o, q6 p/ v# J. q
</property>
. t, U+ P& s* N: P3 i& { <property>& k3 V$ D+ x" |* m0 g
<description>This flag determines whether CPU limit will be set for the Windows Job
8 P, s6 {, k0 X, t' P$ x2 E# ^ Object of the containers launched by the default container executor.</description>
1 y* R8 b( I1 d) @4 \ <name>yarn.nodemanager.windows-container.cpu-limit.enabled</name>3 C3 B- M. g) G4 B g/ Y
<value>false</value>9 P2 ~( m" l* h3 w, S; \$ N
</property>
) z2 ^' {: L7 n3 @ <property>2 o- `* J1 J% P3 F( A' W
<description>
8 ]$ \9 g2 n0 c e M- T Interval of time the linux container executor should try cleaning up
% m/ g; n4 U& b0 K8 n cgroups entry when cleaning up a container., O1 Z; |9 q# V) v5 X5 x
</description>
2 g. _ C# c7 J, L3 `, u- C: j <name>yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms</name>
+ b! k; ^* y! C4 V9 f <value>1000</value>$ c7 O) j- E5 r a
</property>
! _& y9 }& `; W <property>+ j9 ?4 V$ e+ E
<description>+ F7 \* \* O9 n. S$ a( }
The UNIX group that the linux-container-executor should run as./ c/ ]- ?3 `, x, L$ g' F' I
</description>
% [. C8 k8 g# y5 K <name>yarn.nodemanager.linux-container-executor.group</name>% e0 n5 O- k, _8 d, Z
<value></value>+ k/ g& o3 \2 Q4 f
</property># z0 l O& W7 _" v/ Q# x' ~- m
<property>$ i" z4 I6 J2 R) ?
<description>T-file compression types used to compress aggregated logs.</description>& d, `5 X& o$ Z& c0 V# H3 j1 E
<name>yarn.nodemanager.log-aggregation.compression-type</name>
1 h) @( h1 [% g- U) w+ h <value>none</value>4 W v1 h: ?+ D b
</property>
" t) C$ h) S$ C. D0 F <property>
" T4 a1 g# m: @8 w3 q8 J <description>The kerberos principal for the node manager.</description> c8 K a% ?8 l" |* ~
<name>yarn.nodemanager.principal</name>( B1 I, w2 {1 q4 |" l6 q# }! p
<value></value>
' P- H. F8 ~& x1 [+ e5 g </property>" }' R. r) i& s8 t
<property>! r5 K6 j, T/ J9 M6 O
<description>A comma separated list of services where service name should only% r5 [& i% Y' V- O
contain a-zA-Z0-9_ and can not start with numbers</description> c- w3 U3 Q+ V! b4 r# w; ]
<name>yarn.nodemanager.aux-services</name>. x# _6 G' X( c6 ]5 S, Y8 @; S
<value></value>
5 p( p" [1 C+ V* w <!--<value>mapreduce_shuffle</value>-->2 m5 }6 ~+ D% \/ f6 y1 A
</property>) ~0 w9 Q6 s, G2 T
<property>
) M, L- T2 ]/ X$ V' C) p <description>No. of ms to wait between sending a SIGTERM and SIGKILL to a container</description>& W Z/ U4 U$ b( u+ j
<name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
( H' \6 F# D" ? <value>250</value>: @( X9 L+ ?) U
</property>" s+ \2 U9 e8 N2 I0 n5 `+ Q8 P# {
<property>- {* w A% h; H
<description>Max time to wait for a process to come up when trying to cleanup a container</description>' C! A: T3 D3 v* L' Z
<name>yarn.nodemanager.process-kill-wait.ms</name>
3 ~! Y% D& B+ e5 G: H <value>5000</value>
' o$ H* Q6 |5 {- x% j </property>8 l( e) r% O; `0 H u# |
<property>) F7 t4 D" G) U3 j# @+ X
<description>The minimum allowed version of a resourcemanager that a nodemanager will connect to.
# z1 `1 ]6 x& X# ^ The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is
, M9 b/ ~* k. s: x: e" Q" C3 c equal to or greater than the NM version), or a Version String.</description>( e3 R0 u8 S' |" _& F9 w9 M' |
<name>yarn.nodemanager.resourcemanager.minimum.version</name>
) n' H* a, O$ S: Y- W <value>NONE</value>
5 c3 M% L+ R6 w! O2 v </property>
( @8 P0 L) g. n <property>
* c. j4 h: C0 {1 Z" C <description>Maximum size of contain's diagnostics to keep for relaunching5 l1 M% o1 `, J0 H9 i; H
container case.</description>0 R- L1 a/ }2 `! @- e( L
<name>yarn.nodemanager.container-diagnostics-maximum-size</name>
& K7 ]# `$ q" z4 F( E <value>10000</value>
4 X& a7 E, L# `0 k0 K( [* w# E3 n </property>5 U3 h6 [& a# {6 t
<property>
, o* t6 q5 Q* b <description>Minimum container restart interval in milliseconds.</description>. j4 I# q. {" p7 G9 J
<name>yarn.nodemanager.container-retry-minimum-interval-ms</name>
; x2 j! l& D! Y1 s, o/ S- a. F <value>1000</value>
* V, y$ p- o& W( Y* R </property>7 a, |/ ^( s! d' o/ }
<property>
" ^7 B8 ^2 ~: F0 l5 x8 v: s <description>Max number of threads in NMClientAsync to process container
S# u8 w" f/ b. m management events</description>3 A1 q: v/ w7 o" R# M
<name>yarn.client.nodemanager-client-async.thread-pool-max-size</name>
% W* C9 s' s3 Y! o0 K( i$ s" I <value>500</value>& \( `7 {( m0 ]3 k, ~" V
</property>
1 F0 t( c3 q% K7 ]5 p <property>6 L" h4 i6 { M# O% S; |& L
<description>Max time to wait to establish a connection to NM</description>+ [3 }; n5 k: E; [
<name>yarn.client.nodemanager-connect.max-wait-ms</name>
8 C \/ k, T9 \3 `0 k& U1 Q <value>180000</value>
5 n! E5 P+ F5 H. n$ B" N </property>
- C% L' W9 L6 U+ f) \ <property>! F4 j6 p7 ?7 ~
<description>Time interval between each attempt to connect to NM</description>' t* f3 M7 s9 D3 V% u
<name>yarn.client.nodemanager-connect.retry-interval-ms</name>" N+ ?& d7 z6 Q+ M6 W3 O. n
<value>10000</value>8 |; ~: k1 E! O% ]/ \. [- F
</property>; V( g7 m1 X' ^
<property>
. f7 R7 k5 k: W+ F: i <description>$ ^: _6 s3 |6 |( J8 Y. n
Max time to wait for NM to connect to RM.) u* B. d2 P) s! ^9 E
When not set, proxy will fall back to use value of+ c5 U$ g! ^0 Z4 i* ^6 `: c, ]
yarn.resourcemanager.connect.max-wait.ms.
4 g, G$ C* ]# e2 s+ I </description>* v7 ?$ o# C3 V/ f) |+ T0 G
<name>yarn.nodemanager.resourcemanager.connect.max-wait.ms</name>
- {! ?& r r2 v' ` <value></value>
6 C# I! t& H! n) e% ~7 I5 W3 }2 J( G/ e </property>
& `' ^$ Q5 ~+ P/ y; h' P, {. Z <property>" S, B/ D( P& V& x& e+ L
<description>
: @4 w4 ~) l. M- o; I3 } Time interval between each NM attempt to connect to RM.1 H. _* |) \9 v) W+ t& ^# b
When not set, proxy will fall back to use value of! f# b" D n3 h. J, E* ] ^! y3 i* E
yarn.resourcemanager.connect.retry-interval.ms.
* F' v, ?. ~1 n7 w" y# i8 n </description> ?- v+ x' [& Y- G( ?* m# l$ O% m
<name>yarn.nodemanager.resourcemanager.connect.retry-interval.ms</name> Q. ]$ v: J9 M5 |7 W5 I7 k
<value></value>
- p. g" \! F$ ]) S2 \ </property>
* g0 {9 x C( t2 m5 e/ @; X( ` <property>/ [- E0 b; \4 ?
<description>
2 S. K C( j0 V9 P6 m( | Maximum number of proxy connections to cache for node managers. If set( ?7 {2 l+ Q7 N1 K5 g2 l2 M
to a value greater than zero then the cache is enabled and the NMClient
2 C9 {% u9 E9 Q& D# f and MRAppMaster will cache the specified number of node manager proxies.
2 v- Y; b+ W/ V There will be at max one proxy per node manager. Ex. configuring it to a% A1 t& N( ]" ~2 I% x
value of 5 will make sure that client will at max have 5 proxies cached
% M' A! ?# H6 W& B% I1 s2 M# s with 5 different node managers. These connections for these proxies will
/ z6 M& n5 [. R4 t( t& m0 g( u8 D be timed out if idle for more than the system wide idle timeout period.
3 G# y$ T! B/ T0 Y- y Note that this could cause issues on large clusters as many connections: ]% n- u9 z* u; n' s+ ~' y6 S
could linger simultaneously and lead to a large number of connection
, X# t5 U# m5 H3 c threads. The token used for authentication will be used only at
8 H E( L, L' I connection creation time. If a new token is received then the earlier
0 N) U3 ]1 m) S7 N3 D1 b$ Q0 S connection should be closed in order to use the new token. This and
$ Q/ p) Z X! O8 X$ D. X (yarn.client.nodemanager-client-async.thread-pool-max-size) are related; k. d0 O3 o$ `2 Y) v5 U" k
and should be in sync (no need for them to be equal).5 K# |) A8 B* K" H
If the value of this property is zero then the connection cache is
/ s9 F2 D1 [6 ~2 \( D! l+ S2 N disabled and connections will use a zero idle timeout to prevent too
+ T! a5 a; _( i many connection threads on large clusters.
K# S) |2 A" K7 V/ e. F6 _, k </description>3 B6 e: O1 |2 \! q' `
<name>yarn.client.max-cached-nodemanagers-proxies</name>7 i/ v- x8 h7 e+ w z
<value>0</value>
% Z' l2 Y# @7 ?- Z* l4 c; ` </property>
* u6 @/ u" R, X4 u) H4 s <property>
- P& i/ Z% a" I3 h& B M <description>Enable the node manager to recover after starting</description>
5 S' k) Y7 }) x( o4 g$ p <name>yarn.nodemanager.recovery.enabled</name>
% z+ t. ?0 S) u0 c <value>false</value>
( q% P0 @5 [5 ?9 M( S9 ` </property>$ ~% [' b: f0 J
<property>
9 H4 J' ?$ ]; n <description>The local filesystem directory in which the node manager will% r2 s m) s0 B$ f/ w$ [
store state when recovery is enabled.</description># W9 \8 F9 ~1 i0 R% L
<name>yarn.nodemanager.recovery.dir</name>! M3 F3 B8 n, I) @' d- i
<value>${hadoop.tmp.dir}/yarn-nm-recovery</value>) m6 C5 F2 }" i: E8 E% ]
</property>; ]' h9 }6 `! x3 f1 ~* V( y
<property>9 c# F: [: h' W O$ K. y3 p
<description>The time in seconds between full compactions of the NM state
* Z/ Y' `2 ~% Z+ T) f3 g9 S* ?$ `9 g database. Setting the interval to zero disables the full compaction
# _: x* e# e1 P) ` cycles.</description>/ j2 a: F3 U2 ]# i
<name>yarn.nodemanager.recovery.compaction-interval-secs</name>+ m6 Z3 f) Z' @! e" J! E
<value>3600</value>/ G. e, I( Q4 t. Z8 @
</property>$ ^# J r* e1 M
<property>+ |$ A, O9 O) t( s( ^# G2 a8 b$ y
<description>Whether the nodemanager is running under supervision. A
) Q4 @1 Q- I3 \5 X nodemanager that supports recovery and is running under supervision5 p: i- a2 ]5 B4 @* v
will not try to cleanup containers as it exits with the assumption
" F5 ^. B1 ~4 G* G6 d it will be immediately be restarted and recover containers.</description>
# S. y- E* q" y' a# g <name>yarn.nodemanager.recovery.supervised</name>1 P& z7 @7 O1 _$ j$ _ X/ g+ r% E
<value>false</value>4 T+ J3 z: s+ l; M7 @2 n
</property>" m4 d3 t# |$ s8 S' g$ a/ q1 A
<!--Docker configuration-->. K' i! V2 V8 J. I
<property>& F" b( `2 L5 z7 H9 m& L$ d( {; S
<description> ]" l: f4 l/ V$ p" Q! s
Adjustment to the container OS scheduling priority. In Linux, passed3 Q0 v, r" i/ y) v% d2 E/ V$ K( O
directly to the nice command. If unspecified then containers are launched
! x3 j8 }& b( m7 W' e: H" w$ T without any explicit OS priority.
* U! ]) Q( e! ?0 Y$ F( h* J; P </description>! ]7 k4 ~+ k6 n7 v0 I j8 |* H
<name>yarn.nodemanager.container-executor.os.sched.priority.adjustment</name>
7 ~1 {9 }) C' q </property>
& z: h/ _+ h8 S" l <property>
/ }# R, _6 `) D5 M/ E <description>
8 o' Z, D5 u% F. t0 F Flag to enable container metrics
# _+ W7 ?4 U4 b1 X; h9 r% | </description>
) q3 P) g2 U7 v9 g* l <name>yarn.nodemanager.container-metrics.enable</name>
/ g8 t9 L9 ~; f+ |$ p' z9 O <value>true</value>
; J3 v8 j) y1 V3 y </property>5 ~4 D8 e; {6 d& y
<property>
# I5 R/ z" |1 m6 m3 m2 e0 H <description>
z+ Y$ N& {% b Container metrics flush period in ms. Set to -1 for flush on completion./ l: H9 q9 F& ~- v0 H, b
</description>/ F3 b) v: c% I1 Y9 y; J2 @
<name>yarn.nodemanager.container-metrics.period-ms</name>
U4 K+ `5 i3 h% _0 t( [ <value>-1</value>
9 V. |" b2 o9 a4 a </property> }3 E. h% }7 {1 Z5 j$ n' W
<property>% ~. q K6 e* e' ?) A
<description>9 i {' W$ D O1 i0 ]: _) b+ M& d9 i
The delay time ms to unregister container metrics after completion.6 t4 L& o' d9 j4 f$ M0 U, i
</description>) @) I( O' T$ B6 D- `
<name>yarn.nodemanager.container-metrics.unregister-delay-ms</name>: X- q; M: |; X" f
<value>10000</value>% T$ e- x' ^; Z' V
</property>
' E( o9 p; {% k3 C5 @ <property>/ _+ X- D/ o" u% |* e$ ^2 C5 ], C$ T
<description>- f6 f' k* r! M5 \7 P
Class used to calculate current container resource utilization.6 x T7 x0 y$ a; }8 \( k; V
</description>6 ?( k! X3 r/ ~* ^' d9 d$ @. r2 @# n
<name>yarn.nodemanager.container-monitor.process-tree.class</name>
- f/ _5 N8 d1 n, C <value></value>
" `/ d) r" C7 S; Q d5 [ </property>2 Y" s& s! @# Q; Q# J+ H
<property> t6 O* T! Z4 N, Z$ v& R
<description>
, o! r) b+ ^' l" Z* d! { Flag to enable NodeManager disk health checker/ N2 Y, r' y. a1 R3 j& E a" t
</description>
4 R8 q4 ~4 s& H# w* r; P <name>yarn.nodemanager.disk-health-checker.enable</name>
, [$ {; B/ p( M9 a. i7 I; z <value>true</value>
: f* y2 U/ [# Z/ l </property>& ? |9 U% r: S* B, [
<property>
5 [! A) I! P& V* r <description>
! A& y3 ^( m0 j m Number of threads to use in NM log cleanup. Used when log aggregation
8 E( L& l5 d' f9 I8 l$ P is disabled.
) E0 z1 S' v8 Y7 M5 Q; u </description>/ F3 X: h/ R* w) w. |& E& {
<name>yarn.nodemanager.log.deletion-threads-count</name>
& `' \* F O; u" ] <value>4</value>1 v/ s) n. D) L
</property>4 {( V0 y* P- J3 u# |2 e. a1 }4 ~
<property>
4 p+ K* U- }3 l) ^+ z2 n: B& B9 q <description>8 r" k# H; F: l0 c& S! _8 f
The Windows group that the windows-container-executor should run as.
5 ?0 Y- [5 L& ^3 I2 _ </description>* x7 W& K! D7 S
<name>yarn.nodemanager.windows-secure-container-executor.group</name>/ j) m. M# y9 U# O% g+ G
<value></value>; O+ {! h5 C9 d' M6 f
</property>
: s$ } Y$ F( s1 x0 [2 w <!-- Map Reduce Configuration -->
n; E8 U3 Q. p7 M <property>
8 k5 [# ?, X' ]: x, e <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>0 i8 p3 |, M6 ~- ]" ^+ ^
<value>org.apache.hadoop.mapred.ShuffleHandler</value>0 b3 k4 H5 n" v, S) k
</property>
1 k* f* b, N5 ?8 d <!-- WebAppProxy Configuration -->
& ]5 Y% @, @% h, W6 V <property>
/ m9 c. v! h! m' f& v; {0 x <description>The kerberos principal for the proxy, if the proxy is not
/ O4 M% t- m* e1 ]& s1 A3 D running as part of the RM.</description>; f9 ~6 d+ e5 d) E- r+ ?2 s
<name>yarn.web-proxy.principal</name>; L5 D/ s* l# v
<value/>
' w2 Z- q7 P3 C. c0 J. I </property>
4 u* B7 z5 s9 b6 V <property>
) U0 s+ X5 h& B; r X( E <description>Keytab for WebAppProxy, if the proxy is not running as part of
\+ a. |2 W! ~0 t* m the RM.</description>0 K3 z) f- ?4 [1 I4 ]: o5 Z- V
<name>yarn.web-proxy.keytab</name>
4 t1 y: J; W# q+ W </property>- L S; L, v1 J9 j. x$ |
<property>, i9 Z% h4 Y# V- t7 x# ~5 s
<description>The address for the web proxy as HOST:PORT, if this is not* ^" T4 T3 g% k' D+ m
given then the proxy will run as part of the RM</description>. S- ? z$ [, m7 D
<name>yarn.web-proxy.address</name>9 t7 }; l; |3 r6 F6 _: d5 Y, `
<value/>
3 u; Z. ?7 o2 S+ ^ </property>9 b# J* } |( f# s5 ]9 N7 C* ?
<!-- Applications' Configuration -->
! Z; ?, `$ Z' T/ o! G <property>
3 X3 w* d6 r+ S2 ]8 z% m% h6 B <description>8 }1 ]; I) D- t
CLASSPATH for YARN applications. A comma-separated list
& u( J1 r+ |* N+ Q. s of CLASSPATH entries. When this value is empty, the following default
$ h* G1 ~. M4 \* Q& k! A CLASSPATH for YARN applications would be used.
! G2 m' W- `. l( R For Linux:
, m) e! L) c! @& w: I# f1 {2 b, r $HADOOP_CONF_DIR,
0 _$ W, w- C2 ]* T $HADOOP_COMMON_HOME/share/hadoop/common/*,
9 z, C" q0 e9 [ j+ \& { $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,/ E- p2 f b* p& ^5 `
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,; ]- F. S9 S+ }# n6 N
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
/ Y, Y% Q1 L# i: R8 d $HADOOP_YARN_HOME/share/hadoop/yarn/*,8 I' R8 f/ H! M1 k" M4 `- a
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*: _$ X9 T% A% r) g
For Windows:
# S5 g5 w l" E- u7 c %HADOOP_CONF_DIR%,; {4 W1 P4 [1 j7 V, ?4 ^! g
%HADOOP_COMMON_HOME%/share/hadoop/common/*,( ^% b2 g0 E' Y* C% ~* g3 S1 G
%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,
# n( c: s/ _; g) I1 ` %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,
/ _/ o& c9 k l/ H %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,
+ j/ k; K- b6 c1 @* C %HADOOP_YARN_HOME%/share/hadoop/yarn/*,0 J/ A: o: A$ r. t7 P9 P V" ?
%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*/ t/ |. L- @0 C, L
</description>+ I0 F7 w/ ~( O8 @
<name>yarn.application.classpath</name>
; w; N7 f1 n( @4 b+ J, r <value></value>
# M6 B: J" Q2 C# J$ p; g) q' k </property>* Q! l) N$ y6 X* N
<!-- Timeline Service Configuration -->8 O5 T4 p( ?( }: M/ V" U! `
<property> e+ M0 R+ t. `( ~
<description>Indicate what is the current version of the running1 [- P) M2 T6 c( M* o
timeline service. For example, if "yarn.timeline-service.version" is 1.5,
. v6 e: J& q9 B and "yarn.timeline-service.enabled" is true, it means the cluster will and
3 g( w5 V C8 D/ A2 L3 Y should bring up the timeline service v.1.5 (and nothing else).
- k1 C$ `7 D1 U/ I8 T On the client side, if the client uses the same version of timeline service,
' C& F& C% u+ G( a3 `, J7 ] it should succeed. If the client chooses to use a smaller version in spite of this,, D1 i& \ w: ^! \, e7 p4 R" ^' [
then depending on how robust the compatibility story is between versions,7 X# p. |8 d2 R9 i# o
the results may vary.
, n) S0 e6 o: ~ </description>
$ ~* _- z+ M4 F/ _3 s <name>yarn.timeline-service.version</name>
9 Y* _ T/ l9 | <value>1.0f</value>) L. h5 l1 C% b2 Y: e
</property> J( X; r# }3 n, c% I' Y. P( K$ v
<property>
( C6 T3 b7 z5 p <description>
9 H+ S0 b y& ^% n9 B7 ^0 z In the server side it indicates whether timeline service is enabled or not.9 M7 \" {! g( y6 L4 L# C* g4 D
And in the client side, users can enable it to indicate whether client wants
/ m y4 k4 ~, u/ i& n2 i to use timeline service. If it's enabled in the client side along with1 }1 ?# s( j7 h3 l l- L0 ~ j+ \$ t
security, then yarn client tries to fetch the delegation tokens for the- m: D* K, w5 `
timeline server.
. R! y8 T5 A2 u h9 F4 Z- _7 h </description># e& e: z2 a9 ~# W+ [
<name>yarn.timeline-service.enabled</name>
( v, n7 @; {% t/ U, k <value>false</value>
6 j0 ]3 @8 h- U9 H" S </property>' Y) ]' a9 q+ B; H! L; S5 Z9 T
<property>
6 D! G5 K# n% i& A- w <description>The hostname of the timeline service web application.</description>4 Q- m- C3 }& J; e) l
<name>yarn.timeline-service.hostname</name>" N) w5 `; s& W9 d5 s) i
<value>0.0.0.0</value>' A+ R& b6 V4 T- ^" g* W0 \; \" U
</property>
7 B' |& Y4 X9 X <property>! N8 }& y4 A# O" f0 D, x5 \- x/ L
<description>This is default address for the timeline server to start the$ X4 Q- Y3 J( H4 c+ D8 h8 y% T9 @
RPC server.</description>
3 \: m1 t$ c ]3 @, j <name>yarn.timeline-service.address</name>: n6 t0 Y. ?: B9 O
<value>${yarn.timeline-service.hostname}:10200</value>: m* ~. t; [. ]; t
</property>
' x9 C' J/ Z7 F6 t1 n <property>
3 K4 ~( `( i( e" A ~# K. S: _8 [ <description>The http address of the timeline service web application.</description>, ~' J5 I1 F! n5 {3 o
<name>yarn.timeline-service.webapp.address</name>6 q: e6 y& h, h- z
<value>${yarn.timeline-service.hostname}:8188</value>/ m8 `# q. S3 \3 n) q7 }! a3 I
</property>
! W3 z5 S w$ d4 L' @0 n! L <property>4 ?) h7 \8 [& x, E3 b) G- ^
<description>The https address of the timeline service web application.</description>1 D6 W- ^) c. p" ^
<name>yarn.timeline-service.webapp.https.address</name>
$ v5 h9 O8 a3 q; X! {- x) o1 h <value>${yarn.timeline-service.hostname}:8190</value>) c( ]4 D. r* g- P
</property>
- z0 A& O; A( E <property>: a; ^/ W9 {. d, G* w$ x+ q, s
<description>
) X3 G0 }/ d9 _& M( p The actual address the server will bind to. If this optional address is v7 m3 A6 a& j7 u3 a) s4 P- A
set, the RPC and webapp servers will bind to this address and the port specified in
) r' Y% T6 ?% G yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively.9 s3 f* ?' g. d' n1 d( {
This is most useful for making the service listen to all interfaces by setting to4 J0 s. z5 E+ s6 V
0.0.0.0.
" }, L& ~$ y6 M& L2 B </description>6 S0 a4 ?$ m8 [6 |5 f& }
<name>yarn.timeline-service.bind-host</name>. s3 v9 o8 U. t3 O: h8 y5 R+ n
<value></value>. i) Y, n; ?4 C/ c6 C& E ]
</property>
f; u( q' e2 M( h: v! X5 r <property>
" Z8 w7 u& U7 W0 A) {: i <description>6 t; H) A# h4 v' X. k/ y" S$ b
Defines the max number of applications could be fetched using REST API or4 [9 b) [' }$ P: w H* j
application history protocol and shown in timeline server web ui.3 g. j8 u0 ~" J- x& K9 q1 J
</description>2 T' k6 k1 U+ w
<name>yarn.timeline-service.generic-application-history.max-applications</name>& k; y' ~3 C" M' Q5 W, l
<value>10000</value>! c. `% r% F1 X' V; l
</property>" P8 I' m3 M2 q4 u& V
<property>
0 k% l( y) j/ @$ q; t <description>Store class name for timeline store.</description>
2 [( Q9 @0 O( o6 @1 H <name>yarn.timeline-service.store-class</name>
/ E: S R5 m2 ^7 ?2 a* L7 T <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
6 L5 y( Z) N1 F1 b. l" g- Y U1 t </property>5 N/ Q0 E: `' S( d2 |! d
<property>
+ ^1 F' I7 S& r& |" ] <description>Enable age off of timeline store data.</description>
. o+ t2 P, }2 h: N, w, @! Y <name>yarn.timeline-service.ttl-enable</name>( I4 q) W7 H& ]5 }" `7 ^
<value>true</value>7 f) s& Z7 A" e% O' q
</property>
) O1 O: @- C( Y4 t: `6 @ <property>" Q$ N; P a+ H
<description>Time to live for timeline store data in milliseconds.</description>
6 P) s; [! G# S# H <name>yarn.timeline-service.ttl-ms</name>
+ X2 V( Q* f9 U: e. O) Z <value>604800000</value>
0 ]8 x5 T5 w. t- J# r </property> R* q+ m7 S( y9 H
<property>9 x' M/ s( ~: d
<description>Store file name for leveldb timeline store.</description>- `5 M6 ^3 a% R, Y* q
<name>yarn.timeline-service.leveldb-timeline-store.path</name>
: J$ n ^. W9 w5 W9 m <value>${hadoop.tmp.dir}/yarn/timeline</value>4 U/ ]! s! k! n9 a
</property>, d# |& E! ]) _! N
<property>: R( i1 c; g9 e. h) K
<description>Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.</description>4 ^# }. K! e% C W1 ^, D
<name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name>: H) n& K- \7 O
<value>300000</value>
+ M# B- b# y; e7 K7 V </property>; E$ Y$ s9 g7 D/ X' \' r
<property>
1 `0 y! m! i" g, ] P K <description>Size of read cache for uncompressed blocks for leveldb timeline store in bytes.</description>
, I6 I+ B/ m/ ~8 U1 c <name>yarn.timeline-service.leveldb-timeline-store.read-cache-size</name>
$ x+ l% H0 i4 M5 v- F" q: [+ B4 ~ <value>104857600</value># B" j+ h* `. c" D0 k+ r' q
</property>
5 I( Z" M0 I; O( i- \ n2 k <property>
8 j2 k: |8 G( T <description>Size of cache for recently read entity start times for leveldb timeline store in number of entities.</description>
% g+ T+ T* K; m% o. E7 h <name>yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size</name>7 o0 T; Z8 u0 T, q, B- m
<value>10000</value>
& L# \3 x: N9 E# W </property>) I- ?# t( Q& r$ g( ~$ U
<property>
/ X- m7 s N& n9 }( K8 E g) i <description>Size of cache for recently written entity start times for leveldb timeline store in number of entities.</description>
) N# V; @7 f2 l1 [, Q <name>yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size</name>
+ C1 g ~ T+ V8 b; I' _ <value>10000</value># p/ q0 ~* e+ W$ F* F
</property>: d8 K! w) a1 W8 t
<property>
% F- W% Q* e# K/ _- ] <description>Handler thread count to serve the client RPC requests.</description>
, T b- [( o% D7 h. k3 u4 _ <name>yarn.timeline-service.handler-thread-count</name>
4 x% s# d* H G& V: M <value>10</value>% }: }7 F5 W5 c6 M1 g) ?
</property>
( N: r; _8 B+ C, W& \9 r+ w <property>0 M: T( P9 s5 Z# d- ~- g& ]
<name>yarn.timeline-service.http-authentication.type</name>
9 b9 D9 {0 _9 C: D3 z <value>simple</value>0 W- W8 r6 F2 w8 U8 B [
<description>( @5 R5 t8 q8 W0 B7 N+ p4 m9 A
Defines authentication used for the timeline server HTTP endpoint.* J! i- |& n; o
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
: m. e, h0 P9 Y i# |* o </description>+ e) R+ z/ R* S9 X: G0 R
</property>8 B4 \7 {9 L- X8 V
<property>0 x% a' _ w/ M8 L9 H
<name>yarn.timeline-service.http-authentication.simple.anonymous.allowed</name>
" L4 a4 m0 F# C9 l/ x <value>true</value>* A, b$ \$ F0 G' x
<description>
0 C& ~" w8 J3 v+ y" }: E Indicates if anonymous requests are allowed by the timeline server when using
8 p8 v: {' H. ~! v3 Z0 Q 'simple' authentication.
0 \) p0 }! T/ c( r- K/ s6 @# X) a </description>
, f" B/ e/ W: t( w </property>. @5 T, b' [" {' Y* b8 z
<property>2 W% P, M3 d @- F( C: \! {
<description>The Kerberos principal for the timeline server.</description>
' U( \$ k2 L% j <name>yarn.timeline-service.principal</name>
; N$ P! a6 y0 N: g9 B <value></value># K$ N$ c6 O: _( F6 n L# {+ |' O
</property>
" y% d2 s, f7 {! n3 b <property>
/ O; Y8 ?4 O, X) Y4 M1 l4 T <description>The Kerberos keytab for the timeline server.</description>
0 s Q+ v5 d+ Q \7 i <name>yarn.timeline-service.keytab</name>
( I2 p4 Q* ?# e6 Q <value>/etc/krb5.keytab</value>( n2 E+ M% H8 F& j3 j# c2 `
</property>
8 d1 {0 L' S$ I8 d3 O% K( ^ <property>
1 J, q5 \9 f' i& [0 h, f3 q! J <description>Comma separated list of UIs that will be hosted</description>
* |) n& v) {' Q! O. Y <name>yarn.timeline-service.ui-names</name>" Q0 H9 Y" O/ y
<value></value>
8 c2 y& w2 p$ q' e </property>, l$ l0 v& T; J1 V
<property>
+ M n- [5 u4 K1 x. i8 V1 U5 P, J <description>9 w5 C; Y2 N/ {4 [5 M8 `& c
Default maximum number of retries for timeline service client0 g! l. z. B6 e/ m
and value -1 means no limit. r5 I( G7 m+ _0 Z" {
</description>
2 {3 j0 }1 C4 a7 r1 U <name>yarn.timeline-service.client.max-retries</name>
9 s/ X# x( [. x- Y <value>30</value>+ d, s! N: `& J' s/ j
</property>7 @: B* n; u- o7 i$ J5 Y4 H0 i
<property>
" c \: @- O) Q3 D/ A3 _. ^- ] <description>Client policy for whether timeline operations are non-fatal.
# D" ]# G8 c6 q! } Should the failure to obtain a delegation token be considered an application# l% x" t# ?% S. Z' |; O9 Y' f
failure (option = false), or should the client attempt to continue to
* X3 x; E/ ]( L# Z publish information without it (option=true)</description>
5 {* w3 |) K0 T( O8 D <name>yarn.timeline-service.client.best-effort</name>0 V3 Z, P2 W$ W/ E. P8 j
<value>false</value>* L& K8 ]' T- k) ?! L, a8 j
</property>! n: _0 t& F9 j. J& y
<property>
6 |. V; f4 b5 ~5 l- g <description>
, K+ Y1 w- n' |3 c8 I3 w. [ Default retry time interval for timeline servive client.3 u* J2 l: k' l1 C. J
</description>
: B1 l8 J( P7 |$ c& n4 e <name>yarn.timeline-service.client.retry-interval-ms</name>
' K4 u: h$ i* H' |; H9 a% C <value>1000</value>
; K5 }' A- I" K4 N/ n </property>! U$ L3 d& G# Q* J4 r9 P6 h
<property> U+ Y, V0 n/ T$ v. L- x5 Y" d# s
<description>, z, j" n: \+ o$ W) a& d3 l
The time period for which timeline v2 client will wait for draining
# k3 J7 H! a! P b leftover entities after stop.
6 K. Z; f$ W" G" M7 |5 e3 [) Y </description>
; e) Z0 K) i; e8 W- [; N <name>yarn.timeline-service.client.drain-entities.timeout.ms</name>0 h" Z1 Y3 k- R' N ~- ]) f
<value>2000</value># ]/ _3 v7 ]( C2 Q9 h% F4 i/ {
</property>
" l4 K6 {: e9 o3 B. { E$ h& W <property>
6 M* ^& \3 t% L! X1 C7 V7 p <description>Enable timeline server to recover state after starting. If
, f% F9 l" P# q true, then yarn.timeline-service.state-store-class must be specified.
8 F7 L, `3 j1 q </description>
1 n5 }( t- C- e) G; t# O <name>yarn.timeline-service.recovery.enabled</name>7 x/ d' K& I( r5 Y0 }: A% m, `' ^" x
<value>false</value>
+ h& R) i' w7 K! F </property>
# E% Q9 v4 n8 h0 U; b- ] <property>+ D, V6 r& U. y9 ^8 }
<description>Store class name for timeline state store.</description>
1 | R7 q `: ]0 `& L <name>yarn.timeline-service.state-store-class</name>3 h6 X$ M' d0 [: L5 J
<value>org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore</value>0 Z1 _' \. K5 K' J/ z
</property>" [2 p- R/ F5 B1 p
<property>
1 ~: v, D. n- f6 ? <description>Store file name for leveldb state store.</description>
9 K* B2 @1 B+ G. U( U: y' R; R <name>yarn.timeline-service.leveldb-state-store.path</name>
% Q. K0 P4 t4 _, d) T0 I <value>${hadoop.tmp.dir}/yarn/timeline</value>- k" v! e3 Z+ i6 t3 @- `
</property>* G3 `) c4 B% K' s1 h8 K4 O
<!-- Timeline Service v1.5 Configuration -->, s6 ^. _( {, Y. k, ^- r
<property>9 F: {" l1 N3 R4 N
<name>yarn.timeline-service.entity-group-fs-store.cache-store-class</name>
8 a8 ^/ Z! s" z8 D! d% c) T <value>org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore</value>
% c3 ~; V w/ C8 H, W* _' q2 ~ <description>Caching storage timeline server v1.5 is using. </description>
( {0 J7 a2 ~1 N/ N, n/ d </property>8 t% b2 c( |' l- }& f
<property>5 `9 D } r m
<name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
) I, s1 d, R: G% X <value>/tmp/entity-file-history/active</value>/ f8 F! O' Z+ E! p
<description>HDFS path to store active application’s timeline data</description>+ B* V3 L% d; W* z2 F* f
</property>* D: B2 u3 @+ g& V _; z
<property>/ w- ~6 j* ^2 _4 K5 g: r, m: P0 D
<name>yarn.timeline-service.entity-group-fs-store.done-dir</name>! ?. N) z9 e& ~, |! N0 |5 }! K3 N
<value>/tmp/entity-file-history/done/</value>
+ e) ~% K M! N2 ?4 _, d3 K1 ^2 E <description>HDFS path to store done application’s timeline data</description>7 U# m$ ~1 y6 R
</property>
5 o n1 R( S0 W! a4 T7 o' S2 C <property>
; M2 J, K. R( V8 p <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
4 _ E+ x. \! ~" L" }4 z9 k <value></value>5 `, Q" ]3 f6 v3 Z0 Y) }
<description>: y Q/ J1 o& L) X
Plugins that can translate a timeline entity read request into; U% g# Q% J$ W' ]" q
a list of timeline entity group ids, separated by commas.
% w2 T- S/ k m; M4 j </description>
. U( {8 e2 E6 q6 ]. X( U4 v+ q </property>
' U: \' p& Y5 N' T8 x6 ?: h <property>
7 [/ R" b$ S5 ` Z3 V! O- t+ K <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath</name>
! G" D4 P: y8 ?0 a6 k4 F2 e, ?6 P <value></value>
8 c! F& m4 d; H9 @$ Z <description>6 H3 u$ o2 [" a! x+ I6 Z, \
Classpath for all plugins defined in
0 V/ u5 h) o9 O0 T1 F' T. c yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes.
+ {+ l' d2 Y8 }2 G3 J' T U4 P </description>
" }: K4 R- i( A, X </property># f% @! x' g* f+ V/ S0 B9 A
<property>( l3 H5 _; v) U9 N
<name>yarn.timeline-service.entity-group-fs-store.summary-store</name>6 s% t. Y* u) f/ X
<description>Summary storage for ATS v1.5</description>2 _4 ~: _# O5 y' z7 c
<value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
7 a3 Q9 r% [6 m: D$ ] h# u, N' O </property>8 x9 O3 o5 I1 ^+ u- J
<property>
) K+ A2 B: z/ {" a7 @0 F- \3 P- y$ c <name>yarn.timeline-service.entity-group-fs-store.scan-interval-seconds</name>
2 @3 N2 }5 E- U6 y <description>
. n; e z; [: S. ~0 ?( e Scan interval for ATS v1.5 entity group file system storage reader.This: Y' W4 `4 y% L
value controls how frequent the reader will scan the HDFS active directory
8 K( J( j1 l4 L8 H for application status.) _1 d. U' W" r$ k
</description>' x) G; |4 a& {( @7 _3 n
<value>60</value>; m5 E/ |. _8 n6 [! ?+ T+ m
</property>
# F: _. G5 d0 n2 I* C, r- }# i <property>
9 ]- J3 v1 s7 \ <name>yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds</name>6 V: N z$ Z0 X. j
<description>
( b4 U- M: D- J$ e Scan interval for ATS v1.5 entity group file system storage cleaner.This+ \' x4 `0 Y6 s! O- i7 t# Q
value controls how frequent the reader will scan the HDFS done directory
- Z& u/ I7 E$ [1 r! w* F for stale application data./ Y, r8 ~1 ~+ [
</description>
- m, }/ Q" P: }8 c, O+ M: u <value>3600</value>$ O( i8 E9 D0 u3 V! ~3 o3 t
</property>
4 O3 L. Y' Z3 y7 `- t; _0 e6 A <property>
* A3 v* f: W/ G6 F' e. g <name>yarn.timeline-service.entity-group-fs-store.retain-seconds</name>
9 |$ O! V( `4 j2 Q' w" H# j <description>; t; l C4 [# S8 {2 d* w/ r, w! Z
How long the ATS v1.5 entity group file system storage will keep an
" R7 ^# d3 m4 y1 P% V- A- f* D9 \ application's data in the done directory.
+ c. l" }+ }6 p) L4 M/ e% i) Q </description>
2 x$ p1 n J) g. l( q! d w8 l <value>604800</value>
0 ?, O1 p5 l3 l @ </property>9 U3 ^9 b X' V& l- [ a
<property>, x& p0 _) |, J4 Y4 S; F" `1 M6 h
<name>yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size</name>+ F0 c! Q# D: \% F8 ?+ q: ~ I
<description>
7 ^& Q, [, w+ c6 x% N2 A Read cache size for the leveldb cache storage in ATS v1.5 plugin storage.
+ N5 _, A% Q- I* g( n </description>1 ^% g4 H. k, B9 J# Q0 {6 R
<value>10485760</value>
8 ~ [; S& C" [) ]! j* K5 p, d& } </property>1 i2 e( x% {8 S/ A$ j. m! f! z
<property>
8 N2 L' i* c" M6 V! y6 | <name>yarn.timeline-service.entity-group-fs-store.app-cache-size</name>4 f5 k9 x Q- S1 P8 `# V9 q3 N
<description>
6 ^8 Z$ p; D7 i5 L1 D Size of the reader cache for ATS v1.5 reader. This value controls how many2 t( O; |# }7 t. d* c' W# q
entity groups the ATS v1.5 server should cache. If the number of active) A, c1 v$ U) [& @* B) l
read entity groups is greater than the number of caches items, some reads
) I7 ?, G5 X% _ m; n, G: | may return empty data. This value must be greater than 0.
( Z+ @4 v7 L" z8 g, g </description>8 F! _- `1 |0 z0 P$ S6 a( r
<value>10</value>
1 o7 A9 T" ^4 `3 Q r </property>. E" Q X8 V" W% X+ ]1 S
<property>' ^0 a9 Z6 \: \& F- C7 a8 g1 F
<name>yarn.timeline-service.client.fd-flush-interval-secs</name>8 l( U% R! }1 J
<description>
) ]* [0 u! }% W3 J Flush interval for ATS v1.5 writer. This value controls how frequent, |# t" |# x0 R+ p% I) B _
the writer will flush the HDFS FSStream for the entity/domain.. n+ v# d0 O) O& {* j9 p
</description>
$ m% x; K; d Y# _0 Q <value>10</value>
: X9 l5 a4 i1 x; N* D </property>
$ s" C. w$ q5 Y9 r <property>3 W' j( W U0 E; _
<name>yarn.timeline-service.client.fd-clean-interval-secs</name>
( s4 k( r# L# h7 f3 a+ j3 T* @% Z <description>1 a2 L, P/ q% @ @! {
Scan interval for ATS v1.5 writer. This value controls how frequent
( W* O. p9 M: [; ^- Y the writer will scan the HDFS FSStream for the entity/domain.
, k, M/ m, x' Y6 q If the FSStream is stale for a long time, this FSStream will be close.
, M' Q0 d! @" P# O </description>
6 S3 ?' y5 k6 f+ a) f7 ~ <value>60</value>9 `+ z9 Y+ }# K, O& E1 e' w
</property>
# Y: K6 ]. [ K. G. Q/ E0 F- F <property>. r- p( c! T( @0 R) P1 h+ i
<name>yarn.timeline-service.client.fd-retain-secs</name>
' d" P* E5 W) h$ d$ } <description>
6 |" E& q9 J+ f7 \; S How long the ATS v1.5 writer will keep an FSStream open.
" l, ]/ B& W3 j If this fsstream does not write anything for this configured time,
5 A3 ^$ h! @: U8 J% X% A- m it will be close.* {5 U- W( ^9 x: y
</description> K, N* H1 t+ z$ x* e' R
<value>300</value>
. W% C/ f+ I% P- b$ y </property>
# L$ l9 P. d0 t& j5 b4 K) ] <!-- Timeline Service v2 Configuration -->
% v: G4 X" m2 u$ m: j <property>
5 y* f% Z8 \0 O# I! J u, E( k9 j m <name>yarn.timeline-service.writer.class</name>) N, o9 o0 I2 e: t& o# T1 H
<description>
* t5 O; J% S! c" ]2 N8 Q Storage implementation ATS v2 will use for the TimelineWriter service.
: c) e: ^( {. K' E </description>5 v! O* L: i2 `
<value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl</value>3 P5 j0 z- L3 ?0 w5 }
</property>
% Q) L4 ^# I' B; {3 [& M- D) E; Q <property>
* ^( L6 r5 P, \: d <name>yarn.timeline-service.reader.class</name>
; c, W# i3 ^8 o% X <description>
: l$ p& N6 U5 ]4 z' t Storage implementation ATS v2 will use for the TimelineReader service.$ u, U# |4 g. f: F
</description>, j5 x! h$ T5 U
<value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl</value>
6 [$ [. G T7 Y. a) S </property>
" E, o) s1 `7 f! ~ <property>* A; b4 S) K; d" \% L+ i6 A) L
<name>yarn.timeline-service.client.internal-timers-ttl-secs</name>
. J) o( `" H9 r" L1 ?& ^2 g <description>
4 u' q$ T! ^' M7 P3 e s; c How long the internal Timer Tasks can be alive in writer. If there is no! _; l# _9 V* K& M. o
write operation for this configured time, the internal timer tasks will
# w( g$ m# a5 x: _ be close.
# w; d/ q+ \; h1 I6 h$ u- h </description>
& A: ^/ |8 k. e* m6 O9 H <value>420</value>- t3 S( w; t6 @/ R+ H) s" J3 E
</property>
) M7 [& D# Z1 _1 M1 l. [2 g- T9 V <property>5 P( ?9 \2 B9 t
<description>The setting that controls how often the timeline collector2 b6 ]7 k" v' i# ~' X1 e5 w
flushes the timeline writer.</description>
7 I3 P% g2 q% C4 o' o# y4 D <name>yarn.timeline-service.writer.flush-interval-seconds</name>
7 B& C2 ~7 T' j% j% A+ y <value>60</value>; j3 L5 a- k" M% J7 n% o
</property>
. M' Q/ k) g3 X" B' F* V <property>
2 Q$ ^, T- {" l B; p4 M4 x <description>Time period till which the application collector will be alive
$ c# y R9 w1 g! e in NM, after the application master container finishes.</description>9 ]2 F0 H# Z) D* b$ M2 @
<name>yarn.timeline-service.app-collector.linger-period.ms</name>4 B* a/ o. p, U. \; s
<value>60000</value>3 f& w* _2 Y& |1 Z
</property>
6 q, L" s8 W3 E0 G2 f <property>
" N9 h! V3 I! M h <description>Time line V2 client tries to merge these many number of
$ }' f( |0 W/ n% ]; c7 a async entities (if available) and then call the REST ATS V2 API to submit.5 C4 w$ E8 C3 w" ]% X
</description>9 y* z ^( {* g# P# n4 s
<name>yarn.timeline-service.timeline-client.number-of-async-entities-to-merge</name>, g) K4 {& o) J! W. j3 h0 ~
<value>10</value>, A+ Z, ?4 U" O' V: G7 n
</property>3 u G# G) F1 d3 q: X
<property>
% n& F/ h6 h7 j6 X# m: U/ o+ ]& S5 s <description>
; U$ g1 W/ |" `. f The setting that controls how long the final value5 {5 A+ S f' r% ] R9 E3 u
of a metric of a completed app is retained before merging into
, ^ K, R: `' d. `+ s: B: W4 { the flow sum. Up to this time after an application is completed
. R, X$ J- W! v out-of-order values that arrive can be recognized and discarded at the
- ?9 J4 w$ _3 ? V cost of increased storage.* \: e4 L( C8 z* K- p
</description>
3 Y' O$ [( K" T0 Z <name>yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds* ^ h( U) O/ k/ D8 G8 m O# A
</name>2 b' e+ d e3 v1 ^8 a, ]+ ^
<value>259200000</value>
7 p) c& r9 J( C8 |7 t8 ^ </property>5 F+ k& S2 s! F
<property>$ N' X1 G' c+ K
<description># m6 g' i8 h, G+ ]2 O
The default hdfs location for flowrun coprocessor jar.
' K! U8 t9 Z' v$ m8 V8 `1 T </description>
1 p+ o! |& _3 q% v8 a' t <name>yarn.timeline-service.hbase.coprocessor.jar.hdfs.location
3 r- r! z2 N" P& M4 z# }3 J </name># b0 T9 k4 R) F2 O
<value>/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar</value>" T6 t8 L8 s0 ~1 E! E7 z
</property>5 W$ ?1 s8 R. i8 k6 o/ { T+ R
<property>+ @+ h+ M! G( N2 T+ [# V
<description>
# s. y2 A' Z- S The value of this parameter sets the prefix for all tables that are part of" n8 X3 i8 r1 e# P/ ]
timeline service in the hbase storage schema. It can be set to "dev."
* i e7 s/ p1 _. |# { or "staging." if it is to be used for development or staging instances.
* \7 \3 v. I+ f. s3 }8 X" E9 M This way the data in production tables stays in a separate set of tables4 f$ j I7 m2 U
prefixed by "prod.".
- T2 p, J% a3 }- Z6 N </description>- G5 l! a- z5 P- Y; N1 ?% J5 L
<name>yarn.timeline-service.hbase-schema.prefix</name>
! d& O6 b5 h" N6 z: `: Y <value>prod.</value>1 p# s9 x; \5 g) o( d: [
</property>4 Q$ l# ?4 ?& O$ ?2 T. N
<property>! \. {' v+ O7 \; h
<description> Optional URL to an hbase-site.xml configuration file to be
: ^0 I L: W0 k6 E2 {7 W used to connect to the timeline-service hbase cluster. If empty or not" F" A5 y) Y! v7 t/ U! w. M8 c- G
specified, then the HBase configuration will be loaded from the classpath.2 S3 N. X% ^! E- m; R! j
When specified the values in the specified configuration file will override) c3 a( O* L: F/ g
those from the ones that are present on the classpath.4 N' a9 x: @( d/ ]
</description>
& O1 X4 K O7 k" G2 Z/ Y <name>yarn.timeline-service.hbase.configuration.file- \, I- E4 ~9 P7 u a2 p( M
</name>; C8 A6 S8 i, e) }% r/ ^% r
<value></value>
2 W% M8 [! ?$ b5 j3 x </property>
( M* A8 x3 a7 X9 _) s1 \9 p; |7 V <!-- Shared Cache Configuration -->' v3 b; c# f/ _. F# U. W+ y
<property>
2 Y9 Y, a1 u% }( e8 _- O <description>Whether the shared cache is enabled</description>' f! y% g( z* v
<name>yarn.sharedcache.enabled</name>
7 t |8 a4 {- q <value>false</value>
: X; p; ]) U( }2 |! g5 D; i </property>) ]6 a% R. d) `. b5 d9 G+ ~
<property>
7 Q8 M9 B2 i' a <description>The root directory for the shared cache</description>
/ z9 q0 a' J0 s2 z <name>yarn.sharedcache.root-dir</name>6 J! m8 ~( x# v! ^0 Z
<value>/sharedcache</value>
+ v0 \. J# ?& [6 V/ h </property>, X( ]: `$ l# c B+ m
<property>
0 B' J. G6 e7 c6 D# Y4 Y <description>The level of nested directories before getting to the checksum3 t' N7 y/ i1 j* k7 x# s( e
directories. It must be non-negative.</description>
0 Q# E4 `9 t" e2 ?' f <name>yarn.sharedcache.nested-level</name>
4 V& [, r. ]$ T; J% e# y6 u <value>3</value>8 ~* c. C5 K; [0 h1 Z! m9 V. g
</property>' @- j7 U* n$ o T5 l
<property>$ H1 M# y# S7 o: x* |5 X/ S* }
<description>The implementation to be used for the SCM store</description># T* r# m/ W- b
<name>yarn.sharedcache.store.class</name>; K1 v7 ~" E; A, @
<value>org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore</value>
5 y4 \4 E7 d, |$ Q1 n7 H/ x5 P/ g </property>
) N) h/ J1 y+ l" i% E- ^, G <property>( H+ K' ]" F! ^& j2 N9 K
<description>The implementation to be used for the SCM app-checker</description>; g- ]( e# b0 b* `: x, }; C s
<name>yarn.sharedcache.app-checker.class</name>! y( l& d' Z9 l$ [4 N" H
<value>org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker</value>* M; n: K1 p% F) ~
</property>' n. E& \( B$ `7 H
<property>- U" c% R4 Z5 C' O
<description>A resource in the in-memory store is considered stale' B) o: @, D* R4 N* t$ j5 }
if the time since the last reference exceeds the staleness period.
6 m% q9 s5 w: `' x. O! n0 F9 \- I. b This value is specified in minutes.</description>
* C' y+ M+ J8 | _! W <name>yarn.sharedcache.store.in-memory.staleness-period-mins</name>
4 u2 k4 Z% F; G9 a <value>10080</value>: i$ c8 B$ l' [) l4 Z
</property>* E9 [% i C* u& n
<property>
( c- V% h1 i% B <description>Initial delay before the in-memory store runs its first check% ?: t' i0 C3 `5 m- ?8 v
to remove dead initial applications. Specified in minutes.</description>
6 U n/ x# c/ T5 O <name>yarn.sharedcache.store.in-memory.initial-delay-mins</name>
5 \% H0 r: y6 j2 ^. ]8 c, A <value>10</value>! Q; Y+ p- Y( h8 N5 u& U! M
</property>: ?0 l9 L Q! A% j' z
<property>
9 i( t" s7 `1 h. p0 [ <description>The frequency at which the in-memory store checks to remove
9 B* r0 {# T' e5 s. o! P dead initial applications. Specified in minutes.</description>
& \ _; n+ D( E% k/ _ G& D <name>yarn.sharedcache.store.in-memory.check-period-mins</name>
0 D* V: v9 V( r <value>720</value>
( ^2 ]9 V. `1 P0 N </property># n# V# C& y, H. t' \" G8 ?2 W
<property>
7 l& w! [8 J7 E' v- g) F! @8 V <description>The address of the admin interface in the SCM (shared cache manager)</description>* a5 U, d m' D9 Y1 M
<name>yarn.sharedcache.admin.address</name>
; O: C$ f6 @+ ?- ^& q0 [: Q <value>0.0.0.0:8047</value>7 E, x% I- B5 A
</property>
/ [5 P, M; Y5 ]8 G# F# a, P) K# V$ T <property>+ E; u8 }2 b3 T" Z# U" ~
<description>The number of threads used to handle SCM admin interface (1 by default)</description>
" C* n$ S' e. U4 e <name>yarn.sharedcache.admin.thread-count</name>
% p R' N2 g: B) h8 ] <value>1</value>) H0 ]; O& D- v) S5 a$ w
</property>8 ^- A, v! w3 F
<property>3 i. `; r" C; E/ e% _. e3 F2 v/ Q* ~
<description>The address of the web application in the SCM (shared cache manager)</description>
* ]; N8 F2 [! m" \9 ] <name>yarn.sharedcache.webapp.address</name>; j8 Q1 f; o0 I9 q# z6 |1 B0 l
<value>0.0.0.0:8788</value>8 ? z6 n% |9 L* b# G8 o. ^
</property>
. G- i: v/ ^+ B) y! t, g2 _* \ <property>! N# Q0 t, I! s8 f( F
<description>The frequency at which a cleaner task runs.% V8 s. ~. z/ e+ Z k; I
Specified in minutes.</description>7 P8 ]- ?) y. t5 L0 H1 m
<name>yarn.sharedcache.cleaner.period-mins</name>! i$ B) |; \4 U0 j
<value>1440</value>
7 r' R6 G2 r# _; w) L$ H </property>$ o$ n! x% \8 T
<property>, T+ s0 F4 l' ^" z0 M- \0 s
<description>Initial delay before the first cleaner task is scheduled.% Q9 w( q8 R j- e' }
Specified in minutes.</description>( i' K* Y3 X. E- v* @2 |
<name>yarn.sharedcache.cleaner.initial-delay-mins</name>1 P4 u% B w# }4 x$ @7 {
<value>10</value>/ S0 g K2 I0 |$ K1 j
</property>7 B' z5 _7 n) u! p) G( i( P
<property>
1 i2 V, g1 j6 m: _3 }1 I3 S) K <description>The time to sleep between processing each shared cache
1 e3 K n: A, S$ ?- p1 x: W resource. Specified in milliseconds.</description>$ ~7 f# B& E. n+ Q3 |: ]6 V
<name>yarn.sharedcache.cleaner.resource-sleep-ms</name>
, l0 ~. G9 T/ ^% r3 [! e <value>0</value>
) J: p* v3 I) b; T) H0 r# Q2 z </property>
+ V2 w" H' o* q- w <property>& \2 p6 s; A7 n) R/ h# A. Y R
<description>The address of the node manager interface in the SCM1 {5 z1 b/ c" h9 U& n
(shared cache manager)</description>* h% m0 k) ?+ j9 p3 C0 A) M6 ]/ a0 c4 z+ @
<name>yarn.sharedcache.uploader.server.address</name>
& h5 T) Q2 i3 e <value>0.0.0.0:8046</value>
% a1 g# N8 F- x8 j' g1 D </property>( K' M: i; \$ f, H/ w1 Z+ c5 e; `
<property>) s9 E1 t9 b6 g5 V
<description>The number of threads used to handle shared cache manager
8 Z. O# U- L" i* t% A3 W" o" y5 o requests from the node manager (50 by default)</description>
# W+ Z r# p4 f0 ?( N <name>yarn.sharedcache.uploader.server.thread-count</name>
?+ {# m. J& ]" l3 E; }- P <value>50</value>, y$ B z/ B) C1 X: _
</property>
+ x J. e( K9 j <property>
! m' h7 e* Q! ?/ | <description>The address of the client interface in the SCM5 ?" d) V7 V; M" b, M$ K# N( {* _2 a
(shared cache manager)</description>. L$ e: R3 Z( ]" Z
<name>yarn.sharedcache.client-server.address</name>3 K& X, A7 X5 X- `, u- {% w
<value>0.0.0.0:8045</value>
( {6 y( j1 J- n& G1 ~, v </property>
! v: U! u x) z% g <property>
2 P7 V( q. w' b6 n: O <description>The number of threads used to handle shared cache manager# P: L. H+ j: c8 K5 [1 L
requests from clients (50 by default)</description>
+ I" S- K% r' I l( A <name>yarn.sharedcache.client-server.thread-count</name>+ e7 L' q$ m( M: k5 D5 m
<value>50</value>
( a+ r& A- `3 @/ `! `( i </property>
! `$ m7 h3 b% m$ u <property>
" p' Q0 C- N) g <description>The algorithm used to compute checksums of files (SHA-256 by
1 p3 y- m- h' p* V/ g% c: m default)</description>
/ w0 E7 N: Q, r' h) [1 o <name>yarn.sharedcache.checksum.algo.impl</name>
* g, ]/ l0 Y6 f4 x <value>org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl</value>; a# U* Z& b E+ f3 J7 x6 e& q% a
</property>3 H( |* D$ I2 A
<property>% ?- Y' s+ G; V2 a, i
<description>The replication factor for the node manager uploader for the6 _" b% d$ k: z1 h6 f0 n
shared cache (10 by default)</description>
$ `# c# W: p" B3 b4 a+ y7 g <name>yarn.sharedcache.nm.uploader.replication.factor</name>
. y3 Z% @$ l+ q5 i& q <value>10</value>
& s1 i+ {& q: [2 I7 d/ T" t </property>* ?( s5 ]+ |0 @7 Y
<property>: ~4 |, R7 B i! c; k
<description>The number of threads used to upload files from a node manager1 v5 V2 m- z8 h8 I* T
instance (20 by default)</description>
. k; b( i1 v# r U( f' J G2 z <name>yarn.sharedcache.nm.uploader.thread-count</name>
% W3 ~5 R+ z, k; z# R <value>20</value>$ J; l& p1 L C+ m8 w2 O2 h% R
</property>
5 \ x" L. H$ w! I <property>4 D! |2 |# [) G* f' e" Y
<description>' b) ~$ r* w9 A) J- H5 g! r. P& L
ACL protocol for use in the Timeline server.' h* i% Y6 K$ c. y& _! ~
</description>
) ~' P" D! R; L) W" t1 f9 q/ V <name>security.applicationhistory.protocol.acl</name>
. x$ M7 {9 ]- i4 l# U <value></value>6 v8 [+ s. \5 S
</property>
" I8 J5 a1 q) G8 c/ H <!-- Minicluster Configuration (for testing only!) -->
! V6 ~7 ~* i, J0 {. l# i <property>, H. Y& k. f, V$ }, C. z
<description>2 L: k# x3 p' Y3 f
Set to true for MiniYARNCluster unit tests
5 \# `: A+ c6 I$ }9 J3 f </description>3 M* a2 j& q0 l1 R# ?
<name>yarn.is.minicluster</name>* j B+ Q9 R; z. _. M% D$ K/ u
<value>false</value> w' H7 z: _4 ~6 Q+ a7 {# I3 Y; f
</property>" r* [ K. c7 j$ z7 |, S
<property>0 l7 g" J4 e- ~8 o$ e
<description>5 ^& [* V6 h: Z
Set for MiniYARNCluster unit tests to control resource monitoring( k+ S* B" a# ~6 O8 t
</description>
6 Z- ?! r' p0 g% O1 x; g4 X <name>yarn.minicluster.control-resource-monitoring</name>
8 r# O O0 A7 N0 U <value>false</value>) J+ c% m! Q L( c* p
</property>8 ~# }2 R' Z/ d3 b
<property>
: q; K, F7 G, r9 k) o6 B <description>9 V. D3 i+ g5 V" f! h m' h: w
Set to false in order to allow MiniYARNCluster to run tests without1 v7 N9 g; r4 x' L1 E) A
port conflicts.
- [. o+ v8 x: a; ? </description>: q9 V+ [, n: t; D0 p* W" K2 {
<name>yarn.minicluster.fixed.ports</name>! {! S. M% y+ o0 g& [* N& q
<value>false</value>
7 u3 H! d. B7 k7 U4 p </property>
; A6 F) G& {9 _: r* W! z <property>$ P; ~, O x# @1 d
<description>
. \' p0 Z& f9 y" S) S8 G' L4 s' } Set to false in order to allow the NodeManager in MiniYARNCluster to% }9 I. T! T' n7 i4 }7 a% Z
use RPC to talk to the RM.
! V2 _% e/ ?1 l( g </description>
! r" a5 ^' B6 l; U8 z6 w <name>yarn.minicluster.use-rpc</name>+ ?& k& b& y" l: B- m9 Q% M
<value>false</value>
- ~+ p3 [- f* E8 C' e! n </property>
4 c5 H" d6 \, r+ F5 F <property>1 f4 B7 z+ T7 f, t4 ^+ u
<description>
( e% F8 ?4 J/ e9 r, F& ^ E As yarn.nodemanager.resource.memory-mb property but for the NodeManager
: e$ t: q2 m8 p5 G7 ~8 D0 q0 g in a MiniYARNCluster./ s" h' s' I$ x I0 h% i
</description>5 o) u; Z/ U7 d: c0 ^" Y4 s# e
<name>yarn.minicluster.yarn.nodemanager.resource.memory-mb</name>
( Y8 f+ ^) V" f8 c$ |5 i <value>4096</value>
8 G6 z# u" W& `% ^: o: } </property>$ k2 y/ z/ f3 ?
<!-- Node Labels Configuration -->
% {# y3 V1 j2 S) e- W% \ <property>
, y( J$ Q6 m( k# J <description>. P3 i# Q0 M' c( b
Enable node labels feature0 w* x$ a4 A( _6 a3 c
</description>
7 g" P9 x& ~7 u& ? <name>yarn.node-labels.enabled</name>
0 T: u9 ^+ N; b! P <value>false</value>
# D5 o3 a6 m0 j( i; N </property>) ?$ j' l2 v. Y' i! c7 Q" O; J5 r( p
<property>( G& ], Q! f6 m& u
<description>
, x% T2 g) d- a& p7 Y8 D& B* B9 e URI for NodeLabelManager. The default value is
9 }6 i0 j/ k; a2 w# ^- } /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem.* Z# r7 X1 l5 z; @- x: n9 W& P
</description>' q ?9 C t/ R. @
<name>yarn.node-labels.fs-store.root-dir</name>( \4 U) m- ]; B- @* p, Z
<value></value>0 ?: k7 F* L I" \
</property>4 L, K7 g0 C' \
<property>$ E+ K* }1 P. b9 U: m/ W
<description>
, J6 N9 Y' E$ C* `2 D Set configuration type for node labels. Administrators can specify
( n" O: w, _' I' Q8 G7 v- H7 Q "centralized", "delegated-centralized" or "distributed".- f3 p( s: E# w8 O1 o8 Y
</description>' Z, y4 X4 f% y' y* L/ |
<name>yarn.node-labels.configuration-type</name>5 V/ }- Q8 `4 D7 y
<value>centralized</value>
) t) a! p8 F3 D! b4 { </property>
: }* R; C# l, A6 m% @ <!-- Distributed Node Labels Configuration -->
: g- F# ^) i% w4 |% n+ S <property>2 U3 }4 I8 ~6 T7 u0 G+ ]! t( J
<description>
7 u3 N! s( M0 h& y( F' ? When "yarn.node-labels.configuration-type" is configured with "distributed"! ?+ T+ S8 ^8 C
in RM, Administrators can configure in NM the provider for the
& e+ J; l$ P- _/ x6 ^: A node labels by configuring this parameter. Administrators can
- Y5 _3 M/ u: a/ Y/ |8 e" S/ Q configure "config", "script" or the class name of the provider. Configured$ o) l# F1 t2 \# |
class needs to extend/ i' R7 N- U6 {8 M' n; g
org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider.6 }. A. O" Y2 t1 I' P- P1 r
If "config" is configured, then "ConfigurationNodeLabelsProvider" and if
' D9 `/ k L# J! N9 D- r "script" is configured, then "ScriptNodeLabelsProvider" will be used." A- S1 M$ m: p; L1 y
</description>% v1 r2 ~; C7 b+ P
<name>yarn.nodemanager.node-labels.provider</name>4 \9 c3 e: f9 }' q
</property>3 |6 K4 V6 a( @. e; a
<property>/ _7 A' `3 ?& _7 b: U
<description>4 ?# }, |7 E2 i% y. W
When "yarn.nodemanager.node-labels.provider" is configured with "config",
- L( M o) z. @4 Y- G/ C0 [ "Script" or the configured class extends AbstractNodeLabelsProvider, then5 G) E3 G& H4 G9 w4 h
periodically node labels are retrieved from the node labels provider. This
* r* d8 e3 F3 g9 s+ o7 W: k. V8 k configuration is to define the interval period.; t6 b9 Q# @! y" c. ?; C& ^- c
If -1 is configured then node labels are retrieved from provider only
' p( Y8 T% ?+ |1 V4 f- B3 s) d during initialization. Defaults to 10 mins.2 b1 ?( I1 ^+ j
</description>
' F4 E- T& m F J7 @3 S6 a <name>yarn.nodemanager.node-labels.provider.fetch-interval-ms</name>
& A8 Y8 E0 e) ~' k; _ <value>600000</value>7 _1 Z8 H6 x8 _3 r
</property>
6 B! t' [6 N! S7 C' B7 ^" r! ^7 V <property>) _; z6 a) M8 G0 i: T
<description>) y+ ]6 g9 Z! k8 z
Interval at which NM syncs its node labels with RM. NM will send its loaded
& B# X: D! X2 y9 G$ g labels every x intervals configured, along with heartbeat to RM.
( d8 R5 ]7 r* f) A3 S </description>9 E5 |' N5 \ r$ j @; W3 T
<name>yarn.nodemanager.node-labels.resync-interval-ms</name>
# G8 R4 l D- r# W6 { <value>120000</value>
6 h5 M) h. y" Q5 x6 I3 c. i: E8 Y </property>
( e( n6 O( X' e% V; w <property>
3 h: l; |4 ]1 e <description>3 f( I! e$ g( O8 p/ G. G8 [; ] U
When "yarn.nodemanager.node-labels.provider" is configured with "config"# T3 r4 U& C+ U: H1 @9 t1 Z
then ConfigurationNodeLabelsProvider fetches the partition label from this
7 W' T3 _" c* G: ~; x) R parameter.* C/ c! t$ `: P
</description>6 J; b. `: t% T) Q1 g
<name>yarn.nodemanager.node-labels.provider.configured-node-partition</name>
& P: B. @* L6 w </property>& L' {$ n$ r3 Z( A3 w& g. l
<property>
% U* w- _5 J/ F# Y <description>- ^- u {: f+ s( Z: o7 Y
When "yarn.nodemanager.node-labels.provider" is configured with "Script"
3 h; A# [. l+ B7 u& [1 T+ G then this configuration provides the timeout period after which it will
' B) o5 H5 R& I! V' w" ` interrupt the script which queries the Node labels. Defaults to 20 mins.
" i; [2 v( M8 c3 E. M4 G </description>
8 H; X! ~- e* `* Y0 A( Z/ @& S <name>yarn.nodemanager.node-labels.provider.fetch-timeout-ms</name>
: N2 v- n+ i& F0 x* V9 ^ <value>1200000</value>
}" J6 _) j o </property>3 A, L# C v. N
<!-- Delegated-centralized Node Labels Configuration -->0 a/ c$ ]6 V# p1 @7 F8 C
<property>
; @* y3 E* A9 B& ^+ A <description>9 F3 W1 I. c2 G. d9 `( H
When node labels "yarn.node-labels.configuration-type" is
; r* s" t7 \( i, E) S( G of type "delegated-centralized", administrators should configure
% }0 F: F$ A' b6 E& b' P the class for fetching node labels by ResourceManager. Configured
, k4 }# C4 z" @2 L. \ class needs to extend
" w K( C+ d; M4 k8 r, f% `" b: B" V org.apache.hadoop.yarn.server.resourcemanager.nodelabels.1 C' \# c. v! I ]7 M; ?7 ?3 d
RMNodeLabelsMappingProvider.
0 i: E( M+ A0 U. e5 { </description>8 |1 O- _" O |! s# z- x" C
<name>yarn.resourcemanager.node-labels.provider</name>! F% [7 N: A& P# p% |6 X
<value></value>) ?. K' b' S* p! [4 X. q
</property>' [9 c1 Z# P7 z; d- X* a+ S3 K4 j6 e
<property>( W6 N; l5 Q0 Z9 A; A+ p
<description>" e4 U7 i- e( {+ P3 o [: E# _
When "yarn.node-labels.configuration-type" is configured with
; c7 O; w) ~( c* B+ U" u5 Y7 u "delegated-centralized", then periodically node labels are retrieved7 c+ S2 I! k! t- x# B% ^
from the node labels provider. This configuration is to define the
3 p6 l- _- U! B* p9 h, k# @ interval. If -1 is configured then node labels are retrieved from4 F; b( z8 w' F$ S# t, F
provider only once for each node after it registers. Defaults to 30 mins.9 r \' ?0 D' X8 ?. y" E: `; p9 F
</description>5 b1 a1 t: T8 o! l8 |
<name>yarn.resourcemanager.node-labels.provider.fetch-interval-ms</name>+ U0 N4 ?1 J- K- y# `5 l+ c8 V
<value>1800000</value>" B; O p) ^" M3 v5 W- ?' w
</property>. x+ Q5 Z% P6 u, j% Y) f3 y% b2 G
<property>
4 w" c+ F9 M- k. C <description>8 f9 L' Z8 j! a
Timeout in seconds for YARN node graceful decommission.7 P: ?. M7 Q1 g% R+ k
This is the maximal time to wait for running containers and applications to complete$ U! k9 w2 j5 H$ D: \
before transition a DECOMMISSIONING node into DECOMMISSIONED.
+ m% a/ A5 E$ C# E+ [9 X9 u( w </description>
) W: M) p- [! X: x' J0 k0 { <name>yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs</name>
6 j8 {% P4 h/ P! T" P/ F, R+ f2 | <value>3600</value>6 s/ i3 Y! f5 ]; m' n
</property>: W* \5 N; `* A0 f W3 d8 T
<property># a8 x& @9 Z5 T
<description>( g( l- O3 Z) f5 p2 E: D/ ?7 P
Timeout in seconds of DecommissioningNodesWatcher internal polling.+ c D C! O5 F& q
</description>
! c1 m5 [2 v! x+ s, B <name>yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs</name>
6 o+ W4 @) w: N- y <value>20</value>
. g5 ^1 ^) J, o4 O# ` </property>' i: k {7 L; p( e
<property>! [# M$ [( m& ~+ f# Y4 m# e1 |
<description>The Node Label script to run. Script output Line starting with7 _5 k# K5 L8 y4 a0 Q
"NODE_PARTITION:" will be considered as Node Label Partition. In case of
, T; P' ]4 k& ]+ S% ]1 x multiple lines have this pattern, then last one will be considered
( N$ [! ^: u$ p& J </description>- i$ o0 _) |$ ~
<name>yarn.nodemanager.node-labels.provider.script.path</name>7 r1 p6 D5 D. S7 X0 F
</property>
: V/ B4 g: w: u# D) r( K <property>
- g9 u# j, h: S+ g <description>The arguments to pass to the Node label script.</description>
9 |$ k) ?& X$ g <name>yarn.nodemanager.node-labels.provider.script.opts</name>
% u M" ]4 m2 S! ` </property>
' h' o% ~5 d0 t5 w) I( ? <!-- Federation Configuration --> W5 V" } q$ E2 T
<property>
7 M5 s$ ^' L+ a9 l' _5 _ <description>
2 l/ T: }) C( I @4 Y2 t m' e Flag to indicate whether the RM is participating in Federation or not.& k8 L- h% Z, u/ v: M
</description>7 O: V! V' R' z h
<name>yarn.federation.enabled</name>. T) a4 s* ` M( c
<value>false</value>
; U- ^2 G$ _7 q" c </property>
& j7 U0 A# ?7 {) o <property>
# L# H0 ~! \2 R7 x- P1 s <description>+ V$ s# O2 d( {& }3 A* e* ^
Machine list file to be loaded by the FederationSubCluster Resolver) e/ h- N+ Z: A& `$ t! B
</description>
0 R' T% C. g6 w* C6 j <name>yarn.federation.machine-list</name>
# w2 p4 H! A9 E5 J2 _4 e </property>" \; }% `. ~$ R0 v
<property>4 A% k2 \2 C3 I0 t6 g, N
<description>
r* W5 @0 t- |9 X Class name for SubClusterResolver5 k2 G/ I( d# s% L d$ O& H
</description>: r, U U+ Q0 z1 S: U$ H5 ?
<name>yarn.federation.subcluster-resolver.class</name>( J4 O/ x3 h: f0 S$ c9 _
<value>org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl</value>
7 U! L6 M7 t& R- [7 ~ </property>) ^4 j# `5 X; H) R) Z3 ^0 {
<property>( }; f# I/ n0 m3 u; H7 ]6 m
<description>/ f( f' G3 }% O9 M$ g7 b3 j4 o
Store class name for federation state store
! x/ W( S6 X) [) p9 D0 w& E# l1 u </description>
: O2 t7 C* {4 v, J* `, P <name>yarn.federation.state-store.class</name>3 N% k7 [+ C S; K7 t
<value>org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore</value>" C" F; J( [$ K9 k% C8 ~4 D
</property>
2 P2 d. |1 q" U: m! v4 @1 @- n <property>8 x6 K+ { r- r/ P
<description>+ l W! n1 O2 F, Q% j# p# o( l
The time in seconds after which the federation state store local cache$ p4 h' Z! h0 _& p" n2 d2 v' r/ k1 A
will be refreshed periodically
) S1 |, ^; \ l1 f2 @, O7 | </description>9 h& i( e: x* O( {
<name>yarn.federation.cache-ttl.secs</name>+ c7 C1 V3 q" D' u9 m
<value>300</value>
4 K- }8 {) w% F# h6 J </property>
( A* _/ U5 v( @ <property>! C- |) Q' f9 t& o+ m+ o/ R$ `( O
<description>The registry base directory for federation.</description>3 d9 e; O* O3 E# ^1 N+ ^# U
<name>yarn.federation.registry.base-dir</name>' @# |; ]3 p4 y& o& i& j# V8 }
<value>yarnfederation/</value>
6 K" }* o+ |1 S9 q; a0 B% { </property>
# G K% {) x" c) l, k <!-- Other Configuration -->- z' j0 F& a7 `! k9 z& Z- Z: u
<property>
c: J+ ]* y/ |* D q9 \) F <description>The registry implementation to use.</description>6 K1 ]3 i# P8 g9 Z n2 T. {
<name>yarn.registry.class</name>+ u& |( h/ K" t8 l9 ?- j0 F
<value>org.apache.hadoop.registry.client.impl.FSRegistryOperationsService</value>8 n) t* ~8 r7 v5 R7 Z! M
</property>) _' K7 K/ T- G5 Y \
<property>- h: O \! Y8 N& I) P5 q' n
<description>The interval that the yarn client library uses to poll the3 z6 k8 x1 |* j
completion status of the asynchronous API of application client protocol.+ ?2 y3 E5 C$ L! L! D. p
</description>
; A$ g8 K3 E: Z: o- R4 g9 e <name>yarn.client.application-client-protocol.poll-interval-ms</name>
( y* A e& m' h6 w7 \ <value>200</value>
: _. j+ `9 Y5 s; A* _& c </property>
0 C& Z" d2 L) w1 K5 i7 D <property>7 M9 ?, B8 j3 [9 F* E" f& g
<description>
4 A3 `, t# B% Y, T: j+ s& g9 l% g. O The duration (in ms) the YARN client waits for an expected state change
, }% T7 S9 p8 z+ x2 L to occur. -1 means unlimited wait time.
+ z6 [9 G# ~3 ?; U; N </description>& |, d( R' n E" n# M
<name>yarn.client.application-client-protocol.poll-timeout-ms</name>% n, }8 C6 n$ d! k$ M
<value>-1</value>
& M; q- b0 `' b$ O; M) } </property>" j7 \/ F) H' |4 g* s7 ~
<property>5 ^; z1 B, I( c
<description>RSS usage of a process computed via" J+ g5 {1 a3 g; |4 `& ^
/proc/pid/stat is not very accurate as it includes shared pages of a
) }, S- T$ U9 z. @* N7 R process. /proc/pid/smaps provides useful information like
6 l9 @7 o8 B, p! ^6 k, f Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used5 f' T; a* p$ B; M
for computing more accurate RSS. When this flag is enabled, RSS is computed
3 O' f4 i3 u& y B& c3 k) ?. T as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes! W9 G$ K" Q, k7 z. Z
read-only shared mappings in RSS computation.
0 S! H* B' {3 n: N$ v& g4 _ </description>
: r; V9 H4 G1 I" i* F <name>yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled</name>
7 p# m# d: P& o f <value>false</value>5 ^& j6 I0 I/ z e. M" |
</property>
]( J9 W' j' j: u: X <property>
8 D/ S; d A& F8 T; g+ B <description>& m, w* S5 L7 Y% c( Y) S1 \
URL for log aggregation server6 M8 A# H* s8 D4 R6 D
</description>" Q# [. t q' l6 u% ~$ f& f
<name>yarn.log.server.url</name>
' j) Y' @ I5 m <value></value>
+ t. K y* g; x$ a( Z" p1 ~ </property>
1 _2 k' x% v: s9 \ <property>) |4 `/ e8 F" G2 T- ^6 d9 m: j
<description>
$ |, z+ P9 w8 o' I" M* g URL for log aggregation server web service. I% z1 o6 s: T8 j, S
</description>0 t% i0 M3 |7 e1 a3 \4 K4 X( s
<name>yarn.log.server.web-service.url</name>( O# L; t; |! T5 m# W* s
<value></value>
9 m; p0 ~4 L0 j# z( ^6 J </property>
2 p0 I7 i& P7 Y+ | p1 o& L <property>
7 x1 M* D5 ]) e9 N' h8 { <description>7 K2 S; w& y# b9 Q
RM Application Tracking URL
# I7 I/ I j z* B </description>
2 T) o5 \1 t, v- L <name>yarn.tracking.url.generator</name>
( ]4 S( l3 g( m2 Q3 B <value></value> R6 p$ w; }0 j% ]; o
</property># D2 P4 E9 R+ C7 u$ M* \
<property>
( `8 Q: J# \1 S <description>4 P6 ^4 ~5 V6 n6 A6 D0 A
Class to be used for YarnAuthorizationProvider* x0 r- z9 Y: `. K' V
</description>
/ Q: l5 c5 U) V, q <name>yarn.authorization-provider</name> x+ K" P, T% W/ b( I2 _" D
<value></value>
- w: e! q1 u1 N3 I9 z$ E </property>
$ ~$ S6 T4 A- l/ j' { <property>
/ i9 ]7 G, {* i1 r7 r <description>Defines how often NMs wake up to upload log files.
1 L2 \0 p3 x$ Y# E, a The default value is -1. By default, the logs will be uploaded when0 S6 M6 n& g% p8 j" B% [' `
the application is finished. By setting this configure, logs can be uploaded8 `/ x/ I3 F' y0 n
periodically when the application is running. The minimum rolling-interval-seconds7 K8 \% O' f9 _& p, a& C: Z
can be set is 3600. O: e; A3 S S A9 l: E9 W
</description>( u7 @# b* S7 I( _( x
<name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
7 w$ k `2 r% Z0 v# }( @4 e" P <value>-1</value># N% y' n8 j; U9 K/ _: q9 z( x/ x% S
</property>
8 Q1 J1 x- m+ P5 V, g: n <property>2 t1 m+ g9 h% ^0 O: T# v
<description>Define how many aggregated log files per application per NM
3 s# |- Q" A% r5 `6 v% Y' D we can have in remote file system. By default, the total number of5 i+ s. ^$ N0 j1 o% r; v
aggregated log files per application per NM is 30.
, ?, S3 j% Z2 [8 y </description>+ i9 H+ N, r) L$ b/ l* t
<name>yarn.nodemanager.log-aggregation.num-log-files-per-app</name>' n6 ?3 D* ^- i% _0 [# R
<value>30</value> N% j+ B2 x/ S2 U
</property>/ Z. v* I0 T% b2 `* Q5 ?0 x" R7 n2 N
<property>
/ Z4 I. N! v, r7 e4 j% l" G) j' w <description>( ? w$ W- |8 l: V+ \: r
Enable/disable intermediate-data encryption at YARN level. For now,
6 d) }& \+ k; q* m% i this only is used by the FileSystemRMStateStore to setup right
8 D. [6 a% L$ e8 ?8 D6 N- e file-system security attributes./ b! u' [/ g& S' Y
</description>' m( a7 R/ x5 m7 L
<name>yarn.intermediate-data-encryption.enable</name>, I5 U. }& M1 \/ m' L
<value>false</value>: |. h4 T8 U G" h* T
</property>
6 L$ r: ^: t+ z" u, v <property>! r3 \/ N& C4 J/ V
<description>Flag to enable cross-origin (CORS) support in the NM. This flag4 Y7 o% I5 \: p+ Z
requires the CORS filter initializer to be added to the filter initializers% D4 K z1 k( |7 Q' R
list in core-site.xml.</description>4 a0 b* d+ I% W5 g9 d
<name>yarn.nodemanager.webapp.cross-origin.enabled</name>
: f) ]# z7 f) ^0 ` c9 ~8 Q U <value>false</value>6 }; C$ T( ^1 m) a& m( J; L; c
</property>7 K3 K6 I6 M7 ]; O( S
<property>' L' b" m( a) X; Y" G+ ~( U8 K
<description>1 \& K- l# ?/ ]1 `8 R8 [. k3 K
Defines maximum application priority in a cluster.
" Y, g4 i8 }7 a2 H If an application is submitted with a priority higher than this value, it will be
6 ]% }! l7 q) I' `# ~/ d reset to this maximum value.1 n% m; }! d m1 c
</description>8 p& o6 }! W. R
<name>yarn.cluster.max-application-priority</name>9 K/ w; d ?5 K* o; ~4 \
<value>0</value>
4 ~& N& R4 `7 y+ R, T) j) c- z </property>
8 h; D/ `$ D( U0 w <property>
: w/ D& M3 A2 \9 M <description>
6 s( x8 K9 z9 J9 z8 \5 D* q The default log aggregation policy class. Applications can
# e+ B; j. h$ D2 @ override it via LogAggregationContext. This configuration can provide
- O! w' ? x3 d4 d- m U some cluster-side default behavior so that if the application doesn't
: ~2 {* A8 u. V" L5 p5 j specify any policy via LogAggregationContext administrators of the cluster- ] W& I/ @5 T1 r& i9 A6 X
can adjust the policy globally.
! k/ o2 n. l" M& G </description>
; y$ P% @, ~" \3 h5 R <name>yarn.nodemanager.log-aggregation.policy.class</name>
, I! T: C* W9 {3 V5 P& Y/ y <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy</value>4 \8 s* L+ R$ O
</property># I' F3 c7 _- e
<property>: G2 t2 L8 X5 b( u! f; ?- ^
<description>
3 h# Q( E& }' w; ^4 h1 r) p The default parameters for the log aggregation policy. Applications can
( `/ L/ C7 O& k8 G% V- r override it via LogAggregationContext. This configuration can provide
/ D. Q; @2 G8 Q: {4 D some cluster-side default behavior so that if the application doesn't. x6 R. |; f$ {( l% U( _
specify any policy via LogAggregationContext administrators of the cluster
$ H' B$ q! L, E( q- j9 \3 p can adjust the policy globally.: V3 a& S* N+ d a2 i3 G
</description>
5 Q" E; P5 l( X1 |& W <name>yarn.nodemanager.log-aggregation.policy.parameters</name>
7 v, _& d; r9 Z+ m3 @9 B <value></value>
6 P, Z5 @ U E6 e3 f) X% R8 n </property>7 l) _0 F0 n# v; [) G% o6 y
<property>
' v `7 n+ M+ f/ t' A- M! m <description>' x9 D& J: p/ O! D# x% s, D& X
Enable/Disable AMRMProxyService in the node manager. This service is used to) i% ~: ~& ^ z7 k# S1 x
intercept calls from the application masters to the resource manager./ X! E& M$ l _
</description>
0 m& G* N, O9 o2 h2 B$ A: X <name>yarn.nodemanager.amrmproxy.enabled</name>
# P4 G f* g7 c6 u+ @; m9 b4 n" s <value>false</value>
! {% J2 _7 H+ {" Z! O* \ </property>3 l* Y- |% ~; l5 d. c
<property>- y+ [/ A; ], h8 I7 L4 }
<description>
# X- G. g& h+ y: m/ h" K$ u The address of the AMRMProxyService listener.) a# e" K/ @' G9 @" i0 p+ I" L
</description>
+ A/ `1 {/ y% H/ x <name>yarn.nodemanager.amrmproxy.address</name>8 u7 i- R8 ?( A% A
<value>0.0.0.0:8049</value>( H" o7 t& A. K7 B
</property>
* G% X* l5 D7 @2 W <property>0 h8 o/ x3 _9 V5 d. a# l
<description>
9 a( `: t1 S/ {% K The number of threads used to handle requests by the AMRMProxyService.9 s+ E) t$ @4 r/ {: k+ g4 k
</description>
, J; M/ d8 P$ c <name>yarn.nodemanager.amrmproxy.client.thread-count</name>2 n( C- B3 f2 p6 I9 J/ w) d" o3 h
<value>25</value>
& K( s0 F! y, A7 i+ i& A </property>
4 z) b6 D, h: Z. L6 a$ I <property>
! m+ L" h4 f# d! N <description>
! L; L5 I9 G" n( M The comma separated list of class names that implement the
4 L7 T \8 z d6 x RequestInterceptor interface. This is used by the AMRMProxyService to create) f/ B6 P1 b! g# U5 o' y
the request processing pipeline for applications.
( e, k( h3 |3 }# J& T </description>9 Y2 l7 C: L# X
<name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>- J3 T8 c. ~9 @% z4 r7 G7 U
<value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor</value>
6 m$ g% Z, M; C3 n </property>
9 B+ V7 D4 t; q/ \9 W <property>
( L8 R8 E' V! A( |7 U <description>
6 t4 @) u, Y! Q3 v: a/ S Whether AMRMProxy HA is enabled.1 @' H; |4 e$ c+ [
</description>9 E; ` }4 ~9 a; F7 `; n8 Y
<name>yarn.nodemanager.amrmproxy.ha.enable</name>
, n9 s6 o v9 J, o8 [4 C <value>false</value>
3 s8 B1 q( M/ ?* u </property>- O3 K( K8 w! \) Z! ^" b
<property># X- \, t% `8 o3 _, m9 `# Y
<description>
, f. m# |& ^" r Setting that controls whether distributed scheduling is enabled.2 V" a. ?% m5 m: @# _: Q- M# _. F: U
</description>
4 @4 n( ^2 z0 Y$ ~8 H) G1 u <name>yarn.nodemanager.distributed-scheduling.enabled</name>
* X+ o v2 M" h' U <value>false</value>" J% J+ d% N0 d% A. y
</property>! f, J; y2 ?0 u# i, c
<property>4 @9 {* Q# M, t$ g" m' Y
<description>
) Y5 R; `, ^5 C, Y2 U Setting that controls whether opportunistic container allocation) j0 ^1 Y7 P+ l& v
is enabled.& E) I8 ?' W6 M1 A. |; z
</description>* X6 [! g( a1 |) ~0 i6 `: I0 [
<name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>
- L6 t6 x) a1 D# {2 `( Z( { <value>false</value>: m9 [6 ~4 S# t: I O) t3 n( k5 m9 F
</property>! K0 v2 J: D# e1 M) c
<property>
; H* @+ F5 T: }* z+ x <description>
9 b7 g2 \# R) z b) c Number of nodes to be used by the Opportunistic Container Allocator for
& e2 h; D2 [" F) a% U9 z# A) [2 y5 C U dispatching containers during container allocation.
& d Q( q6 }' `" h% L! T: @ </description>% g4 ?; m7 m. z. a& ^3 a2 F4 s. h
<name>yarn.resourcemanager.opportunistic-container-allocation.nodes-used</name>7 ]; v/ t% G0 ]0 }
<value>10</value>
' v2 X0 c! r& W </property>$ `, T6 _- Z6 l8 _5 ]$ w
<property>
; a: \4 s3 i: D9 g" | @- z- N <description>1 X% M. p+ C; [* S& e
Frequency for computing least loaded NMs.6 V7 M6 T% [+ r0 F1 K4 E
</description>
/ g$ J# A8 F$ ~( j. ]7 w1 `2 X* | <name>yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms</name>
, o+ h" j y, A2 m; L/ K% G; q <value>1000</value>
5 ]; Q" M# _+ D$ F </property>4 p# a5 h9 P# r; g" O G( V
<property>* c4 v# v( C' [7 [+ W- {4 U
<description>
+ s1 g0 V& K- V9 p1 U- E Comparator for determining node load for Distributed Scheduling.
- X* o9 R# c+ r4 p/ ? </description>
- X' J# s) }! d+ c <name>yarn.resourcemanager.nm-container-queuing.load-comparator</name>
4 {* @, H: E" j# f# m/ g, Z' j <value>QUEUE_LENGTH</value>
' G# N: S) z+ A# o2 `& K. W </property>* B3 A' z# k* M2 ]! ^. c# ~
<property>
4 E1 d9 V" C0 D: { <description>
3 {/ l; Q' r% h J Value of standard deviation used for calculation of queue limit thresholds./ G3 w2 Q1 i+ ^
</description>
3 c: Z7 L/ h$ n7 J& F <name>yarn.resourcemanager.nm-container-queuing.queue-limit-stdev</name>, o% C/ Q- |$ ]1 ~
<value>1.0f</value>
1 m4 Q7 M8 b; L z3 F </property>
4 C' m& I4 m2 w' ?% _ <property>+ X- b5 L/ T8 S, [& G" [; C
<description>/ w- o$ j$ ~: g, b W
Min length of container queue at NodeManager.* e* x# l% F0 ?. U d
</description>
! ]9 g7 ]( F# p( T+ Q <name>yarn.resourcemanager.nm-container-queuing.min-queue-length</name>
' k8 ~1 A) f, R8 ]2 o <value>5</value>
1 h' {: G4 G8 ` </property>
* B) I5 w; R. w; ~: C+ e <property>
2 ^, G, G: N+ k" C- ?/ E$ H <description>- G& j, |" w0 N& x0 I9 B5 V9 z7 g
Max length of container queue at NodeManager.
: f7 d* d4 i, V9 S {3 ^8 \5 T& q$ o; ` </description>
; D2 ^9 u* ~( a/ V7 q <name>yarn.resourcemanager.nm-container-queuing.max-queue-length</name>
9 ]6 Z& r% ^/ H, D) S7 s <value>15</value>
. e6 ]8 _3 E4 @7 N$ A, t, M8 R& f6 D </property>7 {3 T; p% A( g1 T
<property>
4 k' v9 @- ]+ v9 \+ k6 i$ q3 { <description>" f9 h: k+ b& E
Min queue wait time for a container at a NodeManager.% C1 |: Q; }8 [1 W( X0 e
</description>" A' z" z1 t* _
<name>yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms</name>
4 V/ x+ v% O) J+ P6 l$ h1 P" A* z <value>10</value>2 P. D4 g+ B% V0 x( R, X" f
</property>
; v+ P, ~! J6 k$ [6 ^, G <property>' ?2 e* K9 j& q4 O X s& H! v1 ^& q
<description>" N- S0 F$ o( m
Max queue wait time for a container queue at a NodeManager.& R. ?$ V9 d* I4 E( u% w
</description>
1 x5 u4 Y7 B s) C% A <name>yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms</name>
% d- K1 u% c3 l0 a+ ]! r. h5 W3 ^ <value>100</value>
: B1 x+ s" q5 C; B! B4 O </property>
" e1 M" j+ t2 `, }& u <property>
( V! [8 ]6 l6 p% u: R/ r8 ` <description>. N* Q( V$ ^, m4 A
Use container pause as the preemption policy over kill in the container" g# K# {) _* L
queue at a NodeManager.
; k2 ?7 T b7 H3 W$ z" h </description>( I4 U# e7 e6 a, d" M0 }
<name>yarn.nodemanager.opportunistic-containers-use-pause-for-preemption</name>- o; h7 U# X% r7 u* I9 m
<value>false</value>8 y3 a1 Z. X4 y/ F$ G' h9 u
</property>
$ g% W6 x# {5 I* x$ y- `& c <property>; c8 y; m7 `) f0 Z: Y$ ?3 ~# D
<description>
* F' i4 w" M( m7 i Error filename pattern, to identify the file in the container's; R- }! W* S7 p5 K- H
Log directory which contain the container's error log. As error file
0 n4 K( G! c2 Y4 t; _ redirection is done by client/AM and yarn will not be aware of the error
# w( N% J2 X1 Y- j$ b file name. YARN uses this pattern to identify the error file and tail- s; T: D$ D1 S4 x
the error log as diagnostics when the container execution returns non zero
7 \1 u; y% h! u2 ]1 N, y/ P value. Filename patterns are case sensitive and should match the
3 I$ ?7 F3 B) M3 m+ O specifications of FileSystem.globStatus(Path) api. If multiple filenames
3 k1 V) N4 ` v9 @) a* Z matches the pattern, first file matching the pattern will be picked.
5 A6 F# T# J' C1 ^% {3 g, e/ e8 b </description>
, h/ z7 {4 J3 H4 n% m7 g$ K <name>yarn.nodemanager.container.stderr.pattern</name> g5 h8 H O" C( `+ E" m6 e9 Q
<value>{*stderr*,*STDERR*}</value>
. X$ o1 O* }6 V& e5 o9 u V6 z' e </property>5 m1 I7 D" r8 _ p
<property>9 l& K$ e/ @: a' k, Q% K
<description>
& ?+ w3 V7 K% M9 G& t6 t5 n# ^! i+ y Size of the container error file which needs to be tailed, in bytes. G( M( D* D; ^! B0 ~
</description>6 R1 L, {3 r* g& t/ ~5 x2 E/ D
<name>yarn.nodemanager.container.stderr.tail.bytes </name>
/ b$ j7 R5 [8 ^' N* A6 k) c) Q J <value>4096</value>! B9 K( P- V& D" ?1 u! J" S. _# Q
</property>
& ]% Z% y- o: C0 M3 C0 i <property>
' c- Z% b4 B4 a, o; i <description>3 E! I# X! w$ f, K1 m6 h- u
Choose different implementation of node label's storage0 v8 f& C7 {. A. n/ |6 y! J
</description>
$ J' {$ v( v8 {4 F/ ~ <name>yarn.node-labels.fs-store.impl.class</name>- b. ^ P2 u: o5 f: `
<value>org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore</value>
: r5 O% f: r$ h F5 A/ o </property>
' U+ |4 H( w; E8 K <property>
0 l+ z0 t9 T! t$ z3 D <description>- X' [& M ^( \- i3 D S
Enable the CSRF filter for the RM web app
7 v3 {7 X* z# v3 U </description>$ d( ]6 }6 x7 N. m# Q M/ N/ F
<name>yarn.resourcemanager.webapp.rest-csrf.enabled</name>2 _ p& I3 j3 f7 g6 y5 z }3 T7 n
<value>false</value>8 A% G% h* ?1 J) f r% l
</property>+ P9 m2 T4 |( o7 M5 m, |: H; u
<property>
" b8 Z3 u0 T6 |6 T$ t <description>8 @( D* x5 F) v! l
Optional parameter that indicates the custom header name to use for CSRF
0 x# q' J. B7 ? protection.$ d' x, K& Y, Z' S0 m
</description>' f1 n' j/ H% |! p V2 Z2 W* n
<name>yarn.resourcemanager.webapp.rest-csrf.custom-header</name>3 j. y) u5 |( Y* ]* f% e. B
<value>X-XSRF-Header</value>" U" n1 K8 L# c1 x {$ e# x
</property>
5 G5 W4 u' O. Q <property>$ n( b: ^% Y- ?1 M& t0 q9 |. T
<description>
; Z7 x6 R! I9 c N. d2 i Optional parameter that indicates the list of HTTP methods that do not: r5 i* z+ d$ X) _- s
require CSRF protection
1 V( |7 j7 O E! H </description>. i! |7 ?& z# [7 e' G# d
<name>yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore</name>- j: k* w/ T/ r7 q- q9 Y
<value>GET,OPTIONS,HEAD</value>& G' c% J" @7 h) e. K7 b! [
</property>' [ C8 W* ~6 ], w
<property>
1 d+ K' ~' t0 B- a, t" S- ] <description>7 b' B& f* }3 _! w0 `" t
Enable the CSRF filter for the NM web app I+ D$ u" N6 n; ?4 Q/ J, y
</description>7 S2 b$ k; |9 F) z c! r
<name>yarn.nodemanager.webapp.rest-csrf.enabled</name>8 j: \! d( D1 c( o; u/ C
<value>false</value>/ {. K# Z( \* o# T* I/ J
</property>3 p& }2 E6 |; T& [
<property>& |# T. Y* Q- b1 u# e
<description>
1 r0 l- K( z2 R+ B( z: W Optional parameter that indicates the custom header name to use for CSRF, @8 K( e5 U: n+ b8 \0 o
protection.( m! Z* ]4 U; L. y9 Q
</description>) G) R h- e* O( y9 a, ?, t
<name>yarn.nodemanager.webapp.rest-csrf.custom-header</name>
: u8 ^& s* n8 | t! c <value>X-XSRF-Header</value>
& D0 t1 K' a+ d# [2 a2 i' s& `& A </property>
0 _3 R( o& ?0 Z4 R) ^8 C <property>, P5 s2 }, l9 O/ v* y/ ^; m4 h
<description>" F/ i% {5 U& e1 `
Optional parameter that indicates the list of HTTP methods that do not N- J- x, x5 Z$ i
require CSRF protection9 `9 Z9 Q3 ?, n1 M$ X! m6 |
</description>
: n/ g; F: A' O3 {- r# S/ Q <name>yarn.nodemanager.webapp.rest-csrf.methods-to-ignore</name>) D. \; f% i! m2 t
<value>GET,OPTIONS,HEAD</value>
# m+ z# p" i$ C w! v* ~ </property># y$ E: s: p! V: \
<property>6 D$ v# v. L3 `) m9 q
<description>( n% O1 {- i' Y! X' Z$ X5 Y
The name of disk validator.
8 |3 E, u& S; R$ D( A- V </description>7 L+ V: H9 S' t& t1 ]- @
<name>yarn.nodemanager.disk-validator</name>
8 u1 `9 p% z _, U9 F7 t2 ^ <value>basic</value>
$ u$ w0 Q1 N; o W" |; _0 r </property>% }, Q1 K0 ], U; D
<property>
; o+ d: B( n( ? @7 b <description>
. ]5 u9 T* F6 V3 N& @ Enable the CSRF filter for the timeline service web app, |: {5 p* \4 c7 J1 m! O
</description>" i& c; a% W) y% }2 B
<name>yarn.timeline-service.webapp.rest-csrf.enabled</name>
, R) E7 C) N& C8 O6 |5 r6 a# G! @ <value>false</value>; U: c6 w4 W+ T* e9 `1 z# B
</property>/ j0 M; I9 N, Z2 v1 K3 h6 }
<property>
. E8 w7 m- r6 j. }6 h3 b0 U/ d <description>5 q% I7 @' u9 y1 ?
Optional parameter that indicates the custom header name to use for CSRF! T2 b M$ r& w% y" u* Y- m
protection.
5 |2 F& R( V; L. E* f </description>" V. p+ e9 W0 K& I' ?7 ?7 q
<name>yarn.timeline-service.webapp.rest-csrf.custom-header</name>
# E, l8 ~ r, n6 i! I2 B <value>X-XSRF-Header</value>! E l7 m& B0 O8 c8 k
</property>
$ a) }& W N/ U; V5 E: }: p& I$ Y <property># t5 w- D/ g5 d2 _! k) P) ?$ f* L
<description>0 F: J9 x3 t |
Optional parameter that indicates the list of HTTP methods that do not$ f, } G2 i9 p! F, ?3 n5 Y
require CSRF protection& L/ g/ P/ D1 P6 D) o$ V
</description>
: S% }: p" U7 T/ w8 \. I <name>yarn.timeline-service.webapp.rest-csrf.methods-to-ignore</name>
2 k M$ A. j5 N8 F4 m <value>GET,OPTIONS,HEAD</value>
5 V/ K7 ?- x, N- J- P </property>" D1 S8 V9 d& v6 ]( T6 T
<property>
$ q' y) i$ C% D/ `' R <description>
: U6 G) \; ?1 {" V" P Enable the XFS filter for YARN! X' J1 w+ o& D; M
</description>
F+ [' A& K) x! n# r" @ <name>yarn.webapp.xfs-filter.enabled</name>, v" \+ H$ O5 v# n- u! Q: t
<value>true</value>6 t! H& H4 E' C* B5 |
</property>
- ^% y* K8 G* [5 ^/ O2 b" m) m( g <property>+ h9 _2 d7 P A2 a0 f
<description>
' N/ h. @: w' b: y Property specifying the xframe options value.
6 n7 U; H! @. A </description>( e+ ]+ k! J9 ]
<name>yarn.resourcemanager.webapp.xfs-filter.xframe-options</name>
2 k/ f+ I& B( d8 k8 t4 U <value>SAMEORIGIN</value>
- C& i* V4 V& s </property>
' W: Q" H: H' @6 e2 L0 Z; ^) q <property>
! b. h7 n$ T! w <description>) P* m4 N9 E, l0 S" {$ @
Property specifying the xframe options value., w1 [, ]( @2 M( B% u
</description>2 u) y" D3 [& E1 V2 m5 P. {. ~
<name>yarn.nodemanager.webapp.xfs-filter.xframe-options</name>( U2 S8 C, ]; R* j' z- T
<value>SAMEORIGIN</value> `# v. ?6 ~" I1 k6 `9 c
</property>
1 k' m$ v, O% n6 H4 p <property>$ @. e/ g) \6 ?8 [5 {
<description>
. e0 Q1 w n! h% U4 Q: Y+ a1 F" q3 `$ a Property specifying the xframe options value.
9 J/ k4 ]& U0 d" T </description>
; C. Q) P; ?( Y( R" j( A' b <name>yarn.timeline-service.webapp.xfs-filter.xframe-options</name>% g4 T2 o% b/ ~) A2 E' E' D& |% j
<value>SAMEORIGIN</value>8 `* j6 F% o0 V
</property>6 [, m$ Y- Q" r E5 ^
<property>
; d2 @5 q) M2 Z$ s <description>' p7 p/ P8 Z; ^9 P, Y
The least amount of time(msec.) an inactive (decommissioned or shutdown) node can/ }$ }. m% v2 l1 E. k
stay in the nodes list of the resourcemanager after being declared untracked.9 ^ z4 x% Z: Q+ S" V# j0 s U
A node is marked untracked if and only if it is absent from both include and
; q% I( H6 w% d2 V exclude nodemanager lists on the RM. All inactive nodes are checked twice per8 s' C8 Y/ @- l% D/ L2 P
timeout interval or every 10 minutes, whichever is lesser, and marked appropriately.
9 t+ T U8 J7 g7 G, N& T The same is done when refreshNodes command (graceful or otherwise) is invoked.
( _1 n6 |' t" N' Y2 @; { </description>
+ o9 R" A, `! d <name>yarn.resourcemanager.node-removal-untracked.timeout-ms</name>
$ @8 H- ?7 z. w2 \* U <value>60000</value>% z- m: i& t N7 f
</property>
" Z: ^9 |" p% y/ z P& }6 Z4 |# Y( \7 } <property>
& ^( E2 @* B- {% _0 M1 Z7 ]5 U <description>. x) x& p" G* i' p% I" Z, C
The RMAppLifetimeMonitor Service uses this value as monitor interval
$ H) R4 V8 Q5 d' G5 }! A </description>
1 Q: B5 ~/ z; g& P/ h <name>yarn.resourcemanager.application-timeouts.monitor.interval-ms</name>
3 o t, G( d+ o+ ~/ L8 j: b <value>3000</value>& c3 R2 l S7 k
</property> e8 z% ~0 U$ |" j& {/ |
<property>
% ~9 M8 M/ K4 h) b" R <description>
0 A/ o6 A) A( B Defines the limit of the diagnostics message of an application* {8 [% H) F% B J& a/ r% r
attempt, in kilo characters (character count * 1024).4 ] m- H N- v+ t0 s
When using ZooKeeper to store application state behavior, it's! g5 ]3 n2 R% P
important to limit the size of the diagnostic messages to
8 X7 F/ ^7 ?2 ]* k6 p prevent YARN from overwhelming ZooKeeper. In cases where
+ H6 d W, u& f7 f yarn.resourcemanager.state-store.max-completed-applications is set to
3 O: `1 ~: r; B( r5 m6 _/ ?8 O a large number, it may be desirable to reduce the value of this property; L( {6 E+ E$ u& i/ |
to limit the total data stored." ]# @( B# r8 v8 S5 `$ x
</description>" R; [% V l/ B1 K1 h) f1 ^4 P
<name>yarn.app.attempt.diagnostics.limit.kc</name>
v. |! E; v J4 o, p9 [8 n <value>64</value>; \" q/ k- E9 ]2 _
</property>
* i- ?6 y8 i9 }5 [/ E <property>+ l W: Y% Y) H- F; W
<description>
( A6 N; b" r1 X$ M6 N Flag to enable cross-origin (CORS) support for timeline service v1.x or
; n0 _3 o$ n: u+ R. E6 l Timeline Reader in timeline service v2. For timeline service v2, also add% ?* r% x* Y2 ~) ^4 a7 x9 r5 ^5 j
org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the2 B3 @* o* J# n+ a* z
configuration hadoop.http.filter.initializers in core-site.xml.
& f q: r6 l! O3 c: J4 v </description>; C. y9 d. d. i' u0 Z& |, _/ D. z
<name>yarn.timeline-service.http-cross-origin.enabled</name>
+ L4 S6 `2 Z" }7 ` <value>false</value>
1 j) P# f. `) W$ o' j9 Y) v* o/ X* W </property>
, ?; |2 w% \* ~, e <property>6 m5 L* v: }4 P% Y$ z% ~/ v8 ~+ A# b, [$ V
<description>2 `# C A% K4 S4 v
Flag to enable cross-origin (CORS) support for timeline service v1.x or) j( o8 Y7 |' \9 I3 A8 g
Timeline Reader in timeline service v2. For timeline service v2, also add
7 h1 k/ O$ C" p4 h* l org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the; ^& E5 ?4 e7 |$ o3 [# @9 d6 }, p
configuration hadoop.http.filter.initializers in core-site.xml.
: n& c% N2 O/ n7 P* T </description>. x" L" `7 q6 n% G8 P" @
<name>yarn.timeline-service.http-cross-origin.enabled</name>: D( M7 d9 s4 g) ^
<value>false</value>
1 @3 w8 }6 L, o! e( t3 ~0 Q* z </property>" L: R5 }4 P5 D) G
<property>: _$ K* \+ }' v( T
<description>
8 @: a9 D+ B2 `, I The comma separated list of class names that implement the& e3 M5 i4 D4 D/ ~! p7 v
RequestInterceptor interface. This is used by the RouterClientRMService6 n, j5 J7 ~) b" N2 o
to create the request processing pipeline for users.
3 y) J2 D7 y4 X </description>4 C6 Q/ W3 k; B# S# \1 t
<name>yarn.router.clientrm.interceptor-class.pipeline</name>
/ P+ h' [5 K& q+ u; { <value>org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor</value># P4 k$ [6 |$ c }, [( C) U
</property>: C8 K1 ^4 D1 s0 o( S8 {' M# @
<property>
+ ^. y( g3 Z, z& m <description>; M" \5 @* j9 s q8 E
Size of LRU cache for Router ClientRM Service and RMAdmin Service.
5 h( o! L& h" G* ]3 g </description>: n" Z- i& W& K* J7 ?5 D* ^. J5 |
<name>yarn.router.pipeline.cache-max-size</name>
2 c7 X8 r1 {* T ^ <value>25</value>
$ c* k4 q: p) |% | </property>' S0 \( R9 ]8 ^* `/ b8 T8 K7 n
<property>( y1 r/ i! h' u" U( D
<description>9 y9 Z. K0 {+ Z6 m3 O
The comma separated list of class names that implement the) M2 u& \ j5 X. L% h4 b
RequestInterceptor interface. This is used by the RouterRMAdminService: i4 ]2 p; r* }2 g
to create the request processing pipeline for users., t- ]* I/ b& J7 s" V- p
</description>0 ~# J/ k" i4 h& ?3 l
<name>yarn.router.rmadmin.interceptor-class.pipeline</name>
, H: K( b5 X' u) o! L4 n <value>org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor</value>3 _9 C# }, K* g( R/ ?
</property>
) T2 P" Y6 V# g6 `: J" L7 O" N# [% l <property>
. X$ d# q# Y" q2 l+ `$ g <description>
; N& ] p; c( w2 v' `! v+ g The actual address the server will bind to. If this optional address is
5 D" `0 p+ i* X: M( d4 D; F1 u( B6 P# F set, the RPC and webapp servers will bind to this address and the port specified in
9 \: {' d( b" c! o. A yarn.router.address and yarn.router.webapp.address, respectively. This is. x' B1 C; ^* [% x; }
most useful for making Router listen to all interfaces by setting to 0.0.0.0.
- k# y* Z% `7 V$ s8 W8 [ </description>& q. A8 _0 E( z+ O' K
<name>yarn.router.bind-host</name>% y( W* d+ I9 h5 X1 f5 z) w
<value></value>
* ?' W! u! I% L: X: Z# S ]% i </property>
2 \0 z- f; D% b; |2 r# h- `! D <property>
$ j1 _+ w8 C! j$ k) Y! A& B2 V9 E <description>8 @2 e! O% }$ n$ j) {# I8 \
Comma-separated list of PlacementRules to determine how applications
; s: {) \' t/ u6 Q2 ~ submitted by certain users get mapped to certain queues. Default is
' M, A9 u: c+ t; e user-group, which corresponds to UserGroupMappingPlacementRule.2 B& W$ \! V0 C- v0 [6 Y2 E
</description>6 i- J0 @8 Z& A6 @& n3 }
<name>yarn.scheduler.queue-placement-rules</name>- j% C* C6 g5 u0 r9 m, }1 [( ~. N. m
<value>user-group</value>
, ]0 h) b& `" j" k; l: i </property>
+ L7 u3 u; }: u# O <property>
4 x' ]' k1 X+ |3 X* t# v <description>! m5 v: f4 X/ I: h2 d$ ^
The comma separated list of class names that implement the6 R) `5 X7 H. @# w. t& W4 l
RequestInterceptor interface. This is used by the RouterWebServices* r: B' Q- e T- l
to create the request processing pipeline for users.
" _9 v0 f- z: a' f5 y' r- J6 Q4 J$ e </description>
! \" \5 `- j' g: A1 K, l" g. b# }, W <name>yarn.router.webapp.interceptor-class.pipeline</name>
' s) a3 k! X% G% s o <value>org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST</value>& [' h% g6 D/ g" N3 o9 m# p
</property>
3 S% X% b6 J) W" r& G, [ <property>
% Z8 ^& t8 R* d& j2 c) i1 E7 o8 I <description>
8 {/ X1 O$ @; i' V$ ^1 N% J The http address of the Router web application.3 I+ {/ R9 M$ I! `' K6 g
If only a host is provided as the value,
9 g4 _2 b2 s! |2 }( n- F the webapp will be served on a random port.
# D8 v1 Y r7 R3 c </description>
G2 g* H, R! K$ J <name>yarn.router.webapp.address</name>' d. f; H0 ?8 I
<value>0.0.0.0:8089</value>
7 ^, _. v. W: U% f/ I4 Z, Z </property>4 U2 [1 X& B! G
<property>
# c# H/ o. X7 q; ^( b6 _7 B <description>. a$ D" T, E5 F; r( `% E" t9 D
The https address of the Router web application.
: }6 K) x8 \; D# T If only a host is provided as the value,6 c+ M7 {0 v! s a( _ C$ W& B! B/ `
the webapp will be served on a random port.
3 A! f9 a4 }0 L3 K) e </description>
, L; B3 ~* p9 T+ [. f <name> yarn.router.webapp.https.address</name>
2 p& Z6 X) v. P <value>0.0.0.0:8091</value>
# }9 p( ?! g2 N9 s8 \0 z( P </property>
/ ?( N7 d c% L% O0 m <property>
) X, N$ Y( |/ c X <description>
) C; K A2 ~' n9 C. A It is TimelineClient 1.5 configuration whether to store active z& `: J+ s7 q: [$ N
application’s timeline data with in user directory i.e, v( `$ h S% d( \3 L9 j( I9 l
${yarn.timeline-service.entity-group-fs-store.active-dir}/${user.name}
( {$ T2 {9 T# Q/ F/ Q7 ` </description>
. O0 m O: Q7 t: f- U: k2 p, F <name>yarn.timeline-service.entity-group-fs-store.with-user-dir</name>
' g5 F4 _' J1 o9 U+ n) X, ^ <value>false</value>
% A' r+ r; W+ `; @: b2 o </property>8 n% T5 w) C: @/ u m- F6 i
<!-- resource types configuration -->% A& f, p6 i$ l1 v
<property>+ n5 g4 ]* w! o# q- m6 E: [
<name>yarn.resource-types</name>
! w- w" s4 @- W <value></value>) X, _" p G& x
<description>
. c% v$ I( s& |+ E& h The resource types to be used for scheduling. Use resource-types.xml
5 R2 M1 c8 E( P: ~4 F to specify details about the individual resource types.; L+ S- d' R+ i. Q
</description>' B) X: u: H2 z$ N7 H
</property>* v3 U" I! N4 Z T1 P
<property># [, X" k' l. Z" U" R2 j' i
<name>yarn.webapp.filter-entity-list-by-user</name>3 O- Z& Q: t7 m" T2 C! }
<value>false</value>
2 Y2 S: l2 e3 U: p" U" M <description>6 q3 t* y2 W2 n4 Z+ E8 U
Flag to enable display of applications per user as an admin7 l: j3 l/ `0 Z1 M$ t! _
configuration.. c4 U& S) U, v
</description>/ l; o' p. F+ v
</property>9 k0 }$ B' q5 V% G
<property>
1 r0 W; X9 K1 U' t/ o <description>
- d f7 n4 _2 c, y$ y The type of configuration store to use for scheduler configurations.( t, ^0 L/ U8 x2 z4 M P! r
Default is "file", which uses file based capacity-scheduler.xml to
5 r$ R+ [8 }& i retrieve and change scheduler configuration. To enable API based
" p4 g/ v2 W, d' J4 J! z scheduler configuration, use either "memory" (in memory storage, no
8 j' V T- |+ x8 A4 f: [, \ persistence across restarts), "leveldb" (leveldb based storage), or# }, Y9 u4 ]1 I% T/ i
"zk" (zookeeper based storage). API based configuration is only useful9 K: B7 s0 ^* i# b ], P7 r, ^
when using a scheduler which supports mutable configuration. Currently
4 ^+ P# _9 z! B only capacity scheduler supports this.* q1 f! d' L" r9 Q c, ~+ m
</description>
1 l7 {* Y3 g/ W5 W% @ <name>yarn.scheduler.configuration.store.class</name>$ \+ D) p# E- x: V' i, R; v
<value>file</value>
* |5 L# ? L9 @7 w9 w- ^4 ~ </property>
3 ?( `+ ~8 O0 V. g" R9 b" v+ C <property>
, A1 Y! e- }; y2 W b <description>. x: k# h& R+ ]: v
The class to use for configuration mutation ACL policy if using a mutable
) s3 g+ s- y: C) V2 v( h4 l configuration provider. Controls whether a mutation request is allowed.3 ?( w# c; U S6 w% \
The DefaultConfigurationMutationACLPolicy checks if the requestor is a: K% H' T* A! v* Q4 H
YARN admin.7 j1 R' N7 N8 X) h, G# l
</description>1 x6 K& x) z4 q2 o& Q
<name>yarn.scheduler.configuration.mutation.acl-policy.class</name>
* Z5 |/ M) Y+ B! @' c& d <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy</value>
$ P3 E( B: J* R- ?. y6 M </property>
$ ^. c: o- f* @ <property>
C- j' J, K2 ?3 O7 _ <description>4 g+ A9 z7 E! ]$ \
The storage path for LevelDB implementation of configuration store,1 A% B* ]8 S% i
when yarn.scheduler.configuration.store.class is configured to be
9 ~# \$ j# d- ~5 t1 G "leveldb".
% m- T6 b: h! Z: I4 E, _; Q </description>
1 O6 G$ Z* n7 \* F <name>yarn.scheduler.configuration.leveldb-store.path</name>
) k' F+ b! M6 C. p <value>${hadoop.tmp.dir}/yarn/system/confstore</value>9 } v* E8 Y, @) S8 [0 R( z
</property>
; I, l. @/ c# ~- R: w: ? <property>
3 S4 \% ?, u3 }8 W- L5 P <description> p- U* ?4 ], p& w% x
The compaction interval for LevelDB configuration store in secs,
2 E2 m* g% |6 l- U& t& r when yarn.scheduler.configuration.store.class is configured to be
! Y3 U W0 s# y# u "leveldb". Default is one day.' I( B/ \$ Y* T! u+ j- }
</description>. z/ {& W; u+ A r9 T
<name>yarn.scheduler.configuration.leveldb-store.compaction-interval-secs</name>) R7 Y F/ C. ` a
<value>86400</value>
0 _7 J/ K3 I& \" a1 K </property>
: N) m/ a, b5 N" P! j% C& ~ <property>6 \7 M% m& j T& I/ a% p4 s& U
<description>+ u, Q) P' g; E
The max number of configuration change log entries kept in config( \4 N, K/ ?' R) C& K
store, when yarn.scheduler.configuration.store.class is configured to be. @! _) n2 m4 @0 i# y
"leveldb" or "zk". Default is 1000 for either.
, d. {, o' ?$ \ </description>1 t" n* q% R3 i6 O
<name>yarn.scheduler.configuration.store.max-logs</name>
% ^2 f( [1 \+ I5 | <value>1000</value>
|" l* E! h: e4 |/ }; s0 ?: j) n </property>/ F3 Y1 Q; t: {
<property>
f; U0 g! I8 i! f <description>
. `( _1 u, e( ~+ P( e1 k ZK root node path for configuration store when using zookeeper-based
+ D+ e( {8 n( a0 X configuration store.; L' E+ I; a& x& r/ T. _ E' v
</description>
' u5 A2 d K j# B- X$ b" [ <name>yarn.scheduler.configuration.zk-store.parent-path</name>, X2 q6 j; y* p: k/ ]7 L' [5 X
<value>/confstore</value>
# c9 B1 T+ ^$ ]5 g </property># u g% C2 A. t$ a6 b& f2 _
<property>- \0 |( \- C G+ @
<description>/ u! ^+ _, I8 K7 d P# z3 Z8 H
Provides an option for client to load supported resource types from RM# U5 q) C( O: e+ ?
instead of depending on local resource-types.xml file." ?! b* o% ], g' g: q! |
</description>: h- N6 f; W1 P/ c* w2 a: e
<name>yarn.client.load.resource-types.from-server</name>
! U1 ?$ ?6 y( N- ]7 Q0 `1 T <value>false</value>
, k1 i, g7 Q" |$ O </property>
) {6 ]+ ?9 S1 f" E; A" H0 f. ] <property>1 q1 i9 Y/ \6 F) d- M! K
<description>
, s! x8 V- |; l$ S6 `1 S3 f When yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto specified,4 t" ?6 e8 ]' x
YARN NodeManager needs to run GPU discovery binary (now only support
# n I6 W" |- d5 @# c nvidia-smi) to get GPU-related information.
x: F: m$ h. |; K: a: R8 F When value is empty (default), YARN NodeManager will try to locate
6 T# j8 T u' K# `2 Z, g; ? discovery executable itself.
0 T1 g0 w8 x3 z' o( T, B An example of the config value is: /usr/local/bin/nvidia-smi
, ^5 W }/ [9 }6 g4 l1 C, ^+ G </description>) e5 A. y8 | j. R* z
<name>yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables</name>
% @" w. t" ]6 r w) e& Z- C! v5 X <value></value>
" X8 i+ w& ^/ X R h </property>
3 L: i n! s$ @7 J! P A <property>8 q6 ?1 P6 M. r- U R
<description>& Z I9 F, y. _2 C) R4 G, ~
Enable additional discovery/isolation of resources on the NodeManager,
' o" U# { J* ~/ b split by comma. By default, this is empty., i# u# z" }' _/ p" m
Acceptable values: { "yarn-io/gpu", "yarn-io/fpga"}.
+ S! j' ]- ^3 x </description>
# H+ t( |( g) U* G" J v <name>yarn.nodemanager.resource-plugins</name>
4 d1 c; Q: S4 V% k <value></value>0 s0 \. c+ ?0 Y7 S% x
</property>. |* c+ J9 n4 p7 j
<property>
5 g& ]8 w% ^& P2 |: W, T5 O5 P4 _ <description>
8 |; J% f2 ~: x* ? Specify GPU devices which can be managed by YARN NodeManager, split by comma8 n- X# U4 c0 Z
Number of GPU devices will be reported to RM to make scheduling decisions.
* t. Y0 P4 b5 O Set to auto (default) let YARN automatically discover GPU resource from
) Q& M8 I$ n( | system.* O8 y [4 s9 `/ [, h
Manually specify GPU devices if auto detect GPU device failed or admin
V% d7 {. y' W0 [8 @ only want subset of GPU devices managed by YARN. GPU device is identified
/ `- ]+ Y' T# v by their minor device number and index. A common approach to get minor
) f8 x+ s; l; D2 `9 }1 O& _ device number of GPUs is using "nvidia-smi -q" and search "Minor Number"
X. f- n, }5 L3 _ output.
+ z4 k1 n& X$ L$ @, A! T When manual specify minor numbers, admin needs to include indice of GPUs
) h. R3 Q* Z, ^. U2 C0 r6 g2 l as well, format is index:minor_number[,index:minor_number...]. An example
K; y- O) b* m5 m of manual specification is "0:0,1:1,2:2,3:4" to allow YARN NodeManager to, A, c: z# J- c; s/ `
manage GPU devices with indice 0/1/2/3 and minor number 0/1/2/4.9 U/ M& E, r6 U
numbers .
L/ L- B9 o2 o7 `* `# b </description>
, Z3 B! Y7 k) }/ c3 ^' x; K$ O8 u8 Y <name>yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices</name>) o/ @6 s% `0 a9 B- T: Z$ `
<value>auto</value>
$ b: [0 N: r: |% H' d/ G4 H! _( V </property>& O! ?. i8 ?( D, z' l/ Z' ^
<property>7 s6 [4 d; \2 X1 v' H7 P# Q
<description>
, ]# r# W* P m# b8 i Specify docker command plugin for GPU. By default uses Nvidia docker V1.
6 t9 P2 J8 j4 w( F5 v; M </description>: a: L6 T" ?3 |5 t0 e/ \: A
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin</name>7 [" A( ? s2 p+ P5 [ M
<value>nvidia-docker-v1</value>
T9 ^. z& c$ u2 U </property>% @1 ^0 L; Y+ `+ Q
<property>
) K* l( |0 i7 V f/ b& ?6 G <description>' h9 r2 g0 i/ L* n! ?
Specify end point of nvidia-docker-plugin.4 D. \! J5 _$ p; t1 Z/ P; M* l
Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki
( r7 z0 Q( w) a For more details.* f4 A$ j% @: \! z3 p. N' ^
</description>: C8 C, X6 L3 _6 q6 J+ L. ^. X) }
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint</name>" i k# \- i p% m
<value>http://localhost:3476/v1.0/docker/cli</value># g6 P& _, y0 `+ s
</property>
" q0 N( M. \4 o) c' @7 } <property>
; {! a' d+ R3 B! ^5 s2 ~' ` <description>5 R( S4 u+ t- q' f
Specify one vendor plugin to handle FPGA devices discovery/IP download/configure.
% I8 H) J% k2 g, ~8 T+ m* D, m, C Only IntelFpgaOpenclPlugin is supported by default.6 H$ H- H! s( @3 _$ |% G
We only allow one NM configured with one vendor FPGA plugin now since the end user can put the same3 X% l- d) k& x
vendor's cards in one host. And this also simplify our design.
. P8 d9 C4 O- Z1 g- z5 k </description>
* P) e9 W% `$ [% L! e/ e <name>yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class</name>
3 D7 W+ [8 W/ M9 X& \9 j8 I( @& X <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin</value>
# D/ D! U3 W8 m+ h& i </property>
( c1 [& e6 b! @* k E <property>5 l/ d \ w* W1 g; e
<description>
: E+ X9 m% p0 I% P! v$ h When yarn.nodemanager.resource.fpga.allowed-fpga-devices=auto specified,3 O* X' _- q" O
YARN NodeManager needs to run FPGA discovery binary (now only support
3 v- p$ Y: e$ q. z x IntelFpgaOpenclPlugin) to get FPGA information.
8 y0 A8 r1 W! L4 C5 t L When value is empty (default), YARN NodeManager will try to locate
R9 @2 }5 P4 l1 i( }' h, e discovery executable from vendor plugin's preference; I1 u( t, r# f# T4 s4 v! ?
</description>
- L, n: {, _+ C* n+ Z" l; g <name>yarn.nodemanager.resource-plugins.fpga.path-to-discovery-executables</name>
" F0 e/ {7 G! ?$ ^2 ^ <value></value>( O0 b$ U& M. C8 @/ a+ V+ C
</property>. k H- b* S+ y. \, l
<property>
8 [0 j1 K6 _, @# ^& }; \ <description>3 ]2 l ?% ^7 D% n9 A
Specify FPGA devices which can be managed by YARN NodeManager, split by comma
4 |% F0 j6 o( \9 m Number of FPGA devices will be reported to RM to make scheduling decisions.# V7 P7 j7 ~# e9 c: N
Set to auto (default) let YARN automatically discover FPGA resource from9 u1 C& N3 l w/ X: R& `9 \
system.2 ~+ ]/ c% {9 Q7 B7 k q6 }
Manually specify FPGA devices if admin only want subset of FPGA devices managed by YARN.
# _. _# s! l& X+ @ At present, since we can only configure one major number in c-e.cfg, FPGA device is4 ^0 r3 y$ r% {$ u" x
identified by their minor device number. A common approach to get minor) S1 Z; _1 c* s0 G2 o' C) ]: a) L( X
device number of FPGA is using "aocl diagnose" and check uevent with device name.
4 q: V8 M& H/ `1 Y2 ^9 w6 L( _1 h; t1 { </description># h" t: R, }$ ~& P' W! Y
<name>yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices</name>0 M x9 Z" Z7 {$ H
<value>0,1</value>
7 J6 H% k( ?& f) Y0 j q </property>
4 k- i4 c" r% J6 H <property>
! j* M- [0 K+ b/ A# d# L <description>The http address of the timeline reader web application.</description>
# H0 g! O/ ?! w Q+ p+ j" u <name>yarn.timeline-service.reader.webapp.address</name>, V5 x0 ^+ D2 q
<value>${yarn.timeline-service.webapp.address}</value>
" G% j5 i: ]) D L </property>' A' H4 j3 \# C; c
<property>- d" I j: Y( R
<description>The https address of the timeline reader web application.</description>
2 x+ Q; k: h& X0 L2 P, q <name>yarn.timeline-service.reader.webapp.https.address</name>
1 P4 x" e5 K. G( }' i8 @9 E' S <value>${yarn.timeline-service.webapp.https.address}</value>! ^' a- {5 P( l. p: @2 F8 E
</property>
$ g: }7 g5 |. S <property>
* h* [6 x: a8 x+ g- G <description>7 z, A' @2 Y9 E2 ?* _
The actual address timeline reader will bind to. If this optional address is w( n- U: r! b5 j
set, the reader server will bind to this address and the port specified in% o: ]. W! e8 O* j
yarn.timeline-service.reader.webapp.address.
* w9 K7 ]3 k9 z! J1 A% V# b This is most useful for making the service listen to all interfaces by setting to
8 z- q8 z; U1 S3 j' n 0.0.0.0.
Z# b5 Y$ f. h* N3 W; c' n </description>3 H3 T' ?# F; H: b$ S0 s, U* f
<name>yarn.timeline-service.reader.bind-host</name>3 G5 U. c$ [. p
<value></value>- A* e* V9 d* a$ o) a
</property>
7 P) x2 s& m e8 v% J# @. ? <property>% D9 J9 t; r+ ^; I5 n% h$ n
<description>
5 {% Q( P6 i8 \6 ~: b Whether to enable the NUMA awareness for containers in Node Manager.
% T) ~6 L/ @& j4 b9 m" z) m </description>$ Z7 {" j2 l/ Q
<name>yarn.nodemanager.numa-awareness.enabled</name>
7 a7 `2 |# l/ K3 q0 B <value>false</value> V: w3 x( s& Q# e# h; U5 {
</property>
+ O% ~" }7 I {% F5 D1 F% k <property>
- T; `0 y+ x/ @3 p <description>
) E! n" Y4 `; d3 q) J+ J) Q Whether to read the NUMA topology from the system or from the
0 a0 y- c; ~$ Q5 P, T4 V) a; i configurations. If the value is true then NM reads the NUMA topology from+ C* `7 I5 i l' b, C: ]& a
system using the command 'numactl --hardware'. If the value is false then NM
, T, l9 }1 j$ Q: C reads the topology from the configurations" E- W( q) n' `$ Q* D
'yarn.nodemanager.numa-awareness.node-ids'(for node id's),8 Z; C" r7 D9 B! X/ ]4 T* G& `2 }6 K
'yarn.nodemanager.numa-awareness.<NODE_ID>.memory'(for each node memory), X% V0 G7 d) L& D6 T
'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus'(for each node cpus).
$ @' O- `1 M1 A, G0 d </description>$ \ m. O4 J! s v6 y- t% H
<name>yarn.nodemanager.numa-awareness.read-topology</name>4 M3 v7 i5 K& f5 c7 x
<value>false</value>: ?- t! D9 @2 i: P4 Y* M
</property>
. x7 D, u' S; u9 B <property>% M/ w7 T7 |. M }1 v
<description>
" s ]/ O& b) c2 I) g8 b$ k) g+ K# I, Z3 n$ ? NUMA node id's in the form of comma separated list. Memory and No of CPUs
2 d% a7 `5 Z" d# B* q6 f( s# p) j! ~ will be read using the properties3 E6 S* q6 \4 H5 A
'yarn.nodemanager.numa-awareness.<NODE_ID>.memory' and
; h( O$ {' L* _! n/ y$ N- r. Y2 V 'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus' for each id specified. c5 M( A6 Q# z0 I% |0 G' ?
in this value. This property value will be read only when% i# c# a6 P9 V; C
'yarn.nodemanager.numa-awareness.read-topology=false'., m% W3 j7 F0 i5 h4 N3 D
For example, if yarn.nodemanager.numa-awareness.node-ids=0,1* v( x! _, J% `- W/ |
then need to specify memory and cpus for node id's '0' and '1' like below,
% E0 c4 \2 i G5 W& M( O# z yarn.nodemanager.numa-awareness.0.memory=73717
' r. | g% s( w- I; H yarn.nodemanager.numa-awareness.0.cpus=44 Y7 E5 J. s: G# u
yarn.nodemanager.numa-awareness.1.memory=737273 b& H+ S2 I( r% s( [2 b1 d7 A
yarn.nodemanager.numa-awareness.1.cpus=46 x4 o* w& G) e) i0 I4 S
</description>
! L% \. S5 f2 Z/ s) X <name>yarn.nodemanager.numa-awareness.node-ids</name>
/ v* @6 K- S+ c2 r* z _/ p" _ <value></value>* b* f+ Y! s6 V. i
</property>2 I3 B" T; [! y
<property>8 I# i, P3 Z, _. K) u g6 t
<description>& e/ Z( A0 D6 s
The numactl command path which controls NUMA policy for processes or: Q8 l; H1 t+ m8 r6 m
shared memory.+ Y0 e9 C. N8 o4 k2 p3 R b+ {
</description>
* H1 C: q4 }: U9 h <name>yarn.nodemanager.numa-awareness.numactl.cmd</name>7 {' n ~) b0 J8 O
<value>/usr/bin/numactl</value>
# `. \+ V; m% r </property>" O5 ]0 n5 w3 P @) d+ N% T
</configuration>
, b0 t% j, O' V/ |) w: J# w; b+ L& h5 [0 a8 S. V3 o5 h
|
|
发表于 2022-11-7 11:18:00