|
|
楼主 |
发表于 2022-11-14 13:52:09
|
显示全部楼层
<?xml version="1.0"?>
* u* M- i8 n' T2 o$ {8 k<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
/ l$ B) E4 F9 ^# N+ n \<!--
& `* x5 |' e* q: a7 E Licensed to the Apache Software Foundation (ASF) under one or more# E' `) d! ^1 [+ |2 K7 U$ A
contributor license agreements. See the NOTICE file distributed with& b& d) ^& n* q, m. x" m h
this work for additional information regarding copyright ownership.
1 c0 K# Y8 ] ]" F! T. {6 a The ASF licenses this file to You under the Apache License, Version 2.0+ G- g4 z+ o: i6 J" G3 Q
(the "License"); you may not use this file except in compliance with9 L% ?1 O+ R) ` I, w- S6 m
the License. You may obtain a copy of the License at
2 ^! K2 ]3 v, V) r0 h* ]. T http://www.apache.org/licenses/LICENSE-2.0
2 n/ v V# ?$ u- X Unless required by applicable law or agreed to in writing, software
" `" a3 h" W& u7 p7 l9 X distributed under the License is distributed on an "AS IS" BASIS," T' z/ O& T/ X3 Y6 p
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
4 W5 e! p X- @+ d- p9 \. a See the License for the specific language governing permissions and. P" m5 `1 ^; z$ R; L2 s* K- G
limitations under the License.
& q. y% p" G; U-->
, y* s: l% S7 ?. M7 ^+ a. d% [) C<!-- Do not modify this file directly. Instead, copy entries that you -->/ r. [9 G% B$ k% y. j' R) y
<!-- wish to modify from this file into core-site.xml and change them -->
{( p: \* g8 I% L9 d<!-- there. If core-site.xml does not already exist, create it. -->
A# I0 X D7 G9 F9 [8 {<configuration>6 ]7 }, d( O6 R* H! A y& P
<!--- global properties -->
6 f! | M- [1 {& U' y) l<property>* A( o$ G1 Y# L2 \* q
<name>hadoop.common.configuration.version</name>
3 J# K8 Q. p: O <value>3.0.0</value>4 {6 Y) [ }8 a* m# t
<description>version of this configuration file</description>
( l. `# R9 q/ c' v6 {$ e4 g</property>
. }1 X' g2 A* C3 H: Y<property>
! R4 L) [9 x, K$ y, j6 d/ } <name>hadoop.tmp.dir</name>" d& A( z: y' q# z7 v
<value>/tmp/hadoop-${user.name}</value>& k m/ ]+ Y, b$ S) t# c
<description>A base for other temporary directories.</description>6 A4 t0 a: {3 O& ?# S5 v
</property>
M2 |! {3 v# r/ V<property>4 ?) X$ ^/ N+ H
<name>hadoop.http.filter.initializers</name>9 x( @ p7 l, n! ?' O- `0 A
<value>org.apache.hadoop.http.lib.StaticUserWebFilter</value>
9 w+ o K% T9 A N3 H8 l& t2 n <description>A comma separated list of class names. Each class in the list
0 g$ }3 e- f9 F8 f* f/ p! Q must extend org.apache.hadoop.http.FilterInitializer. The corresponding
( i g9 P( L6 `& t' I Filter will be initialized. Then, the Filter will be applied to all user# n2 ]( A: o# A
facing jsp and servlet web pages. The ordering of the list defines the. z8 X% c( J! Z7 F$ q
ordering of the filters.</description>4 i( p+ @ s0 m* J
</property>7 d7 ^ G/ d2 O& C
<!--- security properties -->
1 ^0 q5 x- q" R( E7 s# V( M<property>5 @0 e$ t7 V- e O- m) X) r
<name>hadoop.security.authorization</name>6 X, n) x( H$ |1 k
<value>false</value>1 W( P% t- v% @/ Q# k" }
<description>Is service-level authorization enabled?</description> u, |. m3 u$ X. ^, n' m
</property>
# F3 v# S; `- A2 Z<property>( g- Q; x6 _$ P8 n
<name>hadoop.security.instrumentation.requires.admin</name>
6 _5 T- G: B8 A6 O0 o5 e5 v/ [ <value>false</value>
+ h7 f6 j0 G. F <description>
8 T5 A8 K4 \0 q" P8 O+ A Indicates if administrator ACLs are required to access3 w7 ]1 R$ Q6 ^7 `
instrumentation servlets (JMX, METRICS, CONF, STACKS).& ?/ R* C. _/ i' I7 @/ ^# [
</description>+ ^4 h+ f' L, Y3 I- y1 b" ` P, D
</property>5 Y. Q2 H" v! p/ Y+ e3 [1 f0 A9 q
<property>: w( j. G1 C4 x, `) J. s; C
<name>hadoop.security.authentication</name>
( q+ D. q5 d0 L, J. b& d5 b <value>simple</value>6 B, r* p. l8 U
<description>Possible values are simple (no authentication), and kerberos% h5 H& n* F0 f( U# @, G& s- b
</description># h6 C0 |# \! j
</property>
. |8 A% s1 m; U' O; w; c$ T2 x# }<property>
/ S- O' \. x" k <name>hadoop.security.group.mapping</name>- W! P2 ? z p" ?: Q7 N: Y h
<value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>8 Z% ?6 l. @: J5 O$ d m3 c+ J$ m. g
<description>
; Q8 B( l" x3 n6 a V; f( E Class for user to group mapping (get groups for a given user) for ACL.
: Q% }/ n8 u4 V1 _+ R% r The default implementation,( W; V( b. G- n: \7 `+ ^: N# a
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback,
$ v- s4 \3 p0 V will determine if the Java Native Interface (JNI) is available. If JNI is
' g, W$ M; z; W2 g available the implementation will use the API within hadoop to resolve a
) D3 o9 O" }6 u6 ` @. r list of groups for a user. If JNI is not available then the shell: V2 q: y3 {7 s0 G$ \+ u6 k
implementation, ShellBasedUnixGroupsMapping, is used. This implementation! t; _9 ^* _* ^) I g9 n, p4 i
shells out to the Linux/Unix environment with the
* @2 ?, c. h" ?& f6 ^: {4 j <code>bash -c groups</code> command to resolve a list of groups for a user.$ q3 G* w1 {0 |3 L* _& V: ]
</description>
8 q% k& L* e( C" q4 I& I</property>+ K9 A- R$ b/ |# i" _9 I* i- I
<property>2 q/ ^0 G( l" B% ?' {5 P9 R
<name>hadoop.security.dns.interface</name>8 Q% F& E" k/ q/ k2 Y; F/ D8 x
<description>
* B6 ~8 L/ p z7 I& ? The name of the Network Interface from which the service should determine
& l5 U) b( P* f+ S1 F' }% R8 {) G- K its host name for Kerberos login. e.g. eth2. In a multi-homed environment,
8 l. }9 v& O z- \" L5 s the setting can be used to affect the _HOST substitution in the service
) c! t) ?# |2 j Kerberos principal. If this configuration value is not set, the service/ U5 u( y8 d4 k9 d2 {% k! Y
will use its default hostname as returned by* R3 F: w4 |6 n+ z* Z9 i: c H
InetAddress.getLocalHost().getCanonicalHostName().
# y; |4 m; `( F: ` Most clusters will not require this setting.2 a2 N- {& y( x- K1 d( }$ a+ d
</description>
' T# Q0 N% b; h- K; T+ d! ?</property>
$ }1 a' {7 u; h6 _<property>
9 i5 e5 P6 W( C2 ^0 T V- X k <name>hadoop.security.dns.nameserver</name>
1 C% E- Y) x0 x! {9 p6 f <description>
! K1 n3 R# C) A; p4 a The host name or IP address of the name server (DNS) which a service Node6 d/ Y$ r2 ~" v& Q' w4 O3 f
should use to determine its own host name for Kerberos Login. Requires
1 H$ M. k' I, k: X; P7 a/ e! F hadoop.security.dns.interface.( N! c9 N; S2 E9 {8 H& Y4 ^
Most clusters will not require this setting.
K& _6 G H0 f" z/ J L6 w1 v; W$ d </description>" [1 w& ^3 L6 Q5 ]$ Z! D
</property>
) l/ Q/ q% V' y/ ]<property>! r! U: H: |! w5 F4 {
<name>hadoop.security.dns.log-slow-lookups.enabled</name>: h' m- E+ V& g y! x: y& J4 n' V
<value>false</value>3 @& d/ G1 e- B6 R6 u: M; s: I7 s; C, ~
<description>* ^7 U B$ Z ^( N* T3 F
Time name lookups (via SecurityUtil) and log them if they exceed the% _+ u( i! l2 V* y
configured threshold.! h H8 o; ~ c* R( Z) h
</description>6 K0 i; i3 n6 }: t' n0 T. D7 G4 z" ^
</property>% O' F e3 t3 h' N
<property>) O/ N) z9 J+ l) q# g
<name>hadoop.security.dns.log-slow-lookups.threshold.ms</name>
$ t5 E- X' M3 `* A <value>1000</value>
* M% e* p3 t7 T' G <description>' Z" P5 u; Q- r" Q
If slow lookup logging is enabled, this threshold is used to decide if a
; C3 [0 S/ q% ?& i* B2 y% m2 u N" H lookup is considered slow enough to be logged.
3 x! p& W9 i. q7 \( J$ Z </description>
1 P# ]% P& w# d' @9 X</property>2 U0 t. u+ `8 W$ ^5 Y( p, n1 s
<property>
5 @# S0 L$ ^+ ] <name>hadoop.security.groups.cache.secs</name>
: o) u% z R& u$ q2 Q <value>300</value>
/ ~( p0 y% q$ D7 m6 i1 l/ I <description>
+ u$ C2 Z5 I" |7 {) q2 r This is the config controlling the validity of the entries in the cache l3 }& i# K2 T% U4 N
containing the user->group mapping. When this duration has expired,5 C% F: N- X5 V" T( s
then the implementation of the group mapping provider is invoked to get
* e+ Y9 G( N: c$ t0 {) }9 T' v8 y the groups of the user and then cached back.9 E3 I0 l! I% ^6 _9 I9 M
</description>0 c5 [6 x! [+ y& m
</property>
. }" q I7 S! q9 n! s<property>$ a* Q7 Y1 s& ]3 n
<name>hadoop.security.groups.negative-cache.secs</name>; s" D G. p- o1 L% W4 D
<value>30</value>$ m$ L4 U; s$ E3 Z: s
<description>- {' m/ b$ K) M; R! k1 g" L
Expiration time for entries in the the negative user-to-group mapping7 U$ I6 E, y5 w# \& @ P
caching, in seconds. This is useful when invalid users are retrying
3 V, b0 P3 K/ R: B( ^: g& e- o frequently. It is suggested to set a small value for this expiration, since0 U; V t$ K0 L, D, [& [
a transient error in group lookup could temporarily lock out a legitimate
9 B8 v/ Z) z. n- `2 S. G( x user.
2 _; u5 w/ @) A4 w9 Q Set this to zero or negative value to disable negative user-to-group caching.
# V. Y; S, M3 `( m# q3 \ </description>
1 k/ D) m5 X) s</property>
3 o! b4 J% f: R0 r; T* z% j3 Y<property>8 A( \9 W/ z! U# Z3 v
<name>hadoop.security.groups.cache.warn.after.ms</name>; T* Q) w) b' ?& p! J$ ~
<value>5000</value>+ P M$ `7 n* S5 @! Z* Y _4 o
<description>! l6 S3 l$ N( H5 ?
If looking up a single user to group takes longer than this amount of/ Z0 v J c, Q/ I" i
milliseconds, we will log a warning message.9 A8 }; K! f7 \' v% x
</description>& T( z3 R0 ?# R6 `2 j& n5 _8 o5 E
</property>
1 l" @% h p& D) p2 g<property>
( v) l& ]: ~& e* R, {1 U$ u <name>hadoop.security.groups.cache.background.reload</name>" s" K ]$ F; Q& l0 l6 o# v( d
<value>false</value>
3 p3 Z3 m9 t% t4 W4 {7 c* o: l <description>1 g) W2 ]* e8 t6 B y, o% D! n, ?
Whether to reload expired user->group mappings using a background thread( M: E- Z8 X; A( g+ g8 d% l, p2 Q
pool. If set to true, a pool of! I" x0 I* g5 q: ^$ R' M. g3 ^
hadoop.security.groups.cache.background.reload.threads is created to1 o! ?2 t: b: i! G
update the cache in the background.7 j7 _5 L0 I7 R- s- ^: h6 B
</description>! @; k1 M" S. i9 U- {8 {
</property>2 r) y+ L: b1 p7 p+ H+ ^+ m
<property>
% n; |/ S7 I6 K2 s2 q <name>hadoop.security.groups.cache.background.reload.threads</name>
! P1 l( g8 ?7 I# E" r <value>3</value>( O* x) H% }* g% Z4 u& ?! z- C4 t
<description>
( ~( `- ?$ R4 S: N( C Only relevant if hadoop.security.groups.cache.background.reload is true.; v$ J. P4 A) e/ b" E. i) z- z
Controls the number of concurrent background user->group cache entry+ u) N7 @! x& X; B7 Z$ N
refreshes. Pending refresh requests beyond this value are queued and
1 |4 _+ v7 v5 W1 \ processed when a thread is free.
3 F( ]+ B2 A' w) v </description>
' g7 E$ k0 `$ d/ b6 z6 M1 S</property>5 |* H- B! e8 ~8 d4 M: s
<property>2 n+ D8 _6 V2 C ~2 v; U3 n
<name>hadoop.security.groups.shell.command.timeout</name>
; m% P$ O! y- `5 l t <value>0s</value>
, K: o. q- K3 ^' k6 W <description>& P8 a0 L$ I6 W6 |# I
Used by the ShellBasedUnixGroupsMapping class, this property controls how
2 a! j" w# {6 O& s% T7 X5 g long to wait for the underlying shell command that is run to fetch groups.
+ C* L q+ F4 e+ [) u' [" r5 P& w Expressed in seconds (e.g. 10s, 1m, etc.), if the running command takes
' N- y* I5 y5 q7 ]% w. a6 ? longer than the value configured, the command is aborted and the groups$ r3 b- e9 U. m e. Q! [6 v( x
resolver would return a result of no groups found. A value of 0s (default)
3 C; q! _5 g7 V! j, ^# W- t6 W* _ would mean an infinite wait (i.e. wait until the command exits on its own).3 { Y! K" a7 e
</description>
* z; z" _) u: }</property>0 [( u' z. i( C6 m5 R
<property>9 j4 t6 Y. e+ Q7 v- k( J
<name>hadoop.security.group.mapping.ldap.connection.timeout.ms</name>) I" S- Z3 {" t* A7 V6 m3 x Y
<value>60000</value>
+ Q9 L7 s4 }# b2 l- r, l <description>7 K; Q) A) |. y( z& f8 X4 E
This property is the connection timeout (in milliseconds) for LDAP
; C: v* ]; X# S |7 q! Z operations. If the LDAP provider doesn't establish a connection within the
& }( b) e" @4 U+ y7 p specified period, it will abort the connect attempt. Non-positive value
/ b8 d8 o( o8 D" R& C8 } means no LDAP connection timeout is specified in which case it waits for the1 D5 a$ O! _6 Z* \; T* {
connection to establish until the underlying network times out.
; | [" E: q2 _! a! a. z </description>4 h( f9 a- Q) k# E' P* C2 a1 j; O
</property>
+ C* O# Q3 ^' ^" V4 I) q* |<property>& q' Y/ C0 ?" a8 p9 u
<name>hadoop.security.group.mapping.ldap.read.timeout.ms</name>
! T C2 f& N/ @% ]( x <value>60000</value>
7 l& C1 K6 X8 j! z4 w# Q9 } <description>
- p+ C3 r" P4 G" m This property is the read timeout (in milliseconds) for LDAP$ W# R5 x& V' r; b# X5 g
operations. If the LDAP provider doesn't get a LDAP response within the+ e' @7 s! O- s
specified period, it will abort the read attempt. Non-positive value
2 {8 g8 s; S% h3 |7 ` means no read timeout is specified in which case it waits for the response; [3 t- Z0 ^) V! b8 i
infinitely.
+ `7 v3 y. C- j3 N* z0 M( D# A </description>
, Z. B7 B! T6 v3 H5 s u% ?</property>
( w+ x& t! B6 J& e<property>3 E. t; N: Y b' M) ^
<name>hadoop.security.group.mapping.ldap.url</name>
5 x& _, v- [- R* n. o <value></value>! ]4 s; L9 q2 S$ S/ J! Q! z2 \ Z, l" C
<description>; n8 Q. @* T6 b' h! V1 t
The URL of the LDAP server to use for resolving user groups when using
$ X' H: T' m: ?$ o3 w5 e+ J the LdapGroupsMapping user to group mapping.1 x9 a% i1 K$ T6 V* c
</description>% n' s+ l, e% c
</property>
" P7 X4 D* b1 O; _2 p, r# ^<property>
3 e. _7 {2 O/ T" s4 v" \ <name>hadoop.security.group.mapping.ldap.ssl</name>6 j \7 R E/ g) z
<value>false</value>* [& h' A+ Q/ |
<description>
) I7 ]( H$ K0 Q g) M, W- z) E Whether or not to use SSL when connecting to the LDAP server.
, K w) Q% Z7 K </description>
3 e* ]$ S- P4 }; @+ e* V0 e$ A</property>
9 v( C' a9 }1 P. I4 Q/ `" I<property>
: s$ q7 x9 L5 i/ R7 x% h <name>hadoop.security.group.mapping.ldap.ssl.keystore</name>1 t5 {* i. F* v- D( M5 Q
<value></value>
) k- P6 q3 y& n5 A <description>/ S0 u" s' N4 J9 c, X0 l
File path to the SSL keystore that contains the SSL certificate required. o, y2 ]5 k; e9 ]0 {
by the LDAP server.
2 C# L+ Y8 c/ ]. ^) C </description>
2 ]; x) R1 K J1 V4 a' V5 n# Z</property>
* P9 `3 y$ N% R<property>
6 J$ N- U: K L( K8 `9 s5 Z <name>hadoop.security.group.mapping.ldap.ssl.keystore.password.file</name>) N! q6 B. N; t8 k9 ^& G& R, {" q
<value></value>
) Y! r# C X* _ <description>1 ?3 }' S. Z# D* _" `5 s1 m. o; @
The path to a file containing the password of the LDAP SSL keystore. If+ u+ u8 q, q7 f& t7 e* H- F' K
the password is not configured in credential providers and the property$ g5 ~2 o+ ~& B$ ` D7 g( Y9 p
hadoop.security.group.mapping.ldap.ssl.keystore.password is not set,
4 w3 b% R) `2 r1 E$ N X LDAPGroupsMapping reads password from the file.4 |, G! S7 v, p' s
IMPORTANT: This file should be readable only by the Unix user running6 P# @3 ?6 N1 S4 B9 G7 B
the daemons and should be a local file.
8 }! Q( N- c7 G! H: i! |9 l% S </description>/ q" c+ D. J' q7 X$ Y; [8 X# R
</property>
0 W& S7 t6 y) k3 b: r+ b<property>; @- D: w8 v6 U/ [
<name>hadoop.security.group.mapping.ldap.ssl.keystore.password</name>
! m; @/ C7 i U* t <value></value>
5 t, Y& A+ }* m; Q! [ <description> f7 S2 E' ^3 u1 V5 o
The password of the LDAP SSL keystore. this property name is used as an1 s3 z" E" _/ k- _9 w
alias to get the password from credential providers. If the password can
0 C2 c2 Z1 J9 n) P9 n9 U' l not be found and hadoop.security.credential.clear-text-fallback is true* g2 [# Y3 J5 }2 Z1 ]9 Y) P7 O
LDAPGroupsMapping uses the value of this property for password.3 h+ j9 I" `4 ]8 {/ b
</description>5 a3 ?! j, u+ ~- i) b8 \% A
</property>
/ @+ K9 O7 G$ E7 V; w' s! g; R<property>; S, O8 u" R( K! h& r
<name>hadoop.security.group.mapping.ldap.conversion.rule</name>
* ?2 C4 v+ ]: I. y$ P. P0 ] <value>none</value>7 F5 o0 Q4 K) J% g: g, i% e
<description>6 Z) E2 g& p% S# E0 w6 J! w5 }
The rule is applied on the group names received from LDAP when
$ F! a/ i9 O% P, K+ D; [& ^ RuleBasedLdapGroupsMapping is configured.6 b$ \6 D4 R* J4 w# o
Supported rules are "to_upper", "to_lower" and "none"./ t% h8 ?0 l, f* `5 S
to_upper: This will convert all the group names to uppercase.1 W- k) B+ _4 A" F) K3 w' ~& H
to_lower: This will convert all the group names to lowercase.) Z6 ^6 a& W5 m' x4 c
none: This will retain the source formatting, this is default value.
' M5 \; ]" J! j </description>
8 ^% ^% a% E2 E! R+ ?' X</property>, J4 E8 B, g' V3 U! K9 F/ L
<property>
5 t9 ^" M+ J! V) o" I; r- T8 c <name>hadoop.security.credential.clear-text-fallback</name>) A9 P0 n1 s6 s$ G+ A+ @ e
<value>true</value>
2 n8 n, B$ m9 V/ w <description>
+ [& [3 o0 p( S- I+ W# q true or false to indicate whether or not to fall back to storing credential
: h: |4 d0 W i% n: H4 K password as clear text. The default value is true. This property only works! b N0 N; V) R2 i- n$ p) h
when the password can't not be found from credential providers.
. n3 D7 x" E \8 y8 K </description>
2 F- w: J. Z% _! c' ]</property>
7 b0 t5 B- A% V: a" y<property> e5 Q* U4 X, k8 r2 o+ T" _( s5 X- b
<name>hadoop.security.credential.provider.path</name>
' R+ v3 J6 x- y' v+ k4 ~2 f' i) t <value></value>0 {6 w4 l/ W# c
<description>
* i" _+ C! f/ x$ e4 n; y6 U A comma-separated list of URLs that indicates the type and" p9 P6 H7 z d: ^- o
location of a list of providers that should be consulted./ w2 H& D T/ Z+ ^7 q- x) p
</description>: m5 @; b0 |9 D& c/ }
</property>
4 X, G0 a0 a6 _: F6 f2 ^<property>
: h0 y$ K0 x2 u& J& Y1 r: d: y# o <name>hadoop.security.credstore.java-keystore-provider.password-file</name>6 T/ h& m- ]' z# q) I4 n
<value></value>7 x* c+ z! e; ]. i. ]# B$ H) i
<description>
7 i- R& k) W' ?! I8 l/ g The path to a file containing the custom password for all keystores$ {+ Q6 J2 c! Z# _
that may be configured in the provider path.
5 k& W6 h0 j% |: R2 {. V </description>
1 e: J' r3 J# |% I, Y</property>
" e( O* t5 \% O! r2 g( ~& W* D/ z<property>
" o5 L- t- u; R- ~, z# V <name>hadoop.security.group.mapping.ldap.ssl.truststore</name>) c0 ~4 \ N' Q L/ L0 V2 j
<value></value>
8 K* ~; u7 ]5 q5 d( d9 i <description>: m8 e/ w2 v S2 s. Z- ~. v
File path to the SSL truststore that contains the root certificate used to
0 O7 F2 n; `5 g3 H6 h9 F. x sign the LDAP server's certificate. Specify this if the LDAP server's
! c) I" z) Z) F* j3 T- w certificate is not signed by a well known certificate authority.
4 k- P* i5 S/ d: x </description>- h5 z& N8 M. l2 U- q& Y
</property>
$ _' d+ y% ^+ N* l$ C s4 h3 u<property>9 b2 n' n$ M9 m/ q& `9 q* a
<name>hadoop.security.group.mapping.ldap.ssl.truststore.password.file</name>
* F+ N7 v# \8 D. u: y5 @ <value></value>) n+ ?) j( Z! D, M
<description>
8 ~: E- r: z2 N7 R; L The path to a file containing the password of the LDAP SSL truststore.0 F1 S b! L$ T9 L! w# V- V7 t
IMPORTANT: This file should be readable only by the Unix user running
) T7 z3 ~5 C, @$ P the daemons.6 z) h9 x+ O- z7 _% s0 p3 p, l5 V
</description>
) s! e) u$ j- e" y" C# n/ \</property>& U$ l: I' t$ _8 W
<property>
1 k |! ^+ E6 M* |5 w <name>hadoop.security.group.mapping.ldap.bind.user</name>
5 d& u- f: s/ \, B <value></value>
) g9 a q, m( S( T <description>
u& p& s. v1 `+ G* @1 s) Q The distinguished name of the user to bind as when connecting to the LDAP
* P& K2 ^4 _& S server. This may be left blank if the LDAP server supports anonymous binds.
: |+ a% A) w0 _ j </description>
6 f k4 p0 G% }( f- [- c</property>
2 i) {4 N' g6 r& h% P4 E<property>6 j% d2 b# B0 B3 i& O% N. c
<name>hadoop.security.group.mapping.ldap.bind.password.file</name>
% _4 M( t `* R <value></value># S- P3 \4 n1 q1 G5 M% K0 \
<description>8 ]3 q# X+ e7 P: ^
The path to a file containing the password of the bind user. If
4 D7 a+ a8 b y0 b7 \ the password is not configured in credential providers and the property
4 m# A d/ N4 ?7 P8 U+ } hadoop.security.group.mapping.ldap.bind.password is not set,
9 Q6 p( q- \3 v' g9 ]; Z3 C8 | LDAPGroupsMapping reads password from the file. F3 o. O6 s7 [ W1 [; ^: U
IMPORTANT: This file should be readable only by the Unix user running& o. W g! F' F: U9 f
the daemons and should be a local file." [* Q! ]) g. N+ U g
</description>3 g' T8 l3 B! F( z# ^
</property>
, P/ Y+ |$ E [9 j' L% p2 y( L7 W<property>% w9 k# Q9 x9 R. I
<name>hadoop.security.group.mapping.ldap.bind.password</name>5 H/ s0 x8 i1 w2 M: q2 u" ?+ v- I7 T
<value></value>8 \0 t; Y& k' h, p" \! H
<description>- G6 S- Q% R6 C, c+ y1 _
The password of the bind user. this property name is used as an
5 M0 D" h6 P9 r alias to get the password from credential providers. If the password can5 d) X6 Z/ ^0 b4 g
not be found and hadoop.security.credential.clear-text-fallback is true
" \( K, M6 f- C2 p, o LDAPGroupsMapping uses the value of this property for password.9 ]4 d& a" |; r; h: K- |
</description>
. L4 s5 @( v. u7 N$ ?. Y1 X</property>
# G3 h7 m/ t$ P8 Y. ^; P/ o<property>
- E. V7 Y1 j% Z* L( K <name>hadoop.security.group.mapping.ldap.base</name>5 z% w2 A7 S* g0 Z- t' w
<value></value>; l9 F/ l. a- O
<description>
2 X8 M# W7 U: J. O The search base for the LDAP connection. This is a distinguished name,
$ l4 h$ |4 y$ r5 l, s# I and will typically be the root of the LDAP directory.
! Q" H2 o+ G9 s" q </description>
0 ?. b) P+ g5 Z7 I</property>
7 M) i0 `, B0 `<property>
+ v# s7 T4 k1 U- c! P8 } <name>hadoop.security.group.mapping.ldap.userbase</name>
/ S# `+ H4 s: A: V: M1 }" o- U <value></value>
2 j% H c+ D& j2 u c& C* H <description>6 y1 E3 X. @" {* n& `4 [4 \0 I
The search base for the LDAP connection for user search query. This is a
+ }3 t. v. I6 x O+ S6 {# U distinguished name, and its the root of the LDAP directory for users.
; A9 D0 x' i* l2 z' z0 g; y If not set, hadoop.security.group.mapping.ldap.base is used.
) s7 @& v; E5 z# w- e9 r6 L% ? </description>" g w- u/ l# ?! Q* X4 m! P
</property>4 L+ ~" |! u' O5 e
<property>0 ~- D* I& d3 ?* m
<name>hadoop.security.group.mapping.ldap.groupbase</name>
4 \, d) W8 T& ?5 s( u: g) O <value></value>
$ s% ? X5 c5 f <description>
* Q) a m5 z a) C9 w6 s5 A The search base for the LDAP connection for group search . This is a
. W, R/ U2 x% i distinguished name, and its the root of the LDAP directory for groups.
* }0 e/ A3 A9 h, W5 A8 X8 ^' M" Z If not set, hadoop.security.group.mapping.ldap.base is used.* d4 y4 a0 n; `: _0 Z- h( X" c
</description>7 O% T: E/ c) B: V; j; s3 m* B
</property>
) |. F0 p# M6 t. f4 V<property># `4 b( P1 c2 a) g) k% p2 g- e
<name>hadoop.security.group.mapping.ldap.search.filter.user</name>* d( r/ D! _$ g. {
<value>(&(objectClass=user)(sAMAccountName={0}))</value>
) Z" e7 k! \0 ]0 z6 y <description>: u1 k7 U1 b5 s, e
An additional filter to use when searching for LDAP users. The default will' e* a6 S M& K* k- S- u$ P5 C
usually be appropriate for Active Directory installations. If connecting to! \$ h3 _: Z( Z! o3 d- k- d, Y5 H
an LDAP server with a non-AD schema, this should be replaced with
3 f6 r7 o( F/ i6 I3 i (&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to
5 T4 g* S) s$ i" f) w1 n denote where the username fits into the filter.
7 N; L" d$ x3 ]! N- Q8 }3 ] If the LDAP server supports posixGroups, Hadoop can enable the feature by8 z4 n/ A3 T% _0 A
setting the value of this property to "posixAccount" and the value of
( U6 q I' I) L! w! B1 F# k+ B the hadoop.security.group.mapping.ldap.search.filter.group property to( f5 m8 S" Q4 |/ j
"posixGroup".
7 i2 _% r# U8 k: j </description>% F- d& _" H& k% _
</property>
) N$ h, ~$ p; x<property>
: w% k! [0 `3 P1 e8 w <name>hadoop.security.group.mapping.ldap.search.filter.group</name>
5 Z4 e9 {" m4 d2 {* l; i) D" [3 V# m- ] <value>(objectClass=group)</value>
2 k3 a4 B1 i- e( s5 I U <description>+ d# D, W0 n) b5 w/ n' t
An additional filter to use when searching for LDAP groups. This should be
. r8 ` F6 d* o% j! T7 Q changed when resolving groups against a non-Active Directory installation.
8 o2 u9 Z6 M2 v/ l See the description of hadoop.security.group.mapping.ldap.search.filter.user8 H3 w' |5 e+ [$ n
to enable posixGroups support.
; [6 u0 s5 _2 ^4 _/ |! ~ </description>
4 }) i& Q! A* h4 r</property>
7 S! n/ a8 n) S' r8 q<property>5 G5 l5 \0 h. `9 t; D3 o
<name>hadoop.security.group.mapping.ldap.search.attr.memberof</name>3 Y% H" J' G1 I" L5 M% E
<value></value>
K0 I7 r& l9 p2 ? <description>
0 z5 p" Q% l9 y6 J The attribute of the user object that identifies its group objects. By
/ a# D `# j: {: H! j! a5 R default, Hadoop makes two LDAP queries per user if this value is empty. If
/ y6 I/ U) w3 X. f/ j N set, Hadoop will attempt to resolve group names from this attribute," ]$ O3 G/ h6 x# X
instead of making the second LDAP query to get group objects. The value
n6 y m" w5 L should be 'memberOf' for an MS AD installation. p, \' J/ H7 L( o8 \
</description>
, r* x+ f/ K9 b1 f5 H</property>
; W- b; C" f- n' M& C% i<property>
0 y$ J: j3 _' n <name>hadoop.security.group.mapping.ldap.search.attr.member</name>
5 x: T3 }& z1 H9 ]1 t( x) ~. a <value>member</value>
0 q( K# T0 u1 P- e <description>5 K" J( `; f9 ]% V8 F
The attribute of the group object that identifies the users that are* D3 K+ e6 i1 ^( k* Z
members of the group. The default will usually be appropriate for
+ `$ D# l+ Z+ q# Y- R% W any LDAP installation.
/ n/ c U# C8 Q' U+ e </description>0 i+ |( Y* U# L1 e; C; J @5 Y. ~
</property>9 U3 ~8 f) @1 ^# H. H0 X% m5 ~
<property>! i* R+ a* v$ t$ l0 r
<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>4 R7 j+ u0 A- R" D H0 K. w5 t
<value>cn</value>
; `+ ~4 O" U$ \7 y1 B <description>
# d9 B& J* B% I2 P: ^3 k [* l The attribute of the group object that identifies the group name. The
- ]4 V. o0 X8 I3 f1 H* b/ f default will usually be appropriate for all LDAP systems.
; f( N# d2 W; y8 V1 M; N# I( U </description>
, y$ s2 Q& a1 x2 s; a</property>
9 D+ P( H; \. o<property>
?9 g$ G+ h3 q5 D <name>hadoop.security.group.mapping.ldap.search.group.hierarchy.levels</name>+ s! q+ ]" C1 w1 s2 `6 W
<value>0</value>7 c$ y& c- U+ K y* i2 W" J: X
<description>. A0 C6 w& y7 h& t
The number of levels to go up the group hierarchy when determining% l4 o h+ }# L& N7 X7 G7 s; ?
which groups a user is part of. 0 Will represent checking just the2 s8 {0 d: Y4 f) c# s% Z8 P
group that the user belongs to. Each additional level will raise the$ R) ?# l/ x# N# P8 l' v% }
time it takes to execute a query by at most4 @- M) U9 x& u1 M; m7 K4 k0 G
hadoop.security.group.mapping.ldap.directory.search.timeout.
" v# B+ h( ^/ f/ w0 W( S5 M/ T The default will usually be appropriate for all LDAP systems.
7 G- c" [3 D" W2 r </description>
+ [! Y$ {. [) y; L& p) V</property>- V$ y. K, d6 d- ?# _+ Z* q
<property>
9 \1 e: \: c$ ]: ~( a" X <name>hadoop.security.group.mapping.ldap.posix.attr.uid.name</name>! [0 B" \9 O7 u( e3 ?: m. C; ~/ V
<value>uidNumber</value>5 g9 @0 h2 t; o4 f& A! V
<description>: o; _0 u* f# e& v2 ^3 K) W
The attribute of posixAccount to use when groups for membership.% n4 |- Z* k( B j7 E7 ^
Mostly useful for schemas wherein groups have memberUids that use an
5 c, a. J- F( i0 B attribute other than uidNumber.3 Y! E1 m3 A: a" \
</description>; ]* d+ e( e. b" Q
</property>, q( p2 r# O) G, w: i- d
<property>
O$ }& H0 L0 d, l <name>hadoop.security.group.mapping.ldap.posix.attr.gid.name</name>3 A4 C8 u# F; N
<value>gidNumber</value>' g& n9 \5 }9 F3 W3 a+ D! x
<description>9 _2 `4 W" m$ ?! ]( i
The attribute of posixAccount indicating the group id.3 [5 V1 E8 a/ u4 d) y9 K# M9 B
</description>3 L7 n. x; ?) y" a
</property>" `( g! M A! ], M" T1 `+ P
<property>
( k, M& J+ H I) O9 c <name>hadoop.security.group.mapping.ldap.directory.search.timeout</name>( O. {, v; f7 e3 Y
<value>10000</value>
3 Y) z& B) T4 J' Z a <description>
- a4 i2 _# [4 Y _( s The attribute applied to the LDAP SearchControl properties to set a0 D2 ]( X' Q; @% U& o b
maximum time limit when searching and awaiting a result.
# N* u5 m' r+ u8 u, {, W* V7 @6 k Set to 0 if infinite wait period is desired. K# t# i& J; @! ?# M
Default is 10 seconds. Units in milliseconds.( o) l* F. X% T
</description>; x1 o. `2 X9 R) v2 {" S2 _7 s: F
</property>4 {; b3 Q& H- m! m7 ]3 g* u- N
<property>" _1 t5 g- S. I ]3 _+ X$ W( ~8 a) E
<name>hadoop.security.group.mapping.providers</name>. E0 `1 `0 C. B! q
<value></value>
4 R$ [0 n. Q& H! d, M% l# {; F <description>( p" s- j: a& [( y9 B5 ]4 V
Comma separated of names of other providers to provide user to group! a, n1 O6 W8 C: |9 f
mapping. Used by CompositeGroupsMapping.% h1 W$ f$ ?6 g2 a! w1 L/ i
</description>0 N: h1 T3 Q& @2 s5 A4 @
</property>8 |( f, D" q& z$ [
<property>
0 i6 {- B- A" ]# h3 V L' K- z <name>hadoop.security.group.mapping.providers.combined</name>/ ?& P* K- p7 K( i
<value>true</value>
2 I7 ^5 K% A* c <description>
& `. r+ u" O2 H; I6 \- p' Y% [ true or false to indicate whether groups from the providers are combined or) G' l) v t2 t l
not. The default value is true. If true, then all the providers will be: E s$ D9 M% C* M7 y+ ]
tried to get groups and all the groups are combined to return as the final
% ^3 q+ _# S& K4 T+ J results. Otherwise, providers are tried one by one in the configured list
% B! q" n6 l# N order, and if any groups are retrieved from any provider, then the groups$ @+ D% S7 w8 k
will be returned without trying the left ones.
7 h/ n, c, q1 e: Z! T </description>
+ C9 ?" l, J4 D( g3 C9 N% @& u</property>. I6 w* U+ \* g+ z, T F
<property>
/ t' q* m/ ~0 ]" y& S8 f9 W <name>hadoop.security.service.user.name.key</name>) m. x6 `' U X2 {% U, d6 K
<value></value>3 {1 R, G% v# N, }: V! n% \
<description>
5 c0 c8 o/ T5 j/ ] For those cases where the same RPC protocol is implemented by multiple
' ]: g3 G- p' V* g; E servers, this configuration is required for specifying the principal* n% o7 f& A, e& O
name to use for the service when the client wishes to make an RPC call.. }- q& ^9 z; d2 ?: M1 V2 a
</description>
2 I) P1 Q2 w% E</property>" y4 ]1 \# z8 t* t; F! M
<property>
6 \; O# b- o' W <name>fs.azure.user.agent.prefix</name>
/ T1 J! C3 J; v4 \& n <value>unknown</value>
& Q7 a) O) A; v: B8 a9 G( w$ _( J7 E9 G9 ] <description>
. d- w1 m- a7 f! V5 a WASB passes User-Agent header to the Azure back-end. The default value
1 I+ I! Y' ?2 o& h- ~ contains WASB version, Java Runtime version, Azure Client library version,
: w* R" ?% C( ^+ V3 v* Y and the value of the configuration option fs.azure.user.agent.prefix., D" p* j6 {. H; O
</description>" z' r6 D' ]9 G/ [7 t3 |0 t6 v
</property>
% \( s" x n( X# J, J<property>, W% G6 c0 G) p, ]/ ~' ~
<name>hadoop.security.uid.cache.secs</name>
: ]: e3 b* W+ W; F3 _ <value>14400</value>) @$ Q6 V' h# y6 v( i/ d+ j, {+ G- r
<description>
1 Y, o' B$ U Z8 e* l2 { This is the config controlling the validity of the entries in the cache6 q% @+ ^1 W. H; P; n. R! E
containing the userId to userName and groupId to groupName used by
@( ?0 n( A8 T: l8 m NativeIO getFstat().0 C4 M9 g" W, Y5 h. B* d# Z1 ^
</description>8 c$ w% h2 k4 q" X7 W% f. F2 A+ M
</property>0 q# u9 c( J/ {9 P1 H8 C
<property>8 \0 d0 f/ I" F
<name>hadoop.rpc.protection</name>
% \; r7 V1 B4 S. r) N+ [+ i/ X <value>authentication</value>
' A$ Z: }/ g' ^5 I; O <description>A comma-separated list of protection values for secured sasl4 H+ D( p/ H$ ^; V% D# u5 j- [
connections. Possible values are authentication, integrity and privacy.8 t" m; B6 {7 t. Z6 Z
authentication means authentication only and no integrity or privacy;
: {$ s# s4 |# M S integrity implies authentication and integrity are enabled; and privacy$ O$ x% ~6 \( I& ^( ]
implies all of authentication, integrity and privacy are enabled.
; L: f# _7 q7 O8 l hadoop.security.saslproperties.resolver.class can be used to override2 U3 F0 N2 h! `/ |- [
the hadoop.rpc.protection for a connection at the server side.
- l" P5 [, u3 r6 ]) l </description>
. r! ~( j7 b8 b</property>7 H5 N- W o, y1 N6 o7 b3 e
<property>3 Q% M% k7 q3 Q- ?' N0 ~. }
<name>hadoop.security.saslproperties.resolver.class</name>- n: }" N; x/ r1 x$ A1 g0 K$ W
<value></value>- D1 b/ [, \* s0 N3 k8 Y# Y
<description>SaslPropertiesResolver used to resolve the QOP used for a
. u/ y8 d0 B! z" ^ connection. If not specified, the full set of values specified in' t% [ S4 [0 y6 M4 _8 R
hadoop.rpc.protection is used while determining the QOP used for the! _+ ~4 _, `5 \: M
connection. If a class is specified, then the QOP values returned by J; l; M- w9 E- \
the class will be used while determining the QOP used for the connection.
, K! c; Y- \( [: i" I </description>
+ E4 S+ K* J* v' j8 P7 g</property>! u+ L% r# p% b+ |
<property>
" u" `& P9 e% ~% l! M. \" l: h& e <name>hadoop.security.sensitive-config-keys</name>
7 i3 E3 q+ t# ? <value>
. D6 {, ]- Q; Y# o4 C- Y7 D secret$
6 ]( z/ m8 v. B) T3 h password$
/ D* F7 o- \( k! J R ssl.keystore.pass$
# ~2 }! n" M7 Z e: m fs.s3.*[Ss]ecret.?[Kk]ey
) I+ H2 D1 H9 X+ d$ v fs.s3a.*.server-side-encryption.key
4 u& A0 m! Z8 @ fs.azure.account.key.*
5 T: h! C0 c5 A& q$ K& d; R3 w2 c credential$1 w% b4 r! I' v9 \* M8 N
oauth.*token$
. x# t* o. j4 S3 c; Z# ` hadoop.security.sensitive-config-keys' X; u% L Z2 O2 d
</value>
! Q% G. x3 v: _6 A; o <description>A comma-separated or multi-line list of regular expressions to, v# l4 {1 K4 B# J1 a. m% \" g& X
match configuration keys that should be redacted where appropriate, for& [; W, k/ L/ U. N
example, when logging modified properties during a reconfiguration,
" l) e6 L4 I3 k5 Y) o private credentials should not be logged.! @9 j/ C' M5 |; W7 m/ `6 u u9 Y
</description>
* G$ K: u5 `2 _. f</property>! ?1 O# E( ]6 B* @
<property>
3 H& |$ N! C5 {: a+ P. z <name>hadoop.workaround.non.threadsafe.getpwuid</name>: P& N J1 v" B! k
<value>true</value>
4 e' I7 j& y H9 a' |8 J7 t' w <description>Some operating systems or authentication modules are known to
& t: s' h+ b2 n have broken implementations of getpwuid_r and getpwgid_r, such that these$ Y1 p" r/ B5 F0 }( n' Z, L
calls are not thread-safe. Symptoms of this problem include JVM crashes
- ]3 n% m# H, e0 I! A with a stack trace inside these functions. If your system exhibits this4 t/ k' E/ y# x
issue, enable this configuration parameter to include a lock around the" @' S0 O; a& q
calls as a workaround.
b# c" ?2 H/ P. o0 ? An incomplete list of some systems known to have this issue is available) I7 p1 @ n3 O
at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations: G" |& @/ } t* ^% u% p' I. S4 m! {
</description>
8 s. `* Y. c6 d% [' G0 r</property>, o7 H. @6 h0 \6 u0 f6 L
<property>
8 j/ j7 d+ Z8 u+ {* V% j <name>hadoop.kerberos.kinit.command</name>. E. ^ \. C* U! r: r- C5 J; i9 f
<value>kinit</value>
5 w0 G- B& x# E <description>Used to periodically renew Kerberos credentials when provided5 ?% [( D4 e; ?9 n% j0 _ X! [
to Hadoop. The default setting assumes that kinit is in the PATH of users, }7 I! ]$ I" Q0 y
running the Hadoop client. Change this to the absolute path to kinit if this
2 [5 V( G3 I9 N2 O0 ^, o is not the case.
, d3 ?. p4 t9 ^% I/ ^6 {6 w </description>
1 I9 E+ j1 z% a) c) Z% T</property>
4 ~ a* }( q X" D2 e" A% a5 e( n<property>) L, r0 l' D! ^8 u6 h+ A2 P
<name>hadoop.kerberos.min.seconds.before.relogin</name>: _' P+ a, U2 ]# J+ v7 \* r$ k
<value>60</value>+ e1 F$ @2 e; r2 v3 v2 l1 ?5 u; g2 r
<description>The minimum time between relogin attempts for Kerberos, in# ? Y: u$ R6 J6 \3 q" B
seconds.! C+ B# k7 \, R, c
</description>
: d$ _) q5 Y: @1 n8 M# C! I</property>
0 `. x7 {( I4 t' R<property> }( t$ R E; p: p9 Y. K" ?$ ^& y
<name>hadoop.security.auth_to_local</name>' w. {0 e: w1 v) }' F1 p7 A
<value></value> R, z! q& C4 ^& g7 h5 f
<description>Maps kerberos principals to local user names</description>' W. D3 y3 b; G3 t/ _" x1 K" a9 p) l
</property>& G) i4 h6 U) T0 d
<property>( i% b9 I3 {2 M
<name>hadoop.token.files</name>
0 @5 w6 H$ u) r D) f( q1 | <value></value>$ @; o7 S+ X4 R1 y$ W
<description>List of token cache files that have delegation tokens for hadoop service</description>) m4 l* Y! S/ R& D/ I9 i% ~* k
</property>
8 F# ~& z! a0 O& i; \, J<!-- i/o properties -->+ z- ~: e+ y' }6 f, n! M# R- S3 i
<property>
9 R9 n- c+ v9 m <name>io.file.buffer.size</name>7 {- Q3 u# T! k G2 V0 c# x! H
<value>4096</value>, R8 U; N1 F- N0 T: R
<description>The size of buffer for use in sequence files.
) y( S/ G1 b6 `5 O The size of this buffer should probably be a multiple of hardware1 }0 z7 ~) E) b2 G8 D6 k3 z6 q* Q" |
page size (4096 on Intel x86), and it determines how much data is: z; x& x( f! s! S; r
buffered during read and write operations.</description>
! g, z6 u) S9 ]% z% r7 Z</property>
( s8 O- S4 M! T<property>" f1 m) u' ]0 M) a1 w. g
<name>io.bytes.per.checksum</name>
7 y1 R7 e% }/ p$ I- [2 P <value>512</value>
3 i& o: ?; ?" ]0 l( X1 i <description>The number of bytes per checksum. Must not be larger than
. x) { y; f2 w: j7 } io.file.buffer.size.</description>5 g& H2 D# M1 Q0 ]' U, B) X
</property>, I. Y, g; u. r
<property>
5 L$ r1 E" L% Q! J7 ^/ B9 l <name>io.skip.checksum.errors</name>: y1 f: x1 W# Z6 f: e
<value>false</value>6 C {! ]1 X+ V( K7 U
<description>If true, when a checksum error is encountered while8 `5 K$ K% d* t G7 s* K
reading a sequence file, entries are skipped, instead of throwing an
# f2 o2 f$ M1 v8 m7 R3 n2 m9 Q exception.</description>* c2 Z9 t1 B% Q* }
</property>; u* p9 y9 w ]* H5 h7 e* X
<property>3 ]! Z H P' W, S) I8 a
<name>io.compression.codecs</name>$ v1 k* Z- ?5 u6 Y) h& P
<value></value>
( V; h/ u# e) k2 B) f3 F <description>A comma-separated list of the compression codec classes that can
7 F0 I" x" `& _1 j* B, N9 @5 m& z be used for compression/decompression. In addition to any classes specified
6 n7 Q% v* P. W; Q8 X% P) o with this property (which take precedence), codec classes on the classpath( k; p% B% ^8 ?0 f0 K5 m
are discovered using a Java ServiceLoader.</description>; k" ~$ e8 J5 j. t6 ]
</property>3 H5 D Y8 {! w: ~, _+ |8 s
<property> |. i7 v. {4 l' U7 Q& l
<name>io.compression.codec.bzip2.library</name>. E: W2 E u3 g R; ^
<value>system-native</value>$ P( H. W. z6 c" Q1 M4 M
<description>The native-code library to be used for compression and! p ~+ M4 f- T: z" o! A+ k( Z
decompression by the bzip2 codec. This library could be specified
4 L* ^/ h- y) f; O either by by name or the full pathname. In the former case, the
3 r& i$ Y- }$ P& l+ Q library is located by the dynamic linker, usually searching the: n# n) d- t$ _9 ~' n6 t
directories specified in the environment variable LD_LIBRARY_PATH.
M& M" b' g$ \7 H The value of "system-native" indicates that the default system
/ q s# V1 k/ }2 ] library should be used. To indicate that the algorithm should* ?* ^* O; }: p; p" _# f
operate entirely in Java, specify "java-builtin".</description>
9 I2 v: L9 C. R! ?</property>
}; _3 f- Z0 U<property>
7 x# l1 E7 b9 H% a: F <name>io.serializations</name>6 y8 B4 N4 Z M0 z* G. l
<value>org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization</value>
1 T } o9 F; E5 Q' ]$ y <description>A list of serialization classes that can be used for
$ M& t# ?& N1 X; V& }1 U obtaining serializers and deserializers.</description> |- ~6 {; c0 g- p
</property>
" G: T8 v: x) _4 X8 Q<property>
, H+ E' G9 l% W. v8 O <name>io.seqfile.local.dir</name>& w/ t& ^5 J( l
<value>${hadoop.tmp.dir}/io/local</value>: G. l: B1 l. W- z% r
<description>The local directory where sequence file stores intermediate: k6 u' V3 {; H% ?$ Z
data files during merge. May be a comma-separated list of5 `5 s" I; L& |1 M3 s
directories on different devices in order to spread disk i/o.2 l# x' |+ _% e1 Q9 y0 n+ T" c) }
Directories that do not exist are ignored.$ y9 R2 q0 Z2 T- z
</description>( \* x- q" {& T6 A/ W
</property>
6 B& `6 c% N, u* d8 N4 W4 }<property>5 F h7 c1 }" A: w# c" [ h
<name>io.map.index.skip</name>( F# C1 c; H6 _: T7 O8 C9 i* X( A
<value>0</value>
9 {7 F2 _. { s& @3 J! l! `" j <description>Number of index entries to skip between each entry.# \8 D* R' g4 z. e. e! s) B# ^5 ?
Zero by default. Setting this to values larger than zero can
9 a% V& Y$ A- F2 D6 e T+ l- z3 k facilitate opening large MapFiles using less memory.</description>
6 M8 f: g* c6 b" E9 p</property>4 Z5 q+ n( t& r, ^
<property>
& p7 P3 d( T m# a4 S! S <name>io.map.index.interval</name>7 \, `- ?- t& t U
<value>128</value>
4 I J3 ` w7 e3 }0 c <description>
3 \! g8 H1 l9 Q' y MapFile consist of two files - data file (tuples) and index file
' @. s) H8 z8 r7 m! R F$ M (keys). For every io.map.index.interval records written in the* C8 M( w$ c( k
data file, an entry (record-key, data-file-position) is written( b5 ?$ _4 Z9 l$ ?
in the index file. This is to allow for doing binary search later
, l {. H' Q9 l# r3 T within the index file to look up records by their keys and get their
- w$ J1 {* ]5 B6 B- n closest positions in the data file." d1 Z4 S* L: {+ u& [
</description>
H5 @2 M2 b3 D% J0 G" j</property>8 n; p8 D2 |; B( q5 v" W
<property>
% g2 U) U7 X0 Y1 O+ g$ x" h <name>io.erasurecode.codec.rs.rawcoders</name>
# ?7 e" p5 X5 }7 ` <value>rs_native,rs_java</value>! Y. j* N$ A! w: U& O
<description>0 i9 W. v `1 c! e8 l, `9 x
Comma separated raw coder implementations for the rs codec. The earlier- D; S8 Q2 f5 d" f
factory is prior to followings in case of failure of creating raw coders.1 ?7 `7 y) _1 k3 }5 F: Y( C
</description>" X% I5 C% K( b' I
</property>9 x, y- L/ Z) {& l3 v: ^
<property>) ]$ r6 ^7 {$ ?% ~6 Z5 ~
<name>io.erasurecode.codec.rs-legacy.rawcoders</name>/ V3 k, e& ~4 s0 U0 D3 z
<value>rs-legacy_java</value>9 O* s8 K e" @ {- k
<description>
7 Y% L+ ?! z; B+ v" U Comma separated raw coder implementations for the rs-legacy codec. The earlier
5 c+ s4 }/ \% P$ J factory is prior to followings in case of failure of creating raw coders.
& m$ i+ R; f, {- Z+ N8 _ </description>
4 R) Z0 H, z0 R) L0 b2 E1 p8 A</property> ^. d3 ^. f8 a( p, ~
<property>
. X' g0 r0 X. j2 M <name>io.erasurecode.codec.xor.rawcoders</name>9 R, x @' Z+ t2 g
<value>xor_native,xor_java</value>% @& g v6 E" V1 q4 @
<description> a- P" V4 L9 c9 V8 W0 p
Comma separated raw coder implementations for the xor codec. The earlier
) q2 x- y& {% B& n: Z) a( q factory is prior to followings in case of failure of creating raw coders.
% }- h* U5 O3 ~7 {4 f7 l* J! W$ E </description>
" `+ g, o5 v+ V( x7 f* G</property>
Y" q3 |% e! Y/ p! {! K% F+ W; w <!-- file system properties -->3 g S+ T8 w: p" T2 k
<property>
. q+ o! ~, L! R$ c) Z- `) l <name>fs.defaultFS</name>
1 n- f$ A& u( u4 d2 o* X+ u9 h- _+ N <value>file:///</value>
& S! i9 l$ k3 Y8 e5 d, C1 n% ] <description>The name of the default file system. A URI whose
7 ~0 j% A3 L2 S+ R scheme and authority determine the FileSystem implementation. The
' t6 `) F% R- t% z( ]% z9 N uri's scheme determines the config property (fs.SCHEME.impl) naming$ B& b: n% k& i4 I/ P, U
the FileSystem implementation class. The uri's authority is used to9 ]: B1 ]3 @/ V
determine the host, port, etc. for a filesystem.</description> J* p3 `: J, R- J* ]
</property>
9 a; Z9 q( _6 C0 o1 y) N2 O<property>
6 Y5 m% ?$ v9 ^ Y1 O <name>fs.default.name</name>
) v: b( n+ T8 Y1 _+ y4 j! ^$ j <value>file:///</value>7 t l9 @4 `8 p- `
<description>Deprecated. Use (fs.defaultFS) property
% o* k- X- N7 [' g- ` instead</description>
. r8 L6 P; ^. O( J7 {</property>
" ^# }$ g/ K# s, l$ Y<property>, K) A; o1 q B' R$ D- B
<name>fs.trash.interval</name>
' P4 |# p" [3 `7 w/ O" V <value>0</value>
# X% w( A- f& \. @: e <description>Number of minutes after which the checkpoint
' D5 J5 B9 A& D; ]% g- q% n# d gets deleted. If zero, the trash feature is disabled.! Q$ Y7 X% ` Y6 M
This option may be configured both on the server and the3 \: E h! D4 V: @' O2 T3 X
client. If trash is disabled server side then the client3 [% [8 r# u6 D6 j0 u8 g) d/ Y, d- X
side configuration is checked. If trash is enabled on the. r9 s( W! v8 ]8 e V3 ^7 k) ]
server side then the value configured on the server is0 R4 @3 L9 K+ Y* b
used and the client configuration value is ignored.
3 M5 \6 \. z8 N& B: I. h) e </description>
) x7 |: |" y/ d7 K5 r X/ t% Y</property>' \2 x- u- |" `. g. Z4 I' t
<property>" C% A1 ^. }; X, T% u8 }
<name>fs.trash.checkpoint.interval</name>$ M. F3 t- O) s9 @/ ^& p0 Y
<value>0</value>
4 B* N. m- l/ X6 m) I' e <description>Number of minutes between trash checkpoints.' x8 y, x* M9 O& J" l2 R3 J
Should be smaller or equal to fs.trash.interval. If zero,
9 A' D8 |4 E; P2 g& P9 V the value is set to the value of fs.trash.interval.+ d! b2 I3 ]! ]% @
Every time the checkpointer runs it creates a new checkpoint8 c4 C& d) W$ a" X0 D% X
out of current and removes checkpoints created more than
( p3 ^' V+ Z" l7 f, L! L fs.trash.interval minutes ago.$ H0 P3 j2 K5 O2 h! ]
</description>
4 L; l0 d$ g) d2 H8 |8 x- }</property>
" t$ k; Q, v8 n5 ^+ J# Z$ H<property>
; ~, E9 i& M) E <name>fs.protected.directories</name>. l4 w2 W, a6 V/ T
<value></value>1 F, f, u6 k6 f6 F$ Z$ w7 X
<description>A comma-separated list of directories which cannot/ z0 e; K2 m) b) X8 o3 j
be deleted even by the superuser unless they are empty. This( y7 M5 a' n1 z8 P
setting can be used to guard important system directories
7 p$ \6 J" O z+ D& k: R against accidental deletion due to administrator error.6 b& O! s( e5 f) r
</description>, g7 j' n9 @+ P1 @, ^$ p2 \5 m
</property>
& p% B% x4 V: e$ r! J7 ?<property>* K' d0 ^ ^3 s' Q
<name>fs.AbstractFileSystem.file.impl</name>* f }: j" a% j7 s1 l, H* t+ F
<value>org.apache.hadoop.fs.local.LocalFs</value>4 R5 A4 ^4 E0 I6 h
<description>The AbstractFileSystem for file: uris.</description>
& y# b3 j& I- L* q/ [1 V</property>
# z. `+ U1 f8 A<property>* d: ^8 h8 s. B( z8 `
<name>fs.AbstractFileSystem.har.impl</name>
" E& r$ H2 O# r <value>org.apache.hadoop.fs.HarFs</value>
" j) o# @& m( s4 k <description>The AbstractFileSystem for har: uris.</description>
1 \% Z2 ~% [9 t) E. L</property> m f4 V: I6 A+ M
<property>/ a% x N1 a/ d7 D5 h& h! \5 C1 [9 Y
<name>fs.AbstractFileSystem.hdfs.impl</name>& t. A: U* d8 h
<value>org.apache.hadoop.fs.Hdfs</value>: R3 U3 T3 u% {8 F( k. }3 i
<description>The FileSystem for hdfs: uris.</description>. g! O2 T6 P8 }4 u W: q R
</property>2 A: j4 L2 Y# s; r3 i
<property>
+ L3 F+ C. M' W. n! N7 @* X6 V [; C5 x <name>fs.AbstractFileSystem.viewfs.impl</name>
3 ?; g. d; ^8 T3 e <value>org.apache.hadoop.fs.viewfs.ViewFs</value>
7 s# N1 g8 a0 k/ x0 P <description>The AbstractFileSystem for view file system for viewfs: uris9 x* N+ @, u( u9 k1 k/ f
(ie client side mount table:).</description>! [* v4 U0 Z u3 n: S+ c1 r8 I
</property>; ]: K' m3 u0 R1 T' A$ W
<property># I! o+ f% Q0 s& V
<name>fs.viewfs.rename.strategy</name>$ ?7 a/ o- M) j
<value>SAME_MOUNTPOINT</value>7 ~' D" a8 K+ F% y! b4 c
<description>Allowed rename strategy to rename between multiple mountpoints.
4 g7 I, I# \$ w" a: e Allowed values are SAME_MOUNTPOINT,SAME_TARGET_URI_ACROSS_MOUNTPOINT and
/ u% J4 m0 a8 a! w2 P SAME_FILESYSTEM_ACROSS_MOUNTPOINT.6 G% k+ @5 Z' h3 Z* ~
</description> Z+ p9 ~2 W% C% Y9 z
</property>6 H! S2 W/ Y' ?0 e
<property>
6 j( O& f" O3 u/ p <name>fs.AbstractFileSystem.ftp.impl</name>9 e% G- H3 k, @5 i K, c! q8 }/ ^
<value>org.apache.hadoop.fs.ftp.FtpFs</value>
# ~) L+ {/ v, c: O' x# d2 V. c; u <description>The FileSystem for Ftp: uris.</description>
1 g" e, L- n- F" Y</property>! z( k9 l( n4 L% g F7 X5 M( S9 ~4 x
<property>& t* K. g. D# l6 M7 ~' ^/ X
<name>fs.ftp.impl</name>
1 Y- Y- Q$ X3 N$ f1 C! A5 `8 p, v <value>org.apache.hadoop.fs.ftp.FTPFileSystem</value>6 P( u' P3 n3 ~2 v7 K6 L/ i
<description>The implementation class of the FTP FileSystem</description>
! j4 R0 K4 M9 P$ Y9 @! V5 _</property>8 w" t, k- n _3 `
<property> l) z8 i% R3 S8 s( ?8 b
<name>fs.AbstractFileSystem.webhdfs.impl</name>
' ?& Y3 `! i% r6 A4 C <value>org.apache.hadoop.fs.WebHdfs</value>& R% l. S8 q* H @$ ?
<description>The FileSystem for webhdfs: uris.</description>
7 C( P/ k7 L& F, h% n& H5 {8 u</property>; m& q- @8 w6 h# J5 A- ]* v/ E
<property>+ Z% q9 \* W! Z5 f
<name>fs.AbstractFileSystem.swebhdfs.impl</name>
8 D, S: P9 G8 x* b5 ?0 r <value>org.apache.hadoop.fs.SWebHdfs</value>
; V5 L5 h; H6 R4 O8 e <description>The FileSystem for swebhdfs: uris.</description>1 Z; |; k8 B& M, s
</property>* D4 R: U+ U/ i7 [5 a
<property> Y3 ?9 W8 g" u E6 Z
<name>fs.ftp.host</name>: _$ j2 }- o U% D
<value>0.0.0.0</value>! M; A- \1 r# Q
<description>FTP filesystem connects to this server</description>7 O9 ^& D' ^% V: y! r6 D1 S
</property>
' X' ~" {& I" l p, g<property>
7 G' _( S. _! N1 [ <name>fs.ftp.host.port</name>( G+ X. M! i$ Q/ W A# i
<value>21</value>
0 j0 p2 V" n; Q4 |6 D2 ]' u <description>
! R4 w3 I8 g( K4 J FTP filesystem connects to fs.ftp.host on this port
/ a9 Z0 F7 ]. p5 n% M2 A </description>
1 O, A5 u( \- g. Y- \4 I</property>7 R, d4 T2 \+ J
<property>
& W. J7 I9 m, t. U, p- {' N2 [: p <name>fs.ftp.data.connection.mode</name>
5 Q$ A, b+ ^) W <value>ACTIVE_LOCAL_DATA_CONNECTION_MODE</value>
# o2 h7 J1 N$ f) d$ q) n9 s- N <description>Set the FTPClient's data connection mode based on configuration.6 y$ c6 `7 T3 I- i0 {; T. a( K
Valid values are ACTIVE_LOCAL_DATA_CONNECTION_MODE,! u0 N' w! n" c# }: K8 P
PASSIVE_LOCAL_DATA_CONNECTION_MODE and PASSIVE_REMOTE_DATA_CONNECTION_MODE.
* y" b. h- n% _ </description>
* e9 {. _: ~, s</property>
# R/ ]# [) |, S0 y8 j" E<property>
- ]* r+ ]+ ~ a% q9 v <name>fs.ftp.transfer.mode</name>
M7 i9 W: w* u$ T% ~ <value>BLOCK_TRANSFER_MODE</value>
5 V5 E& f( I& \* z <description>
* h9 @ T/ B3 C' U. o Set FTP's transfer mode based on configuration. Valid values are5 G1 y- b, m: _+ |1 @8 c; s
STREAM_TRANSFER_MODE, BLOCK_TRANSFER_MODE and COMPRESSED_TRANSFER_MODE.
9 e5 w7 @3 ~2 x& n- \) G: x0 F </description>
6 F, x# h0 e# Z</property>
9 B- t& ]5 f6 @' f, _6 m<property>7 n8 Q! Y% r) J1 r
<name>fs.df.interval</name>
! }# ]" r' s9 y. s <value>60000</value>' B C5 m) e9 y) L- G$ ^
<description>Disk usage statistics refresh interval in msec.</description>" Y# i7 L2 K V: b) K" H
</property>
" |/ o% Y8 [: K, h) W' r<property>
+ x2 |' F/ A- }: g4 m <name>fs.du.interval</name>
* v v2 G; _. g& v5 q <value>600000</value>
. s' i: k j5 a) _( t <description>File space usage statistics refresh interval in msec.</description>! `+ X8 u, W$ Q2 T
</property>% G2 ]8 V2 g' B, v: L: l2 s! w
<property>
0 \6 j: `: q! z" Q# z6 D <name>fs.swift.impl</name>
: |/ E! i3 Q* X" ~3 w, g, b0 c <value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>$ O0 ~( m9 v( F8 ^5 o' p( [: @
<description>The implementation class of the OpenStack Swift Filesystem</description>
& |. ^& R+ q' o) h$ s; m</property>/ t& M: g8 `9 m5 c
<property>5 x2 d$ |7 [! i
<name>fs.automatic.close</name>
1 U) m4 G6 w) G q0 B. S+ Y <value>true</value>
: b3 D! A/ v- `/ O* V <description>By default, FileSystem instances are automatically closed at program( n& z& O6 D! S" K. W8 ^4 ?
exit using a JVM shutdown hook. Setting this property to false disables this
/ X3 ^6 F9 C" y+ B! u behavior. This is an advanced option that should only be used by server applications: O, x4 R1 s! o# J
requiring a more carefully orchestrated shutdown sequence." e8 Z- }7 N! R3 _- E* q
</description>/ e, I) s( z6 u6 q" p$ g$ Q; H' K
</property>! F: e6 G f. ]8 x
<property>
7 H! F9 L- Q5 C$ E( G, D% Z/ h8 O+ E <name>fs.s3a.access.key</name>, W {0 n5 s$ o0 ^9 j
<description>AWS access key ID used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
! u; y9 I1 M3 | y d, M( H! D</property>
* t" \2 g( W, g, R4 S0 x<property>4 b: l7 f- `+ ^1 J5 t9 {
<name>fs.s3a.secret.key</name>; r# l9 `9 B0 r* z5 N
<description>AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
$ c8 ?' s; c2 L, z J# x- l</property>
1 Y# Z2 {0 e4 y& n<property>" @5 l! V! }4 N- X5 u4 ~- u
<name>fs.s3a.aws.credentials.provider</name>
! A4 Z2 L; G6 }' Y7 \, m' ?6 E <description>
$ Q+ e; {' z# A9 l$ b Comma-separated class names of credential provider classes which implement7 B }& D) A0 o B8 w
com.amazonaws.auth.AWSCredentialsProvider.
+ ?* D) M3 w8 J0 Z1 Q These are loaded and queried in sequence for a valid set of credentials.
- m8 J' @/ I/ _5 I5 | Each listed class must implement one of the following means of
" n: U& F3 Y4 L4 M v* Q) ~5 \ construction, which are attempted in order:
" W2 Y9 }* F- I) Y 1. a public constructor accepting java.net.URI and
8 n3 r c' K1 E' V org.apache.hadoop.conf.Configuration,0 K0 p* u- p9 l- S! k( @' E5 O# ]
2. a public static method named getInstance that accepts no
" ~: Q5 \1 ?" N5 n; n2 m! T arguments and returns an instance of- B/ L m' \. m ^
com.amazonaws.auth.AWSCredentialsProvider, or5 K) u9 t0 a% n3 e0 C! e) s. P7 O6 N
3. a public default constructor.
* F6 D5 n$ T, D0 ]% } Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows* K/ o5 X, y5 E0 |1 m+ \: h) V9 }5 f9 ~
anonymous access to a publicly accessible S3 bucket without any credentials.' l, o) I9 A2 G# Z
Please note that allowing anonymous access to an S3 bucket compromises; |! [) V$ F, H; ~0 g) X' `
security and therefore is unsuitable for most use cases. It can be useful
' p4 k, V$ w" t8 @; s. m% H* I for accessing public data sets without requiring AWS credentials.* N: o [* \. g3 v: H
If unspecified, then the default list of credential provider classes,8 x9 S5 O( J4 s2 p: Q
queried in sequence, is:
& G. ^$ f4 a' W, L 1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static
$ {* L7 o- m5 C& A configuration of AWS access key ID and secret access key. See also1 b2 Q1 a7 E) K! R4 }
fs.s3a.access.key and fs.s3a.secret.key./ K. T8 _$ p" b
2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports
+ ~$ }- q0 n4 T! Y2 H configuration of AWS access key ID and secret access key in
+ ^/ q* N: z, L environment variables named AWS_ACCESS_KEY_ID and
, k4 C" T0 E# J3 `) ]2 X [7 F: [ AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK.
v! F) U3 t1 L 3. com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use! T4 [6 N1 Q' b$ R$ D9 \* X
of instance profile credentials if running in an EC2 VM.
" Z& @. Q: ^( e; @' u </description>" S0 h. y* B! S/ ^- X' Y
</property>
2 M4 w" ~1 v6 T, v<property>
4 v H" e6 Z8 E( H& B3 M( } <name>fs.s3a.session.token</name>$ [: @$ Q4 s# @1 r5 o0 N# U
<description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider) k/ B: j/ [: P- g' F3 ]. B2 r
as one of the providers." }) {; K% @4 ?+ G! W6 a4 j; M
</description># f8 I* r6 C# ~/ z5 m
</property>8 J# {3 E* i+ c1 ~7 g" i+ ]
<property>
: m: _. x+ x& _5 Q* a <name>fs.s3a.security.credential.provider.path</name>
$ n0 Q: r- W: V( [/ U <value />
+ V3 Y( e8 y1 \) E1 ] <description>
5 w: {3 h6 O& x* ]/ W, K' t Optional comma separated list of credential providers, a list
: n! S/ L+ }6 L2 t# p: W which is prepended to that set in hadoop.security.credential.provider.path
6 z0 Q5 H" F; d( L9 l </description>' s! ^3 g) B9 t! L" H( }
</property>
2 x- ]* ]9 L: q" m h2 t0 q' j* ~<property>
. N, i! P, ?4 e3 q$ G <name>fs.s3a.assumed.role.arn</name>
}/ [- w+ t* G <value />/ O) X. C1 e6 h; c3 f- H& i! ]! ^
<description> z: k# [8 u+ j
AWS ARN for the role to be assumed.1 M* H4 i! I7 A6 r+ q9 N
Required if the fs.s3a.aws.credentials.provider contains
2 a% V3 m* i( l$ V6 z8 s org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider2 I" p9 q# C, o4 `0 a
</description>
( h% b$ A$ j; U. P- K5 y7 h1 c</property>- R& |" y( ^/ N/ w7 s
<property>+ L8 A" k! V8 O! f2 p1 n. X
<name>fs.s3a.assumed.role.session.name</name>
, G9 l8 L; W8 h3 Y# [ <value />
. O2 ], u- `9 j" [. ~* d$ e0 E <description>+ n& H7 v% J9 v( o r
Session name for the assumed role, must be valid characters according to- t6 ]5 \/ C/ W2 i% _9 G, Z7 E
the AWS APIs.9 r3 j6 {$ G/ z$ I* j# [6 |
Only used if AssumedRoleCredentialProvider is the AWS credential provider.7 r, s8 I6 z3 |2 ]6 X' F( q
If not set, one is generated from the current Hadoop/Kerberos username.$ G1 W, @; r( H$ ~; K. P1 X
</description>
1 w" t1 ]* U6 m* U' D</property>+ m2 Z; [. ] _- z" n! b
<property>3 v5 s, f$ b" h
<name>fs.s3a.assumed.role.policy</name>
8 f- |: [$ D9 i9 s& Z2 y <value/>
& h- Y/ N% n8 s& o2 o2 C0 E <description>' H/ z; {7 l! Z8 Q+ P2 L
JSON policy to apply to the role. E( |2 H% v- C3 `+ k
Only used if AssumedRoleCredentialProvider is the AWS credential provider.) X6 t! t6 d$ U: h
</description>
* S& U# f( j" n! Q</property>
; Y8 m, ?: d" a$ |, ~: D! f# y* P<property>- ?2 R4 J9 I9 J* W' Y
<name>fs.s3a.assumed.role.session.duration</name>
3 H0 e/ s8 k8 _% [; _4 M2 h <value>30m</value>
0 h0 _$ c8 m. g$ f' u( c/ t6 r <description>
- p2 j3 }" g' J4 s9 n Duration of assumed roles before a refresh is attempted.
, A. K8 G' X! k3 ~3 X Only used if AssumedRoleCredentialProvider is the AWS credential provider.& I, E6 v. M" J( h$ [1 G
Range: 15m to 1h
! J! X9 B8 p1 Q! E5 _ </description>* v4 n4 v( q& n7 Z1 n% X
</property>
; T, d# o# [8 v" m<property>
) N1 C! ?8 J1 C- F+ p <name>fs.s3a.assumed.role.sts.endpoint</name>
: p/ g, V( K& s$ \4 ? <value/>9 P5 G; Y( `. O, Q1 k' L. T' M
<description>6 ^! K" X: S/ y6 [" e) Z1 d4 ^# V8 j) G
AWS Simple Token Service Endpoint. If unset, uses the default endpoint.
# M2 v7 o; k* w/ e4 \" J7 F$ B Only used if AssumedRoleCredentialProvider is the AWS credential provider.2 L m: }6 W5 l' V! y
</description>6 L8 _8 _2 z4 }' A! O1 V
</property>& j( }5 d3 ] ~8 o; [1 Y
<property>. Q) O1 E8 E" z( j4 s
<name>fs.s3a.assumed.role.credentials.provider</name>
w$ B6 M& E" {9 | <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
: D% D& A; W0 o+ `# c! S. d7 V <description>5 C8 \4 ^* y. ~ O {& D0 [
List of credential providers to authenticate with the STS endpoint and
2 W7 m* g$ M- T* ]2 T retrieve short-lived role credentials.
$ R w7 K" b2 w9 `0 t/ ` Only used if AssumedRoleCredentialProvider is the AWS credential provider.
0 T' Z# I: L2 e' a8 J If unset, uses "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider".
) i0 q& q! n3 G& m3 p </description>
6 t8 `2 S3 T. i. g3 b# K* b</property>
+ j7 w8 H' \: [<property>' k! V3 y' z% H, B& E [" i
<name>fs.s3a.connection.maximum</name>
" L; J: y! e/ ?; M1 N. p <value>15</value>5 M. p( O, x5 ?* Z
<description>Controls the maximum number of simultaneous connections to S3.</description>" |0 m( E& C o
</property>" B. z8 b8 H4 _0 X& n z0 @
<property>( } B' L8 B! g% w: y5 F7 R! ^
<name>fs.s3a.connection.ssl.enabled</name>
2 O% L' }3 P6 I$ g6 r <value>true</value>" n* i/ Z3 S( g! U: Q# r2 [0 G
<description>Enables or disables SSL connections to S3.</description>
8 F/ k, v4 q5 F+ D8 @</property>
- L& E# H# y# m5 x( D<property>
- V W: j) ~# Q/ |) d2 X <name>fs.s3a.endpoint</name>0 {8 C) `6 e o" U. m% [% _
<description>AWS S3 endpoint to connect to. An up-to-date list is% F2 {' w) J$ q) P6 Q6 B" g0 R
provided in the AWS Documentation: regions and endpoints. Without this
. ^( V. k ?! }& M property, the standard region (s3.amazonaws.com) is assumed.9 [% _( ~0 l3 @" V
</description>2 c, t" p: y! G. b5 l
</property>8 ]4 K) D8 s8 G+ e
<property>. |9 _: @ q7 s) U) u1 Q* e
<name>fs.s3a.path.style.access</name>6 e I: l, Z2 u7 o, S% E- v
<value>false</value># A0 ?' J7 e, A* C @: ]& I
<description>Enable S3 path style access ie disabling the default virtual hosting behaviour.
5 W( {, s, Q% b2 ] Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting.3 Y9 _5 i* E3 D% b
</description>. W& N, `4 ~" j0 _
</property>1 x/ W0 N# D6 |: K) v4 L3 C7 b
<property>! N. S& f# o( Y; z
<name>fs.s3a.proxy.host</name>
4 E) {% s) z9 P4 w, w* b& g! e <description>Hostname of the (optional) proxy server for S3 connections.</description>
0 Q1 ^% R9 f: J2 U6 h3 A0 z$ V</property>
: [, ]7 \' a X! }' S; X+ I) w<property>' H/ _$ r* J2 i G+ i5 q) u
<name>fs.s3a.proxy.port</name>/ Y% ]# g% `9 g. o* C- q6 T; P
<description>Proxy server port. If this property is not set& A% C2 @! \0 i4 C K! ~( S0 l* s
but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with
' @5 p+ i3 R0 s( P, }- F the value of fs.s3a.connection.ssl.enabled).</description>; i5 j5 K) s' P; {- A) e. O: Y
</property>
: Q6 q1 {4 t) t2 W5 P<property># L6 n/ _' P6 V/ b3 O
<name>fs.s3a.proxy.username</name>
0 g; N8 \3 Z! e* C <description>Username for authenticating with proxy server.</description>+ @" P. o1 H8 x3 C
</property>4 c# ?: ~) u% P4 v
<property>; N8 b! d* J4 L) k2 \: g0 m2 E4 k
<name>fs.s3a.proxy.password</name>
0 g+ L1 b& l ?6 N# y) A7 r S <description>Password for authenticating with proxy server.</description>
N1 I3 X' E8 T: _, Y9 a% e, X& S</property>+ }2 D8 s# R' a" B
<property>0 R3 o- ^0 ~2 T# f
<name>fs.s3a.proxy.domain</name>- v5 ?% \; W i4 B& T0 O
<description>Domain for authenticating with proxy server.</description>4 o& F+ Z# p2 i; T4 e) s5 M' r
</property>8 P0 j/ k2 Y' V" h: P
<property>
( Z) z0 [- C0 T! Z# T- ^% B <name>fs.s3a.proxy.workstation</name>
6 n" ^1 g2 }* B E5 D$ E/ k, t <description>Workstation for authenticating with proxy server.</description>. j6 h* s" q8 f6 l, K
</property># y3 r8 l- _3 ~
<property>
$ F5 f5 _- A6 x/ r" ` <name>fs.s3a.attempts.maximum</name>+ a; C7 ?; P, y2 O+ s
<value>20</value>
5 R. O. N, ^" T0 w' @# q* N <description>How many times we should retry commands on transient errors.</description>
) r# d1 {4 ` \/ X: ~; O7 I</property>
" _- z7 U+ X8 y# {<property>7 r( h# l1 y! a8 X9 {* B
<name>fs.s3a.connection.establish.timeout</name>1 ^+ J$ N) N% Z! F& \6 W( S) R
<value>5000</value>; ^7 q; u% ?0 `( h _* R7 h" W
<description>Socket connection setup timeout in milliseconds.</description> o4 a+ N+ d$ x
</property>
, ~+ b1 A) O2 I( x* s<property>
! f3 e4 Z4 v% ?: g }6 g( v <name>fs.s3a.connection.timeout</name>
/ ~& O1 f, ?3 V3 E" l <value>200000</value>5 B0 E' r! Y3 W1 P# M; l, ]& c3 \
<description>Socket connection timeout in milliseconds.</description>7 I2 b. b9 z5 ` ^6 P; V" P
</property>' q' O. G6 k$ U. h8 M
<property>
+ W7 C2 b8 D/ L4 j. N* U3 n <name>fs.s3a.socket.send.buffer</name>
" c0 ~/ n8 h/ p <value>8192</value>: L* }" X" E# A- l/ z: e
<description>Socket send buffer hint to amazon connector. Represented in bytes.</description>
& n! x+ v# T; d# M, \</property>
! l2 `9 M: I# i. x4 K<property>
, {6 s# y) y" I3 Z <name>fs.s3a.socket.recv.buffer</name> `% A2 R& R2 @% p. [3 d; ^
<value>8192</value>
' r5 L. |; g* u9 a: U, d& B <description>Socket receive buffer hint to amazon connector. Represented in bytes.</description>
. w5 ~) g/ a8 L9 V</property>, @+ ^' W3 @! a
<property>! R7 d1 d* [! F& |1 r0 h
<name>fs.s3a.paging.maximum</name>. J8 E# u5 t4 U! L+ d. g
<value>5000</value>: e% J- g( `1 _- J
<description>How many keys to request from S3 when doing# d Y8 x8 w3 v
directory listings at a time.</description>2 b; N$ v. m2 O& v' n8 K1 {
</property>5 @! _- n4 r+ l( I+ P& ]# t
<property>6 a* {! U; g' m, [
<name>fs.s3a.threads.max</name>9 F( t8 C" _; j* N9 t
<value>10</value>/ R. I% o6 C9 j, _$ q" O% _( }6 |2 `
<description>The total number of threads available in the filesystem for data
! ]- b3 I2 o/ ^' w- j: z uploads *or any other queued filesystem operation*.</description>
8 u' S( O K4 X- r" O% M, v0 {</property>
( ~# _* D3 m8 ]: m6 Z<property>
H' u$ {% K4 L7 c) S <name>fs.s3a.threads.keepalivetime</name>
: z0 L& S4 y$ a3 h2 R1 {. m <value>60</value>
' v' \: f9 X& g2 u" C% H <description>Number of seconds a thread can be idle before being' y1 O, o! `, l+ u c0 P& d$ l Z
terminated.</description>9 ?! K: C. j3 {# ]( A
</property>
8 t$ D* k0 j6 I x! H& U% |<property>- z- M" v) y- J# B/ R
<name>fs.s3a.max.total.tasks</name>
2 |4 N) J, W; k u7 I <value>5</value>
2 ], o0 G2 k8 W <description>The number of operations which can be queued for execution</description>
' w6 S1 u7 Y8 |2 d/ c% K</property>. S' f4 l7 I6 T; ]/ w8 d
<property>
( [' l" b. b O- E5 c9 f <name>fs.s3a.multipart.size</name>* r! Z' p- k0 ], e
<value>100M</value>8 G8 P: U- J& n8 _
<description>How big (in bytes) to split upload or copy operations up into.
/ H. p( B7 D# Q' C4 O, { A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
. B1 I. K* \2 J </description>
( o& t3 H. J6 S</property>1 i; d, z3 i8 s% E7 I& U
<property>' e Q. x. _6 d
<name>fs.s3a.multipart.threshold</name>
# |. ]+ t* d. \ <value>2147483647</value>. h2 D6 R& j8 X2 [- u, c* w+ J; A
<description>How big (in bytes) to split upload or copy operations up into.
6 I0 U; K: q6 U7 q Y, ~! P This also controls the partition size in renamed files, as rename() involves
4 n0 l' W5 a d copying the source file(s).9 Z0 N0 ~- m( h6 O
A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. m0 V# L" G! W( A7 I3 q
</description>
! |2 z0 o3 {. K8 P</property>$ U1 I2 v; e( t4 V* h, |
<property>
' f8 ^7 [4 ?2 V# M- P <name>fs.s3a.multiobjectdelete.enable</name>! A5 G# Z3 S% E
<value>true</value>7 \6 T& {& j* j7 @
<description>When enabled, multiple single-object delete requests are replaced by& \% r; Q7 V8 h9 v# C
a single 'delete multiple objects'-request, reducing the number of requests.7 M& S: R& V- L4 d! Y3 a# s
Beware: legacy S3-compatible object stores might not support this request.
( |1 _- S1 b* l" R+ X3 E </description>) J" H% `. B, ?$ l( B
</property>
# d) Q4 }8 [' ~<property>
+ A/ N8 ]; X; D, {4 r <name>fs.s3a.acl.default</name>* K/ V/ C" y, v: O- j; _+ E) G7 I
<description>Set a canned ACL for newly created and copied objects. Value may be Private,
4 r7 w! C2 }9 Y9 \* X; H PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead,
5 @) [( M' @) p9 H" A$ K7 H% h or BucketOwnerFullControl.</description>6 H1 |! c L0 Q ]: C8 d" d
</property>
3 E' \! K' N: K( e<property># `/ r8 W% T- w4 @3 h
<name>fs.s3a.multipart.purge</name>: L. e/ l1 b$ c/ I
<value>false</value>
- H; {6 p1 A2 }8 _2 }0 b. V3 ? <description>True if you want to purge existing multipart uploads that may not have been
5 J% ~& J4 A: M- q+ j completed/aborted correctly. The corresponding purge age is defined in
' l& B1 S* R, ?) l# u fs.s3a.multipart.purge.age.9 K0 M/ ^& ^' [1 ^2 y: U& z! h& V+ ^
If set, when the filesystem is instantiated then all outstanding uploads
5 y2 k+ v2 h) F, J$ v" r! @& M. M older than the purge age will be terminated -across the entire bucket.
. l& y" X5 P! M( _& J This will impact multipart uploads by other applications and users. so should6 r7 u& `) q, z$ A. M+ L0 M3 M
be used sparingly, with an age value chosen to stop failed uploads, without9 z8 f, x5 j, f
breaking ongoing operations.
1 p+ A) ]2 c+ Q. _4 j6 i </description># b& J# e+ u& M1 E" f( P( R
</property>( O! K+ v# r: m$ r, r/ g
<property>- K8 T: {! z& U z; H$ p
<name>fs.s3a.multipart.purge.age</name>* a7 |% z# k6 n; X9 b4 o
<value>86400</value>
0 G: ~) q z& J: G# P0 G$ Z Q <description>Minimum age in seconds of multipart uploads to purge
% @; x9 C) e4 h5 e4 }9 h2 M9 L on startup if "fs.s3a.multipart.purge" is true
7 C& p. T4 `9 U+ X </description>
, V7 E4 l5 Q' k3 b2 N7 W/ `</property>
& K' }3 b( Y) |, E9 S<property>' f. P8 {" k e5 F
<name>fs.s3a.server-side-encryption-algorithm</name>
# N3 o2 I; N# U. M$ _6 H <description>Specify a server-side encryption algorithm for s3a: file system.
3 T! F( |* J2 o. {& i5 c) R Unset by default. It supports the following values: 'AES256' (for SSE-S3),
; W F5 K7 K# ?0 y& \! I 'SSE-KMS' and 'SSE-C'.3 C, J5 ]+ W/ V4 d3 s! F
</description>& b4 i U" e! l1 y: u+ K; o, _
</property>
. l- y, s; i6 |<property>
1 V _' e8 @- G# s, L <name>fs.s3a.server-side-encryption.key</name>
2 F5 V; e. z( f9 n& V4 ^- Y5 L <description>Specific encryption key to use if fs.s3a.server-side-encryption-algorithm! [; L7 b5 {( u9 A, V, c
has been set to 'SSE-KMS' or 'SSE-C'. In the case of SSE-C, the value of this property
$ b2 e! Y' W2 I* P. Q" E should be the Base64 encoded key. If you are using SSE-KMS and leave this property empty,
4 g3 C* Z W7 C! y, @0 ? you'll be using your default's S3 KMS key, otherwise you should set this property to- T6 j2 O, D8 j& M* w4 X h
the specific KMS key id.! Z+ t$ ^2 m: Q0 ]6 E$ X {
</description>, Z- w( e# v V
</property>
, C) ?5 \& L' Q$ W<property>2 u3 Y8 T A+ H! f, f; y+ y: |
<name>fs.s3a.signing-algorithm</name>; X/ N5 C& o0 m# {+ k& z8 y$ v u
<description>Override the default signing algorithm so legacy
' \) Z3 w" C! S5 ] implementations can still be used</description>" Q+ o9 M7 H4 `6 ]) g1 M5 L( W
</property>
3 _% c2 \* A7 y' T' @5 C<property>0 }. j1 a; u* u: \+ e8 d/ Q* s
<name>fs.s3a.block.size</name>
^% w: l4 M% p5 ~ <value>32M</value>
/ R# W5 s' i1 b. z <description>Block size to use when reading files using s3a: file system.
4 g4 T+ O& o/ r- V6 g$ _ A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
) V; Z0 M+ K0 N* g </description>( |, ], O. c' h
</property>
& d2 U. [2 A: i<property>
$ P/ Z7 r# B2 j6 z6 c <name>fs.s3a.buffer.dir</name>: Z) [% |6 t! z A# G9 r
<value>${hadoop.tmp.dir}/s3a</value> h ]7 v/ S8 P9 Q5 b
<description>Comma separated list of directories that will be used to buffer file; }3 i' z& J0 ?3 s" i! X
uploads to.</description>% ~) O8 F5 O. _) b0 x U
</property>5 O8 n4 Q. `0 z1 o
<property>' Z& C( S+ [( `1 g
<name>fs.s3a.fast.upload.buffer</name>2 c0 u# i! j) O7 A3 b4 S3 I0 X
<value>disk</value>
' ]' k: d' ?* t9 H' Q2 a6 s" [ <description>/ u! o7 C# W! s5 ]( F
The buffering mechanism to for data being written.
: X' P# s6 b3 y) m& J* j. g% L2 O Values: disk, array, bytebuffer.
/ s1 S/ T4 f9 d$ {& ?; V F "disk" will use the directories listed in fs.s3a.buffer.dir as8 o5 G) P. R+ X( k* w0 ]# E$ w% E
the location(s) to save data prior to being uploaded.
- n9 \2 b* N9 W- E "array" uses arrays in the JVM heap3 i# j* \3 D& V
"bytebuffer" uses off-heap memory within the JVM.+ u" N9 i. b: \; G
Both "array" and "bytebuffer" will consume memory in a single stream up to the number
" k: S6 Q$ u& s of blocks set by:$ j" K! Q# q+ E2 B% d, P
fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks.
$ L, S4 k- y3 n9 ^# b If using either of these mechanisms, keep this value low3 m2 E& \/ o; R. U, F
The total number of threads performing work across all threads is set by+ h9 g. F" z& M# ?1 i7 t; I
fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued
. Q6 A( y+ @' P. l work items.
) L- u& c0 V$ d/ G" s+ Q </description>% d+ S* g. g" M3 a, W( C$ F
</property>& j: f" R) U* C. \% T
<property>
* ? l+ Q3 Z1 E& j/ l( a <name>fs.s3a.fast.upload.active.blocks</name>
& W4 L9 D5 w. x3 M+ I6 q; Q <value>4</value>6 s+ g* u2 A1 B# i
<description>
- Y" _/ h8 T! u$ ~( Y2 n Maximum Number of blocks a single output stream can have# t" B! }: m' r1 [3 d' E
active (uploading, or queued to the central FileSystem
0 ~( I" n+ r# O- i4 }$ k instance's pool of queued operations.+ a2 J" f7 Z7 s5 c9 u0 G* b) L
This stops a single stream overloading the shared thread pool.
0 [* d" L5 H% ?- o9 _1 V2 ~1 y6 x </description>3 R) @7 t& a. ^ K7 [
</property>
; W) Z: u- P# W: G1 i {$ y* ]<property># S- e+ X8 T! @5 Y
<name>fs.s3a.readahead.range</name>
0 X3 d; C4 W! Q4 o. f <value>64K</value>( j$ N+ p, b1 q8 j7 z$ p) F5 ^
<description>Bytes to read ahead during a seek() before closing and
' T% Q! g+ O& B+ {/ D' Z re-opening the S3 HTTP connection. This option will be overridden if% V" k, e7 V# ]5 g+ v0 P
any call to setReadahead() is made to an open stream.
9 Y* x3 p& E& _- U+ @+ L/ M A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
8 W7 ]) i9 ~+ q. B# h9 [ </description> [1 a: N) o" d2 V/ Z' ~" ?- a
</property>
& e- @' F/ z& L, D$ k& U# k2 M4 y<property>
- }3 T5 N, r. A <name>fs.s3a.user.agent.prefix</name>/ P9 ?8 d- k' |' r3 Q
<value></value>5 c' l1 G4 Y" U5 E6 w4 H
<description>1 F v6 \- d( I" U5 `. p3 N
Sets a custom value that will be prepended to the User-Agent header sent in' z, T( A ?5 K5 s. d- {1 B
HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header% K0 l2 R- \7 }# R, t, l( o1 m
always includes the Hadoop version number followed by a string generated by
3 V* o7 u& X0 \8 \; C7 i the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6".9 c7 T8 _6 i% i/ d) e* v
If this optional property is set, then its value is prepended to create a" k$ O$ T4 q8 O4 @
customized User-Agent. For example, if this configuration property was set
& g) T9 W2 ]' z8 \5 I+ ?/ ] to "MyApp", then an example of the resulting User-Agent would be
$ V* o w( a+ T# G "User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6".$ n) Y Y- c" }9 \3 ~
</description>+ E' x! X/ y' o2 z; J4 ^+ K0 L' `+ v% s
</property>
2 u9 p3 z# S" C# j) y! }<property>
/ q9 [9 P7 S8 Y* A+ K1 U6 A <name>fs.s3a.metadatastore.authoritative</name>
; T4 A6 K6 F- X7 h: _7 m$ O: p <value>false</value>- O" }9 r3 q( ?' V
<description>
0 S& C _- ~5 f; n: d; G/ R# \ When true, allow MetadataStore implementations to act as source of/ E) J3 p* Z- l+ e+ l# d
truth for getting file status and directory listings. Even if this
+ ^( \! Q0 f; Z3 E! C+ d2 s* l is set to true, MetadataStore implementations may choose not to$ b4 f( ~, I: } o7 G: M; l
return authoritative results. If the configured MetadataStore does7 D; s( T/ d- d9 c
not support being authoritative, this setting will have no effect.
1 d' i+ N5 C8 ~5 i </description>
* H( I# Y0 i, e9 a- b# @+ z</property>
5 N# S/ Z8 y2 ~" K& t<property>
% ^ m" @2 Q* U* \# k T <name>fs.s3a.metadatastore.impl</name>
4 c, q q+ }) P; w <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>: R9 ^9 b/ y9 W* d2 O5 _
<description>* F2 H( o3 V" _; ]
Fully-qualified name of the class that implements the MetadataStore' k5 x. ]( Y* q7 d4 U
to be used by s3a. The default class, NullMetadataStore, has no
, w/ @6 V" A6 H, J* q effect: s3a will continue to treat the backing S3 service as the one
9 c; x, h* B: i# V6 w and only source of truth for file and directory metadata.
0 X/ k& T4 Q+ q8 ?% O$ e </description>. h8 `" @) ]/ [ d, i# l9 @, J: a
</property>2 T! j9 `) }* L! {. N4 R7 V. L) _
<property>- e7 \* r1 @- G8 @8 I+ R
<name>fs.s3a.s3guard.cli.prune.age</name>
$ J% X* f; ]* T- y3 U <value>86400000</value>7 n, b! O L: y& z# p& p
<description>; [/ V, a* k- y; W6 U+ L! b
Default age (in milliseconds) after which to prune metadata from the7 j+ t$ E) N4 N8 b5 j
metadatastore when the prune command is run. Can be overridden on the( |# @ ^1 |9 j# y4 U5 L+ g
command-line.: x# V/ U7 H5 ]/ t
</description>3 W+ p# ~* R5 `
</property>; i: J8 J8 F0 o. G8 @$ ]# E
<property>
$ r; F2 p" U. b3 Z% w <name>fs.s3a.impl</name>
0 i! [4 M/ ^! S! R( N <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> N7 m8 W$ S8 G/ C: V: G* n8 }
<description>The implementation class of the S3A Filesystem</description>
$ `( h2 e( h, i( F2 j0 U</property>8 ]$ r1 y: o( S2 |8 ~: I
<property>2 i% g+ h5 }, s3 P
<name>fs.s3a.s3guard.ddb.region</name>* k8 i/ B, g# l, \( L6 n4 c' t
<value></value>
+ F! f& B( ?8 ^7 y; q" J <description>
; b0 S/ C; O1 h0 Z$ Z6 a6 D- U) P* p AWS DynamoDB region to connect to. An up-to-date list is
) w9 A3 d+ |& v V4 {+ D9 k provided in the AWS Documentation: regions and endpoints. Without this
5 B o3 L; i) u7 @! H( k property, the S3Guard will operate table in the associated S3 bucket region.1 F% k8 E8 r# v$ x1 g
</description>
4 ?( s" A9 b/ { Y7 |, z</property>, f9 l# V* J% E* L
<property>
! w: |1 c; {" n2 r1 G4 P <name>fs.s3a.s3guard.ddb.table</name>
$ Z! z0 Y( D& [* f6 y <value></value>* m O& a" L9 u$ i* o- W! V/ r1 n
<description>! n3 L8 l$ W* |) y& e
The DynamoDB table name to operate. Without this property, the respective3 ]* C* D6 ?6 i) q& h
S3 bucket name will be used.9 u; A" E# g5 v
</description>
! @* R9 [! E) S: f</property>
7 s+ M! J1 P" [<property>
# g2 x# \# S9 S* [/ y8 k s2 N <name>fs.s3a.s3guard.ddb.table.create</name>* ^9 P4 |! n) W! T3 F; ~
<value>false</value>
) ?1 }. r. q, L, F& F$ U) p <description>$ \+ D0 S1 C/ u2 @4 e
If true, the S3A client will create the table if it does not already exist.
; i& k9 s& m, d. Y8 t# { </description>
& h1 h) d2 ]. q& o</property>
8 i+ r* }: X) b9 U# C: I<property>0 C% t3 [0 o8 v2 [% V
<name>fs.s3a.s3guard.ddb.table.capacity.read</name>' v8 X1 v% i' z( e: l$ S1 M1 z
<value>500</value>5 f7 _! A9 L% m, G/ X- t
<description>' k" {' n4 f' O/ y/ U; O j, ~
Provisioned throughput requirements for read operations in terms of capacity+ j% `! S$ A, e
units for the DynamoDB table. This config value will only be used when
, y3 C4 w% _ X6 M' Y, s creating a new DynamoDB table, though later you can manually provision by. C! K3 n2 x" `
increasing or decreasing read capacity as needed for existing tables.
$ I. N& u( D" e+ E See DynamoDB documents for more information." ]+ X' w& V" @& e4 s8 q. N
</description>
9 E& _$ N" L# N {; f9 l. T</property>& |) p( B1 w8 i
<property>
4 R$ e2 m/ R; m <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
/ D5 ]2 H8 M# C/ M. h. {7 r$ W <value>100</value>
0 w4 }' ~3 i' c/ L* d <description>
: [6 G6 x8 d! z3 _ Provisioned throughput requirements for write operations in terms of
8 n" d7 l( k- w( V$ e: y/ U capacity units for the DynamoDB table. Refer to related config9 t) }' p$ J' A& p" \# Z( g' @
fs.s3a.s3guard.ddb.table.capacity.read before usage." F7 b4 G6 q3 b$ U3 J0 f5 x) Y
</description>
; C# D- Z" Y# I" {1 X7 v. z: Y- O7 i5 y</property>
, _/ p* @' p1 u3 J4 y( [0 a# L<property>+ v# e1 E4 {0 y& t% P' Z/ h: D
<name>fs.s3a.s3guard.ddb.max.retries</name>
9 @) h( @1 Q, Q <value>9</value>
) i( E- F3 ^& k& |$ q <description>8 `; }# ]: \) V7 X I
Max retries on batched DynamoDB operations before giving up and+ M" y' L, K v1 e/ R% l
throwing an IOException. Each retry is delayed with an exponential0 x7 B( J5 }2 \
backoff timer which starts at 100 milliseconds and approximately" D( |4 Z; P/ _! R9 ]! w
doubles each time. The minimum wait before throwing an exception is
% f# E( `; Q) ]2 U+ {8 z/ }0 m sum(100, 200, 400, 800, .. 100*2^N-1 ) == 100 * ((2^N)-1)* x; [2 R1 h% B
So N = 9 yields at least 51.1 seconds (51,100) milliseconds of blocking, i1 o+ g7 I+ c ^
before throwing an IOException.
3 K" a$ s+ t9 t& L/ ^5 e6 [ </description>
2 y1 J1 n" |- {* d3 W7 S$ V</property>
! _+ A, D2 T# J! S. t4 g3 b<property> ]$ p# J" e- M5 U+ Z# \& u6 V. }
<name>fs.s3a.s3guard.ddb.background.sleep</name>! d, C# A2 `5 }+ f6 Y. m t
<value>25</value>& [8 o( u; V u6 K$ b" P: ^/ C
<description>9 x8 ~% k# X. r
Length (in milliseconds) of pause between each batch of deletes when' U0 Y6 t- ?( u* S
pruning metadata. Prevents prune operations (which can typically be low
Y* Y' I o7 i, ?5 Y4 I9 f- S priority background operations) from overly interfering with other I/O
* P3 x" h9 B: d/ l operations.
2 E! Y% C8 N6 e8 c+ b' } </description>
u5 B& j% p4 j( {" d2 U! {9 }/ h</property>4 e6 }7 R) k9 p4 [& }1 ^2 v" ~$ Q% N
<property> d: f0 {8 g4 x; |8 P) a7 x( W; N
<name>fs.s3a.retry.limit</name> |+ ] `) t7 L4 u/ j: M
<value>${fs.s3a.attempts.maximum}</value>
2 a$ A+ z- |1 R% Q <description>3 P# T) C, g. `. S2 d" D
Number of times to retry any repeatable S3 client request on failure,: t5 M: v% H/ ?% w) \% c
excluding throttling requests.
& u# w: U( q7 x+ r </description>* b- s5 x3 e7 J W- _. H& n1 p
</property>
2 `, N2 M7 a5 I' M+ x1 e: Z<property>
1 \. I$ d& Y3 ^( |3 {. ~ <name>fs.s3a.retry.interval</name>! p( r& o2 Z+ R9 Z
<value>500ms</value>
7 J' o* j5 d. t; o C% V! w <description>
4 B& V* l8 \4 P& ^+ z( l4 p0 }2 y Interval between attempts to retry operations for any reason other
2 U8 G0 C [3 y. Q7 ?, S than S3 throttle errors.9 G# z! S' C! B) \) C
</description>
2 {& c7 k) n/ d1 t2 L$ C2 s</property>+ `/ g$ X. J( V. z
<property>4 D1 f$ v4 i$ r* }! z
<name>fs.s3a.retry.throttle.limit</name>
6 V0 b& h6 Z1 b& H9 b3 p <value>${fs.s3a.attempts.maximum}</value>0 A$ s" L/ r b+ U" x$ y; W
<description>
; Z9 [ u6 C* ? Number of times to retry any throttled request.
8 ~0 j7 z6 y M! `, g! `' e& _& } </description>
+ `8 I& b( N0 l% s$ n N) N</property>
) j: I# ^5 u/ N- s2 i<property>
7 X4 R L9 N1 ?; t <name>fs.s3a.retry.throttle.interval</name>
2 X/ ~6 J7 m I z <value>1000ms</value>
8 k; @7 _8 o! Y, e1 H <description>
( Q6 k3 U1 R. e# e7 k6 b1 H Interval between retry attempts on throttled requests.
# I# |) Z7 ~8 T- v6 s6 `8 D </description>
/ @6 M, j2 h. ?6 u7 f( W6 L0 N</property>; Q' m2 v" k1 h& Q) f: A
<property>3 I# y9 k! D5 z! Q. f& v( b
<name>fs.s3a.committer.name</name>7 P$ f9 f9 c2 N, x- A* Y# z/ z4 X& M
<value>file</value>6 d1 d& G; k/ c9 b( |% t
<description>
# i! h# b& G" n( { Committer to create for output to S3A, one of:
; n" E1 @/ a1 j "file", "directory", "partitioned", "magic". C$ T2 _8 [8 n7 h. c
</description>
( {& [2 i. H6 J( [+ z* ~8 e2 w</property>
! e3 b) t( N! t5 J" s4 [ l<property>
2 P8 u5 B6 e; R. ]% D! j. [ <name>fs.s3a.committer.magic.enabled</name>! m# ~" g7 I6 x0 m2 E0 s5 v
<value>false</value>
& P. @7 v) p/ L5 S9 f$ _3 q/ | <description>% D# |6 | q' n
Enable support in the filesystem for the S3 "Magic" committer.7 y+ P2 y1 e1 W6 a
When working with AWS S3, S3Guard must be enabled for the destination5 n7 B1 L* t5 m: Z- n
bucket, as consistent metadata listings are required.1 @) g( I! ^& i3 h) e8 Y3 P
</description>
: W- n6 }+ `" I2 p% z. `5 f& a5 R# }</property>: W4 p1 e u9 n" |4 o- T
<property>
! Z2 Y r3 P; a1 {. Q <name>fs.s3a.committer.threads</name>6 g. T- S3 d( m
<value>8</value>! g' E1 ^$ p( k Y$ X7 i! Z
<description> o! k4 X. p: G |7 {5 H; i6 U
Number of threads in committers for parallel operations on files
2 T& c9 @4 h0 E (upload, commit, abort, delete...)
0 {# ~. u6 f6 ?; E </description>
0 a1 _! r# o0 L( _9 X/ V</property>9 T7 Q/ Q8 J& S# d# E; {
<property>9 f5 |+ o A2 m4 \( M0 c
<name>fs.s3a.committer.staging.tmp.path</name>
0 X9 U# m4 {# m0 @) e <value>tmp/staging</value>
8 K' i3 y. i. L9 ^- @" i' w; r <description>( o: Z: P* I v. ?
Path in the cluster filesystem for temporary data.5 G$ n$ e9 H8 m2 _9 M6 b2 p) m' K
This is for HDFS, not the local filesystem.
2 O# \" T9 `+ U( C3 ? It is only for the summary data of each file, not the actual
2 b9 U* n% K: s' M data being committed.
( F/ S/ x/ h0 a* h' {' x1 k! \ Using an unqualified path guarantees that the full path will be6 j& o1 F G5 x5 ]) n% j
generated relative to the home directory of the user creating the job,! M; L, j6 m _6 ~+ ^" j# @! G
hence private (assuming home directory permissions are secure).9 T0 u# F3 [" L( J5 t F' _
</description>
8 [- k5 r4 S: d4 H0 T* o! c4 K5 i, S</property>
0 r( r! ?% X, D/ Y* h+ h' R<property>8 y/ y3 d) w R9 Q5 d. G
<name>fs.s3a.committer.staging.unique-filenames</name>( L7 w3 z, _+ p% ~) J2 n
<value>true</value># A( ]6 B- V/ ~% B
<description>
6 {: I- i5 y- z9 ^0 F& h Option for final files to have a unique name through job attempt info,
: A8 B8 o6 D' x" o( G5 E or the value of fs.s3a.committer.staging.uuid4 W! S3 d0 Z5 z
When writing data with the "append" conflict option, this guarantees; D* o5 g4 @- |- j. e
that new data will not overwrite any existing data.
1 J) G, B; G2 l) m+ g1 O T/ j& y </description>
. G- K3 j2 W1 w5 C0 L/ o' U4 V</property>. s4 H& q# ]; W
<property>
5 Q1 C- t' x$ B9 a& w8 y <name>fs.s3a.committer.staging.conflict-mode</name>, p8 C. V V/ C7 I# k' l
<value>fail</value>+ Z2 a# \/ J$ a y8 E2 l& f2 `* v( B
<description>
3 N$ v9 u" O$ l N" |) n5 l% o2 G Staging committer conflict resolution policy.* i! v, X8 W. i0 C5 d' j" ]
Supported: "fail", "append", "replace".
2 d' r7 V# S" R& a! Z' @8 | </description>* L& c0 h( N, `7 ?
</property>
+ ~: k6 O: q& F<property>, d0 n$ _/ k" ?7 L* A; T
<name>fs.s3a.committer.staging.abort.pending.uploads</name>
: ?4 f$ a" _# C) ]. z0 G9 p <value>true</value># l4 I, q& w( w
<description>
# A+ [6 T- E1 d1 ^0 B* _; S6 h Should the staging committers abort all pending uploads to the destination
: i- R8 H, ~ Q. C1 d n directory?% F( ?( J+ I) U, t
Changing this if more than one partitioned committer is; p+ X. N: U* ]+ A
writing to the same destination tree simultaneously; otherwise2 C9 D/ d2 }$ A9 `4 `% w9 x
the first job to complete will cancel all outstanding uploads from the
) a$ O) K5 T' e3 C! w. F others. However, it may lead to leaked outstanding uploads from failed+ c+ @. e1 q' O) l1 I2 E7 S
tasks. If disabled, configure the bucket lifecycle to remove uploads; d9 w* Y* r1 w u" L
after a time period, and/or set up a workflow to explicitly delete
$ Z8 K( V. w- V+ _2 G* m m W entries. Otherwise there is a risk that uncommitted uploads may run up' P, m& q& j, ]9 y! v7 g
bills.
6 f+ p6 f) A0 e6 _ </description>" B! O5 C: I+ ^: U5 z) j K
</property>; h& ]. Z6 i$ z/ Z# J2 Z. V8 k, e
<property>
! q) Y3 r: h& Q6 J+ |# B: r <name>fs.AbstractFileSystem.s3a.impl</name>; a8 L: m. I/ z v/ V& k5 \9 P' z! w
<value>org.apache.hadoop.fs.s3a.S3A</value>: r! n) B" v* n) ~: {3 L
<description>The implementation class of the S3A AbstractFileSystem.</description>3 x+ ]$ d1 m" N
</property>
5 E4 _" I g: N+ O3 ` v<property>0 F% M$ Z8 u5 B9 R; \+ l W n
<name>fs.s3a.list.version</name>7 U. z' |- t# m# W* n" T: w7 W
<value>2</value>* f# h0 X6 s! M5 ~3 W
<description>4 h( n/ ?* B7 t' r% L
Select which version of the S3 SDK's List Objects API to use. Currently
7 n! V: b% G: |3 d1 R" V support 2 (default) and 1 (older API)." s3 ]4 `- q# L& q: i8 L
</description>
' b5 r& K6 N/ f! P- w9 t</property>9 S) P2 q6 N1 j( G6 f; F
<property>0 b& K1 P6 Q+ y& M1 F5 [
<name>fs.s3a.etag.checksum.enabled</name>
4 r( n, t6 t1 n# p6 q- l <value>false</value>( P- r( `9 y! h9 x( ]# F
<description>
0 Q. T2 T! X; t: y+ A Should calls to getFileChecksum() return the etag value of the remote6 D; b" j# O- k8 E1 I& S
object.
' b' Y5 ~! A3 h; U+ o! D WARNING: if enabled, distcp operations between HDFS and S3 will fail unless
( J( g( u3 e$ P -skipcrccheck is set.
: B- z* k3 [ V& k! f </description>
8 t9 k8 }/ L( x0 d( \</property>
8 U/ _/ j$ [" |- W, `5 [2 L<!-- Azure file system properties -->
: p2 J$ I9 x2 Z O0 i<property>( c* N9 a% P2 B2 R' d! g" b# M* }: p
<name>fs.wasb.impl</name>* n' ~: [' H/ T' q; W" s7 b M j
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
2 N0 U2 Q5 F6 e" S <description>The implementation class of the Native Azure Filesystem</description>0 B; S* g! ?5 y9 B
</property> ^$ N* F. J0 S2 ?( K; W
<property>
% B4 p2 Z# R+ ]( _7 E5 Q. ^ <name>fs.wasbs.impl</name>
+ T; A- x: a) V3 P2 l <value>org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure</value>
- [ t+ b7 I1 w0 S z. d6 [- w <description>The implementation class of the Secure Native Azure Filesystem</description>0 x3 i- u M- r2 O) B
</property>
K, v( B0 ~" z( W9 f& o! y! Q<property>
8 w1 F- v) v; U" w <name>fs.azure.secure.mode</name>
8 W* w/ c C; T* c8 M <value>false</value>4 A. F3 ^1 j9 |* v7 |
<description>
2 ?* v1 q- B5 m9 p: O; j3 F Config flag to identify the mode in which fs.azure.NativeAzureFileSystem needs
U5 B* J' V$ q/ J to run under. Setting it "true" would make fs.azure.NativeAzureFileSystem use/ K% g( o0 |" `! ]/ G7 g
SAS keys to communicate with Azure storage. n. I+ C% O. E' f; p& Q
</description>
1 O# K( Q6 m" L$ T</property>' j0 |% u* K% @( m Y
<property>
9 A7 m7 i' \# p+ a/ A/ X; c <name>fs.azure.local.sas.key.mode</name>
" U" F: i& [1 o; h* D- | <value>false</value> Q9 J" w/ i9 U- R, ^) M, R
<description>
( s3 w* B5 J9 b7 b0 c. n# @ Works in conjuction with fs.azure.secure.mode. Setting this config to true) z. L& y# @) [$ M9 H: J
results in fs.azure.NativeAzureFileSystem using the local SAS key generation: R4 e) ]4 t/ s- p$ O$ O" m
where the SAS keys are generating in the same process as fs.azure.NativeAzureFileSystem.
& Q5 ~) \/ s# _! }1 w* O) l: { If fs.azure.secure.mode flag is set to false, this flag has no effect.
0 l; a4 N) ?. D- I" Z, W </description>; ~: [& _2 p4 L0 l
</property>. L- x. u8 h; T9 U. D; \2 V& T
<property>+ M: \/ R4 B8 R4 j6 m& T3 w7 q3 {
<name>fs.azure.sas.expiry.period</name>8 z: W; j8 \- y9 l; Q/ g
<value>90d</value>- w' t$ \. D% A q A, ]
<description>
% r) E4 q7 f6 w0 L The default value to be used for expiration period for SAS keys generated.
8 L+ I, I: T4 C Can use the following suffix (case insensitive):7 ]7 X! F4 b( v9 A' x, r0 K' I
ms(millis), s(sec), m(min), h(hour), d(day)4 B9 z$ o, F+ o) N
to specify the time (such as 2s, 2m, 1h, etc.).+ A9 u) `( z f; g; d1 M" y
</description>
9 [. \$ r# v4 M3 n, P</property>
' c( S+ {9 A% W5 f) J<property>
# R: L0 L/ J9 z4 n- Q' w <name>fs.azure.authorization</name>
2 L/ L3 b$ R2 y& N- A) j0 w8 T2 z# F <value>false</value>5 E8 Q5 y3 c3 H7 n4 B, w& k0 s
<description>
/ h! `' R& a! G Config flag to enable authorization support in WASB. Setting it to "true" enables5 P: \, s; _0 X8 }) @! l4 x
authorization support to WASB. Currently WASB authorization requires a remote service# G1 u: G, u E2 N# U
to provide authorization that needs to be specified via fs.azure.authorization.remote.service.url
& Y) a1 Z) x% z7 w$ O+ t* H* d configuration
* q- F7 D3 }; {8 B% Q5 r( L </description>
3 ?7 M( F% K3 ^$ j</property>9 j1 c) m! ^/ c5 _
<property>
+ K) e+ m. S. {; M; v$ g5 m8 k <name>fs.azure.authorization.caching.enable</name>
7 W6 `9 m+ N# r; Q) S <value>true</value>8 o: p7 B3 d" j
<description>& |/ ~. ?4 |2 x
Config flag to enable caching of authorization results and saskeys in WASB.0 U0 {8 C- U9 a" P$ t. L
This flag is relevant only when fs.azure.authorization is enabled.
6 D/ h( `1 C7 D1 {' a- X Z </description>
. I* t5 @" j$ { R& ~( V% @: o- d6 }</property>
9 Z! Q# p4 ]' O* z! H<property>
2 \! V: G( o* e+ Y/ v <name>fs.azure.saskey.usecontainersaskeyforallaccess</name>
- \$ A* G: y, Q7 G$ G! U <value>true</value>
7 b3 e2 T0 c' m <description>
6 z+ p6 f2 v! n8 k2 n& g7 C2 K Use container saskey for access to all blobs within the container.1 S( U! a; D! _- \ x! A
Blob-specific saskeys are not used when this setting is enabled. j& N+ A7 a3 h* C* _, F2 |& W" h
This setting provides better performance compared to blob-specific saskeys.. n9 x+ j: V, D
</description># h: [9 L; N' P9 X- f! o8 @
</property>0 s- [$ i) o5 N4 D0 C" P) l
<property>& F2 W( l& V* p/ U. U
<name>io.seqfile.compress.blocksize</name>0 E' h4 {: j' M
<value>1000000</value>
u2 M, H" V+ l <description>The minimum block size for compression in block compressed5 Q& w! a! k( |" B5 n) O
SequenceFiles.: l; f0 @ }/ G% V1 A; N3 S
</description>
: }; f4 J& V2 b0 T, r; p- i</property>
) l4 Z) k* a0 i; ]( y' U <property>( G1 }8 P- J# ]' B* A" R8 E* v
<name>io.mapfile.bloom.size</name>
% @& X- F+ \+ ~/ l) A4 D+ ^0 E <value>1048576</value>) R. h' f; \5 h p
<description>The size of BloomFilter-s used in BloomMapFile. Each time this many
. B; P( X4 d3 d( ^# k9 q/ C9 K keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter).# O x) g4 l2 R3 p2 d+ i) w
Larger values minimize the number of filters, which slightly increases the performance,: E, B5 F# }; q/ q5 t d3 }
but may waste too much space if the total number of keys is usually much smaller
% P K/ S5 V; ^# Y2 y6 B) Q than this number.
) j. \" T2 R) l! l </description>& s8 S* S; B: d% J$ B
</property>
: K; o7 U0 x. @<property>! T# K% |- p8 m( l) S0 y% u
<name>io.mapfile.bloom.error.rate</name>
% C8 g8 r2 ~; J/ b2 y3 r4 J# ^ <value>0.005</value>
/ |/ X& O5 j1 I4 p! H <description>The rate of false positives in BloomFilter-s used in BloomMapFile.. ^! A0 k6 c& F4 c! @5 G
As this value decreases, the size of BloomFilter-s increases exponentially. This
! K& s- [: F" J; N value is the probability of encountering false positives (default is 0.5%).: o, n0 p: P5 E7 U2 i
</description>; h% i; M* Z: y* b' M. T
</property>2 y, ~. y7 S, U' K$ t4 ?
<property> K1 Q/ F/ r) @* r
<name>hadoop.util.hash.type</name>
- L/ j6 V& Y- z <value>murmur</value>) U& H6 [/ e: F" u
<description>The default implementation of Hash. Currently this can take one of the! o: I5 a+ s6 V3 c' V
two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash.
8 ^' N, b5 F4 @( o/ m </description>
6 U/ d. ^+ T4 Z% p</property> H) a" q1 z r4 _) G4 T8 q
<!-- ipc properties -->
. A! k. V7 |( }, _6 K; V<property>3 G# `* ~, s2 N' N/ ^9 F$ j) Y
<name>ipc.client.idlethreshold</name># { X4 i3 ]4 T
<value>4000</value>
5 \3 k7 ~2 H6 B& I8 b1 j <description>Defines the threshold number of connections after which( ]6 L1 Z0 r8 v% F+ r5 o3 |
connections will be inspected for idleness. S: A2 s# C6 X8 ~) f/ [- d
</description>
6 x r8 [8 |; d. _1 D</property>
$ h; ?# |2 o- O. g2 Q. U4 p' M<property>
: q/ S+ B% T7 z5 Q, p <name>ipc.client.kill.max</name>' p! \+ ]/ F+ V6 K. T# l8 w! m0 @* y, f
<value>10</value>
* N' `8 T) d; l2 `3 i <description>Defines the maximum number of clients to disconnect in one go.) P5 {2 A9 M" q% U5 u. s
</description>6 g4 J) x6 i0 L: V1 t
</property>$ h, o% { t+ K) k) X# K4 l& e$ e
<property>
# r3 ]' a3 I1 K. s% L9 C <name>ipc.client.connection.maxidletime</name>
1 s% U; y/ o) `% a <value>10000</value>
4 ~2 e9 j8 `- q+ c2 i% D <description>The maximum time in msec after which a client will bring down the
, j/ t( t" u8 N# f" X: I connection to the server.8 n5 b5 t# l2 m! I/ ^9 Q: o& A
</description>
( C/ e4 w% s$ {- J</property>
7 G2 M0 w8 I5 T5 l2 o<property>
4 A" n/ W3 y2 d3 t <name>ipc.client.connect.max.retries</name>
# s; q% ?" \; \& ~0 O <value>10</value>
# E: n( U; w+ y/ ~( T: r7 \ <description>Indicates the number of retries a client will make to establish
/ }, C+ | V( U( U2 }% ` a server connection.! B/ H) o- K. R3 p- W' q( i, I- I( U
</description>2 @# k9 d* \0 b" @0 V9 M1 i
</property> c$ u" U6 p; \2 n) J* }% F
<property>
, O. P$ g8 e5 r. u) N$ Z" b <name>ipc.client.connect.retry.interval</name>0 N. y7 l& r6 R
<value>1000</value>9 y& U# `2 g# ~; c, B; c
<description>Indicates the number of milliseconds a client will wait for
0 V) o, _1 c0 G before retrying to establish a server connection.
8 G/ M/ i( ?2 o3 E4 `' u+ q4 Z </description>8 w. P. A2 I8 c# z# O
</property>- i8 d* f3 C p, G& u: a& M9 @
<property>
' [) @0 E( c, l( t <name>ipc.client.connect.timeout</name>
: E: J& k3 A6 [! { g <value>20000</value>
2 j7 w$ m* a* K O% a, h <description>Indicates the number of milliseconds a client will wait for the: e9 \2 A# ?& }2 s5 H2 X
socket to establish a server connection.( F `- X# H- }4 n+ G ^; @ m
</description>' r4 H' Q( n! e2 S2 d# z7 z
</property>
; \" V6 M d- s# y1 \<property>
1 n; J( U1 q# V; b6 j. X <name>ipc.client.connect.max.retries.on.timeouts</name>$ t$ G0 F% O% H9 X% E1 C
<value>45</value>0 ~; M. u6 T$ M" M$ G
<description>Indicates the number of retries a client will make on socket timeout
! \! m- R. D- H" r p to establish a server connection.
4 i( r8 n1 d7 O2 E/ C8 u </description>( W4 h- ]/ h9 J' n+ T6 M
</property>
# a" H7 p' {1 U<property>$ c- i" i1 S5 w/ [
<name>ipc.client.tcpnodelay</name>
9 Z U- O5 l2 s# l" Z3 \7 J <value>true</value>% J# u, ^- I2 G2 p0 J* n
<description>Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays.3 j, g% M O ~
</description>) H* E8 o( z/ [' I- g5 V s& _
</property>
5 L1 G Z5 N, n- T' ~<property>
+ r A9 T3 S9 `$ D; L0 ^( ]& m8 f <name>ipc.client.low-latency</name>
0 q6 h6 [0 q0 e <value>false</value>* N# A* l6 _! H6 D5 z
<description>Use low-latency QoS markers for IPC connections.
' O1 X* I+ D/ [4 x; A4 G </description>6 s$ j! G# W8 M! F
</property>
2 b0 o, b {% j! r4 J5 a; @2 W6 d, i<property>6 W" p( O) M; \0 C9 n J
<name>ipc.client.ping</name>
2 m0 G$ U/ Z, H+ r <value>true</value>, b3 p9 B( u, O3 B: f- L
<description>Send a ping to the server when timeout on reading the response," K x2 w [2 o* I- d
if set to true. If no failure is detected, the client retries until at least
5 a- M1 l3 J! a5 i2 p a byte is read or the time given by ipc.client.rpc-timeout.ms is passed.+ S; f# k$ Z: V( `
</description>8 a+ F0 e) E7 M2 q0 I
</property>
. w e" u8 h) f) u# N1 O! j<property>
. Q( y+ I; M, L$ S& J. K8 m <name>ipc.ping.interval</name>
" D9 _+ U9 Z! Q6 H <value>60000</value>
) i( J d b5 E; ] <description>Timeout on waiting response from server, in milliseconds.
& ?% b+ ` |6 o8 Q The client will send ping when the interval is passed without receiving bytes,
/ a) F8 f8 g' V4 l if ipc.client.ping is set to true.( ?: [; Z" n# n7 t( I4 [
</description>
+ v; f1 u1 n- t" C; ~</property>
& g. h" Y/ o4 b2 w! F<property>8 V& q. W7 I- g- p( I; V n1 ?* N
<name>ipc.client.rpc-timeout.ms</name>, P5 K% S/ C( @5 G- E+ p
<value>0</value>6 @: m* ? J/ @, ^! ^* v
<description>Timeout on waiting response from server, in milliseconds., `9 X3 T3 p3 p' ~8 F7 v2 M
If ipc.client.ping is set to true and this rpc-timeout is greater than, a9 p& s# k: _) G
the value of ipc.ping.interval, the effective value of the rpc-timeout is: V" ^3 J; @ O5 r( T e( A
rounded up to multiple of ipc.ping.interval.9 J6 l2 p' I5 h) M/ E* {/ K6 G+ D5 a
</description>) }% L5 O7 [1 o
</property>
* h5 I0 u2 N2 G; T<property>
) h2 a( R7 Z, g3 ^( K4 I. D- N <name>ipc.server.listen.queue.size</name>
4 S5 L. A/ ?! X' g( ] <value>128</value>
+ m$ r: r9 j0 {& J8 D/ B <description>Indicates the length of the listen queue for servers accepting
4 }+ r( _ O% L$ Y/ w client connections./ ?$ J0 q/ p9 T# D6 k$ Q+ d
</description>* b! @8 I( M2 D3 m1 F% S
</property>% _. k9 V; p' X1 \) b
<property>8 \ F$ n8 k. S* E4 X
<name>ipc.server.log.slow.rpc</name>
9 \! h' L3 Y A7 |5 b <value>false</value>
0 |9 S. e$ U% F <description>This setting is useful to troubleshoot performance issues for
6 N2 e9 H) o0 w( }# m z6 A1 V+ A9 H various services. If this value is set to true then we log requests that% N. a* O% @( @& q+ b
fall into 99th percentile as well as increment RpcSlowCalls counter.
& U4 k$ K ]3 z9 O </description>
9 i+ O# A {0 Z8 K9 R4 J4 x</property>$ P$ b. u- @ J2 O9 q/ L, a* L
<property>+ S5 g! i! D. b
<name>ipc.maximum.data.length</name>" y$ Y" ?8 u1 W& M
<value>67108864</value>
# U; K5 R0 ]. l: J. B- M, P3 T <description>This indicates the maximum IPC message length (bytes) that can be# P# {+ x) B8 v# g% V; j
accepted by the server. Messages larger than this value are rejected by the
. _0 A4 K2 a. a* p immediately to avoid possible OOMs. This setting should rarely need to be3 m$ O* g$ O: ?) [
changed.
& j b( i8 M; }* }/ p) L </description> _) _( ?5 h; {! J( k5 L
</property>
( j8 w+ c( {* j$ c7 _8 M. a" g<property>6 S4 { [) `: L8 W/ ?" \4 x7 T; T& v9 D
<name>ipc.maximum.response.length</name>
% f" j K4 J& w _ <value>134217728</value># l5 c+ i$ d% ?6 I
<description>This indicates the maximum IPC message length (bytes) that can be
0 f7 j% S9 [. Q accepted by the client. Messages larger than this value are rejected
. P. C: Q1 n/ d) ~ N immediately to avoid possible OOMs. This setting should rarely need to be
. `7 X1 T" J1 m3 n5 |" @ changed. Set to 0 to disable.& t9 b: j: [- u6 b N) W
</description>
: R8 b6 h7 V( A# @7 d1 r</property>
F" d7 _# f4 ~( b9 l( h$ z, P<!-- Proxy Configuration -->- \. A9 d, I. E& }/ h2 Z5 x
<property>( C9 w3 P) e' \) Y# V
<name>hadoop.security.impersonation.provider.class</name>
! j7 z- J3 U; C4 Y$ ?5 A <value></value>
& S: Y- V5 y' K- r5 M+ W <description>A class which implements ImpersonationProvider interface, used to; M4 }, o# W; J U. n: r4 G
authorize whether one user can impersonate a specific user.
0 l6 x* a& n( `& N) x. U# r/ Y If not specified, the DefaultImpersonationProvider will be used.
! B6 L; h7 c) u' }. g If a class is specified, then that class will be used to determine: m( o7 S6 l3 C t* q0 t
the impersonation capability.! w4 Z0 g) E8 d# S5 Z+ E0 K
</description>
X8 Z4 k+ [) h* m5 v3 f, D</property>1 v- |) \; }: N
<property>0 ]7 E7 u9 j$ P; o3 U
<name>hadoop.rpc.socket.factory.class.default</name>) \ e& |$ U! X+ C
<value>org.apache.hadoop.net.StandardSocketFactory</value>, T4 g% K, E4 w7 B$ ^
<description> Default SocketFactory to use. This parameter is expected to be
- Q6 }& v0 |$ ^, s4 S! ~# X, t formatted as "package.FactoryClassName".' |3 W& X/ g7 i- E; ?
</description>
, f M5 `2 m, L" |</property>; X% E# \9 N1 c* @% K. Q C8 M4 n
<property>
7 e% R' D4 v1 X, i& O4 z& _6 O5 O0 j <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>
' T, J4 a8 A# M; v% P <value></value>
! O; p+ f& l( O) u' N! x <description> SocketFactory to use to connect to a DFS. If null or empty, use U( d+ i j, C1 \9 e
hadoop.rpc.socket.class.default. This socket factory is also used by
% R$ l, Y: a1 u, Y DFSClient to create sockets to DataNodes.
4 N+ d- W# ?9 F+ c# K; o A p, S </description>3 t6 x" Y( T! D# D" p
</property># `( v: `9 h3 |
<property>
* T+ D k F$ Q7 k" h. Q1 a <name>hadoop.socks.server</name>$ ?+ x+ |1 `( n0 w" x
<value></value>
6 z/ A7 a- D/ [ <description> Address (host:port) of the SOCKS server to be used by the
4 C. ]$ X+ ^( v. Q& n n SocksSocketFactory.+ Z8 |, \- P4 ~' |9 K
</description>7 [( m6 R) p+ L. ^9 C
</property> \8 q0 g! w& Z1 {% C: z5 D* U% b }
<!-- Topology Configuration -->9 M3 c1 v0 Y( s& `8 X
<property>
1 p1 f- t: C/ y8 {$ j; M" E <name>net.topology.node.switch.mapping.impl</name>
* y: ~9 K! g9 T* g9 W: D <value>org.apache.hadoop.net.ScriptBasedMapping</value>$ o* D. E4 \4 t( J' f# M6 k P
<description> The default implementation of the DNSToSwitchMapping. It
/ @$ I! e3 Q/ f, j( G1 J! x invokes a script specified in net.topology.script.file.name to resolve
1 s# q5 e% c6 t1 \, `/ g3 @ node names. If the value for net.topology.script.file.name is not set, the% `; m+ {' S( m: N& D8 o. {1 k
default value of DEFAULT_RACK is returned for all node names.
( V2 H# b% S# Q3 [ </description>
: p f& D8 D# B6 @ X! V" c</property>1 o; s& o4 h' P
<property>
$ a9 U" q* H5 { <name>net.topology.impl</name>
7 _) _3 r- _% b$ P {' Z. c% C <value>org.apache.hadoop.net.NetworkTopology</value>
, b# s5 e) v+ r* t7 j) J <description> The default implementation of NetworkTopology which is classic three layer one.7 S0 f0 E0 K' G0 z& H$ {8 V
</description>! \' N+ R g' _6 ?1 d: a4 }/ ]
</property>
: r) w+ `: W6 o6 `9 u3 ?<property>
* Q0 _0 N8 S' |. o9 b' O& R$ x <name>net.topology.script.file.name</name>
' I: _1 Z6 n0 k o <value></value>1 W7 K! [9 Q/ b# m* J
<description> The script name that should be invoked to resolve DNS names to
& {; V: u$ Q0 P$ L! | NetworkTopology names. Example: the script would take host.foo.bar as an
( z0 V5 I, T3 t) C9 e argument, and return /rack1 as the output.
) K3 _8 n* \9 ^. B- y. U+ H- T" A </description>8 ^2 q" H% {5 a/ B# ~$ u
</property>
) H2 h5 L- R" B5 b<property>, ^% q" [) }! r- [, X: s6 B
<name>net.topology.script.number.args</name>5 {( k( w2 I4 |8 A
<value>100</value>
9 `- q7 X6 W5 T5 C; w/ V1 e4 r6 l <description> The max number of args that the script configured with" C) z2 h; m4 p% B0 @! o" {
net.topology.script.file.name should be run with. Each arg is an+ t% Q: I$ z5 s( p" `: p& c; K
IP address.
" s1 k- U; g; ^# [ </description>
8 z% P+ v A" D</property>5 M Z8 k0 Y' [6 y: P
<property>
! ~2 V9 X2 b! h <name>net.topology.table.file.name</name>9 [; g) ]: m* B4 D9 t
<value></value>, n) E" Q6 L) F' t; f+ R
<description> The file name for a topology file, which is used when the/ v X+ U( n, S* y
net.topology.node.switch.mapping.impl property is set to
( [- k6 P2 K3 [ org.apache.hadoop.net.TableMapping. The file format is a two column text2 o# e! x j' `( n' D: `/ u
file, with columns separated by whitespace. The first column is a DNS or
% K# M! m/ g! Q$ W IP address and the second column specifies the rack where the address maps.. }6 F0 h- H$ |8 {) j
If no entry corresponding to a host in the cluster is found, then# l% w4 @+ M2 W# A
/default-rack is assumed.9 u6 [: v0 p+ ]4 X# z& h
</description># m$ R4 ? w4 p, Q h
</property>8 A! V& d2 M+ ` \* D% U
<!-- Local file system --> k4 O1 M+ C3 L- l: J4 m6 [
<property>3 R4 E4 b& ]/ ?0 T/ R- d
<name>file.stream-buffer-size</name>
3 f$ U9 o$ Q/ \4 T2 K <value>4096</value>; G; h+ k1 Q0 c# w, e7 _2 B
<description>The size of buffer to stream files.
* C( n4 U1 [* W, T: m/ W The size of this buffer should probably be a multiple of hardware
( n7 E% k6 K/ Q/ U; ` page size (4096 on Intel x86), and it determines how much data is
* h( V, p, L6 N buffered during read and write operations.</description> p" Y1 D( }* x2 m$ \7 w
</property>
1 R' U) Q) s. x6 i" |<property>! x$ f, E% e, {
<name>file.bytes-per-checksum</name>: ~3 [4 z0 d( h( T# ?, {
<value>512</value>; ]; s7 v( y! r. Z1 C+ } ^1 U
<description>The number of bytes per checksum. Must not be larger than, B2 L; O; L" n. D. ^
file.stream-buffer-size</description>
) N3 f' e' i! M. ^</property>
& d/ L) n& b) e* y7 P$ ~( N( u<property>
" A1 U G( d9 O- s- h% S <name>file.client-write-packet-size</name>4 C9 |! w: U8 R1 w
<value>65536</value>' K) T' R# a! M4 M6 w2 f2 X# x C
<description>Packet size for clients to write</description>9 W" g8 U2 _/ E0 x d0 F f
</property>
) D3 F% F8 F' Q$ H5 t6 n. }<property>
9 p; |) X: u+ f' J; q <name>file.blocksize</name>
5 Q0 _+ B! q- w1 Z+ p <value>67108864</value>
+ h: j( X, K' V; z- Y% a% v4 x <description>Block size</description>* x/ ^8 q, T7 B$ H* o8 {9 Q5 V
</property>
# D, A# l5 A; i5 j% h<property>
9 Z- H K. G. {2 P$ Z <name>file.replication</name> B0 G3 V1 r# \6 z9 X/ ~
<value>1</value>
3 {+ K4 S+ G } <description>Replication factor</description>. k6 j0 F6 |0 C8 M9 K; S
</property>
/ ?3 ^3 |* n1 Z7 z- t<!-- FTP file system -->
8 a; t" O* W' T$ F& U<property>' {7 h# I, x5 I$ S# q( c# }' J' Q
<name>ftp.stream-buffer-size</name>
( v6 z5 i$ P- K, \ <value>4096</value>8 y$ U' ^ e% f0 `
<description>The size of buffer to stream files.
7 U& l( B ?: B0 C The size of this buffer should probably be a multiple of hardware# u# T0 T G# E: ?! q
page size (4096 on Intel x86), and it determines how much data is
) j( T& s2 v3 }9 n& q4 s buffered during read and write operations.</description>" R3 I$ J2 ?2 i4 K m# i; f
</property>$ J/ Z$ `0 ^# \$ x( M
<property>
9 z/ a- D# L/ o, y <name>ftp.bytes-per-checksum</name>3 B$ R J3 Q' h M4 F
<value>512</value>
$ l. F( G, d( u3 B- L6 ?% l0 { <description>The number of bytes per checksum. Must not be larger than# \% a0 b0 t: ^8 \; ~$ D
ftp.stream-buffer-size</description>8 A2 r0 g, R- T7 w
</property>
9 f9 ?4 L! `$ e6 l3 m7 R3 F<property>
* N+ g# f" E. @8 U: D0 u+ K9 B <name>ftp.client-write-packet-size</name>
1 e1 S! u8 v! l [/ |0 t* S <value>65536</value>6 H6 P, A# i9 a6 ^: H$ S
<description>Packet size for clients to write</description>
/ d( s1 D& f* l" M) K2 _$ A! \</property>
! E) s% Q6 B1 p. K9 h. _4 {<property>
- Y4 `# U5 V1 ^ l. o <name>ftp.blocksize</name>
0 ^; ]- y1 d" |% g' H- { V <value>67108864</value>
. _" F5 f) S0 `( l <description>Block size</description>
& g" K$ P: @! D</property>
6 R. J& j8 y& R* X<property>
0 }8 z% Y9 O2 y8 I9 e' K3 a- Z <name>ftp.replication</name>5 X& v' j+ ]5 h8 W) _8 \8 U
<value>3</value>" i. B! t& K0 f b
<description>Replication factor</description>/ D8 e$ f; \ N8 F3 `; ?5 d0 m
</property>
|* q7 \4 L2 y* |9 ~3 f# l) m9 e<!-- Tfile -->. A2 f0 o: R5 x
<property>. R) t$ a; y% ^8 b! a& Z8 r" h
<name>tfile.io.chunk.size</name>7 ]. J! j2 u, p8 L# g
<value>1048576</value>, C K8 L* s1 q) _1 H9 r
<description>
" N# p* e- l7 ~& Z) i& K: W Value chunk size in bytes. Default to- J [0 F- W3 v$ p [$ P1 T
1MB. Values of the length less than the chunk size is
; X& W1 C" i+ v0 v& H" B1 u guaranteed to have known value length in read time (See also1 H: [+ X0 q* m4 [4 C( M8 c' S
TFile.Reader.Scanner.Entry.isValueLengthKnown()).8 L% m7 q" o7 N7 R0 b7 S
</description>: e$ P( m9 e& f. j
</property>
4 v" I+ D/ @& A0 ^3 a<property>% H1 U7 c; n1 O+ P
<name>tfile.fs.output.buffer.size</name>9 G* V, P- k+ j1 U# E
<value>262144</value>8 U3 B8 d% N4 E" z2 f
<description>
3 p8 s! W2 z. @" U r Buffer size used for FSDataOutputStream in bytes.6 i- Q/ k& H/ O1 R1 C9 }1 w% [
</description>
0 c4 C; p4 y' s' F0 S* |5 U</property>4 X5 |# x% v" [
<property>
! R8 |& A! x# x5 Q: S: D* d/ [ <name>tfile.fs.input.buffer.size</name>4 R& B* H# t( `' j" T
<value>262144</value>3 u/ v& o9 ^; e1 A/ ^% T X
<description>* j4 x( t1 B% b6 d7 ]
Buffer size used for FSDataInputStream in bytes.' k6 J" X, j* e% ?1 V
</description>
i8 r m5 U! z, y o/ y1 t</property>$ c% \: w/ V6 j& `/ @' ^/ L9 z4 p& v
<!-- HTTP web-consoles Authentication -->3 h2 r% e6 b6 c M7 ]; s
<property>
9 f# t8 Y% E9 s* @! K <name>hadoop.http.authentication.type</name>
6 x) K8 Y4 R1 N( W& S2 T7 R <value>simple</value>' c' ]5 a% R$ K' \& k5 e
<description>
6 ?0 @/ U3 S3 }4 C6 w; _- w- \ Defines authentication used for Oozie HTTP endpoint. ]5 u0 q' q9 `3 d& G) p
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME## s3 ~1 ?# F% @. z% {
</description>
, }% X+ d* Q4 P% a5 X3 B9 Y</property>& k1 B0 _0 x6 r/ }9 u9 k0 i
<property>
" P: h! F& d/ h8 } g! M) `$ ?4 V <name>hadoop.http.authentication.token.validity</name>
6 q9 l% h' r! `) V( I$ E; ]9 n <value>36000</value>
4 u5 A0 p) q, M# [" F9 ~ <description>
9 s9 w' c& B o! T) W Indicates how long (in seconds) an authentication token is valid before it has
4 v+ N) s. x% [' {" s- P* d# \ to be renewed.3 ~" H3 J7 }/ R/ q2 e$ W, I
</description>
9 H/ f! q" }: t! x3 O</property>6 Y5 G, Y' l4 A7 C& a9 e; ]- \
<property>
2 \/ h% {( T3 B: F5 `7 N <name>hadoop.http.authentication.signature.secret.file</name># X3 t4 E W3 X& ?, L
<value>${user.home}/hadoop-http-auth-signature-secret</value>, h/ m* W1 f; `8 J8 ?1 c4 l: F
<description>
; l, Y2 t! p, C% P! `, A; r9 I The signature secret for signing the authentication tokens.) f/ |$ `+ A. a9 o* v7 u+ [4 `
The same secret should be used for JT/NN/DN/TT configurations.& j5 w& e# \) Q' Q* w! a$ D* [
</description>
; J& l" A- A) f+ ?: S</property>
# Z4 S8 c9 N! @: Y) K<property>. i! ^: m( j1 G9 {4 ~3 @
<name>hadoop.http.authentication.cookie.domain</name>+ ?6 i6 N- R2 q
<value></value>
% X; Y% f9 X, X; V4 X4 s9 o( u$ }' ` <description>) y9 I* u" Y4 @" f3 M! K- }
The domain to use for the HTTP cookie that stores the authentication token.- S1 T* Q& c: h& {3 K" ^8 f# \
In order to authentiation to work correctly across all Hadoop nodes web-consoles: L. s5 K, L1 J
the domain must be correctly set.! E+ g3 L" C2 s8 S$ U: B$ k* P
IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings.
) c3 Q" b# e+ _: `4 u6 X& B1 K For this setting to work properly all nodes in the cluster must be configured7 t* f3 I3 `' p% E
to generate URLs with hostname.domain names on it.' Q3 I1 l) B E4 s0 C
</description>7 X3 Q& w7 c5 a7 v* n: }* j" r
</property>
3 C* b7 Y# X% ?! b/ b' F/ T<property>
& r1 k6 m5 R! K <name>hadoop.http.authentication.simple.anonymous.allowed</name>7 K) x" u3 i2 i4 A( q
<value>true</value>* z6 R1 l6 x7 S
<description>
3 Z( i; z4 D: b2 @: m# g Indicates if anonymous requests are allowed when using 'simple' authentication.3 q/ J- j, ~/ y3 x" s- l
</description>
3 v% Z; B! j( T8 K4 \</property>
5 n3 p2 k* a o3 N _& h- {4 X<property>9 p- N% _1 v% z2 g2 ?) A
<name>hadoop.http.authentication.kerberos.principal</name>
; A/ q5 M0 |6 i% D2 g <value>HTTP/_HOST@LOCALHOST</value>
- m* j- ~( _# y. y3 o' o, P6 V <description>
# x, m8 G! v0 I% ` Indicates the Kerberos principal to be used for HTTP endpoint./ ^0 ?. p7 D" L
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.0 ]0 i6 o, R6 u' j$ V U
</description>' n8 w) d" E& d9 E: A- m$ T1 V
</property>
: i8 i& f) G/ F( }. V<property>
5 `2 o) p4 {, C& {! N <name>hadoop.http.authentication.kerberos.keytab</name>
- W3 ~# s6 C+ I+ I <value>${user.home}/hadoop.keytab</value>& O2 P2 L0 H1 s4 e2 L1 }9 v, Q% @4 M
<description>7 m& l! E/ x6 e1 e2 z
Location of the keytab file with the credentials for the principal., @1 f& T7 h- N) Y9 g- A1 r
Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
+ [8 A+ ]/ k ~: N F4 a4 S# E( [ </description>
( q& J- X8 Y8 C</property>1 c2 ?! R# }& ]) j% k
<!-- HTTP CORS support -->1 J0 H9 K8 b5 G F7 m9 y
<property>/ c; ]. S3 x, `0 D8 S K9 [3 m% k
<name>hadoop.http.cross-origin.enabled</name>
0 }, p3 ]% a D5 r- J q) M <value>false</value>, M3 g5 q$ Q9 B6 q
<description>Enable/disable the cross-origin (CORS) filter.</description>
( A% w5 `9 ]% S- T. z4 K</property>
3 Z( \. U. P6 s<property>, {8 D; K6 n# q. D3 d9 G
<name>hadoop.http.cross-origin.allowed-origins</name>
! p0 S) Y9 `: i* m$ S <value>*</value>
9 Z7 u, j) l- R& L: }% }6 C <description>Comma separated list of origins that are allowed for web services! B8 l: T& P7 }6 {* c
needing cross-origin (CORS) support. If a value in the list contains an
1 L8 v4 B- j2 {5 B asterix (*), a regex pattern, escaping any dots ('.' -> '\.') and replacing9 r& D2 w/ T' Y0 U/ H5 ~+ {1 J6 k
the asterix such that it captures any characters ('*' -> '.*'), is generated.
$ E2 R; k r* l Values prefixed with 'regex:' are interpreted directly as regular expressions,8 P% I, e3 m4 _
e.g. use the expression 'regex:https?:\/\/foo\.bar:([0-9]+)?' to allow any0 a% j2 v% K; D2 F, e
origin using the 'http' or 'https' protocol in the domain 'foo.bar' on any K I9 d" a$ |0 U' J2 x
port. The use of simple wildcards ('*') is discouraged, and only available for9 r7 A0 `3 k$ S/ a
backward compatibility.</description>
7 Z( H3 }2 a9 p' g</property>
, m# c" ]# l: [: B7 A" F<property>$ k3 T7 ?! P. f
<name>hadoop.http.cross-origin.allowed-methods</name>- v( U+ K s/ A' J
<value>GET,POST,HEAD</value>- X( v1 \2 Q. A& z8 R) ]
<description>Comma separated list of methods that are allowed for web( W9 V: ]9 c8 N- H) O
services needing cross-origin (CORS) support.</description>: _- h9 b% y+ M8 d7 a0 Y' k0 c
</property>2 B7 n" {; z" b( _
<property> R1 y& [4 l. N9 X; i0 e/ o5 u
<name>hadoop.http.cross-origin.allowed-headers</name>; l& |' i" g* L7 b% f2 _* e% @2 s8 ]
<value>X-Requested-With,Content-Type,Accept,Origin</value>
2 X+ |# J9 q# \( |! k( T& P& X; Y5 U <description>Comma separated list of headers that are allowed for web% y ?1 l- N: w2 `/ g
services needing cross-origin (CORS) support.</description>
& Q/ F) F. K: t, s</property>( @' [2 r, L* C* a
<property>
! \4 F/ S8 y! q/ A' A$ M9 G <name>hadoop.http.cross-origin.max-age</name>- X, d8 g/ W$ E9 C* P$ x
<value>1800</value>
8 e1 H* I7 M* L Y O+ l <description>The number of seconds a pre-flighted request can be cached2 S" r- C( j0 y1 v
for web services needing cross-origin (CORS) support.</description>6 T9 a! T+ ~- t1 d1 q, h
</property>0 O% _; L5 `5 W; A" W; b
<property>" t$ ?) C+ p, |& A. {) r" i
<name>dfs.ha.fencing.methods</name>! v9 w7 E5 ?9 J! {# ?- V d0 d
<value></value>! M. O+ ?/ I# n* C ~
<description>! x9 Q: Y3 J5 u) |+ W* }3 P+ H
List of fencing methods to use for service fencing. May contain
1 a9 z0 ]' a" F9 z9 D! ` builtin methods (eg shell and sshfence) or user-defined method.
5 h& q# P! C7 u+ y# | </description>0 y( }) k% W; O C( u/ C! D
</property>
2 i* Y U# l9 @( w# p! _<property>2 ?5 I- A* z, l0 F( O ~/ z
<name>dfs.ha.fencing.ssh.connect-timeout</name>
$ F5 x* B% T8 \& {" I/ s+ v" ~ <value>30000</value>
2 C' p& T9 |- g" Q7 f <description>- |1 A. I$ R6 z
SSH connection timeout, in milliseconds, to use with the builtin
9 s* G' ], ?! W# }( B sshfence fencer.
8 X* r5 T5 o3 ~6 q. o! `' @6 A </description>
0 f: `, g+ o7 `</property>1 q; R& u) r$ p- h- Z! W, O
<property> b" F& b8 |/ N' J9 e& f
<name>dfs.ha.fencing.ssh.private-key-files</name>
9 `5 z& h. `5 X/ j9 N <value></value>
! E" d3 V! v) [ <description>
0 l) K6 v( O; y1 ~9 a" F The SSH private key files to use with the builtin sshfence fencer.9 @! @6 q7 y# c( ?8 l
</description>6 K* D7 u, u6 U+ ?: P9 { y1 [
</property>1 ]+ C0 W& L" C/ o
<property>
1 j# o$ w o4 e <name>ha.zookeeper.quorum</name>
" ~) E3 v F( N <description>
1 {$ Z6 u5 }8 Z8 J A list of ZooKeeper server addresses, separated by commas, that are
- b7 A+ d, o& ?) A/ P to be used by the ZKFailoverController in automatic failover.! n/ R! n% W0 r* \+ Y+ P3 C* h+ ]
</description>1 t, Y% r/ ]& ?4 \* m9 p
</property>" g. x0 n+ u8 A" ?, N8 R
<property>
1 z4 U5 }6 x+ I1 q4 x% K) A <name>ha.zookeeper.session-timeout.ms</name>/ z" N" |# K l! }* {! d
<value>10000</value>
+ O' _- B+ D# ?: { w7 z) ] <description>( l( p% U( M& r2 H; X8 X- t
The session timeout to use when the ZKFC connects to ZooKeeper.+ F7 {: L- f" m$ j! x& H& g
Setting this value to a lower value implies that server crashes3 G I! T/ r3 F/ ~# S, W; ]
will be detected more quickly, but risks triggering failover too
) e; f/ J5 p- T" O/ n aggressively in the case of a transient error or network blip.1 a8 y0 o( f% D
</description>! K0 z* k& |+ W7 @2 Q' I
</property>
6 ~. Z7 q& r/ d& ~# @<property>
6 A0 K, [3 v; d7 w2 x! H: @ <name>ha.zookeeper.parent-znode</name>6 m& @. Y4 M6 c4 b) ^; ~) N' U
<value>/hadoop-ha</value>
: j% P5 R& P" f# a <description>" |9 K- m8 b4 @% r
The ZooKeeper znode under which the ZK failover controller stores+ q6 y& ^1 \0 W; }1 @' w3 `
its information. Note that the nameservice ID is automatically
2 g- m% d9 `8 K- `8 C- U+ S0 O+ { appended to this znode, so it is not normally necessary to8 b8 y; i/ B% z$ M
configure this, even in a federated environment.
6 S4 `3 j: J- u5 v# C2 v, i* U </description>3 G; P. p& p5 K( A* Z: T
</property>8 J( x X( Z0 {- s" i
<property>
6 P% Y- ^$ M+ E5 n# } <name>ha.zookeeper.acl</name>
; m- C8 A- _- B! t2 I- \$ W <value>world:anyone:rwcda</value>' Z+ w' O/ ]: D2 I
<description># O4 r b" [9 [; x _' A6 n3 T: E0 s
A comma-separated list of ZooKeeper ACLs to apply to the znodes
8 i$ ^" M% |1 k1 R; {& b used by automatic failover. These ACLs are specified in the same
9 U4 N9 P( _. d- ~. {7 ]8 [, A" | format as used by the ZooKeeper CLI.. B5 {7 N; g4 E
If the ACL itself contains secrets, you may instead specify a
4 J, a" w2 Z! L; e+ x, p3 O path to a file, prefixed with the '@' symbol, and the value of" H: J0 m3 S) v/ o7 z
this configuration will be loaded from within.
g" G# [/ `, {& p% a' i </description>
: m3 [$ W/ z) o6 F, r8 y( M</property>% z! k) s+ W2 i7 c! S) N4 J; I% P
<property>
, t4 A8 d: A- {* g% P& T/ F) ^' K* Y <name>ha.zookeeper.auth</name>
: V" E* a. d+ R' ?' W& e/ u6 F <value></value>
& s# p# J2 [8 j3 m0 h E <description>1 z2 n1 D4 J0 k
A comma-separated list of ZooKeeper authentications to add when* l5 e/ Q% Y7 K4 n
connecting to ZooKeeper. These are specified in the same format
4 t; `, z7 H/ ~" | as used by the "addauth" command in the ZK CLI. It is
( N. [& E0 ?! M/ K: \0 R+ r0 ]" {- \ important that the authentications specified here are sufficient
% G# M5 O6 [% i# |9 A to access znodes with the ACL specified in ha.zookeeper.acl.& m* T8 _. A6 S. _4 m! ~( M \" z. C
If the auths contain secrets, you may instead specify a
3 S. k2 `/ A% d7 d( R, k path to a file, prefixed with the '@' symbol, and the value of
' c- G& t, B8 O. g8 D5 e this configuration will be loaded from within.
# G3 H0 L& ?% Y" j: _ </description>! Z; p1 i1 {; g8 p
</property>- V- s- ?; {- O
<!-- Static Web User Filter properties. -->
$ P% Y! @: z5 m( N, ~<property># g/ Q0 i, E2 ^, g J$ ~( x
<name>hadoop.http.staticuser.user</name>
8 |7 B- K+ k# T* }& _& W& i <value>dr.who</value>
K ^# O5 J% l/ J Q <description># i }" [, Y9 i$ y3 `- [0 W
The user name to filter as, on static web filters
* G5 Q `5 m5 N1 K while rendering content. An example use is the HDFS
/ E' U( Q1 P- U- Q' i- O: G, O0 L web UI (user to be used for browsing files).9 q. |5 U" m3 q+ x" L8 y
</description>/ l2 e3 A8 a. Q# A3 f& C9 a( d
</property>5 E$ P6 i- d- X1 f) \& R
<!-- SSLFactory configuration --># G- U u( B. j3 p2 s
<property>
* Z' w, ^/ j5 Z9 W$ ~/ H <name>hadoop.ssl.keystores.factory.class</name>& Y2 u. M1 f1 E) L% z
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>* }9 b8 b+ Q1 d
<description>
, k4 E+ A3 H" L$ p0 J The keystores factory to use for retrieving certificates.# b L. `1 K! S8 X+ o
</description>
4 f0 j; E/ o6 V! e0 T</property>" H+ b+ C, h& O2 M: @/ K% ]* e6 {
<property>1 U. n/ p# [5 X( I( @$ x
<name>hadoop.ssl.require.client.cert</name>' u4 {/ c0 I) k) N' y; T
<value>false</value>9 f1 A1 x4 m( r H! X0 M
<description>Whether client certificates are required</description>5 S& @! H: p$ N0 ~/ ]
</property>
8 d+ m- f) f6 l. u3 G3 A<property>0 N S& U+ [/ v1 F* P3 f
<name>hadoop.ssl.hostname.verifier</name>
1 O: S q8 f1 }0 Y! J <value>DEFAULT</value>
3 x' x1 b, [1 y1 ]( F$ X <description>; n$ `' C( ~4 Q
The hostname verifier to provide for HttpsURLConnections., E4 x" M* f' m5 y8 O
Valid values are: DEFAULT, STRICT, STRICT_IE6, DEFAULT_AND_LOCALHOST and" w( M1 n& E/ m, B
ALLOW_ALL
/ S B3 }1 B$ { </description>9 _7 Y0 X: K4 e( U. S
</property>. ]% H5 z! f/ |1 X
<property>9 z" x1 }& v% q1 x7 F5 t7 D" e7 Q
<name>hadoop.ssl.server.conf</name>' D. e2 Z! ]% w: q% J( z
<value>ssl-server.xml</value>. h' P, W6 E9 n% G5 y" N( m
<description>2 f. q) i( N; }, }( d
Resource file from which ssl server keystore information will be extracted.
0 |% z& ]# U+ c3 {9 Z& j This file is looked up in the classpath, typically it should be in Hadoop3 j5 ^4 O# @# M. x$ h
conf/ directory.# T6 O* `7 E8 c% [6 ]+ F' ~
</description>
, @; `6 U% T5 m* K5 H</property>/ ]/ }3 z6 N6 P+ P" Y7 R2 |
<property>
$ x( s4 `' ~$ R- T' ` <name>hadoop.ssl.client.conf</name>
5 g, T( q" r* }3 A3 Q <value>ssl-client.xml</value>' a% R3 Z! Y' o' a
<description># }' ^! T9 v8 e5 C- `# W& u
Resource file from which ssl client keystore information will be extracted" c5 i$ a$ S: B) v
This file is looked up in the classpath, typically it should be in Hadoop
3 p% D* o! U$ P% ]0 ^( Z: z conf/ directory.
1 P9 A9 k5 Z) N* r </description>$ }: Q8 ^8 k# K6 F2 M/ n- I; W0 F
</property>
j; y" ?, _5 ]3 M; y7 _' z<property>+ W9 j; z9 e- n: f8 i# V
<name>hadoop.ssl.enabled</name>
1 a. y4 Y3 j Y7 h <value>false</value>
% H3 x. R" ?5 `* q, W5 A; f <description>
" ^ k8 ^, m2 G Deprecated. Use dfs.http.policy and yarn.http.policy instead.4 Q3 e4 {) a- Y8 A% M
</description>% D- y2 y. f8 q# W* q, j+ g5 m
</property>+ N$ w) @: L* |, Q6 k' Y! ?
<property>3 x) ^, |/ Q4 l% ]$ L- }1 z5 Y
<name>hadoop.ssl.enabled.protocols</name>/ d5 g, P( F) [
<value>TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2</value>% V3 C. v6 R% N# {" s% M3 y
<description>9 _! `" L9 u% }/ Z( N6 a$ P/ y8 v9 X, m
The supported SSL protocols.4 D: W1 e { s+ K# g# J
</description>
+ U/ q9 J8 m% I8 N: m S</property># v8 z. B- t( ^1 G
<property>6 Q- D% B! `4 F% d% V( l- \
<name>hadoop.jetty.logs.serve.aliases</name>
6 G2 I t8 N# T4 M7 k <value>true</value># H( V" @; P& i# ~9 o
<description>* i" X( b p8 o
Enable/Disable aliases serving from jetty7 f' ` Z3 u+ b
</description>4 D9 A2 N2 l" _2 j8 ~4 A
</property>
/ g, j7 n$ a6 q0 J# Z5 x% S<property>
) ?5 d5 T2 f# ^) }2 A/ r! s4 w <name>fs.permissions.umask-mode</name>' R0 o$ H: U X
<value>022</value>/ ~3 W. b% S9 ~5 R/ ?/ z/ w8 v
<description>
* v- `' A U8 G1 \) e, L" U The umask used when creating files and directories.
$ p" h1 O! l% B7 P) J! D- | Can be in octal or in symbolic. Examples are:
T! i v/ ?% y4 j. O "022" (octal for u=rwx,g=r-x,o=r-x in symbolic),
& f+ c, m G% \( O& Y A- r or "u=rwx,g=rwx,o=" (symbolic for 007 in octal).: c; k. r" Y$ ^* g3 w0 M
</description>
& [; C H' f" ^( g! ^7 b</property> E9 P& _3 i1 \8 e1 R4 q
<!-- ha properties -->( F' n% w8 p% u$ N4 z
<property>
- {% J; z6 f5 P$ b <name>ha.health-monitor.connect-retry-interval.ms</name>- K+ i% F+ b" O/ U7 w6 Y
<value>1000</value>
0 L( p3 m- @, p <description>
. F9 w1 s- X, V6 C9 M7 P L How often to retry connecting to the service.
- U/ r V! Y& B& I, r+ c$ _/ e2 [( @ </description>/ _2 g4 e0 n- T; Z6 R' f
</property>
( k. W( [' z% ?5 y3 M- G d6 `<property>% E+ p$ {% m7 T" V3 u
<name>ha.health-monitor.check-interval.ms</name>
: `2 O1 [9 A. {5 U4 T <value>1000</value>
5 a/ d- K; `. J6 K; R& ~ <description>
6 b. u( ], J1 e8 | How often to check the service., V, t7 ^+ w# V
</description>
' u4 T. _2 I* @" X1 u, }</property>: b% G3 p5 v2 O2 Z9 W$ s
<property>
8 T" \# A7 i( V# r$ H% ~9 w <name>ha.health-monitor.sleep-after-disconnect.ms</name>9 J% ^! y0 Z6 a2 C
<value>1000</value>& e5 J6 e P( K
<description>
: ^% E* K3 b- D9 @ How long to sleep after an unexpected RPC error.. C! `0 J8 n9 W$ s2 v+ i
</description>
* [% m& v! d4 H$ c2 [</property>
( B4 i) z, A+ f. U$ Y1 e, I<property>
' K2 z# T. S* }# S% [, ] <name>ha.health-monitor.rpc-timeout.ms</name>. g! b/ m9 ]0 N6 _) _
<value>45000</value>
3 T0 c; N, J: I- h; j% d <description>& Y% N, { I% R/ C- H& f" [+ t1 W/ V
Timeout for the actual monitorHealth() calls.
4 d+ h) P q/ i5 m </description>
" X; O V' _& u4 X h+ r</property>) N7 ~7 T. Z2 D# w3 M" O
<property>: O& h: ~6 \# J! G' r: m5 x
<name>ha.failover-controller.new-active.rpc-timeout.ms</name>
8 Q* N k' p8 e9 Q) i I+ M <value>60000</value>6 Q" p+ y. i$ D+ B* d7 |, I( f8 @
<description>
. n7 W0 G, a1 E; x( n7 ` Timeout that the FC waits for the new active to become active; U( ?; Y' o2 ]
</description>: B6 s( u% Y+ |' v- Z
</property>
. t& F+ a" u: u- G<property>
5 B( _: M3 Z( P/ S) w3 ^) b0 S$ U) i6 r <name>ha.failover-controller.graceful-fence.rpc-timeout.ms</name>0 q" |" x) i6 x/ y* Y
<value>5000</value>
- l- R$ ?4 U3 y! J0 R* Z0 B <description>
. U+ s* m, m: w2 M* f- \4 | Timeout that the FC waits for the old active to go to standby4 Y# F9 }7 m- p3 f- l; k2 [7 R# Z
</description>" H) l5 I8 w- D$ N0 h. v
</property>
/ _& w' Y0 W! C' n8 x! s, J! g<property>
- Q7 w: j# W$ T0 S+ x% I" p: j <name>ha.failover-controller.graceful-fence.connection.retries</name>
* ^; B# h- S) ]% t <value>1</value>
' ^5 f7 O8 T3 j7 e5 S: w, O# g$ _ <description>" w2 a0 W$ h/ B8 [
FC connection retries for graceful fencing
, X& ^0 _9 {5 ]/ O1 X0 O& `; D3 c </description>2 z- v0 g6 {: k7 S) k# o6 d1 a ~
</property>
" R/ \ p6 |/ ~; H, q# {* F2 b<property>
( @/ p7 r U B <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>
# P- c3 v! ]( \. T1 g <value>20000</value>
1 |, Q) Y2 C$ u <description>7 L# `0 t1 k0 b2 _6 O; i
Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState
8 f9 B3 Z" R9 b% }+ T </description>$ R7 M4 z7 ]4 z- {# t& \, W
</property>
8 X& M# }* i5 [3 t* h5 v, z8 a<property>- H7 W# G- e L3 N* a; ]
<name>ipc.client.fallback-to-simple-auth-allowed</name>
5 P. f; H) l. y! z( C <value>false</value>
! X8 t( b m) V3 p' T5 q$ X <description>
4 Y1 u* o* e% b5 m9 G$ u% N, h! I When a client is configured to attempt a secure connection, but attempts to
9 Q1 ]! C' v1 F- q7 v connect to an insecure server, that server may instruct the client to# p5 Q1 k |0 ~( a8 @
switch to SASL SIMPLE (unsecure) authentication. This setting controls
$ O0 _" w0 r% g3 Y' o" A whether or not the client will accept this instruction from the server.6 [! C J Q0 x
When false (the default), the client will not allow the fallback to SIMPLE( y7 k6 u& a- N
authentication, and will abort the connection. d0 z0 C; N) H- }' w, U- p
</description>
8 N" _5 G. @5 I& u</property>: h3 Q9 l! S7 X) R! S, n- z
<property>
6 R6 m$ x! H: u& U& b* d <name>fs.client.resolve.remote.symlinks</name>4 J) v6 C: l, y# ^% y* W( |
<value>true</value>
: u3 j- @3 n8 }% L; b& b <description>
/ ]5 M# s' g2 ]1 O% C: V8 H Whether to resolve symlinks when accessing a remote Hadoop filesystem.% n* i9 f* V6 d/ {: Z
Setting this to false causes an exception to be thrown upon encountering
, _9 p; w6 _4 h1 X9 i1 k5 g2 J a symlink. This setting does not apply to local filesystems, which
9 b3 J* d1 B: h, w automatically resolve local symlinks.- R8 I. Z% N$ n8 F: m6 J1 V
</description>6 Q( G1 o' F# d: L4 v+ i1 j X6 w
</property>; n! U/ A5 |" G! Y* r: q. a9 e4 b3 R
<property>
. w8 r/ q" a4 q4 ]* O <name>nfs.exports.allowed.hosts</name>
$ n; X4 D3 g: a! N <value>* rw</value>/ ^2 e$ S* @; ?/ F9 V% \: P
<description>
# i3 C7 U0 G3 x( n( ]2 ]# u- w; ~ By default, the export can be mounted by any client. The value string2 Z$ t0 w! }. Q. m2 f
contains machine name and access privilege, separated by whitespace/ f) s g1 _% l+ }1 `/ U! k
characters. The machine name format can be a single host, a Java regular+ F7 ^: ]# v4 W4 G. P
expression, or an IPv4 address. The access privilege uses rw or ro to
* R2 L4 Z5 ^7 J+ z% |+ C specify read/write or read-only access of the machines to exports. If the
# x2 J6 f' r1 M6 {0 ^* z0 |+ q access privilege is not provided, the default is read-only. Entries are separated by ";".
# x* ~9 p3 _$ j- { For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;".: w! G- [1 o8 l/ o6 ]1 q3 K
Only the NFS gateway needs to restart after this property is updated.
, ]' J: H ~/ a" \+ d </description>
, G- L. R! |' v</property>' l% U( A$ _3 ^: Y/ t$ X8 F- x# E
<property>5 z1 K* m7 k# L# N
<name>hadoop.user.group.static.mapping.overrides</name>$ y3 s% K% E+ {$ i3 A
<value>dr.who=;</value>
4 m9 n) |% U) b6 a9 B8 L P/ o <description>/ a0 r$ _( z. @# O) T- E" h- e
Static mapping of user to groups. This will override the groups if5 a, w h4 l' y4 \' |
available in the system for the specified user. In other words, groups
6 i! @0 _0 i* M6 | look-up will not happen for these users, instead groups mapped in this" }1 p! j! C& t, F6 u, V' s
configuration will be used.8 l) m, U* b3 Q- ?% t9 L1 N8 l) I
Mapping should be in this format.
, V4 [" @, ?" i6 t H9 [1 ^ user1=group1,group2;user2=;user3=group2;
+ b0 H, l7 I0 I$ y Default, "dr.who=;" will consider "dr.who" as user without groups.
& J# u. J v5 {! F1 w </description>
6 P6 s* r5 s- v</property>; e, R& y9 H8 R5 ]2 w/ u+ C3 A! J
<property>
; k' C# [! D' S( f <name>rpc.metrics.quantile.enable</name>
3 \, P5 b: k' {7 _- `/ Q <value>false</value>
/ d/ t1 B x, E/ O <description>
& j% c: D- ]1 W Setting this property to true and rpc.metrics.percentiles.intervals
9 ^( R" h! }4 a3 P; ^% A to a comma-separated list of the granularity in seconds, the
. `9 x0 }9 v; d; M& v 50/75/90/95/99th percentile latency for rpc queue/processing time in _" ?% _5 @2 H% u
milliseconds are added to rpc metrics.
. W @6 n+ k: M+ o/ k </description>
6 ]5 X& {" h% B: q" e9 H</property>5 P" k9 a) u5 ?9 {0 ~
<property>
# D9 a+ R/ J9 s z# g; R <name>rpc.metrics.percentiles.intervals</name>& _4 y$ @& g6 P- k& G7 L4 `! o
<value></value>
% R' K7 v3 ]9 N; @: J8 A <description>1 b; P i; ]; N
A comma-separated list of the granularity in seconds for the metrics which% y0 w7 [8 V% {+ U; [7 R
describe the 50/75/90/95/99th percentile latency for rpc queue/processing
6 ~, N8 T* l. s3 l7 m, R6 T8 J time. The metrics are outputted if rpc.metrics.quantile.enable is set to* D# B$ [3 X: }( s6 S
true.
7 T6 ~( g% E0 u3 r </description>( i: |7 q) h7 Q- k& W$ c4 i) D: s
</property># _. b+ I3 P( e v% v w1 ?8 N8 }8 s# b
<property>
+ v1 [3 v/ ]4 [- Z1 m* p6 | <name>hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE</name>, K- O6 B) c: q5 k0 P! Z
<value></value>
0 l7 I+ G1 R' v6 @* c% } <description>5 ?; O0 J5 @* R5 S1 g" |1 w
The prefix for a given crypto codec, contains a comma-separated6 ?( x2 p5 S& i0 P# X
list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE).. l) k& F9 R$ J9 b2 I: w
The first implementation will be used if available, others are fallbacks.
- i% l0 J5 K+ q8 ]0 I2 x) @ </description>
) @) G7 ^! ]- C/ r* W' ~6 c</property>, J9 N0 c( U+ Q9 ~
<property>
* s5 T2 M/ @0 u <name>hadoop.security.crypto.codec.classes.aes.ctr.nopadding</name>
% ~0 I# t) W( ]5 w; O <value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>. h0 |. ^' \$ B/ Y, w+ N
<description>
0 e2 ~% l, n9 x Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
$ U# B3 W- R* [" W, C The first implementation will be used if available, others are fallbacks.! K* }9 q) [ k# Z9 P7 f
</description>
* C; I# K" a# K0 O( w+ [</property>' c+ \/ r1 S1 S1 l
<property>% d. z3 u: V3 n) }9 D+ y5 Q
<name>hadoop.security.crypto.cipher.suite</name>
0 ~# a9 r0 @1 u$ w5 B# V3 K <value>AES/CTR/NoPadding</value>
4 K9 A t6 O, u% k, ^8 T5 h3 x/ S( p <description>9 [/ p; v. T; y- l2 @$ W9 E2 g* |
Cipher suite for crypto codec.- s- Y- r$ \+ W0 T
</description>
2 E7 D7 I5 R' f7 @</property>4 s& W8 W! R6 J6 R: c) B
<property>
" S7 w1 c5 E' Q C5 t3 q" R+ ] <name>hadoop.security.crypto.jce.provider</name>. I/ @. w; m: T& p) U% ]' R
<value></value>
) ]- X5 F. X; T0 ^1 D <description>5 w, k: w1 @) Y& R* V# v0 _
The JCE provider name used in CryptoCodec.
& o& p9 O2 e$ G* T) {! o </description>
# o* {; c# i. {. ?3 P, p</property>
( b S3 }. P2 M0 }. o$ R5 C<property>
) _: d, o) k9 m/ ^' ~ <name>hadoop.security.crypto.jceks.key.serialfilter</name>" O& R0 V; E2 h; W H; K$ p
<description>
8 i8 R7 r% O, g: j a Enhanced KeyStore Mechanisms in JDK 8u171 introduced jceks.key.serialFilter.
4 G ?" Y* a; W3 [ If jceks.key.serialFilter is configured, the JCEKS KeyStore uses it during
: H$ c/ e+ |. P! A" ~) G+ c the deserialization of the encrypted Key object stored inside a
- E& r: @. T6 b1 p SecretKeyEntry.
" q- A9 P$ j7 x2 S$ y If jceks.key.serialFilter is not configured it will cause an error when& c0 V3 J. Z: h: E; n
recovering keystore file in KeyProviderFactory when recovering key from- S# x5 x- E/ |
keystore file using JDK 8u171 or newer. The filter pattern uses the same! G& K, j- s; G! j2 l
format as jdk.serialFilter.% f8 U& g* u" U
The value of this property will be used as the following:
8 o. l) u) x; z. |! ] 1. The value of jceks.key.serialFilter system property takes precedence) D4 L, c$ v. r
over the value of this property.
( g; @+ K0 O" ]' ^7 w1 ~ 2. In the absence of jceks.key.serialFilter system property the value of
2 U" H0 d( G6 ^- X& f9 n this property will be set as the value of jceks.key.serialFilter.
( c7 L1 x$ Q& r 3. If the value of this property and jceks.key.serialFilter system
( {/ |8 T8 d6 v! R7 m% e property has not been set, org.apache.hadoop.crypto.key.KeyProvider
% Z6 F6 A/ T: H9 K( E# K2 ? sets a default value for jceks.key.serialFilter.
( z7 F+ C0 C- T </description> D3 ?5 w+ C! o$ l: l- t
</property>" D' |5 ~2 ~' u3 A+ e: j
<property>' r: C% o/ }$ I+ _
<name>hadoop.security.crypto.buffer.size</name>
! z! r8 w: h" |2 S% W7 r- {; l <value>8192</value>
; n, y. h3 a( G. n6 I3 j <description>7 L% D" J$ x' n: R- s7 V* W3 \
The buffer size used by CryptoInputStream and CryptoOutputStream.
/ v& B+ [5 u; y( g) l </description>8 m4 h& U0 Y- ^: _$ J; ^
</property>
2 x0 e1 g+ N% b+ S<property>
( p( I- L1 ~0 S! g' f% { <name>hadoop.security.java.secure.random.algorithm</name>/ B: o- O; f* J3 c9 z$ N- \
<value>SHA1PRNG</value>
5 r6 x } a* c <description>
2 A5 Y! e: F% V" r The java secure random algorithm.7 {/ M3 Y, P% n( Q
</description>, b/ _" t) W. v: Z0 g' N; K+ W1 b
</property>
, V Q% A' M N4 Y# z<property>
9 T( c/ N+ C W& T" D8 s <name>hadoop.security.secure.random.impl</name>3 {; e- m/ d/ A+ U ^; N
<value></value>; d7 p& Q1 D- ?5 E! W& i
<description>
7 A! t5 L& E1 q G) z Implementation of secure random.& d1 P, j e' q+ C6 ~8 z
</description>
! z5 \7 \2 {3 F, }- l</property>
6 M$ E$ {; e6 @' |( l<property>
% P& t5 }( z+ |1 l <name>hadoop.security.random.device.file.path</name>
' l8 @$ N# j* D, j u% a2 J; ^ <value>/dev/urandom</value>0 ^- W' }' q' ]
<description>. Z% V% U# Z6 ~! m& l' Z) ^ X
OS security random device file path.
7 Q5 L$ f+ h/ I. d9 ^! U N: p8 H </description>
) V% j2 J' a2 ` B</property>3 c# A6 x6 o! l2 d
<property>6 y+ X' n& i$ Q4 c1 @
<name>hadoop.security.key.provider.path</name>
2 d2 q) q! m7 j% s: f4 p2 k <description>
( ~! [3 h5 g& ?; o+ H/ s' p The KeyProvider to use when managing zone keys, and interacting with
% W- t0 P D y encryption keys when reading and writing to an encryption zone.
( `# Z3 I9 u) D- H* a- k# X' i) k For hdfs clients, the provider path will be same as namenode's
; R9 m) q( u: v; A' c provider path.( ~8 ~$ y; w: r7 Y/ G- x9 M0 x% V
</description>
" `0 L. {8 |9 Z% h. K! K1 e% P</property>
' e6 `: r/ n2 P<property>' |& c% C6 q8 g8 G. E6 R6 }
<name>hadoop.security.key.default.bitlength</name>/ x2 z. A3 C1 Q( x* e
<value>128</value>
5 j+ R5 M* e9 |3 q9 }' Z <description>
/ T. ?9 A2 }, G( q" F5 G/ R The length (bits) of keys we want the KeyProvider to produce. Key length
8 g* M- N. g! I* E defines the upper-bound on an algorithm's security, ideally, it would# o7 V' @3 t* v- B! I8 K. V
coincide with the lower-bound on an algorithm's security.
6 z8 E7 ~$ A) l </description>
- g2 l$ ^& T) S1 I</property>$ l- v4 B3 D5 q, q5 y+ Y
<property>- \* D5 @9 v3 u
<name>hadoop.security.key.default.cipher</name>! K! P* ^& w( j) B5 G* j
<value>AES/CTR/NoPadding</value>3 p' U1 X8 L- Q5 k& q* K, ~
<description>: @& ^% w6 E% K/ S( ^% [
This indicates the algorithm that be used by KeyProvider for generating, g, U$ J" h( n5 u/ ]5 Z
key, and will be converted to CipherSuite when creating encryption zone.; l2 T2 t+ x$ s5 M, ~
</description>
8 n: V3 H6 ^6 g5 J: x) Y7 O; X</property>
7 v6 v3 ?8 K( }6 a<property>
9 J" H: X" m4 b% D/ _2 X. | <name>fs.har.impl.disable.cache</name>2 X6 B0 u: h; D7 z
<value>true</value>
5 b2 N+ N& W: Z <description>Don't cache 'har' filesystem instances.</description>
p$ I0 Q2 K; D$ ~( E</property>
" h- p \* u9 I0 C! _+ K2 w* ]$ |<!--- KMSClientProvider configurations -->' b0 M& B+ Q4 |( s
<property>! M- ?7 z. T) {4 k4 b. j& i5 m2 S
<name>hadoop.security.kms.client.authentication.retry-count</name>
: p( x: h6 T, j7 Q: } <value>1</value>0 e0 L! @0 v: Z R/ D3 I. V- o
<description>5 ?5 U! N( Z0 v- j
Number of time to retry connecting to KMS on authentication failure! b# k1 J# }( J! k/ O: i+ u" ?
</description>) D: [% X, ]& P
</property>
' H d' i0 ^ |( ^1 c* C" D2 M<property>* _1 T3 ?6 ]+ S q" A4 M2 u6 ]% B
<name>hadoop.security.kms.client.encrypted.key.cache.size</name> ]" @8 l8 i% r: |, K
<value>500</value>
4 y3 l* T1 i. _7 q% Q- L <description>9 C6 \$ O& B' s1 x; M- F5 g
Size of the EncryptedKeyVersion cache Queue for each key
" |9 L& D, ?9 o+ t </description>6 r" @2 \) x' S3 f, ?. W
</property>
- f1 S- c2 b# p) Z2 P: k- L7 _<property>
; q: U' u+ F( p7 w- s <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>: v2 v+ K: ^9 R! |+ R
<value>0.3f</value>
# s) i! `! O3 A# H( `$ ~ <description>' V% N% B9 {% A& q. b
If size of the EncryptedKeyVersion cache Queue falls below the; f4 i" q0 k ]- n+ H9 h# g
low watermark, this cache queue will be scheduled for a refill/ N* N. S! t) c
</description> a5 I; x4 y# t) m/ g" i
</property>
6 {: m1 c2 s8 y* D: _<property>: q, |8 @- c# T8 o) \$ C
<name>hadoop.security.kms.client.encrypted.key.cache.num.refill.threads</name>
- E4 H) }- t$ Y0 S5 n) d# @2 ^/ K, z# e <value>2</value>& W5 k2 d9 b. T) ^
<description>) q( W. P6 a, _4 s, B0 P
Number of threads to use for refilling depleted EncryptedKeyVersion
& |2 b! N4 I: d0 y cache Queues( W/ z B, X% B0 A5 i( h4 ]3 Y! K
</description>! K' h# ~* X' h
</property>
Y* A" N0 @+ i) d4 q7 `4 d* U ^<property>
4 g( `! m. S- a) u4 p <name>hadoop.security.kms.client.encrypted.key.cache.expiry</name>+ l% R& s b# c3 b# z$ `
<value>43200000</value>
; Z, N% d, w2 r6 y* l <description>2 H% J7 c2 |4 j7 H3 R
Cache expiry time for a Key, after which the cache Queue for this+ q% t6 a# m% U3 j h
key will be dropped. Default = 12hrs
, N$ h; u- f+ a9 @ o6 N </description>: L& Y+ |. i7 l) `. Z0 H; G
</property>
; }. u) v& N" f" W1 K9 I* l5 V<property>
0 z g1 Z( \6 M" l <name>hadoop.security.kms.client.timeout</name>9 X% x; |2 r2 R- y: ^
<value>60</value>5 V$ w: d- B& ]% H* d; A% U
<description>
4 A4 y. a/ |$ M! P9 f Sets value for KMS client connection timeout, and the read timeout( o. K0 Q& {1 Q* q& V% s
to KMS servers.- i) m+ N8 _- Y8 K- [- D
</description>
* s5 F* D8 g6 S' c</property>
$ K" l- t8 A9 i; g3 ?<property>
1 y( V3 F0 {+ a* B <name>hadoop.security.kms.client.failover.sleep.base.millis</name>9 {! d: a, K" v' Y( K- y
<value>100</value>
3 p' e2 U2 |! p' v) F- P <description>
. w2 U4 s' o( P4 k& X. O! X( W Expert only. The time to wait, in milliseconds, between failover
) ?0 t7 {8 X# N7 s/ Y attempts increases exponentially as a function of the number of
% E% Z3 R: {. I; P6 I1 J T7 a attempts made so far, with a random factor of +/- 50%. This option
3 k8 X/ f. w! W8 @. V5 V9 T5 v specifies the base value used in the failover calculation. The/ U9 r/ g8 x' C
first failover will retry immediately. The 2nd failover attempt: @3 h s' x* g9 d: M8 | ?
will delay at least hadoop.security.client.failover.sleep.base.millis
' J$ [1 o N& C+ t# x7 b milliseconds. And so on.
9 \1 m$ Q6 n6 Y) w </description>
1 C) w0 E( `6 Z& W# J2 b</property>
$ H- E( ~( o) O" c; B<property>( T" b5 ?. K' |3 A
<name>hadoop.security.kms.client.failover.sleep.max.millis</name>9 l$ I3 t7 I. [+ _% ?3 v1 H
<value>2000</value>
. P% C' C1 P, Y! Q/ t+ g. y <description>% ?" p/ ?8 G, k% C6 J
Expert only. The time to wait, in milliseconds, between failover5 d& h( L6 ]- a) ]
attempts increases exponentially as a function of the number of
! G; t) a% O7 J5 W attempts made so far, with a random factor of +/- 50%. This option2 O9 ?, ^$ x& p y) e4 W' w3 G
specifies the maximum value to wait between failovers.
) m1 X( V- l6 j$ M; r Specifically, the time between two failover attempts will not
' R, @* G/ M. M2 @. @0 R exceed +/- 50% of hadoop.security.client.failover.sleep.max.millis
) }5 j5 b7 R, M$ c milliseconds.
5 h5 M f; P8 D </description>
& |4 |0 t$ @' |" X</property>
2 L6 s. _- F; q( M% M2 _1 E) | <property>
& w& O) _) o9 m <name>ipc.server.max.connections</name>
. s* L, z5 O( b j0 l( V( k. Y <value>0</value>
5 X$ \$ Z$ Z" g7 h; q, u u6 p9 r <description>The maximum number of concurrent connections a server is allowed
/ c: A1 x* z# a0 L+ V! b to accept. If this limit is exceeded, incoming connections will first fill
/ b ?2 m1 ]' Q2 B' F the listen queue and then may go to an OS-specific listen overflow queue.
$ M" g( s8 S+ S4 K The client may fail or timeout, but the server can avoid running out of file
" _5 x! }" v2 {: z descriptors using this feature. 0 means no limit.
, e) w: ~- }. l* q5 p5 Q! U/ _" l </description>
8 n; e; r6 `, A9 K" Q* c. Z</property>
: ]( S: b! A5 _0 Y. G <!-- YARN registry -->! I7 v/ T. X" s5 K/ U! c
<property>
% k- b" P$ f7 |4 J. y B <name>hadoop.registry.rm.enabled</name>
4 n- K/ S! o/ p/ X <value>false</value>
! S {' F, ?5 O( K, B9 `% l <description>( v2 i* r ^& d" ~+ y% d Q
Is the registry enabled in the YARN Resource Manager?6 j6 E2 V4 `; T* n0 N7 q Q) D
If true, the YARN RM will, as needed.5 j2 [& c0 ^ o. n4 H: k0 T
create the user and system paths, and purge
. [& {+ F% c" K9 Y- f service records when containers, application attempts) {) k1 s8 N; n z
and applications complete.
# T- t, ^) B, y; @8 X9 H3 b If false, the paths must be created by other means,
. N& B; r0 M1 a2 r; S; K and no automatic cleanup of service records will take place.4 j+ \3 k5 u0 L& b9 w! b; g; |1 E4 S
</description>- Z; c/ D! x' b# C: ~3 g
</property>
: P: I# t2 ?. c; o6 h1 Z: c <property>0 n* l+ N! }" W' G- O0 \( r) `
<name>hadoop.registry.zk.root</name>
: Y, g4 [" A8 X1 X/ p <value>/registry</value>- x% U1 Q1 {6 S q
<description>1 F7 U; n: v0 n' b
The root zookeeper node for the registry1 S8 O, V% ^5 Y6 \! j, W, Z2 M
</description>
! i6 m: N1 P" n1 ~9 Q8 P: l- C </property>2 D- U" F' |& k: ?9 w5 [
<property>: O, k1 \: E. ]$ i+ @% j7 x
<name>hadoop.registry.zk.session.timeout.ms</name>
8 Y9 I- d# a& E: `: | <value>60000</value>
6 s' p! p) ^2 f t <description>6 u! a' R1 H) I. o. a6 U7 t/ x# c
Zookeeper session timeout in milliseconds/ m. B. _, b6 N+ N- u- K
</description>
0 l, \7 N) Y! v9 p2 p </property>% B7 z' Y6 ~3 ^% m" W7 T, P1 Q1 d
<property>" t7 B8 Q3 K. X! Y( g1 s V0 H
<name>hadoop.registry.zk.connection.timeout.ms</name>
3 G7 ` M$ f" A! X; M* C) L <value>15000</value>
8 w" A* a. y& ] <description>$ m& x. N9 N$ W/ J( K
Zookeeper connection timeout in milliseconds( C6 @) L7 `$ n8 @
</description>
8 n: u$ b+ o( ] </property>* N& ], v% ^& X6 s7 ~: n( i7 p; \
<property>
+ w$ {! U0 r( [) ?+ [ <name>hadoop.registry.zk.retry.times</name>
5 V/ |0 |% F9 m! P# f+ S <value>5</value>
2 E; n* n$ v: l <description>
( y) f. t# ~& E+ f2 E Zookeeper connection retry count before failing
6 J" e! e* ^' ~2 u7 |7 v: Q </description>
' M; q3 D: ]. h" B: Z </property>
0 u/ r4 z$ a1 h' z4 B <property>4 @* R- u! H: p3 B; f5 x
<name>hadoop.registry.zk.retry.interval.ms</name>
! S9 z2 i! T p9 i# Z <value>1000</value>
' V7 S% m4 U' n: }4 J2 | <description> Y" Z# y3 ]" l! O8 }8 ^
</description>
4 ~& |) ?. w. d </property>0 }. q3 ~' X% R/ k
<property>
3 a6 r2 f/ {7 }6 [& ^& n; G! p <name>hadoop.registry.zk.retry.ceiling.ms</name> m9 M* ]! H, \% A
<value>60000</value>- M+ M5 t4 t3 Z3 X$ U5 v# w3 K+ p1 c% A
<description>
5 S: l! \+ a4 B4 i4 H6 { Zookeeper retry limit in milliseconds, during' y& Z- a: x- C% u8 x2 S4 O
exponential backoff.) H& {4 B! T3 m% ?9 R% v8 l1 F" X3 t! Y
This places a limit even9 T) f$ ]% g) I( i3 n
if the retry times and interval limit, combined9 p; R& ^2 R m# a2 G
with the backoff policy, result in a long retry: ]$ v+ y( S. @
period
9 X0 G* D& `# `6 E: e/ } </description>( r( q! Q$ F2 M/ R. T6 ~7 _, c3 s
</property>5 i) C: m2 G- v, Y$ H4 p# G
<property>
& ?4 @: S6 @ c. L( s- D <name>hadoop.registry.zk.quorum</name>
' C4 |# i7 x8 q7 b# Z' I <value>localhost:2181</value>
- g6 \. c6 B- n" ?6 f <description>
3 u0 q! k- A3 R+ a! e/ m List of hostname:port pairs defining the
5 E; B2 h: J' Z) J2 e. c& ]8 C zookeeper quorum binding for the registry; Y5 T( u- ~( X- Y
</description>" A1 E; C1 y6 B1 s
</property>3 F9 L0 I D- n1 f) b8 V& q
<property># T, x/ \( ?( I) Z. c2 u
<name>hadoop.registry.secure</name>
& D9 f5 z+ H* t! M: }8 f/ @ <value>false</value>* i* ]" D l8 i& {2 g4 b
<description>! A/ F1 F' c) B3 r v" H* E% D
Key to set if the registry is secure. Turning it on
: s. w. K7 g0 ^8 a3 o changes the permissions policy from "open access"7 r8 a' J( F" f
to restrictions on kerberos with the option of% n5 m& Q& S! X& d. x6 M5 i
a user adding one or more auth key pairs down their% Y6 U4 A8 A$ Q# W: N6 O, |$ ]
own tree.8 ^+ J( d: s/ {3 p; Z, ^5 I
</description>9 I( M8 _: A; f4 p6 w* q: v. ?, ^' }
</property>
. x, t$ q9 s4 u e <property>
* `- L( ?9 c/ C6 j0 g; K <name>hadoop.registry.system.acls</name>: \3 T" }0 _4 }7 {; ?) x
<value>sasl:yarn@, sasl:mapred@, sasl:hdfs@</value>
+ e! _2 A" x0 f$ p <description>
i. p* I! Y3 H- K4 a A comma separated list of Zookeeper ACL identifiers with% p4 N, P: A5 ^0 [! t
system access to the registry in a secure cluster.
n4 z2 u$ \6 V: B These are given full access to all entries.0 C' W# [& g( V/ U4 |2 f
If there is an "@" at the end of a SASL entry it
4 I8 I; q+ S- }2 k instructs the registry client to append the default kerberos domain.
5 D& D& A, T7 m </description>$ n$ E5 F! q5 R) [! H8 B& X7 ]
</property>
/ t/ j$ x4 C8 s' i- G <property>5 T& }: a" i3 i/ T& T
<name>hadoop.registry.kerberos.realm</name># Y1 V+ N- ?* E* Y# B2 h
<value></value>
! d- H, u% |4 m/ C <description>6 O" g+ R/ w+ O" g' p
The kerberos realm: used to set the realm of
0 p9 v6 G+ R) c) y) [& ~ system principals which do not declare their realm,( S7 o6 W- K9 E3 A3 z
and any other accounts that need the value.
3 s; w: ~4 j; H/ i If empty, the default realm of the running process
3 t6 ]' {/ w& N is used.
! ~7 H j1 k- ^' m If neither are known and the realm is needed, then the registry
3 I$ i' h( D1 L5 O: W service/client will fail.
6 F4 q1 p1 N. t J* C6 t </description>
& c5 d5 ?/ u; \( r* E </property>
3 ~% n9 F, s L# O/ { <property>9 P+ ^ {/ ^2 o2 F
<name>hadoop.registry.jaas.context</name>
1 R# d0 s" g4 Q9 A <value>Client</value>
, J* z8 v/ R5 R+ L0 m' w <description>
" t. x" J( L7 K8 k Key to define the JAAS context. Used in secure( O. q% x: o( |# {
mode0 [# y9 }, N. h g4 u0 P
</description>6 T: Z- P6 i1 t' r$ o
</property>, R; g& I! `: g: u
<property>" a9 N' m' q; ?( Z/ k: B: n
<name>hadoop.shell.missing.defaultFs.warning</name>
8 _! A8 f. m! j/ T9 I# W <value>false</value>( L9 a5 _& [/ k4 l" {
<description>% Y- A" r# a0 k# a# ~* E" u
Enable hdfs shell commands to display warnings if (fs.defaultFS) property& R0 a2 k" q2 y1 ~, a/ ^
is not set.
' s! \5 s4 y- b </description>
1 h$ U- ?2 B: {) ~* P4 r </property>
2 P/ w0 T* J4 b! H <property>
7 h' @/ K! w2 e3 B6 i <name>hadoop.shell.safely.delete.limit.num.files</name># i6 ~6 f$ a0 P7 x
<value>100</value>& f$ J1 m5 z A
<description>Used by -safely option of hadoop fs shell -rm command to avoid
+ w3 u: M: E, m: i! w accidental deletion of large directories. When enabled, the -rm command5 ` o X7 F# t4 z) B d7 f+ c
requires confirmation if the number of files to be deleted is greater than
7 t) J0 @' }+ V% ]6 s this limit. The default limit is 100 files. The warning is disabled if
4 G, g' w& A, b; y the limit is 0 or the -safely is not specified in -rm command.! l7 u: ?' a2 a4 \7 Y% a
</description>+ P7 U! d$ ]/ t! L
</property>
$ k1 ~' x+ \5 d) _* ] <property>
- n* F/ Y2 o, T( `' ^3 l <name>fs.client.htrace.sampler.classes</name>
* j* W3 Z% f+ K <value></value>
$ D- { S8 H+ `: K5 U h <description>The class names of the HTrace Samplers to use for Hadoop: G8 s% p: {7 x
filesystem clients.
& G d0 j z, T# M2 V) w3 l$ \ </description>
- x A& m$ n/ D. e2 z </property>
$ } f6 E$ P. I <property>% X6 h) d4 T) Q& U: L! u* C
<name>hadoop.htrace.span.receiver.classes</name>
# K# ~" l$ [4 M2 ~3 c* j <value></value>
1 T/ D& D' {8 H5 f <description>The class names of the Span Receivers to use for Hadoop.7 K" k/ v; K9 L
</description>
% ]) u7 ~1 P6 a1 A </property>7 W# X6 M# E- `$ O8 O& F* M: M# k
<property>! [0 n, Y6 h7 z! f) Y1 I8 k' N9 d- n2 b" R
<name>hadoop.http.logs.enabled</name>* \. i9 r* N0 h) B0 F; {4 G
<value>true</value>
5 ^4 {5 W& i1 M: J% ^: f4 m <description>& ?# Y5 Q" ?( C2 z
Enable the "/logs" endpoint on all Hadoop daemons, which serves local
$ {3 j# `! ?7 T, k5 S8 [ logs, but may be considered a security risk due to it listing the contents% X" y0 c' }. ~! `
of a directory.
8 ?' {1 Y8 {% K1 f7 u1 L, V' F </description>
( R6 c/ [ V# |6 D </property>
) m( `0 f. X6 U1 c$ q( G <property>5 c" w0 l; J4 |2 N0 I+ K
<name>fs.client.resolve.topology.enabled</name>" R# M/ x: a! J7 Z2 l+ j
<value>false</value>
7 b9 R l- ?3 f8 y. A1 A( i <description>Whether the client machine will use the class specified by g2 K, Q+ |" G& [# h+ m
property net.topology.node.switch.mapping.impl to compute the network# t. v, K- E W: P
distance between itself and remote machines of the FileSystem. Additional. X' D7 K m- U' y- z
properties might need to be configured depending on the class specified
. K# j1 X, L; C& V$ Y' ^% S in net.topology.node.switch.mapping.impl. For example, if
- V {, c& T8 l2 q org.apache.hadoop.net.ScriptBasedMapping is used, a valid script file' k, u; _" |6 ?* y6 u
needs to be specified in net.topology.script.file.name.6 r2 q Y; ~ H& q: H$ \' @
</description>) s8 F- _1 F% l% W, C0 ^3 Y% T U/ [
</property>
2 m6 e' Q5 V% Z <!-- Azure Data Lake File System Configurations -->6 W4 q3 Z5 h+ O4 |" u( F( o4 c
<property>
* ^% N4 {" M9 |$ {& L: x <name>fs.adl.impl</name>% t+ X3 p1 j4 a% j7 N2 ]5 G
<value>org.apache.hadoop.fs.adl.AdlFileSystem</value>" {& Z6 z: D# y; e% j
</property>
/ G- ~ D( r6 l" A* N# ^ <property>
6 e3 j: V1 Q( q1 \ <name>fs.AbstractFileSystem.adl.impl</name>
( O5 z. h& A! z* N7 z4 Z8 q <value>org.apache.hadoop.fs.adl.Adl</value>7 Z- h! }2 I! H1 B. |, k
</property># C5 s5 L- L* c @- a; N$ M
<property>* }2 n0 R, { i C
<name>adl.feature.ownerandgroup.enableupn</name>
4 k! K" [/ ?) p( G" A <value>false</value>+ B( w4 N1 t7 X" [7 N: E: @) C, ~
<description>( e9 b2 N# S3 e0 C
When true : User and Group in FileStatus/AclStatus response is
6 @- \% l$ o+ d" F% j8 Y represented as user friendly name as per Azure AD profile.' V$ I8 Q! ~. f6 r) b+ Z
When false (default) : User and Group in FileStatus/AclStatus% j3 X5 t0 `4 i# V* @
response is represented by the unique identifier from Azure AD! |/ B! u$ z7 y2 A) G+ _' @. q
profile (Object ID as GUID)., T9 z$ K, t6 h
For optimal performance, false is recommended.
' p8 w1 i) T" h) l7 ^ </description>
* F& H' _; y! e3 ~ </property>3 @6 x* L( V2 v" X2 ?
<property>
0 H( G6 m5 I# ]9 H9 ]4 | K w" h <name>fs.adl.oauth2.access.token.provider.type</name>% J3 u. x6 `* @8 P) r5 s( T* a
<value>ClientCredential</value>
x4 k8 h! k3 N. t <description>
& ~/ Z" S/ O6 N Defines Azure Active Directory OAuth2 access token provider type.
# a0 V' _' B/ C- ^ Supported types are ClientCredential, RefreshToken, MSI, DeviceCode,
8 o: a- D, P; X; w; `. ? and Custom.
/ R7 `: N* e j3 c* {. |2 ? The ClientCredential type requires property fs.adl.oauth2.client.id,; H# T% f3 H: \! h7 M$ A
fs.adl.oauth2.credential, and fs.adl.oauth2.refresh.url.1 d9 L, l3 P, H4 ?$ O0 X' U
The RefreshToken type requires property fs.adl.oauth2.client.id and
' q9 L0 ~( L0 y# Z+ a1 j; } fs.adl.oauth2.refresh.token.
- {6 I$ p8 h4 e0 z The MSI type reads optional property fs.adl.oauth2.msi.port, if specified.; R6 n: G' a* U' Y2 o6 \
The DeviceCode type requires property9 |+ Y @% E% P6 I: y
fs.adl.oauth2.devicecode.clientapp.id.; M" e0 Y9 l9 e% W) Z* w5 Q
The Custom type requires property fs.adl.oauth2.access.token.provider.
2 ], i% ~8 r5 t- C, U0 c </description>
* V! l H4 `' |7 E </property>
4 t) q4 a8 v2 u, l$ A <property> c1 k" h+ n& c9 `5 S- m! c0 l
<name>fs.adl.oauth2.client.id</name>/ l9 L6 p% N% O* ^5 @: ?
<value></value>
. v/ _4 }9 I2 i, D7 d, h8 n. U- B <description>The OAuth2 client id.</description>1 W2 c9 A8 A& ?5 Y; Q
</property>
/ S+ M9 i( ^5 Q4 g1 o9 m( l$ r: X <property>. t) w7 ?6 t- }' t- T
<name>fs.adl.oauth2.credential</name>
. y" {8 ?9 z1 m1 H* O/ V5 |" \; } <value></value>
$ T1 S( I( u$ t7 R" _ <description>The OAuth2 access key.</description>
6 i4 y' h. Z/ d1 C. y7 N( b& Z </property>5 M5 F% r1 Z+ h
<property>
- }" h, d. A& u <name>fs.adl.oauth2.refresh.url</name>
3 K- z2 K. m4 e# \ <value></value>
: ~! Y! X3 V. s/ L$ s <description>The OAuth2 token endpoint.</description>, s/ m- X! F7 `" U
</property>) g8 d/ @% S/ \, C& _) x
<property>8 O% G5 s$ o" l( \/ E! R6 \
<name>fs.adl.oauth2.refresh.token</name>- E+ D" J- j V `9 ]$ n
<value></value>6 Y4 D$ z5 N! V5 `8 Y
<description>The OAuth2 refresh token.</description>
( E8 v( z/ a* f0 E </property>8 ~- @* k8 h8 ?' M, b* v; v: f5 ]; }
<property>
, Q/ p8 o+ R7 u& w1 O; e; a <name>fs.adl.oauth2.access.token.provider</name>
2 g- m5 {, B: ~# N0 ^- W( d <value></value>- z# C0 R% u9 g0 r
<description> [, f) W0 F3 F4 [
The class name of the OAuth2 access token provider.
0 o6 W0 F- b: d( h- @ c </description>0 x# ]# D( O1 k: D0 E- ?
</property>
; N3 C( z0 | {( x <property>* e. ?6 K" |+ k
<name>fs.adl.oauth2.msi.port</name>/ {7 }, P9 N5 G2 ]8 m9 T
<value></value>; |5 A5 p, F- d% N7 `
<description>% \4 t# [; V; f4 \, A$ w4 V
The localhost port for the MSI token service. This is the port specified) t8 C2 J+ [: ?! F+ H
when creating the Azure VM. The default, if this setting is not specified,
9 i# W# p$ s e1 I2 A. g' Z" y is 50342.
2 c, \- g# A$ Y* A) A0 \1 h, z p Used by MSI token provider.% }& k9 {( M* b0 |2 f
</description>' j- u I/ ^; W, g5 v, s" n3 E
</property>$ m! n* g- B0 r# d5 I! G& c7 I
<property>
7 ~' Z& [* |- D$ | <name>fs.adl.oauth2.devicecode.clientapp.id</name>5 I# n8 ^1 G( h
<value></value>! W9 q8 c( C' D& L
<description>, p* l- l: @. q2 c: p0 q. U
The app id of the AAD native app in whose context the auth request6 j0 X( {& z3 T# W
should be made.
$ S, z9 a8 N8 p* R* m Used by DeviceCode token provider.( [4 C- G& b( |4 s" u- }
</description>
. J- k7 O: y) x/ l6 H; h4 a/ C3 w </property>
5 z* ~" j( a4 q8 O8 `. h <!-- Azure Data Lake File System Configurations Ends Here-->% Z, ^* T% H/ n
<property>
% | d5 b, u1 O+ V <name>hadoop.caller.context.enabled</name>
$ ^2 J# K# n7 Z' G <value>false</value>
+ Z$ J/ Q5 }6 K+ b5 D$ X& v <description>When the feature is enabled, additional fields are written into2 O7 z7 f2 [/ O$ a
name-node audit log records for auditing coarse granularity operations.
/ g' L) g+ I( Y: e+ O' J </description>. b) H1 k/ z3 }7 H+ Y7 E
</property>3 P1 b. [1 f; K6 z5 k5 P- k
<property>
# ]4 R, Q& R' S' ~( i% S <name>hadoop.caller.context.max.size</name>5 @9 C2 G% v+ E, P: w3 X o1 B% `, d; J
<value>128</value>
, U& e' K$ C0 b, S9 {1 ` <description>The maximum bytes a caller context string can have. If the2 e7 `5 s l5 U6 f- Z! @- B
passed caller context is longer than this maximum bytes, client will4 X4 G! D; v. o1 {
truncate it before sending to server. Note that the server may have a
5 ~$ E$ H1 ^7 T* S) f8 l% ] different maximum size, and will truncate the caller context to the2 p8 _0 s) `0 O) L: G5 V- p0 l
maximum size it allows.
* O' U- ^0 B5 p& O4 p3 E </description>
G' H: Q5 y. O8 v/ n </property>
6 l U8 j7 X% i# O <property>9 Y; p. D- t$ @0 W$ b
<name>hadoop.caller.context.signature.max.size</name>, r. D1 p- N) c; _) |3 n; k# C* h
<value>40</value>0 I/ t6 b* Z' ]! ^2 }' I& t. Q
<description>7 _0 X9 Y2 x0 F" y- s6 j2 H* e
The caller's signature (optional) is for offline validation. If the
$ K- M9 f' V `. N signature exceeds the maximum allowed bytes in server, the caller context
! R' H6 b6 ^7 z9 J& k5 M will be abandoned, in which case the caller context will not be recorded
0 f* s. _" {% D* N) P- R4 h# m in audit logs.6 O, }1 r( R; M9 Y# r
</description>/ b8 M0 j: w2 Z5 d
</property>
7 S# D) ~$ K# u6 W<!-- SequenceFile's Sorter properties -->
+ w/ w% c; |: I9 ^% }( U <property>
8 @0 O" ^" y6 F, c+ m6 d <name>seq.io.sort.mb</name>) ]) \4 N7 f! n
<value>100</value>
% l2 e% V/ a/ }: w X4 [ <description>
, O" R& s6 T% Q& w* f5 S2 p The total amount of buffer memory to use while sorting files,7 ^! d C# i7 Z% ~, U
while using SequenceFile.Sorter, in megabytes. By default,; C* t1 E$ G2 D0 S+ s$ k4 D
gives each merge stream 1MB, which should minimize seeks.
0 N+ [# \1 b4 x/ X* e' \ </description>
* D8 g% }1 e" I3 A0 `: Z, n </property>
: C G! o# s0 r) b4 v <property>
# p( H( v W: S; J2 c6 ] Q+ d! ? <name>seq.io.sort.factor</name>( ?8 ]# k% M/ G6 \4 K. }* T# S
<value>100</value>' r( @; _5 a: T2 N& h
<description>7 _8 ?+ W# B# |& f
The number of streams to merge at once while sorting
2 K' R& T3 C: `8 F; h' I! X files using SequenceFile.Sorter.5 q. E9 Z' Y1 A Z: X
This determines the number of open file handles.2 M2 S; A) x( l5 } Q/ C# ^: u
</description>
7 v# m# l* S/ B+ p </property>
M% K* i& C# k: L- p" x3 Y& d- | <property>
( L k7 ~/ l0 M% d1 Q8 B4 o <name>hadoop.zk.address</name>7 _; {/ ^) d3 |9 R( S
<!--value>127.0.0.1:2181</value-->
3 X5 F/ J: L( _% }1 b <description>Host:Port of the ZooKeeper server to be used.
: o: _; d1 F* E( D/ x </description>
1 w, h0 Y9 [. t6 l+ _7 z </property>9 m! b `! o# u0 h
<property>9 @8 Y0 I: q$ U+ b- V2 f
<name>hadoop.zk.num-retries</name>
# }, Z8 Z9 _! t5 p3 ?2 T6 l <value>1000</value>
8 F* l$ R c; N2 e+ X* n( |! Y _ <description>Number of tries to connect to ZooKeeper.</description>; h6 p, k4 z/ \. y; C
</property>
/ `- S* A9 r9 @' V& b- z7 {/ }0 j <property>% X% ~# L. ~' c5 l; C
<name>hadoop.zk.retry-interval-ms</name>) u, S& ?5 ~ S0 Q
<value>1000</value>
, |: ]9 T* r& n4 n& |- O <description>Retry interval in milliseconds when connecting to ZooKeeper." f" S- r& r) s9 y$ X# q; S
</description>% ~% H x& \: O2 c0 Q
</property>
0 F8 D. e- X8 b3 g7 p <property>
1 F" N4 i8 H! l <name>hadoop.zk.timeout-ms</name>7 X4 C- i0 n6 ~: @, E$ A
<value>10000</value>
. _5 W, `8 b2 J <description>ZooKeeper session timeout in milliseconds. Session expiration
y% y& i8 }6 Q3 ~! c" [ is managed by the ZooKeeper cluster itself, not by the client. This value is# L) G. T* V% J o* E# ?& U
used by the cluster to determine when the client's session expires.
$ u U/ g6 G8 g Expirations happens when the cluster does not hear from the client within
' a4 }) d; C; i# I$ T/ x the specified session timeout period (i.e. no heartbeat).</description>8 u3 T" V$ @/ H; Q# B) X; t
</property>$ T2 C7 Z: c( f- p7 |7 c6 F6 ]
<property>
' W4 G% h* ]5 ]. v9 g8 I5 N" x <name>hadoop.zk.acl</name>& H* l' |4 c, K' G0 G& p3 S
<value>world:anyone:rwcda</value>
$ O& A) e# H3 s2 _' E4 ^ <description>ACL's to be used for ZooKeeper znodes.</description>
, F4 h, n2 w1 N& Z </property>
( b' i. @ `' U) D3 T <property>0 M8 a9 I, x# ?* C2 Z
<name>hadoop.zk.auth</name>2 v3 N1 K9 Q6 `" L8 m; Y' E
<description>
) e8 Y6 G) d" M9 l Specify the auths to be used for the ACL's specified in hadoop.zk.acl.
3 j3 W1 E2 \: T9 _1 k+ i This takes a comma-separated list of authentication mechanisms, each of the- b5 A, r9 C( D
form 'scheme:auth' (the same syntax used for the 'addAuth' command in
; A, t' Y x. d; ^- p1 o9 K9 P. \ the ZK CLI).
N$ N. ]9 D* [3 I/ g </description>
& {3 m( q- D% v* L* L </property>* Z+ Z- }) M3 o9 v( A* p
<property># [9 X( {6 M# Q# ?8 f
<name>hadoop.system.tags</name>
[/ l2 \, ^, x; v# z2 \% @/ S. n <value>YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT
1 {6 n3 A \& L9 c9 T7 Y2 V2 L ,SERVER,DEBUG,DEPRICATED,COMMON,OPTIONAL</value>% M G2 ~; E( S6 L1 K
<description>0 }! x7 M& r( F
System tags to group related properties together.
9 `' \) k9 x4 T+ W$ s </description>
, k6 x W) @$ W& f7 F9 v( I% f( Q! d" T </property>
3 p. n' B$ z3 ^# E6 A <property>
' P5 C9 P8 \( z' C, o3 u <name>ipc.client.bind.wildcard.addr</name>
5 w# S$ M& c1 u2 k) a: G& H <value>false</value>
0 d, Y# Q9 l; v, X# D- Z <description>When set to true Clients will bind socket to wildcard. J9 s6 s0 }6 Q4 e
address. (i.e 0.0.0.0)
; h7 H7 P$ d4 a2 ]* u </description>. K% F* d- z) B+ |
</property>
: ~" e- W& }! B4 H% e8 B</configuration>
6 M1 {5 Z) t; i6 @7 w0 v+ y% }$ M) F" ]2.hdfs-default.xml
! L& y" H' p: ?* N5 q$ B/ K* m0 U' J' _ m3 j, ~
<?xml version="1.0"?>
4 r) w: w- H- b( F6 r: M2 I. Z% h) J<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
7 y: K+ e7 A. \# P; |" B1 Z: l<!--* V( }! C7 g, \8 Y
Licensed to the Apache Software Foundation (ASF) under one or more
& q; s7 B3 E# _' h contributor license agreements. See the NOTICE file distributed with4 S# Q% K8 }* p! F/ j( W
this work for additional information regarding copyright ownership.
" _% s, K1 U& h The ASF licenses this file to You under the Apache License, Version 2.0
2 ?0 r5 P8 X7 H* V) g- j0 Z0 W/ k (the "License"); you may not use this file except in compliance with
9 E, @# K0 P9 S! o3 y" _) J the License. You may obtain a copy of the License at
/ b! v1 n8 Y; G/ o% e0 G* D+ K. a http://www.apache.org/licenses/LICENSE-2.09 C, j9 Q6 x( P; C, m% d- r
Unless required by applicable law or agreed to in writing, software5 v& r! P* T s; s0 I+ k9 m
distributed under the License is distributed on an "AS IS" BASIS,
) H8 u7 L) _6 z WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.% k$ N& {+ s/ o& {$ N8 L
See the License for the specific language governing permissions and7 C& ?3 S0 F& N, d( W( H, k
limitations under the License.; U7 q4 ~) k5 D$ k: B3 s$ v8 y
-->
/ t" c! g' g+ o/ d4 F8 N<!-- Do not modify this file directly. Instead, copy entries that you -->1 C9 D o: T) Y: A1 Y9 x* R+ V
<!-- wish to modify from this file into hdfs-site.xml and change them -->. F$ d5 s6 h& K/ G l' z
<!-- there. If hdfs-site.xml does not already exist, create it. -->- `8 H% ]+ K. [. S
<configuration>) ]2 [) S3 q+ \' N* @
<property>
: I% b7 v% K$ u9 M: m2 m <name>hadoop.hdfs.configuration.version</name>/ K5 o" Z& @, ^7 s9 \
<value>1</value>
) v& U$ G6 c( |* {0 F <description>version of this configuration file</description>
% s; @; b6 D1 j& l</property>4 w6 Q3 f: h/ Q2 c6 Z$ V
<property>
5 R$ ^0 }5 J6 E! q, L# G2 x1 ^ <name>dfs.namenode.rpc-address</name>1 S3 b1 X% R4 W# k7 B( ~2 A
<value></value>1 z2 U" V* m' M9 ]5 j3 ~* M
<description>
% [0 m9 y/ [9 [; B- ]- @( ^$ h' b RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,
?4 @. W) e& Z+ ?( Q2 m the name service id is added to the name e.g. dfs.namenode.rpc-address.ns11 y2 [8 X! a5 \$ ~* g" H4 b
dfs.namenode.rpc-address.EXAMPLENAMESERVICE
' B+ G) ?: U6 C! o* o+ g/ a% o The value of this property will take the form of nn-host1:rpc-port. The NameNode's default RPC port is 8020.$ i' A: D j/ D1 g+ t+ U
</description>& j! w7 Z# i# J2 U! k8 L: t; P* ?
</property>0 l- k4 V# A2 u6 o' z' g
<property>
3 a+ m# L! z, w6 `* X( D+ |& J4 p7 d$ u <name>dfs.namenode.rpc-bind-host</name>
- \# l ^ h; ?3 p. q <value></value>: W# C. T: d6 y+ C1 U
<description>. p% W3 ]- A# H
The actual address the RPC server will bind to. If this optional address is4 }& ?! Q- a T6 [: h$ o
set, it overrides only the hostname portion of dfs.namenode.rpc-address.
3 `2 z5 |/ x* m+ A8 _+ @ It can also be specified per name node or name service for HA/Federation.% X- |! n' m' l) _
This is useful for making the name node listen on all interfaces by
7 a; w2 T6 b. R setting it to 0.0.0.0.5 [. Z: c. M; `6 K5 `
</description>. k# T: ^) F2 o+ n" ~! o; a% G
</property>
3 p5 T; f: y" U<property>1 Z8 a/ N1 N! `9 P7 N& f% o; G
<name>dfs.namenode.servicerpc-address</name>
8 { h& t" L+ p <value></value>! d' _. G" y& _" D) M# f% A. H
<description>/ F/ s, u, o; N$ f' s
RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be' e8 e! c k$ A
connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, [; b, ~) e3 \) O X0 W7 a/ n' z: J
the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns14 u: m1 ?9 c" {5 Q' }
dfs.namenode.rpc-address.EXAMPLENAMESERVICE7 C+ Q6 u0 x' t# X! y
The value of this property will take the form of nn-host1:rpc-port.. P) H8 y( W& g2 { ?
If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.
" K9 _; T1 j: I) t `" d </description>
9 \1 @' \! @; y( t- H" O3 r6 {</property>
# W' k* X% x. V0 _<property>
9 O0 Q' T7 f& O* R <name>dfs.namenode.servicerpc-bind-host</name>
5 `3 z: U8 T/ F9 P& q, h3 M <value></value>0 W' e0 ~+ u# p' `4 U0 x7 Y2 s7 r. U# t
<description>
% s0 \" ~4 v0 V4 j9 n. {+ s The actual address the service RPC server will bind to. If this optional address is3 A0 ] J% s3 F4 a" M* |( ] J
set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.& l4 ^, c2 I9 E6 ~* y( e- d
It can also be specified per name node or name service for HA/Federation.
* p5 l3 o# _! Z: ~: a6 f; { This is useful for making the name node listen on all interfaces by
4 b* y3 ~2 u T setting it to 0.0.0.0.
* S. m' G; d( w2 l; D </description>
% j% o/ I; M u e. D) }</property>
; t9 X; G1 @% J$ l<property>
& t# [" j7 p2 U3 {# S; ^. [' t <name>dfs.namenode.lifeline.rpc-address</name>8 P. d$ D( H- B5 I& O+ C
<value></value>; `& P0 s" \9 d6 Y2 s
<description>" S0 ~3 G! w$ ~5 y; S% J2 E
NameNode RPC lifeline address. This is an optional separate RPC address
g; N- Y+ e0 h$ o& m0 x* {& S that can be used to isolate health checks and liveness to protect against6 i+ C2 N4 n3 \! T7 A. ]3 n6 x7 W
resource exhaustion in the main RPC handler pool. In the case of
1 X4 y! x0 [$ ~ V) T$ J: K HA/Federation where multiple NameNodes exist, the name service ID is added
$ _9 L2 [" h" i+ W& H to the name e.g. dfs.namenode.lifeline.rpc-address.ns1. The value of this
7 Y# P# e9 P. i property will take the form of nn-host1:rpc-port. If this property is not
- M( ]- r2 N- j$ F defined, then the NameNode will not start a lifeline RPC server. By
0 r( T Z' }0 @9 S' s- q+ R4 R. n default, the property is not defined.$ K8 e; d# B* n6 N
</description>- s# A' a* E9 G! g3 n- |/ _% f% V, i( {! [
</property>
+ ?4 c. R- `8 z8 z<property>6 ^, p5 a- R0 b7 _
<name>dfs.namenode.lifeline.rpc-bind-host</name>
# l3 l; Q O) X7 J- t <value></value># Z& a1 S2 i! ?! H0 @9 T) h
<description>
1 G: F, H; p! ]8 m. ` The actual address the lifeline RPC server will bind to. If this optional; v& @5 `4 J9 k( i' Y% Q
address is set, it overrides only the hostname portion of9 V: J- b7 W3 i1 z) z0 b
dfs.namenode.lifeline.rpc-address. It can also be specified per name node# S/ A" h2 ~+ I1 X8 {( H! h4 a
or name service for HA/Federation. This is useful for making the name node
! W; S/ O. c) C listen on all interfaces by setting it to 0.0.0.0.
& Y2 h6 U; ~, J' |. T </description>* R9 W9 A1 c0 W) H- E4 j3 s+ u' e M
</property>
Y& V$ A! ^6 [ O<property>
1 {- e0 L4 [3 C <name>dfs.namenode.secondary.http-address</name>
) P( n$ [3 b9 {8 b% \4 t% r <value>0.0.0.0:9868</value>
! F4 X+ W8 ~% k8 `! K5 {- G <description>! A7 I d( p/ O% O: \) j1 R6 e
The secondary namenode http server address and port./ L: A4 p$ v8 G1 h& h. q
</description>
+ w- b8 ?. t+ k# F' {/ A</property>; ]9 ?- g$ P$ V; y
<property>
! ]3 P- A: B* q, j0 z6 t7 O <name>dfs.namenode.secondary.https-address</name>
4 k( S1 j; L6 _# X* ` <value>0.0.0.0:9869</value>
# V8 A+ T8 x7 V$ R <description>
3 F( o) Y, k7 {% _1 n+ @7 G0 Y5 u The secondary namenode HTTPS server address and port.
e, s3 ] |0 m6 N: R: E* q </description>( R; ~7 o# T4 z$ ~0 X4 b
</property>+ Q1 n& v, a9 m% }
<property>; G; Y; M: Z; {
<name>dfs.datanode.address</name>
% Q9 s+ @* p% |! ^ <value>0.0.0.0:9866</value>
2 t7 [3 c" K9 w4 K& V <description>
) I# v% B3 r7 Q3 f! R7 I* e1 C The datanode server address and port for data transfer.2 ?7 |& P. o' l, s0 u1 s* R# X
</description>
$ @: m, @. s% @7 M: d9 r</property>' P) e, X5 a! y
<property>
, M6 y% k& T" Z0 k- ]* r: G v% g <name>dfs.datanode.http.address</name>4 `1 ^% G. p( m+ c
<value>0.0.0.0:9864</value>" W' a8 a- W. I/ M
<description>- D4 L& x: q) s# X, \' N, p( x
The datanode http server address and port.% D+ x2 C u1 Z7 `* U: _
</description>
' |2 ]4 V4 K2 ?9 @</property># y9 }2 F/ F# y) V* O" n; E
<property>% \: C# |. N4 B% D4 N
<name>dfs.datanode.ipc.address</name>+ g `# m- K* J
<value>0.0.0.0:9867</value>$ z3 ^# L `; }3 ]8 C
<description>
2 p$ E5 l& h5 k2 H; [6 S' B The datanode ipc server address and port., I# y U4 _) T5 H4 m
</description>5 J; _5 A: k- V! B$ ?6 `
</property>
: A7 l6 }9 ~( f$ \7 x9 c# b<property>7 H- h! d" p4 @! x
<name>dfs.datanode.http.internal-proxy.port</name>
$ F# n/ G. U9 G0 S <value>0</value>
; w( `8 J6 y2 u8 Y3 M" H% i, S6 k- z <description>9 C" d, M# o6 @' ?& A8 g. Y
The datanode's internal web proxy port.0 X' x* S2 N" R/ Y* o, G( m
By default it selects a random port available in runtime.
7 o" J7 {6 i- i: Q; t# |1 V8 K4 Y </description>) s2 q/ p5 \* ~& a6 y$ R
</property># E& p. c5 x) r: E' ?7 v
<property>
. L* x9 I2 k' o2 E <name>dfs.datanode.handler.count</name>* R9 j- B4 K, n) |: K
<value>10</value>
: {' y; x3 t9 c9 Q W: p/ g <description>The number of server threads for the datanode.</description>
6 j4 A; I: P* y* u) J7 B</property>
8 F& J2 K( c# j<property>: J `; M: r5 z* E# i2 ^ j- n
<name>dfs.namenode.http-address</name>
! v+ e0 Y$ t. v. Q' ?7 ^# G% D8 l <value>0.0.0.0:9870</value>- I9 ~6 z, N e; a% j1 \( E4 }
<description>
" u" x7 L- m) R6 l The address and the base port where the dfs namenode web ui will listen on." L- c9 ~& a& R2 E+ T( j& Z9 n
</description># v( p* Q C+ Y) |
</property>, O) K6 e- I' x
<property>) [$ y; D) _+ i& j4 Q7 [
<name>dfs.namenode.http-bind-host</name>4 @& n* ^$ s w/ t& u) P
<value></value>+ B: d, q( g2 R+ \5 s
<description>
. s! p: z& t/ P The actual address the HTTP server will bind to. If this optional address
% A# U2 ]! t( {9 p! [" g/ J is set, it overrides only the hostname portion of dfs.namenode.http-address.
- o' p' _0 W, ~6 G" L- j6 L6 z It can also be specified per name node or name service for HA/Federation.1 p( p2 j9 c" N2 G
This is useful for making the name node HTTP server listen on all& l+ X( q2 ~* A. W6 q _& v" S8 K$ v
interfaces by setting it to 0.0.0.0.; O- ?) ~3 U# p
</description>8 D1 z, o5 B' Y& z( x; ]" [9 s
</property>
, K, X. L8 k# @$ O) L/ w( _% U<property> a! R5 r2 N {& P
<name>dfs.namenode.heartbeat.recheck-interval</name>
s/ x" s4 j) N2 r0 z& b* j: g <value>300000</value>( B% B" j- w1 h# h; r- M
<description>! `- D0 t) Z8 F; i) M
This time decides the interval to check for expired datanodes.
. |% l q* M# ^( ]' g6 i With this value and dfs.heartbeat.interval, the interval of* U4 L1 V( G0 ?' b& B2 G* F
deciding the datanode is stale or not is also calculated.
4 R3 Z0 L4 D1 ~9 e4 w* M. ]* S. v The unit of this configuration is millisecond.) b' {. P) g% E9 u
</description>, M" g/ p/ Z3 k: l
</property>
% D1 D' _6 `* N% T% p+ {8 M+ }<property>5 ?$ _) `! `5 o' B" J
<name>dfs.http.policy</name>/ V- }. A( C' S# @( |
<value>HTTP_ONLY</value>
, L+ X; F2 @- @/ F8 T* x <description>Decide if HTTPS(SSL) is supported on HDFS: R: b7 ~: j- x, Q1 t7 A3 ]
This configures the HTTP endpoint for HDFS daemons:
# b5 ? d C$ `1 s% R The following values are supported:" `- k3 a/ c N( U5 t7 R! X, X
- HTTP_ONLY : Service is provided only on http
; F7 _7 g6 z8 k. ^ - HTTPS_ONLY : Service is provided only on https& U8 Z$ g/ b1 t( M9 X' M
- HTTP_AND_HTTPS : Service is provided both on http and https
; K8 c- T2 I! u </description>, A. o2 |3 W5 h+ }' J9 p0 ~
</property>9 O- e- m, f) J6 U {$ g( ~
<property>$ k) E# p( E& j
<name>dfs.client.https.need-auth</name>+ R G9 y! s0 c
<value>false</value>0 r* h! r7 q1 L" O k, w% ?
<description>Whether SSL client certificate authentication is required4 i# _3 M. U% Y5 b5 j; P
</description>
- n [$ M5 i! Q</property>
6 ~+ h( i) V9 i# z0 g<property>
^. I" |7 l& a8 V$ H1 W5 h v+ o <name>dfs.client.cached.conn.retry</name>
" [6 T3 v* @1 P" O, }3 ~ <value>3</value>
2 b) t- j% q9 T3 S w6 x8 n7 C, i <description>The number of times the HDFS client will pull a socket from the8 F, E' T0 Y2 s2 j/ M8 b$ S
cache. Once this number is exceeded, the client will try to create a new
4 }3 {/ o; ]3 R: s3 F' m, S% }! j socket.0 {3 Q; F. P6 B# g
</description>
2 o" H9 H& z3 \( E</property>
" R& i4 [# z4 k$ p2 c<property>
( ^& c2 s; i& P5 G% J9 ? <name>dfs.https.server.keystore.resource</name>- H6 z# K3 V- B/ t* x9 v) P( S
<value>ssl-server.xml</value>
0 w, M+ ?# z- Z! P% `! Y <description>Resource file from which ssl server keystore
?9 x1 Z' L; s; D information will be extracted5 v* X: ~+ \# U R8 C: U& ?) t
</description>
9 u- D: Z, k! F9 S0 z0 n& |</property>
# J2 W2 P- h P" ~. V<property>! [2 g1 a2 r. q. D" p! o* C# @
<name>dfs.client.https.keystore.resource</name>3 q) \ U5 f: o% j2 `
<value>ssl-client.xml</value>9 u5 N5 {: ^; [4 W
<description>Resource file from which ssl client keystore
$ v% v' Z( }+ Z9 r5 c4 s) y! l7 @ information will be extracted. C* ?& [$ W5 h: c! @4 ]: |
</description>
! z1 E% W, X* q</property>( C6 q1 l/ ?0 M3 b
<property>
+ E2 B2 ~2 M# _: W+ ] <name>dfs.datanode.https.address</name>
" D. s9 Q5 d" Z* [5 o$ L4 z <value>0.0.0.0:9865</value>
% J# I$ B- g, f6 Y <description>The datanode secure http server address and port.</description>
0 @! ~$ t6 j1 W* i' r+ V</property>) O4 X T* t' f5 o7 G
<property>* R0 v0 f7 Z+ z/ ]
<name>dfs.namenode.https-address</name> E" J4 Z; r) B# C7 q2 t u
<value>0.0.0.0:9871</value>
: E* U: [7 I; u! r+ {1 _+ L) ^4 m" T <description>The namenode secure http server address and port.</description>$ Z T5 Z" S- Y. V0 N- y
</property>. H+ v3 f5 Y) b, p6 R/ Z
<property> e, j/ |* \. a" n
<name>dfs.namenode.https-bind-host</name>5 s) r6 r8 s5 P2 p3 u
<value></value>
( Y) l8 n2 H2 u& `) G+ G. L' s <description>
- P9 X) \& v2 q! Z+ h' V The actual address the HTTPS server will bind to. If this optional address
+ l h( R8 b( _1 M- l2 |$ X5 P is set, it overrides only the hostname portion of dfs.namenode.https-address.% Z/ w# M9 Z t, x
It can also be specified per name node or name service for HA/Federation.1 R2 q% t6 z* O0 J6 G, v& v! P- p
This is useful for making the name node HTTPS server listen on all Q7 s [' x: i5 O- {
interfaces by setting it to 0.0.0.0.( ^% j; L' e' A o' j
</description>" r& p9 A% l% b( a4 b) A2 T
</property>' E3 U8 z, j# n+ o% i- ]
<property>3 e9 }2 [, I) u- E/ E5 _
<name>dfs.datanode.dns.interface</name>
& ~+ z( s5 x* o1 y0 W# F. @ <value>default</value>: T9 L3 B I5 |4 x1 o8 `
<description>' {0 l* Z- W# R e1 M
The name of the Network Interface from which a data node should
* l" {4 Y | T7 B3 { report its IP address. e.g. eth2. This setting may be required for some( D* C& o7 ?% _; W9 M6 v
multi-homed nodes where the DataNodes are assigned multiple hostnames
8 W' I" J% q. j: V/ M4 R7 Z q and it is desirable for the DataNodes to use a non-default hostname.. j- Z: e8 D' J h% j" r
Prefer using hadoop.security.dns.interface over
5 Z% m1 i& }9 P dfs.datanode.dns.interface., f( m8 M1 o# X5 i) Z4 y
</description>' s0 Q; F4 u% u! ~1 p
</property>
9 p8 N( u& m/ ]. u1 u/ Z* L* h) t<property>) ~. e- M: A3 S9 O
<name>dfs.datanode.dns.nameserver</name>9 U2 J) R( s$ T. w% N$ ~$ T2 a
<value>default</value>: |' j3 D( j6 W7 q5 ~, J- c/ E
<description> f& _9 A9 Q# i* b2 [
The host name or IP address of the name server (DNS) which a DataNode
* g# v3 H+ L1 p" L4 q. R should use to determine its own host name. B j+ _0 }& g0 s0 W# S* V- q Z
Prefer using hadoop.security.dns.nameserver over
2 x) R' c, N/ ?# G* Q q dfs.datanode.dns.nameserver.
$ I8 ^, g/ v/ j6 [1 m </description>& c; \" o$ n- q H8 w) O
</property>
' t5 b; y" i1 h8 |4 J! p: f <property>
9 a8 C9 U3 |5 q" D <name>dfs.namenode.backup.address</name>
5 m; o4 s/ {( |2 `: v9 K: z* C <value>0.0.0.0:50100</value>$ Q2 P c+ _ W' _4 S
<description>+ A8 X4 X3 e& x4 K+ i
The backup node server address and port.- S9 Z' d, g% y' p( u3 y! F% y
If the port is 0 then the server will start on a free port.8 b# e$ E. e; \1 B( G! Y3 n: Q
</description>; b' @0 A5 [! x& L. n& V
</property>
# B$ p3 C2 t) |6 W) Z2 w! s9 P <property>' V3 ]& P7 M& W; `
<name>dfs.namenode.backup.http-address</name>
+ y0 s. a7 X7 d$ v <value>0.0.0.0:50105</value>' h4 G# x% v' F( y0 I! N
<description>
. q, m& ]4 P/ Q! B. I3 O4 \4 L+ | The backup node http server address and port.7 w7 Q% O8 F3 J' P# S+ }% @' g; [
If the port is 0 then the server will start on a free port.
+ R# P% a! k2 q </description>
) c# C( l" l6 {1 z! H</property>
# x. N2 d+ g3 H8 X4 Z6 ]" R6 O<property>4 O7 H6 v1 L) W6 q+ c- i
<name>dfs.namenode.redundancy.considerLoad</name>/ \+ G j1 Q. p/ ]& H l
<value>true</value>" Z; Q& C' C' D6 O
<description>Decide if chooseTarget considers the target's load or not4 ?" B! N6 x5 [1 B0 H
</description>( n/ a" U9 f$ J( S* R
</property>
' \5 Y+ I2 v \. |' R* ` <property>$ Z% w a' U' Q; \! m
<name>dfs.namenode.redundancy.considerLoad.factor</name> M( L: a" K: V! h0 O: l
<value>2.0</value>
3 B0 ~% G& O8 A: h3 ]: r. q F+ n <description>The factor by which a node's load can exceed the average
+ v% u8 W: f9 s6 v) f% v6 f' G+ i before being rejected for writes, only if considerLoad is true.6 w8 `/ \% b% E# Q! ?, I5 X
</description>8 f1 A4 G" i( `. x' O
</property>; e y1 i- R4 k \( l6 {
<property>
8 A2 h7 T; Q( ?$ F! b) j <name>dfs.default.chunk.view.size</name>- W n" ^2 t: l) c
<value>32768</value>
. q. \( t8 u$ [5 `" @: e% |9 G <description>The number of bytes to view for a file on the browser.0 W' T6 t2 I, ^3 F$ ?
</description>- H, r$ p! X2 [( \! S" ^4 X E1 i, @; F% U
</property>7 i8 C) @6 G, a, Y) p) |: p
<property>
* Z# A" c6 y, d ^# S <name>dfs.datanode.du.reserved.calculator</name># m5 o# K7 \% u# a
<value>org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorAbsolute</value>
; {( V6 ?% t1 I/ b <description>Determines the class of ReservedSpaceCalculator to be used for- j- G4 N. m. x# O# h5 A& {
calculating disk space reservedfor non-HDFS data. The default calculator is
0 [6 F& [- m4 r, {; x- R ReservedSpaceCalculatorAbsolute which will use dfs.datanode.du.reserved
% I4 B- @, Z B. k, S Z for a static reserved number of bytes. ReservedSpaceCalculatorPercentage6 }6 x! [- u& W' a3 B8 _
will use dfs.datanode.du.reserved.pct to calculate the reserved number! f7 r6 O+ z- f& s
of bytes based on the size of the storage. ReservedSpaceCalculatorConservative and
& d2 e8 G. E4 c2 a7 Z! ? ReservedSpaceCalculatorAggressive will use their combination, Conservative will use- n( t/ W2 l) c w" g
maximum, Aggressive minimum. For more details see ReservedSpaceCalculator.0 r9 _& N6 S+ h' b: S2 G3 X
</description>
, S: Y$ w% i+ N1 l6 g/ z' C, K2 Y</property>7 o" \0 K: d5 J' q) }- W
<property>
' ~- {$ {4 k, s8 t* X& {. k <name>dfs.datanode.du.reserved</name>
" g* x) {9 j& P" ^! o <value>0</value>
2 J7 Z; Z6 H% a <description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.) [" G. [" d& m5 b! R b; ~
Specific storage type based reservation is also supported. The property can be followed with
' E% O- P$ M1 ` corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.: r9 l1 C+ f r; ?3 N3 Y
For example, reserved space for RAM_DISK storage can be configured using property
$ c% U. O) W: Z3 ?3 t* t! `- ~ C 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured+ v4 U, p- o8 S7 l
then dfs.datanode.du.reserved will be used.
0 m3 f+ D: P; T g </description>
! [5 o; ^" e: O! t</property>& M3 r4 `0 y6 M" p9 @0 s! }( D7 W
<property>
# O. c5 K- t5 O' a: A- i- Y <name>dfs.datanode.du.reserved.pct</name>' k" Z' S% r5 T! G7 B& i
<value>0</value>4 _& ^" j: }8 x5 b
<description>Reserved space in percentage. Read dfs.datanode.du.reserved.calculator to see
# _' z6 f6 ^) C8 _% x0 r3 m/ q' V when this takes effect. The actual number of bytes reserved will be calculated by using the$ s' Y$ i; V/ a
total capacity of the data directory in question. Specific storage type based reservation
% ~! t8 \' D& M is also supported. The property can be followed with corresponding storage types
* F7 {! @! w1 z; h% c& o! E ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
1 S! t" u! s; ~" H For example, reserved percentage space for RAM_DISK storage can be configured using property
9 X# e# N% `; Z0 ^$ O1 w 'dfs.datanode.du.reserved.pct.ram_disk'. If specific storage type reservation is not configured
, Z7 z/ _, j9 ]/ n, ]5 s: v e then dfs.datanode.du.reserved.pct will be used.
! _; n# g. G& P </description>
/ f9 n3 k: p3 k9 q8 ?</property>
' `; |' x: ?1 D: z+ ?- Q- B/ B<property>
X( p3 P% i! G1 ]; ~ <name>dfs.namenode.name.dir</name>
+ F7 |% ~0 o9 y, ^3 m <value>file://${hadoop.tmp.dir}/dfs/name</value>
! \& Y- P1 A3 _5 k5 D <description>Determines where on the local filesystem the DFS name node" N+ |% u0 t$ a* H# s4 h
should store the name table(fsimage). If this is a comma-delimited list5 M1 y$ Y5 _3 {3 n. }
of directories then the name table is replicated in all of the
$ X8 ^, J6 D n. U, V, v6 T9 t directories, for redundancy. </description>
. @% o$ U( O2 _; }" k2 p</property>" s4 L5 n7 `1 S2 {' t
<property>% x7 m! P; O) Q# A
<name>dfs.namenode.name.dir.restore</name>1 @2 h* A/ I! j4 {3 P/ n
<value>false</value>* X2 E S4 {% M2 D6 e
<description>Set to true to enable NameNode to attempt recovering a
- L6 ^0 h3 m" h2 j previously failed dfs.namenode.name.dir. When enabled, a recovery of any, i/ p- ~$ z8 k5 F$ j# t4 W2 C
failed directory is attempted during checkpoint.</description>
" N, a/ y t* W$ x( o- U</property>
8 Q# j1 I7 K0 A3 N6 x7 ^<property>9 B& { r* E3 F2 c+ U! l
<name>dfs.namenode.fs-limits.max-component-length</name>% E* t" Q& G2 h I6 c9 y0 n# V
<value>255</value>4 } N4 O0 u8 s$ M, L) R
<description>Defines the maximum number of bytes in UTF-8 encoding in each' R! V8 v: @ {% Q$ C
component of a path. A value of 0 will disable the check.</description>
. d0 f- l+ _4 {</property>
% {- H2 E3 z( V0 t( p0 [( M+ F<property>- L9 W& p7 @' m# D8 O
<name>dfs.namenode.fs-limits.max-directory-items</name>
" V, m% Z( o: B: K3 c <value>1048576</value>
/ F6 ^9 t- e& w) p/ t3 ? <description>Defines the maximum number of items that a directory may* b2 C3 d/ _6 n" G: m3 u& G
contain. Cannot set the property to a value less than 1 or more than- i3 X3 R N6 P2 v0 o! G
6400000.</description>
/ P1 [- J& S( j) `) N</property>) O4 U i: D. O. E
<property># q4 Y5 ^! s( h4 c
<name>dfs.namenode.fs-limits.min-block-size</name>
, m$ K+ P$ L' W ~( F <value>1048576</value>
$ o' j$ }, n( A6 L <description>Minimum block size in bytes, enforced by the Namenode at create
: ~# j: T: T G7 Q9 X time. This prevents the accidental creation of files with tiny block1 E7 w2 Q4 m4 L/ n) r# `
sizes (and thus many blocks), which can degrade. Q1 R, P1 V' K( k; A
performance.</description>1 s+ T7 f) K- b/ {- [- r+ U+ [
</property>
# G6 q4 {6 F% ^ `+ A, b+ R<property>5 |& {0 e4 y& a$ n3 c. o2 ?$ p
<name>dfs.namenode.fs-limits.max-blocks-per-file</name>3 V0 M: G0 R# k* k0 z3 s
<value>10000</value>
! X5 n- ~8 \! [$ E; M <description>Maximum number of blocks per file, enforced by the Namenode on. q- L0 _7 ~! k5 u
write. This prevents the creation of extremely large files which can' u: i2 A5 U" j) x
degrade performance.</description>) H/ y4 c ^) \: f. Z) A/ o: c
</property>+ ^7 u' j; M/ G" S/ C7 T [6 q+ x
<property>
_1 _6 s5 x8 j, S+ R <name>dfs.namenode.edits.dir</name>
) C- x- N2 e/ n' k. f, U <value>${dfs.namenode.name.dir}</value>
" [( \1 e' H( r6 J <description>Determines where on the local filesystem the DFS name node
' A, X; p# J8 v1 l! k' n; ~ should store the transaction (edits) file. If this is a comma-delimited list
1 S7 d0 h1 `/ ^" b3 \ of directories then the transaction file is replicated in all of the
* p6 s5 R1 [5 z+ P9 M directories, for redundancy. Default value is same as dfs.namenode.name.dir
& p) d' D9 j; d9 c4 ?- k </description>4 S3 e1 ?; ?( c/ ^' N
</property>) ]$ B% T4 w6 b
<property>$ u2 F6 D3 h7 ?" y& J: H
<name>dfs.namenode.edits.dir.required</name>
# K4 b5 j$ F9 z+ @) P2 O <value></value>
3 F# K/ ?) G8 R4 s) W/ \& O <description>This should be a subset of dfs.namenode.edits.dir,
3 d5 \" q* |) m to ensure that the transaction (edits) file
. Y) X2 s. W4 `, Q# V in these places is always up-to-date.
3 m4 N7 x! C/ \ </description>/ ^9 N0 n6 K0 G3 L& S! F* D
</property>' o3 y j0 i, P k4 C
<property>
2 M% `0 R$ u9 ]! c* D# C! L <name>dfs.namenode.shared.edits.dir</name>- y i0 X7 m' x. a* c
<value></value>0 r- B: W# P7 s
<description>A directory on shared storage between the multiple namenodes
4 O- @4 x1 N( j2 e' N+ Y. O @ in an HA cluster. This directory will be written by the active and read
0 K5 V" l% c" K; E by the standby in order to keep the namespaces synchronized. This directory8 E( H$ z2 Z( S$ K% ~* X' C* S
does not need to be listed in dfs.namenode.edits.dir above. It should be
- C" }( s% h& {. e left empty in a non-HA cluster.. L$ M5 s) {% U; Z( Q
</description>3 }7 O/ X7 Q Q2 y" l
</property>
3 b9 J; j8 t' `; w<property>6 F( n& O+ Y, g$ P# H, J7 z) n' k
<name>dfs.namenode.edits.journal-plugin.qjournal</name>: _' ~8 X5 h6 S; H i" [. M
<value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value>
9 E+ i; W0 G, N5 X8 k</property>4 U4 M) m! F! a! [- _ T. t |
<property> m- m- {" b" v' D+ q, k3 W
<name>dfs.permissions.enabled</name>
( I) @, L' t R' O* N <value>true</value>
* V7 A/ r, n' X, X4 ]8 _ <description>* S( N" F2 d- w
If "true", enable permission checking in HDFS.( J& m! i3 `! q- [! A
If "false", permission checking is turned off,% ~1 M9 c& e1 M. b0 G, v" C
but all other behavior is unchanged.( j' Q2 k; o- `' ?6 w
Switching from one parameter value to the other does not change the mode,
8 _) [3 k( e: \% t; d0 e* l: m owner or group of files or directories./ d2 ]% s! e- D( [+ v/ v
</description>
' u& M+ e9 w" L. M) N. \</property>
5 I$ p E# r$ j" I3 G4 x7 z<property>8 F d9 _0 T3 w% n$ ?" a9 E
<name>dfs.permissions.superusergroup</name>
+ V" N: s+ N% U <value>supergroup</value>
- |- s6 B" ]2 D! d+ e <description>The name of the group of super-users.
2 Y' ~# R2 x f0 T1 [% e The value should be a single group name.
4 S* V1 M: \5 ^& @ </description>
4 l! n# e( _0 s ~</property>* u0 Q- R* L1 l2 y- ]
<property>
# R1 ]8 @9 y8 d <name>dfs.cluster.administrators</name>
' P% a& X/ i% u/ i <value></value>
8 K: E+ y% K1 K" Z. ^, \* j5 y. [ <description>ACL for the admins, this configuration is used to control
6 f3 d& C2 V1 m& ^/ z! G who can access the default servlets in the namenode, etc. The value: P) v; } X: t" @4 U0 }
should be a comma separated list of users and groups. The user list
# D+ P0 N R5 B3 G+ I comes first and is separated by a space followed by the group list,
J& W7 \) f' y3 g8 } e.g. "user1,user2 group1,group2". Both users and groups are optional,
?: ~' g" _! } so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2"
7 \0 c& l# Z2 c' a) B8 h1 ^ are all valid (note the leading space in " group1"). '*' grants access
+ {8 y* J" [" W; \8 [& q to all users and groups, e.g. '*', '* ' and ' *' are all valid.; S9 M5 K( E A: W. m
</description>
/ o' Y0 \4 T0 P' t2 `% _</property>: u" W" t1 P$ t: q( H
<property>
& I! K5 h' u7 S/ ] <name>dfs.namenode.acls.enabled</name>6 E* C3 f+ ~7 E5 O( t. ~ }' E
<value>false</value>+ h1 q, q) ^/ M2 l c' [6 O N- a
<description>" M: J' m4 q/ s6 q0 r" {9 l
Set to true to enable support for HDFS ACLs (Access Control Lists). By8 {6 G( u, b5 R7 n* f/ H
default, ACLs are disabled. When ACLs are disabled, the NameNode rejects+ w7 d2 |, L, S. O" [% G. D7 E% @
all RPCs related to setting or getting ACLs.7 w9 N; ~* Z; Z0 C
</description>% T4 q1 N! d8 k9 v9 ^
</property>
! D+ _$ W( B1 S' l) t <property>
/ t# z7 D# p! c6 W/ ]% K <name>dfs.namenode.posix.acl.inheritance.enabled</name>; h/ d7 v2 s$ c3 n# [
<value>true</value>4 |5 l1 p* t( S
<description>
6 T" k5 a! P9 W Set to true to enable POSIX style ACL inheritance. When it is enabled" Y- M/ W/ i. w6 ^4 Y, ^: y- N% r2 [
and the create request comes from a compatible client, the NameNode
2 W. A M/ n0 N9 a) y! B will apply default ACLs from the parent directory to the create mode, f$ K: Z/ N) o
and ignore the client umask. If no default ACL found, it will apply the
' G$ O f: ]4 G client umask.8 L5 w" c) q0 S, l+ E4 a
</description>6 I) K( M$ I# g H$ v
</property>
9 i3 d0 \+ q: _* K4 k& b <property>
2 x6 i' m8 h0 S+ F; S/ @5 P <name>dfs.namenode.lazypersist.file.scrub.interval.sec</name>
; r4 e6 H: W9 ?& ]! N2 d% M <value>300</value>
2 l1 u H) Y( X+ V <description>
$ R4 ]$ s w1 T6 F' e" s8 Y The NameNode periodically scans the namespace for LazyPersist files with6 s. ?8 J' h, k7 ^
missing blocks and unlinks them from the namespace. This configuration key+ b* ?% V) H- G2 S' N/ u9 j# I+ B
controls the interval between successive scans. If this value is set to 0,5 q2 x1 y8 z O( ?4 n" e) ?
the file scrubber is disabled.
M7 a) e& g* y. j9 v+ W0 l/ n </description>6 R0 x5 D( G$ T: M6 T
</property>+ ` P! a1 Y+ w0 s% C2 H( N7 R
<property>
# D$ S7 K- P6 E; o2 u! H3 x <name>dfs.block.access.token.enable</name>
. K+ n( U, }$ R. m! F! U" |- D% _+ w a <value>false</value>7 X" ]# U) l; N/ F- K1 D" x
<description>
: d3 s7 \+ B Y# Y If "true", access tokens are used as capabilities for accessing datanodes.
( d- K- \8 {* ~8 c# K: x8 s If "false", no access tokens are checked on accessing datanodes.
2 K& g' K- H7 n3 A" @( }( _' z </description>4 ]+ U A, e8 y' S' ?% |
</property>
9 p0 l5 \2 G z! c" W<property>
7 k8 f8 b, e9 T- W& ^ <name>dfs.block.access.key.update.interval</name>
" v( j! ]6 | A7 j5 G3 b/ r <value>600</value>
" ^" ?/ v' y* D; a <description>
. B# n* Z6 v( A: A k$ k0 N Interval in minutes at which namenode updates its access keys.
- P' \4 y9 \" Y# ]) G4 d4 g </description>$ ^$ D6 I+ A2 V
</property>
6 |' N( X9 ~) {) M0 H<property>0 ?; k& M' O$ T) b
<name>dfs.block.access.token.lifetime</name>' M7 J* v6 ]$ c) P
<value>600</value>
, `& x G" h; m" n# z <description>The lifetime of access tokens in minutes.</description>) A- y+ T2 I2 p# Q" B9 e
</property>
2 ]: x: a$ Y0 E$ b<property>
8 v: `, h* y2 Z( Q <name>dfs.block.access.token.protobuf.enable</name>
" c4 V' W1 L7 R5 Q' C <value>false</value>
7 R+ t* s- |$ n/ F* s+ E+ `) E }7 ~ <description>+ \9 e) ^( ^$ H# O4 p. i
If "true", block tokens are written using Protocol Buffers.
/ N3 E! W5 N- H3 u; ~. I$ h$ c' O' @ If "false", block tokens are written using Legacy format.
" E( x+ j% A8 g% n0 w3 y5 A( I </description>
) g& }- _ g) w</property>
: I h- d9 S0 L9 K; `$ z M/ l4 |- |<property>
2 i7 P) a- l/ j1 t <name>dfs.datanode.data.dir</name>
2 o) O3 t4 U+ `4 i+ R. G$ d N3 O; q <value>file://${hadoop.tmp.dir}/dfs/data</value>
" w0 U6 s& e# [' j9 a <description>Determines where on the local filesystem an DFS data node
]3 m) w' w- o0 ?5 U should store its blocks. If this is a comma-delimited( [- u. c/ ^6 {1 H# k' f
list of directories, then data will be stored in all named
( t9 m2 a s1 q0 Z! F8 _+ U z5 Q directories, typically on different devices. The directories should be tagged( H( ?7 x) @( E9 Z2 S
with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
0 b3 J* [, W$ b storage policies. The default storage type will be DISK if the directory does: R! Q8 g- k6 K& C4 O0 N
not have a storage type tagged explicitly. Directories that do not exist will
1 E# v8 A# u( g2 N/ U+ ^* G be created if local filesystem permission allows.7 A8 K# ~ }# F$ y- x/ x' a
</description>
% m J. v& \1 I8 @, i. \ G1 a</property>) j7 R0 A _0 P1 k
<property>1 y3 L% `( o( t( z; ?4 |
<name>dfs.datanode.data.dir.perm</name>' P$ s: ~* N7 M- F
<value>700</value>! i5 l- V/ H: W2 `0 f& ~
<description>Permissions for the directories on on the local filesystem where
0 W( [2 w- @& O8 v$ M( B+ e the DFS data node store its blocks. The permissions can either be octal or. e( p: _. V" ~5 U7 ?7 S3 i
symbolic.</description>! l( d% J( {5 u1 k
</property>" ^ U, p+ O4 r' R
<property>) h& D, ] Y% a7 m" k2 N5 Z
<name>dfs.replication</name>5 o4 R! \1 H+ s% I
<value>3</value>) F; A' A0 X! a Q5 ], W0 s7 E2 j
<description>Default block replication. - y3 H7 T2 \, i& V, a
The actual number of replications can be specified when the file is created.
! \9 @& Q. b' N4 p9 t. L$ a The default is used if replication is not specified in create time.
3 ?; z! ^- Q3 Z6 P </description>
j0 t" t7 N! l' ?: M3 i U</property>
; f. D3 V1 b+ x8 f: t r K<property>
- k2 ~+ F3 {, H2 [/ z6 a <name>dfs.replication.max</name> \0 p/ v9 J7 x! ~7 p; y: W% S+ J6 o2 Q
<value>512</value>
; Y+ ^: y& k h# I <description>Maximal block replication. 4 c( f) E/ b# @# V& W6 e. V
</description>3 S" M, E! {9 q( [
</property>
9 D$ o) G P4 Y1 o8 v+ ]<property>, M' o2 \; P( m5 S; c. R
<name>dfs.namenode.replication.min</name>
7 F2 l( Y! Q; \3 ~( h <value>1</value>8 @# |# T3 l4 I. F+ d k
<description>Minimal block replication.
" M( N" Q" e& y </description>( f8 I# o; y* ?6 [- Y8 L
</property>
( J8 K4 i3 b4 Y7 d \5 n, u/ f<property>
7 j+ J! b- w1 E, e3 H) i: E <name>dfs.namenode.maintenance.replication.min</name>
. w: R& t0 e8 D' ~5 \ <value>1</value>
/ J: ?8 U* ~3 ]/ _ <description>Minimal live block replication in existence of maintenance mode.1 |4 O' v# p# F9 O
</description>5 ~1 Q/ v2 g, D) `$ |* n
</property> J, H5 y$ U9 T6 x" z
<property>( Q& Q/ X& G- @/ O3 @" E
<name>dfs.namenode.safemode.replication.min</name>7 c/ B1 x8 w5 T5 L3 F' }6 [' d1 y
<value></value>
; G h" ]% @. b( T2 h <description>5 t0 G+ I3 c% P% a, C6 O/ {/ I/ D2 J" L
a separate minimum replication factor for calculating safe block count.
3 L1 R$ N) ]9 _ This is an expert level setting.
B% P+ F; k& J Setting this lower than the dfs.namenode.replication.min2 w$ g' b0 e/ f6 @, l
is not recommend and/or dangerous for production setups.
" z4 v& J5 L2 x" m1 S+ z9 P When it's not set it takes value from dfs.namenode.replication.min, l7 F& o4 _0 s3 `4 f
</description>- G8 m; y7 _/ }* c0 J5 ^
</property>4 F( ~2 b' W- o% N
<property>
, H1 M, o/ E* m% z <name>dfs.blocksize</name>
) D0 t9 s2 b% ?& C o7 [ t <value>134217728</value>
) n5 Y' X- R8 g' Y9 k <description>5 c. }( h0 v6 X6 C9 ^4 s+ E
The default block size for new files, in bytes.
* Z$ r* q9 T: T& y You can use the following suffix (case insensitive):
0 M0 h9 c7 A/ E/ H k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),; H- T M- k4 r$ b: N+ `4 P
Or provide complete size in bytes (such as 134217728 for 128 MB).
5 q r8 T$ G. l1 t2 i Q </description>
3 D+ X; p# v; y2 ?</property>% d- _( t" c2 ?# R
<property>
" V: T% y! r$ G+ v# }* t <name>dfs.client.block.write.retries</name>
5 i4 k) _) k% i7 K) ^ <value>3</value>
$ E0 P) U7 M3 I1 q2 r; V <description>The number of retries for writing blocks to the data nodes,
3 x1 ?1 D/ B2 Z1 b8 _2 Q- X2 D before we signal failure to the application.
$ ]: N1 d2 H6 f# r( S" f- X, l </description>
u- }- a5 d$ R% U2 w% z</property>
: G; R" x# B! j( w5 | P<property>/ T9 Z0 e! _' T8 r+ m
<name>dfs.client.block.write.replace-datanode-on-failure.enable</name>9 N: a' ^, Q4 W& ]0 C+ d3 W, g* q6 M
<value>true</value>
" n$ {: a5 M1 E( {1 X <description>' ]* ?9 p& Y7 a. D! M8 |3 J/ Y1 q% }
If there is a datanode/network failure in the write pipeline,% K6 Y6 x* N" d5 j6 }( x: l8 K' O7 X- I
DFSClient will try to remove the failed datanode from the pipeline& s8 ]: Q; d. _/ x4 A# {( \
and then continue writing with the remaining datanodes. As a result,
q/ G/ `: V, p# t the number of datanodes in the pipeline is decreased. The feature is
; G; R& V' N$ l7 Q* ^8 e to add new datanodes to the pipeline.4 w( s$ v: k1 y6 H+ j* |9 ?. u: b
This is a site-wide property to enable/disable the feature.
" G( N, |4 q. \7 g When the cluster size is extremely small, e.g. 3 nodes or less, cluster7 W% S9 W8 I( a2 ~* O
administrators may want to set the policy to NEVER in the default
( V) i. ^3 f g: G configuration file or disable this feature. Otherwise, users may; e3 a3 l: v- L/ F C3 p
experience an unusually high rate of pipeline failures since it is
3 |: o" L" m2 I impossible to find new datanodes for replacement.
+ u& J% F1 q& i( F- C2 f See also dfs.client.block.write.replace-datanode-on-failure.policy* \6 S$ a* O" O+ C1 z4 C; b5 g3 x1 s
</description>. b( }; u- J3 b6 w+ E3 h
</property>0 A5 s7 X- I0 a# c9 l0 \8 m
<property>8 `( a& f. C6 i( k' T
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>1 D0 b+ b" I* w1 b
<value>DEFAULT</value># P) X' l2 Q% }0 m9 a
<description>
$ n# K4 x( H2 z/ ~- ]/ J8 T o This property is used only if the value of/ i& n k6 `7 Q* |
dfs.client.block.write.replace-datanode-on-failure.enable is true.1 x9 E/ ]) h8 j0 I
ALWAYS: always add a new datanode when an existing datanode is removed.. e+ I& H+ @$ y0 d# S% g! }( b! x
NEVER: never add a new datanode.% ]% J9 i$ z: J" u# r' t. j& E
DEFAULT:
; F2 Y+ C4 G+ K3 E5 n# W- i Let r be the replication number.# {6 O8 @; M& q- N
Let n be the number of existing datanodes.# s4 i; R, c2 D. r2 j% K
Add a new datanode only if r is greater than or equal to 3 and either
$ p1 p5 `; w1 i, X+ ? (1) floor(r/2) is greater than or equal to n; or a3 {+ [- V2 @
(2) r is greater than n and the block is hflushed/appended.
$ S6 a6 m8 P- y2 ] </description>
[; k0 ~- U4 J4 C1 K9 h/ a</property>' ~# o4 }7 A* v) l
<property>4 I- i3 J. W7 }' E7 C& t6 R/ F* ]
<name>dfs.client.block.write.replace-datanode-on-failure.best-effort</name>9 t9 |4 q2 ?7 |, G# d! ^8 e: o7 v
<value>false</value>+ `# N* ^: {1 m3 U4 b4 j3 o
<description>
! {, ]0 W7 V ?/ h+ K/ O& c% \ This property is used only if the value of
" v5 `9 `/ ^' ^8 [0 u$ T dfs.client.block.write.replace-datanode-on-failure.enable is true.
6 ]/ h8 O- k( T; I Best effort means that the client will try to replace a failed datanode
" D! l9 o8 D4 t* ^ in write pipeline (provided that the policy is satisfied), however, it
) ]5 a6 G+ r; a7 P6 O continues the write operation in case that the datanode replacement also
. p1 X. f; g6 s' z& {4 q5 G fails., }1 x' @- I: g4 P4 o4 `- {* r
Suppose the datanode replacement fails.; A7 l( U; w& J, `8 e ^/ w8 F
false: An exception should be thrown so that the write will fail.
* D8 c* @5 L! @4 ? true : The write should be resumed with the remaining datandoes.
* ]2 z+ q" v+ k: f, k6 ^& r: b Note that setting this property to true allows writing to a pipeline
9 i' s) B; s: o2 F7 V' o. Y0 H with a smaller number of datanodes. As a result, it increases the
/ z# ~% J& Z' V" T1 Z probability of data loss.
7 G' r7 {! L; I9 P </description>
* m! s( x1 L, b/ a; ~</property>9 R" }6 L3 O6 N. g) J
<property>
8 Q7 U( X& _+ x2 b- X <name>dfs.client.block.write.replace-datanode-on-failure.min-replication</name>
# C0 M% `" z% N2 t0 n <value>0</value>" U7 i8 a# q+ ~9 N3 C9 E& c& S
<description>
: k: c$ v& K+ g The minimum number of replications that are needed to not to fail
J" l$ Q8 {/ s3 V- q& V the write pipeline if new datanodes can not be found to replace- t; _6 S9 {; h3 V3 E7 T- R
failed datanodes (could be due to network failure) in the write pipeline.
# z. T, M: D5 S If the number of the remaining datanodes in the write pipeline is greater
. q7 L4 H" [, O5 ?$ I) L) p than or equal to this property value, continue writing to the remaining nodes.: h- p8 Z5 i3 _* {$ B/ {
Otherwise throw exception.( {- p) d: W6 s7 z6 N P, K' x
If this is set to 0, an exception will be thrown, when a replacement
8 b; O/ Y( r1 h& j can not be found.- F: G& c, \$ G$ R4 V
See also dfs.client.block.write.replace-datanode-on-failure.policy
% X3 f7 B. |- s; @4 O! I+ k </description>. e; v! o4 c* l1 L. Z1 K7 U5 m
</property>1 r% b9 o4 a* {9 k: P& O O
<property>" I3 r2 r3 A) G0 F u
<name>dfs.blockreport.intervalMsec</name>6 P7 D8 f5 \& s9 W7 E
<value>21600000</value>
6 o- V3 Q/ @/ H4 f3 _- i4 m3 I! U& ^ <description>Determines block reporting interval in milliseconds.</description>
5 }" ~7 |( k8 X1 @& P- c</property>
- S ~9 |! p# \3 l<property>
. e( H0 R6 _! P, t* F+ I3 ~4 T <name>dfs.blockreport.initialDelay</name>/ k. X/ n4 c4 B0 V- t2 E
<value>0s</value>2 I' i/ A, h) {
<description>
) A* T( A8 o6 `& [# |3 E Delay for first block report in seconds. Support multiple time unit
& |) m" B( T5 Q( [8 V5 U, A5 [' ] suffix(case insensitive), as described in dfs.heartbeat.interval.+ M& T9 g4 q% w; M0 j& l9 h5 F
</description>
3 n/ a) j9 ^* @) t" A1 N/ I; n</property># K2 E* a+ ]& [
<property>0 T$ o7 Z# b; t4 A# L; _, b
<name>dfs.blockreport.split.threshold</name>
. M# Q, p( x, `7 I/ U' R <value>1000000</value>
6 s( u' |8 r9 g/ Y4 h <description>If the number of blocks on the DataNode is below this2 O1 W% w7 R; U0 b1 l; S* P% W
threshold then it will send block reports for all Storage Directories
, G7 q9 `7 }# e" F% A; A8 @ F in a single message.
4 }/ l. j% J2 H If the number of blocks exceeds this threshold then the DataNode will/ i4 }& _. b! f+ e
send block reports for each Storage Directory in separate messages.9 |" W- E4 u; }# F% h
Set to zero to always split.$ E8 e9 a; Z, @0 ^9 a0 R' X: y
</description>3 `- R- Z) x6 M2 C
</property>
% f5 t3 Q# g; f# v4 D3 r( m/ o( ~<property>
4 |: r4 }5 _' T; T( z <name>dfs.namenode.max.full.block.report.leases</name>2 T. I4 O6 k* x
<value>6</value>
5 K4 R4 c( j8 V& _ <description>The maximum number of leases for full block reports that the
4 A5 p. b2 ~7 R8 h$ Z- y NameNode will issue at any given time. This prevents the NameNode from# O3 ~7 k( J5 [- k/ z" ^
being flooded with full block reports that use up all the RPC handler, l8 M% i; b3 k0 i. x3 V9 m5 w8 j
threads. This number should never be more than the number of RPC handler
& J3 ?2 t0 v' i: Z+ t6 b1 ^# c threads or less than 1.
: ~! \/ S% H/ \- U9 D' T </description>
* z- A6 x6 k3 D: E+ t" S</property>" Q3 \1 q4 t8 d& h( `+ F. g3 c* K
<property>2 Z! Z, }6 a$ ]# y. X8 V
<name>dfs.namenode.full.block.report.lease.length.ms</name>/ r. }5 E2 J9 z% \ P9 N9 O, y
<value>300000</value>9 K. D0 \% \4 P" z
<description>+ _) q6 H; L: i& O4 I7 u/ A; F
The number of milliseconds that the NameNode will wait before invalidating
7 v, Z+ A: ~+ c: |5 R a full block report lease. This prevents a crashed DataNode from
: {5 q- B7 J. n! V permanently using up a full block report lease.9 H9 P8 U: H$ q7 n. \5 ~6 e# J1 A
</description>
/ P. e- q3 u E k# }5 M) R</property>
0 }5 {( O4 \( R/ K<property>: L/ h, v" r9 h
<name>dfs.datanode.directoryscan.interval</name>- ?8 T. a1 x8 X
<value>21600s</value>
) Y% R. A+ l* Q2 v1 q4 \ <description>Interval in seconds for Datanode to scan data directories and
% i! B: M) \. Y reconcile the difference between blocks in memory and on the disk.
' t, A8 X" R7 P5 c B; O5 c5 H Support multiple time unit suffix(case insensitive), as described
. {' O. F" \) r4 H& ]' ^1 n in dfs.heartbeat.interval.6 [) j% I3 a- B. Z8 k I
</description>
% W8 o" p) T# q</property>
% |. [9 C5 m$ D) [9 F4 m<property>6 ?* \' w1 K& }+ _( C
<name>dfs.datanode.directoryscan.threads</name># F" i* h1 X1 K0 P$ j
<value>1</value>5 G4 [6 g+ q! o, Q. N1 h' y
<description>How many threads should the threadpool used to compile reports) c( I7 ]! k* e
for volumes in parallel have.2 n* x+ u" ]" A9 R! c# {) k3 D
</description>6 I! S" Q2 J- L0 F( A" g, C" X
</property>; g' M8 I7 O0 E" d1 i G, w) g! I
<property>
9 n. h& R L% q" X; X& X1 Y <name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>
9 v! X! I) J7 I. _, C <value>1000</value>6 G, w- Q: m4 i0 w' Z9 R K7 V
<description>The report compilation threads are limited to only running for
1 g4 t. y0 ]* h" c& P, b a given number of milliseconds per second, as configured by the' _* f1 g+ C- i. B& n1 D) D; x( `$ d
property. The limit is taken per thread, not in aggregate, e.g. setting
' y( P3 W# {& d# S# f a limit of 100ms for 4 compiler threads will result in each thread being L% o5 `. c. S( v
limited to 100ms, not 25ms.
; n( j2 ]/ {( g! U0 Q Note that the throttle does not interrupt the report compiler threads, so the
7 z* V* p" Z3 w actual running time of the threads per second will typically be somewhat- t3 I2 t P% |& q2 \: H
higher than the throttle limit, usually by no more than 20%.
( a4 _5 a8 b) @* A; A Setting this limit to 1000 disables compiler thread throttling. Only- P2 [6 q, R6 r. k' F2 R( c, ?& w
values between 1 and 1000 are valid. Setting an invalid value will result0 n" L0 K n" z2 C1 E1 d8 Q R0 {
in the throttle being disabled and an error message being logged. 1000 is3 F+ s* _& X6 G0 a
the default setting.& |$ t3 M: |! A6 F7 V
</description>3 E% @; I3 c6 t A; e
</property>
1 `! H, c/ t1 g" g E, f9 E<property>
' Y' }( Y" a2 t8 ^: _7 `/ C5 b* B <name>dfs.heartbeat.interval</name>! @- k9 P+ @7 d8 m- {% K
<value>3s</value>5 Y( k, f7 D8 d$ E- o# ~! G
<description>6 H! [8 f$ W1 ?
Determines datanode heartbeat interval in seconds.8 E* o- Q- J' d: m$ k7 x
Can use the following suffix (case insensitive):
6 T) r8 C! I1 y( } K _ ms(millis), s(sec), m(min), h(hour), d(day)' B& c5 k2 e, G6 L# r3 A
to specify the time (such as 2s, 2m, 1h, etc.).2 O4 l- L1 a8 R S3 v! E
Or provide complete number in seconds (such as 30 for 30 seconds).
# ^. o! k% v5 M, v+ p4 j/ x </description>) y. P: ?2 t* e+ ^) D. S+ @
</property>, J1 k* _; Q5 X2 g9 T& d3 }
<property>
: T+ _& E! [$ c' A: P1 g" j9 ] <name>dfs.datanode.lifeline.interval.seconds</name>
8 c Q z+ B# n+ H3 b3 L, P <value></value>( k! g$ L, D) e2 `
<description>
; h, X/ R/ t) _" y7 T6 T' n Sets the interval in seconds between sending DataNode Lifeline Protocol
+ H" ~, |9 l' \7 l" b messages from the DataNode to the NameNode. The value must be greater than
( e8 I q) J1 O3 c6 ? the value of dfs.heartbeat.interval. If this property is not defined, then
2 ^8 M: @/ l+ I1 [+ e9 l/ t0 o the default behavior is to calculate the interval as 3x the value of4 }. P6 x/ d* S( p, n
dfs.heartbeat.interval. Note that normal heartbeat processing may cause the9 `! I$ l' g0 Q l6 X9 ?
DataNode to postpone sending lifeline messages if they are not required.
8 W+ t5 M" E) C* X4 u Under normal operations with speedy heartbeat processing, it is possible. u7 T! h5 j- _% z4 @
that no lifeline messages will need to be sent at all. This property has no; r& d2 j: ]: ^9 ], c! x/ E
effect if dfs.namenode.lifeline.rpc-address is not defined.# F3 |+ r6 S$ G7 x0 e
</description>) p0 A4 b( _; ~* X6 w
</property>
3 j1 k+ O% K+ h9 j, X* O6 d<property>
3 r7 f% a! o8 p4 f4 g% p <name>dfs.namenode.handler.count</name>
! {. X6 h' D- j% a# @! Q <value>10</value>
% ~! ?3 i6 M# }! x. L0 c7 ] <description>The number of Namenode RPC server threads that listen to+ G. W) ~- m( @
requests from clients.
+ E( h. M1 v& q4 R! D+ b# G If dfs.namenode.servicerpc-address is not configured then
7 V/ u7 v& k) \, S& e Namenode RPC server threads listen to requests from all nodes.
" x! j6 y- ?* t4 \( ? @ </description>8 q- _6 I" |- a, u, E
</property>5 l$ o; X0 L8 f n
<property># O. U' T& |3 A" W/ D
<name>dfs.namenode.service.handler.count</name>
2 Z; [. j5 k$ m. w4 o) Y2 T <value>10</value> v+ ?" l6 @) B8 E% ?
<description>The number of Namenode RPC server threads that listen to
# U. {9 F* o* ~' x requests from DataNodes and from all other non-client nodes.2 _. s# z: c; Y! e+ x8 W% h
dfs.namenode.service.handler.count will be valid only if# a2 N. V: n K& W! M* L+ ?
dfs.namenode.servicerpc-address is configured.* p* N' T! S1 _% G- j4 Y0 M# A2 i
</description>( N, u0 r2 u2 R( H* U3 Y j+ b+ Y
</property>5 D/ h, v6 B; x& G3 s$ `$ A
<property>
z$ _: Q( f; b# ]( T6 P <name>dfs.namenode.lifeline.handler.ratio</name>
! s& s# r. |( h( T' Q& H6 b <value>0.10</value>
' `3 l/ D) `+ }3 h& [: b3 q8 P) v <description>
4 ^' @7 _ M2 d A ratio applied to the value of dfs.namenode.handler.count, which then
. Z6 B7 H, h; S$ p$ P) J" P provides the number of RPC server threads the NameNode runs for handling the4 @; `4 ?/ }! m: ?4 \
lifeline RPC server. For example, if dfs.namenode.handler.count is 100, and5 k, R2 y0 {; g" Z* T8 e
dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts
2 z1 I5 w/ i( e) d6 J. R 100 * 0.10 = 10 threads for handling the lifeline RPC server. It is common5 F( \% ?3 k( z9 G9 N# O, t# [
to tune the value of dfs.namenode.handler.count as a function of the number3 f% o/ I+ T R' l, y1 O) s
of DataNodes in a cluster. Using this property allows for the lifeline RPC
( @. ]2 L9 a0 e5 @, P2 C) B4 j/ o* @/ R! A server handler threads to be tuned automatically without needing to touch a$ q# `' f7 ~" N: l4 N: u
separate property. Lifeline message processing is lightweight, so it is' N& X. |9 |3 _( l ]" I
expected to require many fewer threads than the main NameNode RPC server.
% x7 S4 G ~: S This property is not used if dfs.namenode.lifeline.handler.count is defined,
1 o7 z7 a( k/ n) e2 [: m! ? which sets an absolute thread count. This property has no effect if! B G, H; x5 K# x2 C7 _; U
dfs.namenode.lifeline.rpc-address is not defined.) h4 m& a$ k0 [ G4 _+ `+ G; t1 ?
</description>
* p& z8 i8 v7 k" \( Z* o</property>
7 w$ i4 f Y) I) E" h. P/ G+ |$ X/ r6 t<property>' O9 G; j n2 K0 x8 F6 Z+ E
<name>dfs.namenode.lifeline.handler.count</name>' U+ @' }* }$ E% Q( ~ X
<value></value>9 s/ I5 t) k% y, N
<description>+ n2 {+ p c0 Q& u3 m
Sets an absolute number of RPC server threads the NameNode runs for handling* M7 z' N2 L+ A, G, R- p
the DataNode Lifeline Protocol and HA health check requests from ZKFC. If
" p8 x6 I+ Q" n this property is defined, then it overrides the behavior of- N; q' [8 H" t9 d
dfs.namenode.lifeline.handler.ratio. By default, it is not defined. This) O I" F' S1 Z
property has no effect if dfs.namenode.lifeline.rpc-address is not defined.
( U# H2 t' g# L) ?: i7 Y& x( P </description>
3 t A, `* h! G' U% {/ `6 o</property>
1 w* v2 m+ [. F/ V8 Z" }<property>
: ~ G2 P# k( q" a! d <name>dfs.namenode.safemode.threshold-pct</name>( {4 P" B2 G/ T4 ^! C$ Q
<value>0.999f</value>
3 Y Q& e1 ?6 H; a <description>, E- d+ u0 x3 m( j$ p
Specifies the percentage of blocks that should satisfy
3 e( a9 }7 G/ ]! V. t7 N the minimal replication requirement defined by dfs.namenode.replication.min.
% ]' ^* W- P. I: ] h! K- I Values less than or equal to 0 mean not to wait for any particular# c5 g5 m9 X; Q" u- i# A# `
percentage of blocks before exiting safemode.
. U1 d9 Y* T6 o# C. o Values greater than 1 will make safe mode permanent.
! o) \ b* u _, p% R4 j </description>+ R2 f- j- k8 t; \# X9 n
</property>& \# j8 t1 G8 [7 \8 f
<property>
" [2 R8 f+ Y0 r <name>dfs.namenode.safemode.min.datanodes</name>
% X- {0 A# `' u <value>0</value>, L* K1 Q; p; Z8 \( \2 ?! B. S0 f
<description>
: [- B c8 z0 ~ q. x Specifies the number of datanodes that must be considered alive
, Y; u' a5 L, o* A& d before the name node exits safemode.0 f3 T( v9 ]( v$ C# s0 Q
Values less than or equal to 0 mean not to take the number of live4 u! d$ _+ Q: g
datanodes into account when deciding whether to remain in safe mode4 ?2 w7 o6 D( W T0 Y7 S2 s
during startup.
( s' V; ^( r( Q# N# V( `3 @ Values greater than the number of datanodes in the cluster7 X5 {. ^+ E6 F, z5 K5 W* ?3 i
will make safe mode permanent.
# N3 A$ C4 H& ?/ j1 W( [# h3 q </description>. Z) f% A* ?2 M8 a( A: Q3 ^
</property>
( L; H7 s) i- K% f6 F$ }<property>
; z8 n, m+ x4 q i X7 Q <name>dfs.namenode.safemode.extension</name>3 H E) U- Y6 |! H" u' C
<value>30000</value>
; ?% \: y1 ^% g5 ~4 W <description># L3 y6 W/ P+ \
Determines extension of safe mode in milliseconds after the threshold level0 v+ B( v+ s/ X+ n" R6 o
is reached. Support multiple time unit suffix (case insensitive), as9 J6 x8 e/ f, ~$ c: I4 j( t/ f* \
described in dfs.heartbeat.interval.1 J! T0 a) x# p. Q* m
</description>
6 ?& s* ^8 u) q7 v</property># Q) m; g( v+ E7 ^5 D, u: d
<property>7 M# }. `! s$ }5 }3 c- B/ Q
<name>dfs.namenode.resource.check.interval</name>
: I j* Q0 g& u& M& j <value>5000</value>: _6 e, ?% d! s% A/ z2 [( j$ v
<description>
9 P' T# _; P" a9 {+ M- z- N3 b% c The interval in milliseconds at which the NameNode resource checker runs.. D9 J& C- J" W5 ?6 Q" P' @" \& O
The checker calculates the number of the NameNode storage volumes whose
4 Y% s1 D! _6 h' p available spaces are more than dfs.namenode.resource.du.reserved, and: h; x; \8 H% O( t( k( N1 a
enters safemode if the number becomes lower than the minimum value
6 G- B' B( p" l; k specified by dfs.namenode.resource.checked.volumes.minimum.
% l' V8 `- k1 M; T/ A q i$ C </description>8 p$ J) q" e' E$ U$ K
</property>; w* j- I% P8 h* R
<property>2 `' t A4 C$ F; g0 s
<name>dfs.namenode.resource.du.reserved</name>
% F$ e5 O3 g/ H- W( ] <value>104857600</value>' t/ f4 M- d" }% A/ b" W
<description>
+ B: M0 a3 Y* e( b# C The amount of space to reserve/require for a NameNode storage directory- S9 @ H5 b. P, T/ e
in bytes. The default is 100MB.
( @8 t5 e: I$ s0 N </description>, }% R. U. n/ p/ L
</property>
% G/ y: t2 i' v* c% J# r<property>: t) Z2 q7 }! A# [
<name>dfs.namenode.resource.checked.volumes</name>& q& [+ t. r2 _5 c, u
<value></value>4 D3 H- X+ ]0 r7 U" T" o
<description>
8 z4 x8 F; d5 e" \( P A list of local directories for the NameNode resource checker to check in
# { x5 U/ P( B2 x! X, {1 D9 ^ addition to the local edits directories.
1 |" o( E M$ p; W </description>' N/ w1 f7 T! Z& @. q7 t7 p4 I
</property>
( c9 F6 f, w9 S# v% h2 N& T* U<property>
, Y+ B0 [# c6 Y <name>dfs.namenode.resource.checked.volumes.minimum</name>+ P4 c$ F9 O$ n) h Y" F
<value>1</value>* H6 t: Y; V$ X1 k1 O+ O
<description>9 P( h* i0 m. i" R
The minimum number of redundant NameNode storage volumes required.. S+ @3 E& p4 o
</description>& \0 Z4 N }' a& t4 C9 l
</property>
$ s2 A5 G9 O$ k<property>
`! R: Q1 u T' O, A8 b <name>dfs.datanode.balance.bandwidthPerSec</name>. ~& G) g9 q" k# \/ B6 B
<value>10m</value>8 f, F5 d2 G& ^- _# G
<description>
; N) x$ P0 H' E% R; R Specifies the maximum amount of bandwidth that each datanode* W! T! w6 W1 E; B) s, e+ j- q
can utilize for the balancing purpose in term of
2 G4 M( l6 z, O R2 s/ s$ i the number of bytes per second. You can use the following
$ m$ t* o8 P! D( l' d suffix (case insensitive):( @0 j' N( n, t5 f
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size
' T5 L# M& P" U- ~1 Y( D (such as 128k, 512m, 1g, etc.).
L3 v8 j6 Z! q8 d) p5 |: w- U Or provide complete size in bytes (such as 134217728 for 128 MB).
" Q! b7 {* f# e# i </description>
9 u1 W( g, _% }! j3 }7 o( c9 F</property>
1 L% Q8 z3 q7 d# m<property>
. b% W" _3 ~+ y, [% W5 H1 n! b <name>dfs.hosts</name>& s7 n1 H' a4 \" W5 S3 G
<value></value>: k. X" F. j6 [' p) T6 P" Y
<description>Names a file that contains a list of hosts that are6 W5 ?9 S6 h( @2 z* H; C
permitted to connect to the namenode. The full pathname of the file' }$ [4 U" H: o! N
must be specified. If the value is empty, all hosts are- v% A/ S, v4 b/ C
permitted.</description>
% [% ?! L0 Q4 g! r3 `! }</property>
! S3 K7 y; j& \3 ]2 a<property>
4 y. P l) |# K8 N# ]5 j <name>dfs.hosts.exclude</name>9 t* @* E* h* j, g$ u
<value></value>
* X y0 O$ N' ~4 S6 f <description>Names a file that contains a list of hosts that are1 F! c# n: H' J- J: \- Z
not permitted to connect to the namenode. The full pathname of the( x) C; S m+ \8 E9 e
file must be specified. If the value is empty, no hosts are. c ?4 ]/ t3 n t/ x2 I" }+ p$ ^
excluded.</description>
. @' `7 V8 x w% ^3 ~# a</property>
. B6 j2 [% c; x, q: @/ v- t7 i<property>
# O }$ @; l4 p! u. w <name>dfs.namenode.max.objects</name>
6 t9 J7 |* q, j <value>0</value>) J/ y/ A9 b' _3 @) G- ?; u! M7 @
<description>The maximum number of files, directories and blocks: t/ h) [) J- X$ V) L6 A7 E5 f3 P
dfs supports. A value of zero indicates no limit to the number+ Z9 i5 l5 q& p* d
of objects that dfs supports.- O) N/ N; X& Y" H- p
</description>
) R- r9 a/ J6 M</property>* K* }4 r/ _6 a9 O2 G5 p; n
<property>
7 Y+ T8 y+ i$ `) L+ s <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
2 J) T' w5 L8 t( m <value>true</value>
1 ]' p4 _, p( Y8 [$ |+ J+ q R <description>7 P0 F! ?1 e4 `
If true (the default), then the namenode requires that a connecting
- k3 U/ i6 Z: {: r datanode's address must be resolved to a hostname. If necessary, a reverse
+ r! z' Y9 [4 ]3 B8 w& h) N DNS lookup is performed. All attempts to register a datanode from an3 E" L( _" G$ V, h# A* h
unresolvable address are rejected.% e4 }+ x0 Q. e4 j1 c7 ~
It is recommended that this setting be left on to prevent accidental
8 X6 @& n( ]4 ]+ F$ {. n5 r5 }4 w$ _ registration of datanodes listed by hostname in the excludes file during a$ h, x3 t3 W5 c
DNS outage. Only set this to false in environments where there is no' p9 `1 y3 X% z: r# G3 T5 Q, v( |
infrastructure to support reverse DNS lookup.8 Z5 Z* F, F- Z1 t" l* L! l) D
</description>4 m7 A( P. X+ ~2 j
</property>
0 ?9 z3 \$ P+ u/ W; ~+ A<property>. q+ ~/ S6 ?5 i' T: @: j
<name>dfs.namenode.decommission.interval</name>
; q- m" i! d( ?7 ] <value>30s</value>5 h1 n [" C' T$ a2 B ^
<description>Namenode periodicity in seconds to check if# c# H5 [9 q- ?. a, U- ~% u
decommission or maintenance is complete. Support multiple time unit
0 _, m" q; [, J suffix(case insensitive), as described in dfs.heartbeat.interval.9 F* G2 I/ V3 e- ?1 Y# \; m0 w% d1 H
</description>5 e; C, ~. C6 D( D
</property>% j* {# E2 }, y) T$ Y4 v( D
<property>
5 K1 M7 f8 I- C( s' Y <name>dfs.namenode.decommission.blocks.per.interval</name>6 i$ W7 h4 s/ S
<value>500000</value>
3 K3 `2 Y" I/ u" }% G& X N. q <description>The approximate number of blocks to process per decommission: t9 x9 y n9 n' M: [0 ~+ v- u
or maintenance interval, as defined in dfs.namenode.decommission.interval.
( U5 a! u7 V1 j2 S7 H </description>+ _& T- y% B ^8 U: H% X
</property>
, `6 V2 [% a- X6 x! v" x7 M; E/ ]<property>
1 u6 n, X' U1 Y2 o/ H0 K4 P1 E4 ~1 A <name>dfs.namenode.decommission.max.concurrent.tracked.nodes</name>! ^9 @+ N+ {$ ^5 B) [, O! W
<value>100</value>
9 ~' E' J. w* a <description>
6 v! w# b d7 q5 {2 a& L The maximum number of decommission-in-progress or" H: f& g8 [7 e( I- X
entering-maintenance datanodes nodes that will be tracked at one time by R6 W0 H( ]( ~2 \6 s, k
the namenode. Tracking these datanode consumes additional NN memory
+ T4 [6 ~+ _' Z- d proportional to the number of blocks on the datnode. Having a conservative
# k ~3 ?9 p, p* \: v X limit reduces the potential impact of decommissioning or maintenance of: g# ?- o4 M# a6 A; W
a large number of nodes at once.
; l1 p0 ]" G9 w% O6 H) k A value of 0 means no limit will be enforced.
, U$ p' A; s8 J* J/ [' C& z) | </description># y) k2 v2 L. q, o/ C
</property>
) o5 [, `% ^. }) [<property>
& x& y, n! `7 G' _ <name>dfs.namenode.redundancy.interval.seconds</name>) f0 d5 a' N) \9 L# ^+ z6 o
<value>3s</value>
( v I. |1 f# }, l c <description>The periodicity in seconds with which the namenode computes $ H5 @! R5 S9 k9 O3 U9 \# t9 Z
low redundancy work for datanodes. Support multiple time unit suffix(case insensitive),7 p, M) Q; {. G% M
as described in dfs.heartbeat.interval.
" ?$ \2 v F# K </description>
, `5 a* M0 [3 V: v8 I& Z</property>
, S0 q8 x: @! j8 U) _/ T3 P/ ^# y7 t) V<property>+ H3 v2 @$ d! v2 z, r
<name>dfs.namenode.accesstime.precision</name>4 m7 Y6 U; k$ G2 \ o* x% ?
<value>3600000</value>
& Z9 p5 p% G9 Q; ~% I <description>The access time for HDFS file is precise upto this value.
. s3 O+ J( j. t The default value is 1 hour. Setting a value of 0 disables1 X6 A0 o/ z9 n# F |( F( R* z
access times for HDFS.
. d- W& J- F$ h </description>* @5 k2 V( u' P: R. }) I! o9 Z
</property> M. o7 y# d. [8 W3 Q* v. L
<property>
t, k' l2 x* h" l3 M <name>dfs.datanode.plugins</name>, `' Q8 I4 U5 P! H+ D1 A
<value></value>
, J7 t7 h6 Z C2 m <description>Comma-separated list of datanode plug-ins to be activated.
+ D1 l; k7 K& V. V5 \, p </description>8 C% [$ m8 n# T+ F4 h. A/ z0 | l
</property>3 d4 H0 f! i8 l& `5 f
<property>
( G3 I# W. x, v1 } <name>dfs.namenode.plugins</name>
+ r- `9 H5 j0 b! K+ i, x <value></value>
, p/ E3 F3 k& c; k6 u <description>Comma-separated list of namenode plug-ins to be activated.
/ t& O. ]) l" V/ m8 N& O4 A( g </description>
- M% K" @' Y p</property>5 }3 e1 s. b" C0 }% u1 m) `
<property>
' ^+ y' @' t! `5 F) l <name>dfs.namenode.block-placement-policy.default.prefer-local-node</name>
0 f9 `9 \: J8 I6 h; @6 G& I, W2 `: L <value>true</value>
* `8 V6 a- N+ @! r0 V; b <description>Controls how the default block placement policy places
6 U$ G! `7 y7 @' n" O the first replica of a block. When true, it will prefer the node where
2 @5 P6 c$ f+ `5 o the client is running. When false, it will prefer a node in the same rack
. C" }' D/ n" Q; q# `7 I; y as the client. Setting to false avoids situations where entire copies of
5 B9 @) ^0 s, [# H' A g5 s Z large files end up on a single node, thus creating hotspots.$ N% B# z, T7 {8 X/ l
</description>
' f! G( J- {6 }) ~. {- Q, O+ |</property>
9 U3 Z1 l1 x3 w9 p$ G+ O<property>. ^+ a/ H- |, W$ S+ a$ @7 Y5 U
<name>dfs.stream-buffer-size</name>" Z; ?; t6 d& [1 ~6 |2 }' k; N
<value>4096</value>8 Q8 ?4 n7 d# p$ N9 v; o
<description>The size of buffer to stream files.
) G: _" a i# S The size of this buffer should probably be a multiple of hardware3 [$ E. k$ b: _1 J
page size (4096 on Intel x86), and it determines how much data is2 V z# t& v6 m* U
buffered during read and write operations.</description>, ?; m0 ^! ~ v8 C+ {' F
</property>/ A# d" C8 u" W6 }! s7 M5 [# }$ l3 I
<property>
- l, ]+ u& J% C4 D. w) Q: O <name>dfs.bytes-per-checksum</name>/ f9 F6 N" X0 Q$ |+ I h" w
<value>512</value>
7 X' H6 A( I( h+ p) e0 i& t <description>The number of bytes per checksum. Must not be larger than
% A/ b Z# k( o' {* f8 D dfs.stream-buffer-size</description>
, R2 q0 g: P2 e* E7 a</property>
% j2 H+ N& p# m4 x. P. h2 w<property>
/ e: o3 ^6 p6 B" |1 U: K" C6 ?: _8 r: p <name>dfs.client-write-packet-size</name>7 @# ^; G. M' K* p
<value>65536</value>9 ~3 I" l+ b. o4 T+ G
<description>Packet size for clients to write</description>
J/ H8 h) S2 L; k' V* Y8 G</property>
0 h' J' \: i+ z- k- y" o<property>
( P @3 O6 R& n8 A6 D% \ } <name>dfs.client.write.exclude.nodes.cache.expiry.interval.millis</name>, X* x( f- h# W
<value>600000</value>- m5 k9 \1 B/ C- M2 k9 a0 h
<description>The maximum period to keep a DN in the excluded nodes list+ Z# D/ [, s4 c# b. L
at a client. After this period, in milliseconds, the previously excluded node(s) will" G" H. H, e& r* v8 i3 c
be removed automatically from the cache and will be considered good for block allocations
7 s- m, [( F2 f$ s8 H# h again. Useful to lower or raise in situations where you keep a file open for very long/ l: C- M, c; z, z6 J
periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance
. ^" k" W' j/ a! j* x restarts. Defaults to 10 minutes.</description>9 f; B" w6 t; M
</property>, T3 v$ D8 h$ H4 W+ `9 I
<property>6 Z1 t/ C) R+ c' M
<name>dfs.namenode.checkpoint.dir</name>7 N4 k" R. R2 M0 L! b$ ]
<value>file://${hadoop.tmp.dir}/dfs/namesecondary</value> ^0 o+ w, E5 ^ L
<description>Determines where on the local filesystem the DFS secondary
) q, J t& i5 }) Z/ Z2 ~3 z5 t name node should store the temporary images to merge.& K8 v/ _8 c- c
If this is a comma-delimited list of directories then the image is
/ T. i1 A5 ^6 U5 D. I replicated in all of the directories for redundancy.' N y- x# M9 K' l. |
</description>5 C) L% v5 f9 m/ [ l* V; f% _
</property>& A# C f8 \ n
<property>
1 m2 y# D8 |6 V+ n) V4 V5 s& A* q <name>dfs.namenode.checkpoint.edits.dir</name>
- g3 C j( p# C0 R" b7 H+ b. ]% q Q <value>${dfs.namenode.checkpoint.dir}</value>; J# J. W/ M7 r
<description>Determines where on the local filesystem the DFS secondary: `* ?2 @! N' `( y' W/ `
name node should store the temporary edits to merge./ A( t7 J' ? S
If this is a comma-delimited list of directories then the edits is: H' \, A j0 _& b C$ k4 B4 _
replicated in all of the directories for redundancy.( t# l7 F: w3 ]! f% s
Default value is same as dfs.namenode.checkpoint.dir- m+ ^& Q- Q4 a1 q9 _. d# d$ h9 W
</description>8 m' ^7 Y, N% I
</property>
1 d# A. }" h8 G9 Z<property>: h8 ^) V( c7 X9 k
<name>dfs.namenode.checkpoint.period</name>6 m0 p# ^7 z6 K4 L' Q1 H
<value>3600s</value>
2 u/ B+ j$ X6 q' F( @ <description>3 n/ t) ?+ J0 M6 r6 w
The number of seconds between two periodic checkpoints.
7 R! w1 }; b8 a% ~$ ?1 x Support multiple time unit suffix(case insensitive), as described
5 l) h, C/ a4 @5 ?4 v0 z) M in dfs.heartbeat.interval.
* }2 ]- E% e/ l </description>
E/ {: i+ t/ U8 g6 G</property>
! i0 k7 b5 Y' \0 C<property>
' _% Y3 O4 f- Q/ M7 m <name>dfs.namenode.checkpoint.txns</name>9 ~4 Q$ R2 s( ?8 c: H6 o; ~- e
<value>1000000</value>
7 Y2 u; I7 D! Q6 l; Z( R <description>The Secondary NameNode or CheckpointNode will create a checkpoint
$ y" x/ Y5 E0 r% v of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless
4 T3 i9 O& A$ n7 G6 t, R. h0 E of whether 'dfs.namenode.checkpoint.period' has expired.
' T& f1 X8 k+ |: e) Q </description>
$ |9 E' a: l( h) t- @</property>: W( k; ?' ]% w, A1 [) S# u* r
<property>
, x: V& e5 d8 {0 B0 O; l! z <name>dfs.namenode.checkpoint.check.period</name>* z9 l5 u4 \4 E
<value>60s</value>
! }7 E9 V: }# z3 ]# L" i. w. v <description>The SecondaryNameNode and CheckpointNode will poll the NameNode/ K% [, b- r& k. b' z2 i
every 'dfs.namenode.checkpoint.check.period' seconds to query the number9 |2 h% ]4 Q2 \2 j8 ^/ s
of uncheckpointed transactions. Support multiple time unit suffix(case insensitive),
& o3 r, r# F5 J as described in dfs.heartbeat.interval.. `2 a# ?3 I* z$ C1 V
</description>5 y4 c" m! q. M1 v) N# d: ?
</property>
7 q" D$ h! { d( p& [1 S4 k<property>! c$ K5 X' p9 [8 e) L7 c
<name>dfs.namenode.checkpoint.max-retries</name>
6 u0 P+ o% Y5 _9 _8 s <value>3</value>
2 @: {/ A3 D+ R9 \% F6 @% } <description>The SecondaryNameNode retries failed checkpointing. If the 8 M7 k4 u0 v9 x0 n l+ w- o6 A
failure occurs while loading fsimage or replaying edits, the number of7 d+ i' v4 I- p' }6 T8 W7 C7 V: o
retries is limited by this variable. D7 Z8 i$ q! s" q t
</description>1 Y% Y/ w9 d6 r2 h1 R/ i
</property>
# }/ }+ M0 G" d, D, t y: s<property>
U1 c+ z. y4 R- ~0 k9 I <name>dfs.namenode.checkpoint.check.quiet-multiplier</name>
% e5 s) [; t. }$ Z6 r4 q <value>1.5</value>) J+ u0 y3 x+ r
<description>
# }3 W5 B1 ?! a2 ~ Used to calculate the amount of time between retries when in the 'quiet' period
3 T: F: }. W: u9 r. A9 [5 o/ x for creating checkpoints (active namenode already has an up-to-date image from another1 t& s0 e' r1 |# |
checkpointer), so we wait a multiplier of the dfs.namenode.checkpoint.check.period before$ R4 J+ W# x" r6 I! d1 s
retrying the checkpoint because another node likely is already managing the checkpoints,6 n) M* k" U9 S. s) v0 x. E
allowing us to save bandwidth to transfer checkpoints that don't need to be used.
3 d6 R1 f- c' D+ s+ t </description>
# m9 g! c! m+ }0 }7 h</property>$ K4 z9 j) v9 k, S
<property>9 @ Q V* i$ j, B2 }
<name>dfs.namenode.num.checkpoints.retained</name>
9 ~! A- F+ H% }: U <value>2</value>% j& m0 `( T0 N% a! }! p: A2 X
<description>The number of image checkpoint files (fsimage_*) that will be retained by
1 I9 z6 s% f+ Y5 I$ U$ s6 `* K, }2 S" ` the NameNode and Secondary NameNode in their storage directories. All edit
9 ~, \2 b' f" R+ E; F logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
' h5 |/ B. X; E checkpoint will also be retained.
/ |& g5 Q+ x' S2 I7 v4 F: [/ g </description>: f3 _$ T% J3 c' m
</property>
9 `3 c+ U6 b+ w D5 q, @ w: Q& j<property>6 z: i3 L# N1 \# {0 Z; I
<name>dfs.namenode.num.extra.edits.retained</name>
! w& {. L9 r; w <value>1000000</value>& o0 J- P! Z& V) t+ m+ V
<description>The number of extra transactions which should be retained
! f4 c& {4 w: K- Q- A! | beyond what is minimally necessary for a NN restart.( v; S+ n% I# }9 v
It does not translate directly to file's age, or the number of files kept,
' P _+ `! v1 W; P! [, G but to the number of transactions (here "edits" means transactions).* F- K8 x* \: d; z+ `! m
One edit file may contain several transactions (edits).2 n, @/ P7 l0 ?5 u" o3 Q
During checkpoint, NameNode will identify the total number of edits to retain as extra by
0 s: A/ q& c% x1 c checking the latest checkpoint transaction value, subtracted by the value of this property./ Y4 R& i# a4 d! x2 _/ N" x
Then, it scans edits files to identify the older ones that don't include the computed range of3 q$ J% r+ ]4 {* m! B
retained transactions that are to be kept around, and purges them subsequently.
% q' P3 W) K) J1 @ The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have e1 W' w9 r; E, o, s
been offline for some time and need to have a longer backlog of retained. K; H/ y% J$ N3 M
edits in order to start again.0 H1 B3 o; O M. k2 y
Typically each edit is on the order of a few hundred bytes, so the default
& s( _! t! O/ H7 A. p' H of 1 million edits should be on the order of hundreds of MBs or low GBs.7 E! V9 h: C- m9 G
NOTE: Fewer extra edits may be retained than value specified for this setting
4 N. N% S+ N) W if doing so would mean that more segments would be retained than the number
5 D( w9 w% E: w. q configured by dfs.namenode.max.extra.edits.segments.retained.5 b: q0 c F5 f3 B' x" j
</description>5 U% S# t& l- K4 ~; y$ N
</property>
2 ^8 D3 d) {0 o<property>
9 E1 U( b2 J" |' j x <name>dfs.namenode.max.extra.edits.segments.retained</name>
! ?& a, s3 S% [# d0 i9 ?: y <value>10000</value>9 s j/ ?2 e1 k) o6 s
<description>The maximum number of extra edit log segments which should be retained
3 ]8 o# n' D4 }( I/ |/ A beyond what is minimally necessary for a NN restart. When used in conjunction with
9 |) N& x G9 ^& ? dfs.namenode.num.extra.edits.retained, this configuration property serves to cap6 a" @, Y/ O, m& ~+ F/ O
the number of extra edits files to a reasonable value., f- K( l9 Q; n4 M1 F; L3 r- `
</description>3 s% g0 {# Y* l" Q: e, D- X
</property>
- ~0 D( K% a; {/ s) j, E7 u7 B<property>
7 k6 a6 ^) [2 a2 h6 u <name>dfs.namenode.delegation.key.update-interval</name>, n1 I8 R9 m2 [+ b0 w
<value>86400000</value>& ]% `# m# G$ V* e! B
<description>The update interval for master key for delegation tokens & m# e% l8 h* D+ F+ U3 s4 l
in the namenode in milliseconds.( L. z. ~# D, r* ]
</description>
. E0 q/ ?9 N5 W; e+ |</property>
2 e' D& p& P1 U0 o# |) v; }<property>
& l/ l5 X! I) V/ j <name>dfs.namenode.delegation.token.max-lifetime</name>
1 A6 }2 m5 r% @' L& Y( e: V t <value>604800000</value>
& _* p! ~: q) ^+ m5 F <description>The maximum lifetime in milliseconds for which a delegation 5 a( t8 w" m* [! P
token is valid.
- ?5 r# Y+ @' { </description>( ^/ X& q1 W7 V1 X! W
</property>+ _* f8 A& \% v8 D' O) J" W
<property>" Y* c$ q+ t) J8 g. [3 W: X( M
<name>dfs.namenode.delegation.token.renew-interval</name>5 c0 L" W* r/ M
<value>86400000</value>
- F# S v5 l2 v <description>The renewal interval for delegation token in milliseconds.( t! _# x. H+ w# ?; E
</description>
$ s# o0 \" ]# F; ]( q. r2 R</property>
8 c5 g) k( @+ g2 v8 l' y<property>
+ S1 o; B% U0 J! I/ a+ I <name>dfs.datanode.failed.volumes.tolerated</name>
! o& L+ ?' a, _$ h& O! j6 V <value>0</value>! I$ n9 H3 p3 W9 q/ U9 j, {, ~
<description>The number of volumes that are allowed to9 V5 T% @* Y" y* l# O4 Q8 Q- }
fail before a datanode stops offering service. By default ?; a, J; }* ]5 _6 u) E4 M
any volume failure will cause a datanode to shutdown." [6 e0 q% k' o4 l4 E
</description>: a, [1 L( l' _, o. P
</property>
" b! b! o8 z- h o<property>9 K0 u! i8 p1 Z; Q; L4 d! ?7 R/ l
<name>dfs.image.compress</name>
( I8 l2 R! J- N <value>false</value>
?5 ^+ X3 F' @' a; V <description>Should the dfs image be compressed?
, f; G/ W$ l" }1 i8 G </description>
3 k: @* s7 c2 {& W</property>
$ H6 M4 T$ B% I9 F4 {- f<property>4 G2 @( T4 Q1 R) B6 k/ ^
<name>dfs.image.compression.codec</name>
# ?, e/ j( L" l0 h& } <value>org.apache.hadoop.io.compress.DefaultCodec</value>
- _$ N |7 f- _! _/ D <description>If the dfs image is compressed, how should they be compressed?
: o7 s# H$ m/ V d$ `7 I This has to be a codec defined in io.compression.codecs.: ]5 P0 q5 b# J7 I( B
</description>
0 ^: P0 l7 P, `6 U; y. D* g; L2 `</property>; a, {/ `+ R0 r" _
<property>
. c# S2 l. [& j$ v* a <name>dfs.image.transfer.timeout</name>2 H& S0 }& v1 ]( W; M
<value>60000</value> p+ u9 v$ Z3 S6 a+ R4 A8 {
<description>
% t& _- A$ K @ Socket timeout for the HttpURLConnection instance used in the image
6 `2 B5 ^8 W6 |* I, ]; j, } transfer. This is measured in milliseconds.
; s8 z* Q+ }7 { This timeout prevents client hangs if the connection is idle
" Z! E! `2 S, F' A, v for this configured timeout, during image transfer." n4 ]) J: W" h. i% m6 }
</description>0 @1 y- j5 @7 |3 k# r6 W
</property>
3 | {: }" c) j3 P( X<property>
: {7 U1 n: x, L4 p) s <name>dfs.image.transfer.bandwidthPerSec</name>, ?0 I0 m; n- V0 b
<value>0</value>/ R* x" ]6 ~# H3 n/ Y* r* I
<description>/ I2 d& S) O o( |4 a- g
Maximum bandwidth used for regular image transfers (instead of
& b7 \4 c! ~7 _6 x8 g bootstrapping the standby namenode), in bytes per second.0 W. W% `$ F' _' \. Z
This can help keep normal namenode operations responsive during
( U# p3 M4 ]7 {# ^; j1 Q checkpointing.
; U/ c: r/ {# K0 T A default value of 0 indicates that throttling is disabled.
+ \1 k: J) z: b* x3 `: v4 m, w7 D The maximum bandwidth used for bootstrapping standby namenode is
3 U) V ?' Q& T* H" t configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.
5 V" [2 j# x3 a2 U </description>
7 E7 J" M d0 } R( |( Z5 G</property>: Z* r7 ^4 ?. H1 c. D7 S
<property>+ [4 ~, a) |0 Z
<name>dfs.image.transfer-bootstrap-standby.bandwidthPerSec</name>
7 `5 ^ n5 `! | <value>0</value>
; p# O4 `" v- H# i. E0 K* @& r* _ <description>
% F; v' Z8 j/ i! z: v9 z Maximum bandwidth used for transferring image to bootstrap standby+ K& ^$ `# ?, F- E* N7 n7 s. s
namenode, in bytes per second.
2 Z1 z% { F+ p- D. p A default value of 0 indicates that throttling is disabled. This default# B' _; S8 J2 u5 I. b1 p5 |# k5 C+ V
value should be used in most cases, to ensure timely HA operations.$ Z2 u3 b: L) c6 S1 ]
The maximum bandwidth used for regular image transfers is configured, c3 g2 L9 r) F, G* ^2 E
with dfs.image.transfer.bandwidthPerSec.
/ ^+ P" b0 n2 p) Z, h: U& K </description>
0 Q+ t- {8 x" g: L8 G8 k </property>
" k% O) J4 B) Q2 J9 Q! Q! _' y<property>
" U% `3 N3 ^. F- ?; o <name>dfs.image.transfer.chunksize</name>
" m: T9 `1 C G) C <value>65536</value>, _0 c J! w; |2 w5 P
<description>
. P% ?- i0 U4 }8 V- w" B; ?3 g2 U Chunksize in bytes to upload the checkpoint./ z' I6 |1 E; e5 V6 |
Chunked streaming is used to avoid internal buffering of contents* V+ m0 A$ a5 ]) K, c8 K$ M$ K; \
of image file of huge size.
/ g" R+ Q. _! j$ m6 i </description>4 B) B( T( ?; h. N; E
</property>
) L2 O2 g8 [7 g<property>
6 F8 e/ b6 P* U* O: U+ V- P <name>dfs.edit.log.transfer.timeout</name>
! ]; m# R; W5 F& i Y <value>30000</value>% S3 w! W' T5 V; a. M( a
<description>
; `6 L, X! | k' Z- C6 ~. N" h Socket timeout for edit log transfer in milliseconds. This timeout, l8 L' y0 P/ m/ ?1 V/ ]% k0 Q
should be configured such that normal edit log transfer for journal
2 ^8 k8 g5 X2 m Q node syncing can complete successfully.
' [; w! Z& S- m% z( n </description>. X& q' k+ l0 L; x
</property>8 s; s) w/ J/ _' v9 M
<property>/ n: a/ s3 i7 E8 B* A4 @4 O$ ~0 u
<name>dfs.edit.log.transfer.bandwidthPerSec</name>
: k/ X; L$ E) Z& T* p; p: } <value>0</value>( j. b% C) Q) [% x; X
<description>6 _; t( i' ?% _9 w+ P
Maximum bandwidth used for transferring edit log to between journal nodes
) ^: U( V1 l, z$ n" Q8 } for syncing, in bytes per second.
: h% Y8 S6 H' B$ ^) _+ w A default value of 0 indicates that throttling is disabled.* t" H( x+ }2 ~0 V$ `0 O& Z: p
</description>( _! F6 e3 e& A8 c6 |
</property>
- @7 [ ^" @0 f8 }<property>
+ [6 N6 u% v1 b6 o" R <name>dfs.namenode.support.allow.format</name>
' e2 r% M0 y' f' `7 M7 p <value>true</value>5 U% Z. f. Y8 a; d
<description>Does HDFS namenode allow itself to be formatted?% z, A! {5 J \9 o3 D, \
You may consider setting this to false for any production8 |' x" m* s3 ? Z
cluster, to avoid any possibility of formatting a running DFS.% u* o1 {5 |7 g
</description>
. O. T% V( u. `7 N</property>0 h6 \- T9 G8 n
<property>7 [- P3 p5 o6 w+ A, y0 ]' k
<name>dfs.datanode.max.transfer.threads</name> n3 E$ D+ ?- A- s0 l
<value>4096</value>
8 j$ I% ?* e- {; i1 ? <description>
( M$ h9 }/ S- F o Specifies the maximum number of threads to use for transferring data
! U2 D8 O. h2 s" z$ k0 L% p& J9 r in and out of the DN.4 Z% b+ j3 }) n
</description>% B& e% E* g8 {! F' {
</property>
% T" \2 G5 a' X. `<property>
" l8 W5 N+ l1 E' n' `4 {" T1 z <name>dfs.datanode.scan.period.hours</name>
+ Q8 m$ V3 x$ k; f3 ^ <value>504</value>
# n0 |* u- w. P" U <description>! P& b' ^/ ~& r: x% \$ ~) v) P
If this is positive, the DataNode will not scan any
- G2 t- f; U; a7 J individual block more than once in the specified scan period.
& I7 c4 Q' r9 c9 N" X* h If this is negative, the block scanner is disabled.' q: H4 V# |' ^5 Y
If this is set to zero, then the default value of 504 hours
! t4 L9 s# P! `5 N: z* r or 3 weeks is used. Prior versions of HDFS incorrectly documented( x$ N7 w* j: R2 w! Z7 x1 w
that setting this key to zero will disable the block scanner.
0 R" ]4 {# Z9 Z+ c7 X </description>4 f* t7 A( p% b% W7 z# S m% Y
</property>& }' G( D& Q6 r% v4 g
<property>
5 r6 P1 g' @3 ]4 T, ` <name>dfs.block.scanner.volume.bytes.per.second</name>
& A- \; ^6 r4 O4 M3 ~ <value>1048576</value>/ [3 n0 q0 S, m( m8 c
<description>
" j3 N3 T, l) b If this is 0, the DataNode's block scanner will be disabled. If this
$ c, Z+ U# c7 x1 a/ X- c is positive, this is the number of bytes per second that the DataNode's, g1 z( F5 l8 M. \" g
block scanner will try to scan from each volume., q2 `7 G5 p. H9 d& ?8 g
</description>
/ j# f3 F& r( v' h- U</property>
$ Z( I: j7 w2 t) S3 p" V4 M$ s<property>
& R6 j: O3 T( m2 @ <name>dfs.datanode.readahead.bytes</name># a# u) V$ \! _ o" j
<value>4194304</value>: U; m2 ~* f. W
<description>
& d$ J) j9 y1 |" [' I While reading block files, if the Hadoop native libraries are available,
0 H% c7 V4 C# B3 w6 w% w the datanode can use the posix_fadvise system call to explicitly" i1 h! ?9 R" E& [1 o( e
page data into the operating system buffer cache ahead of the current
9 _9 M$ U* Z9 }8 r reader's position. This can improve performance especially when
3 Q4 ~4 L+ f b& T disks are highly contended.
?) L8 l% w) m1 L+ Q0 p This configuration specifies the number of bytes ahead of the current( e+ ~+ Z" d% K# h7 m
read position which the datanode will attempt to read ahead. This
1 {+ p7 B O( n0 o4 p# c feature may be disabled by configuring this property to 0.
; c! s: _) g8 r/ o* p' L) y8 @ If the native libraries are not available, this configuration has no
) v6 f. Y( W' t" o effect.' \: B2 N2 l' \+ [9 W- J$ `1 }1 [
</description>
4 w! M) ?: e: K; q/ {7 B</property>
( j- c' T% I$ m' M" S5 ~<property>
) H8 J' d) c0 H: e4 I <name>dfs.datanode.drop.cache.behind.reads</name>
# ^( t- t& u, M9 N9 I" A7 s% F <value>false</value>
" h+ u2 A( X3 N S/ j6 r' x' Z7 d <description>
1 c7 [4 W5 F7 P$ w5 Q& @$ _; O0 x/ y In some workloads, the data read from HDFS is known to be significantly7 t5 _0 P( @. w( T$ X9 w/ v6 ^
large enough that it is unlikely to be useful to cache it in the
' n% G# a2 u" U8 Z7 V$ b operating system buffer cache. In this case, the DataNode may be* E- R; Q# Q% u9 P4 n& r5 Y Z
configured to automatically purge all data from the buffer cache% l& [+ E" y+ |" ^
after it is delivered to the client. This behavior is automatically
/ r9 O% ^8 x/ Q* B$ n- P( x disabled for workloads which read only short sections of a block8 a6 U* l1 T( [1 w* ?! }% P' P
(e.g HBase random-IO workloads).7 j' j; o# O, f* s
This may improve performance for some workloads by freeing buffer
6 A; Y+ N. ]( ^ cache space usage for more cacheable data.
! c5 N/ a& C N0 O I. w4 s- ^ If the Hadoop native libraries are not available, this configuration
- E4 m, u- a! }+ a- B3 c has no effect.
6 J1 w7 M$ {% d </description>
. y# q# Z% ?" y</property>3 r: U; m0 E4 }
<property>% @+ g" k$ P! A! c( I
<name>dfs.datanode.drop.cache.behind.writes</name>
8 Y7 | e% V0 Q <value>false</value>
4 O( [2 b: m! G# U7 M8 J* J$ V <description>
" Y' E4 i; x1 ]0 e8 K In some workloads, the data written to HDFS is known to be significantly
0 W: ^; [$ k3 H0 b$ n& t/ s large enough that it is unlikely to be useful to cache it in the
' n( T, V$ H8 s operating system buffer cache. In this case, the DataNode may be
0 }) L; o, Q7 G+ q% G configured to automatically purge all data from the buffer cache9 o7 _# z$ A8 C9 o' S
after it is written to disk.
9 ?8 H" `3 J! Q3 E( i; A; k7 M3 j This may improve performance for some workloads by freeing buffer
* s o- c& N$ q* W2 U cache space usage for more cacheable data.7 j" @( ?, }5 V. c) `
If the Hadoop native libraries are not available, this configuration
O8 f' n: o/ F4 {( I* k has no effect.- S" `. e: M1 t, [
</description>: o9 l9 O3 p! K0 N
</property>4 y4 [ u) D# H7 n( _$ P' S
<property>3 V' c, L) b$ u* q$ `
<name>dfs.datanode.sync.behind.writes</name>% a0 Q7 r7 ^6 g7 G* m
<value>false</value>
6 D1 a s9 H5 f1 m4 D* @ <description> r$ K* l: x. B2 i1 A. G& F
If this configuration is enabled, the datanode will instruct the
7 `; m( q9 K/ j* V) w, P% b+ H operating system to enqueue all written data to the disk immediately
H. d( U6 T1 Z9 J i4 W after it is written. This differs from the usual OS policy which* a, H; ~/ o: v. K/ s0 S
may wait for up to 30 seconds before triggering writeback.8 f4 |3 k' N& T9 ^1 N
This may improve performance for some workloads by smoothing the
7 \+ i* @; L) v2 b! e4 n IO profile for data written to disk.9 ?" B6 x+ r7 s( e
If the Hadoop native libraries are not available, this configuration
; M! G7 Z4 f Z2 y, k8 F4 [3 ^ has no effect.* K5 @3 h) |& ?
</description>
( t% i; h& _3 H0 c8 g</property>/ j( v3 y3 n+ `: Z1 y& ^- u2 S' f
<property>; Q! W$ ^: ? ~! T3 ?
<name>dfs.client.failover.max.attempts</name>3 e8 B: B4 p( k F2 ]( V" m
<value>15</value>
. \% s% r3 C& X% e+ X <description>) o: M7 c3 v7 o5 ^* p4 @
Expert only. The number of client failover attempts that should be
$ n+ R# y0 h& n0 V# R made before the failover is considered failed.8 ^7 ~1 L1 l* l; x# s
</description>& z) g/ F% W' U$ O8 O
</property>( }- X) `. r3 t' q
<property>
. q- Q Y0 n/ O- S/ s <name>dfs.client.failover.sleep.base.millis</name>3 I7 h7 L/ Z$ h* E- T# j( q
<value>500</value>7 e8 P# r! C" ?- E U' G4 I9 ?
<description>0 [7 h1 \# ?3 r8 r7 `1 A
Expert only. The time to wait, in milliseconds, between failover
( ?# w6 t5 U; M( K. Z0 X attempts increases exponentially as a function of the number of
; q8 J+ _4 N; N- @ attempts made so far, with a random factor of +/- 50%. This option
5 v# D" x* {) N V, G- m6 `' v" x specifies the base value used in the failover calculation. The
5 \" J. K6 Y# y: L P, v first failover will retry immediately. The 2nd failover attempt. J( n$ x' c& J" W. k2 d& D
will delay at least dfs.client.failover.sleep.base.millis- t$ |+ z% s. P
milliseconds. And so on.
. p9 Q6 v* h2 Y/ a1 k5 p4 e0 T. b5 e </description>
8 Y- O% Y E0 H# ]) y</property>* T8 ?5 M2 _- x# J. x
<property>! k# r7 R$ ~: t; a. X
<name>dfs.client.failover.sleep.max.millis</name>
) j# S( o3 B3 P& | <value>15000</value>
6 w9 ], I- z4 ~. G <description>
( _9 ?! X$ h! _) S( R% Y! R+ U Expert only. The time to wait, in milliseconds, between failover
' L* F# a( k8 }/ Y attempts increases exponentially as a function of the number of4 ]7 _0 K7 k3 j! l5 }6 D' \2 G$ o% r
attempts made so far, with a random factor of +/- 50%. This option9 a7 }* C, B9 e( j2 k- A! [) U( G
specifies the maximum value to wait between failovers. 5 E" c5 s" k9 _, I d; Q/ }; H
Specifically, the time between two failover attempts will not8 S# O) z0 G' W5 D
exceed +/- 50% of dfs.client.failover.sleep.max.millis
! A9 Q- o+ ~% Z1 O v) } milliseconds.
; l/ K' B; u7 ~ </description>
! a. V6 y# \" H& j* H) \. l</property>! s8 o4 l2 @3 {* v
<property>! ?0 r* S* e/ ^0 Q2 `/ n @
<name>dfs.client.failover.connection.retries</name>4 H: ? d$ j I) S$ U9 c; R
<value>0</value>0 ?7 r$ @9 ~$ [6 e6 B9 c
<description>1 B6 B0 g( l! L6 ^( ]. O
Expert only. Indicates the number of retries a failover IPC client+ \( n0 V# D' v; U
will make to establish a server connection.- U9 r, x, l; e( F8 [5 q& W$ v& {
</description>; U, s6 P& Y9 f% W7 B* Y- X
</property>4 e! H% o) z0 J& K
<property>) Y1 g* b i, U6 T3 l4 M
<name>dfs.client.failover.connection.retries.on.timeouts</name>4 \! O$ ^0 {+ q! G
<value>0</value>
+ C: h7 D- ]$ q/ d: a <description>3 v8 _$ S9 u, a$ r
Expert only. The number of retry attempts a failover IPC client
. [3 I. u; g" T' G7 ^6 d will make on socket timeout when establishing a server connection.
9 B0 K' f/ A+ W& |! t$ c </description>. |. x% Z) r" x! x+ E7 j3 [& f1 o
</property>3 ?& p: K% I7 u" C' L8 z
<property>8 O2 p2 q' y3 F' }: v6 b, D
<name>dfs.client.datanode-restart.timeout</name>
1 a+ n6 D) I% ]! M6 A4 S <value>30s</value>$ B5 q8 W0 r9 _2 d
<description>
j! |0 x1 Q: M" q Expert only. The time to wait, in seconds, from reception of an* Q; I; i; u P. _0 ~/ t3 O! j
datanode shutdown notification for quick restart, until declaring
2 [- I& L: E& A1 W( m, e the datanode dead and invoking the normal recovery mechanisms.
( ?* ^! M8 J1 ^5 x7 Y, n) Q The notification is sent by a datanode when it is being shutdown
* |. A) K2 A1 t" N, `1 C using the shutdownDatanode admin command with the upgrade option., X8 X8 H, u4 ^+ U* S
Support multiple time unit suffix(case insensitive), as described
# F( h: h4 X& k in dfs.heartbeat.interval.
( @6 J g3 s3 W& E </description>
% F6 e/ D6 W6 w% g# R</property>7 z2 [: l# Y- c- n, R
<property>4 G* x. q4 n+ v$ v! F0 \
<name>dfs.nameservices</name>5 M/ |: o6 H# N8 j5 L5 J% O. u
<value></value>
! n/ q% b& N% C3 a) P" T <description>$ N; c9 U- j% D! |3 y* y6 c
Comma-separated list of nameservices.
$ I3 F1 Y/ ]( u4 _/ X </description>! I+ N" a. ?1 R0 m5 u, {0 G- _6 F) k* r
</property>7 f0 v, I+ y, W" I
<property>/ w% f; a1 S# T) l8 ~
<name>dfs.nameservice.id</name>
8 q, o2 O$ L& F+ B& G <value></value>
7 t& r3 u$ F1 W$ u, {2 \6 O <description>" E$ k0 V' w3 P9 I6 Z- {2 a3 }
The ID of this nameservice. If the nameservice ID is not
( P# r R d: Q* u configured or more than one nameservice is configured for
, b/ }( f; S4 D- g! G dfs.nameservices it is determined automatically by- O' F+ i( l& L4 w$ `% U
matching the local node's address with the configured address.) T+ U9 E/ s( Z
</description>3 |3 x2 ^; y* i5 l6 p
</property>" t3 ^( z7 N& ]3 t* @+ s/ R
<property>
$ H" j7 e# ]9 X' K U2 Z1 O# [ <name>dfs.internal.nameservices</name>
- O0 d1 x5 g$ i8 O3 j <value></value>
* A! K h+ h/ a! \ <description>
6 v0 O# p8 p0 h9 ]8 h& g0 S+ R Comma-separated list of nameservices that belong to this cluster.
8 T3 B' X# [" t Q Datanode will report to all the nameservices in this list. By default
' A' b" F* @& M, `: z! s* Q this is set to the value of dfs.nameservices.
4 I( v Y2 S) I9 K- R( I </description>6 Z4 O, v s( t3 F7 r8 G' \
</property>8 v; P0 d: f/ f* x4 y: O/ ~1 |
<property>
4 J% S" q$ | n* _ <name>dfs.ha.namenodes.EXAMPLENAMESERVICE</name>* }9 s" F2 [: _8 D1 k
<value></value>6 l1 ?0 l6 C0 l2 I ^0 G4 K8 l
<description>
4 A" G) F" C) {& h7 }& m, H The prefix for a given nameservice, contains a comma-separated" t: p0 O0 P- X4 C% L" @
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).- }8 d% ^* d" A0 q
Unique identifiers for each NameNode in the nameservice, delimited by( @$ e1 x3 t( O9 Q I7 C
commas. This will be used by DataNodes to determine all the NameNodes/ K* k5 k; k3 O/ p
in the cluster. For example, if you used “mycluster” as the nameservice* {4 X/ z3 W3 ^& ~2 Y" r
ID previously, and you wanted to use “nn1” and “nn2” as the individual m, H% x! ~7 O, C0 j% z7 |( i! i
IDs of the NameNodes, you would configure a property
3 c) D& J7 o. Y+ ] dfs.ha.namenodes.mycluster, and its value "nn1,nn2".$ S5 O* ]- [; b' j" z7 S5 t# q
</description>
" m L6 n) u9 L* g q</property>
: K ?5 o6 F/ n9 S- q9 Q) ]<property>8 s* j& u3 L) Q. \' a" d9 S
<name>dfs.ha.namenode.id</name>3 w8 Z. i/ }2 {: ^
<value></value>% V# D1 g" t- [# ] V
<description>- a7 t# M7 W) b1 j
The ID of this namenode. If the namenode ID is not configured it0 B) a8 ^9 R" ~" f
is determined automatically by matching the local node's address5 X2 c1 L2 G. Q8 q
with the configured address.& e& ^4 l; K3 e6 I/ L; V) D/ o
</description>
3 K& S, ~# T: i) O6 S+ K</property>
! O5 _3 h9 b' I7 s2 w( G) t<property>
/ p( Z5 t1 d8 f4 p) I" ?7 H <name>dfs.ha.log-roll.period</name>/ y: L6 M4 F& Q
<value>120s</value>( W/ B ^8 |/ o
<description>& G7 r2 X/ _( _% A3 d$ R
How often, in seconds, the StandbyNode should ask the active to( _1 Q3 o6 C( i. C1 Q
roll edit logs. Since the StandbyNode only reads from finalized; t" T6 a1 d# A- ^7 W l% H) h# ^
log segments, the StandbyNode will only be as up-to-date as how% @3 d6 A, e4 \' H' i [- f
often the logs are rolled. Note that failover triggers a log roll
% ]0 u0 g2 }, K0 a so the StandbyNode will be up to date before it becomes active.
3 G: b0 B- D7 g! |* j Support multiple time unit suffix(case insensitive), as described C9 A' x/ |' z. @& I
in dfs.heartbeat.interval.
! o9 y% I, a% C+ y) | </description>9 O H, S2 f3 |8 h" c/ [2 k, z
</property>$ s4 \+ d( y+ E( {% p- T
<property>
5 S. M4 O e5 b <name>dfs.ha.tail-edits.period</name>& e6 {5 Y* t1 q3 ]; M3 [# O
<value>60s</value>8 i6 G2 e, R1 u/ A2 |6 V
<description>+ `5 t1 y; H3 L8 j) z
How often, in seconds, the StandbyNode should check for new
A- h$ F/ \% h2 m finalized log segments in the shared edits log.4 l8 S* G; T3 E/ ~* q5 I9 w
Support multiple time unit suffix(case insensitive), as described$ g$ }0 l3 I5 c* o
in dfs.heartbeat.interval.; Q' Q) ^; S( ~% N; }( J0 P
</description>- P/ E. a5 E! m8 _
</property>8 i- ?2 l" q+ f8 b; X
<property>
5 a n' T0 d+ t1 U+ ^! I <name>dfs.ha.tail-edits.namenode-retries</name>6 ]: @; f: {+ n- h; O- `" s
<value>3</value>: c2 Z- Z5 O8 L+ K
<description>1 I3 E9 i h! P1 q& K
Number of retries to use when contacting the namenode when tailing the log.
! m( \: x* V; _7 Z </description>
. C; }! k9 z, _5 v; @6 p</property>
* U% z* |1 ~! p<property>
0 _) Y' Q; u6 O4 M: z0 `8 M <name>dfs.ha.tail-edits.rolledits.timeout</name>
7 g6 [7 K: e& N7 E <value>60</value>) S6 f9 f+ l$ z4 T
<description>The timeout in seconds of calling rollEdits RPC on Active NN.8 n* R: }: V) e, ~ {/ L
</description>
( L7 \" A9 B* m% Q1 O3 D</property>
* i0 \, g0 e" Q0 I& J' B3 k O<property>3 f$ o( z3 M4 X. J8 t
<name>dfs.ha.automatic-failover.enabled</name>
9 C4 u G/ s2 v* d5 r8 ^ <value>false</value>* b' ^' s6 W: X" u- y" M2 {% Y
<description>, b5 i) v1 [; j. o+ y7 R4 \5 E
Whether automatic failover is enabled. See the HDFS High
7 h2 L4 U3 g+ h* ^ [9 @ Availability documentation for details on automatic HA J- W) y6 M! F! D$ }7 a% h
configuration.2 s/ i- p& v$ I/ Q: U
</description>
; K, A- I1 S/ ?" S7 k- W2 {) q</property>
( J, \& K0 ^3 P# B6 E! F<property>
- U3 ?6 |6 E2 U+ B <name>dfs.client.use.datanode.hostname</name>
; i) g6 o' W5 f+ K# r( V <value>false</value> s0 s" C5 |( N- f& z" y0 j
<description>Whether clients should use datanode hostnames when- o2 V- f* c2 b n7 ~1 [9 F
connecting to datanodes.
9 t! l: a" u5 \2 m: N v6 v </description>
3 m8 g* A; g" z</property>+ S3 Q j3 o* E2 w2 G
<property>5 j" l. q- R* G, Q7 ?0 R
<name>dfs.datanode.use.datanode.hostname</name>
+ u1 s2 Z* B5 Q! T! A Q <value>false</value>
( L& Z2 ^% Q$ V; i( X <description>Whether datanodes should use datanode hostnames when; p4 g+ ?' _$ ]
connecting to other datanodes for data transfer.+ P$ e( G* K* R2 b
</description>
! y1 O6 Z- H. n4 \1 [</property>: T1 f$ D, Y; y; w
<property>+ H* ~$ B8 D1 i" a# E- _
<name>dfs.client.local.interfaces</name>: c! Z: f2 @% w
<value></value>0 y* S+ D( k4 n' O, ?& w
<description>A comma separated list of network interface names to use/ E$ e, p4 Y! D% t1 h% m
for data transfer between the client and datanodes. When creating
# q: o/ z8 c" o+ x; {$ b a connection to read from or write to a datanode, the client9 v7 J5 @4 G7 o* G/ F* ~6 C2 X8 T
chooses one of the specified interfaces at random and binds its
. v7 t4 B( ^8 ~5 l socket to the IP of that interface. Individual names may be
6 a! b. Z0 t+ D0 T$ e+ e specified as either an interface name (eg "eth0"), a subinterface* b7 Y$ D7 `4 A9 g
name (eg "eth0:0"), or an IP address (which may be specified using4 T' r1 K+ r0 [. Y+ A5 ? S% n* @
CIDR notation to match a range of IPs).
! x7 p+ E$ U: x4 T# E2 n2 ~. z </description>1 X2 `0 U# [$ p# m6 b3 ^
</property>
6 q$ d4 N, @5 m. {' S<property>/ x, |7 s. Y" M0 |' V
<name>dfs.datanode.shared.file.descriptor.paths</name>7 d4 ^; B7 G; s0 T- t5 g! E
<value>/dev/shm,/tmp</value>
# w2 z" [/ y$ o, I( \ <description>
4 o+ K/ g9 j# @; t A comma-separated list of paths to use when creating file descriptors that2 D6 ^3 s0 a% z' y4 p
will be shared between the DataNode and the DFSClient. Typically we use6 M6 e6 r& s) a |7 u2 ^
/dev/shm, so that the file descriptors will not be written to disk.
5 X j9 U/ o i+ n; p3 V) S7 ? Systems that don't have /dev/shm will fall back to /tmp by default.
# Q0 M) W1 l7 _& c& d' E </description>/ F! _: ~: T& D8 m {9 w. ]
</property>
7 ~; w O- a6 o9 U<property>9 i2 |9 A0 i2 E0 R) h
<name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name># S5 P) P1 d) F
<value>60000</value>
4 q- J. f0 G3 o, F, K1 K' i <description>4 g6 a( t- Z& d- ?) ]4 I8 f2 [
The length of time in milliseconds that the short-circuit shared memory
- |$ b$ e Z+ Z i1 O- E+ A watcher will go between checking for java interruptions sent from other
3 L1 _$ f9 @/ v: N: V$ C n6 @ threads. This is provided mainly for unit tests.' b/ L, J4 V+ }
</description>* d3 T' \6 H8 D
</property>% l2 V; i* m$ A l; u
<property>% A, s" }9 _) t
<name>dfs.namenode.kerberos.principal</name>1 Y: T, y) N/ d2 i$ m( o% f
<value></value>
$ N- W# d0 i q" d: ~! V0 G <description>" f+ b$ E P+ z" E( o* x
The NameNode service principal. This is typically set to5 n% q. ?* |: f. y' e3 H
nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its) w0 p0 q: U2 F8 D+ D
own fully qualified hostname at startup. The _HOST placeholder* P: [6 }3 Z, v7 g: e; _$ \+ ~3 X, j0 w
allows using the same configuration setting on both NameNodes
6 j* P, l, |8 T! J! k/ d in an HA setup.% ?1 v5 d. ?+ P
</description>
+ K( O( g+ P- E0 _9 e</property>* [# }* ^2 [/ w6 ?5 Z: g. u
<property>
, n8 i7 n) U( j <name>dfs.namenode.keytab.file</name>1 a4 m! ]6 O( C7 c# k2 `. ? O
<value></value>
4 Y: ?$ e6 |. r. M% O t <description>
: M7 M$ \/ m, W% [! z* i& l# d8 p The keytab file used by each NameNode daemon to login as its
V R% Q4 A! H# u. ^/ v service principal. The principal name is configured with3 F/ L. @) @+ r. C# F
dfs.namenode.kerberos.principal.5 G* c3 c/ h' V
</description>) [ d& \6 @% `7 D9 f0 S
</property>% g7 Z0 k( t* D% k5 p( X& @
<property>
( ]" e9 Z0 c8 B1 U2 P# t <name>dfs.datanode.kerberos.principal</name>
( P7 L$ {/ c+ n( e. z <value></value>
& B& C0 m$ }& ~7 _0 B <description>& W6 e A3 \6 n* Y& X( {0 G
The DataNode service principal. This is typically set to, [7 T! S1 f$ f. T: k8 q- N
dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its6 _6 B% P. y- n; X. o
own fully qualified hostname at startup. The _HOST placeholder
% a2 z( C A; l7 l) p1 @' O allows using the same configuration setting on all DataNodes.
4 K9 U x r% ]/ {; N" P </description>& B0 [6 l1 ?1 a
</property>( ?+ [5 o2 p2 ]( T r
<property>
3 ^. u+ e6 g5 j; c <name>dfs.datanode.keytab.file</name>
5 u; c6 y8 u+ Y5 N# E2 t) v) ]5 h7 g <value></value>
& t4 R# L# I1 ~: y; r0 K <description>$ m# \3 K9 ]" x3 M" {% F0 E0 X
The keytab file used by each DataNode daemon to login as its
$ {- N. I6 u+ { service principal. The principal name is configured with& z5 V" R! a) t) E* {" o! G p
dfs.datanode.kerberos.principal.
% n) c0 h/ F. Y" |- ~ </description>& s" C( _: Z; q: ]/ f1 j
</property>% p6 [! j8 n, K* |0 M
<property>5 ?' G( N# w' M& Y3 y7 {
<name>dfs.journalnode.kerberos.principal</name>+ c- b! B8 W1 ?5 {$ U" ]
<value></value>
, k! n$ G' Q- _0 V$ r0 g <description>
" o3 I9 T [2 t/ _7 ` The JournalNode service principal. This is typically set to/ O8 j* k. ^) j8 u
jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its
- a* u2 N W$ ~' i* ]% x2 _% w! U own fully qualified hostname at startup. The _HOST placeholder
, L/ v' I5 v4 k: _% C: K allows using the same configuration setting on all JournalNodes., U `. ~( Y8 x1 z
</description>
/ R% R) b% \+ o3 [* y' l7 Z</property>3 H% T* G9 `1 ~# C
<property>& x% r* e/ ? l7 u7 @; `9 Z
<name>dfs.journalnode.keytab.file</name>
& }6 C/ N2 ~% t7 } R <value></value>
3 R$ F1 g! @0 a* j- d <description>2 [# M I1 K5 ^4 [: F* c+ h9 |
The keytab file used by each JournalNode daemon to login as its
- n; ?) W7 L+ A0 ?1 Q service principal. The principal name is configured with
- |+ h4 L, N6 D G dfs.journalnode.kerberos.principal.# ?! i! P5 B* _3 Q- d( C2 W
</description>
# ^6 v3 C/ e: d2 _( c</property>4 q) `/ ? i3 @3 N" P1 O
<property>
5 g- ^' R/ W" w7 b <name>dfs.namenode.kerberos.internal.spnego.principal</name>2 y, v* ~' m5 E4 L4 Z4 n
<value>${dfs.web.authentication.kerberos.principal}</value>
* S6 n# ?& Y; f* C* x$ D <description>
+ v4 W4 X. h5 n, v, a: h The server principal used by the NameNode for web UI SPNEGO( M7 Y1 `. [/ y
authentication when Kerberos security is enabled. This is
9 K8 J) C* D- c; O4 v8 i- K8 p typically set to HTTP/_HOST@REALM.TLD The SPNEGO server principal
5 n+ \3 F+ S8 I y% ]9 B, F begins with the prefix HTTP/ by convention. Q. E6 g) L! Y' x, e; Q
If the value is '*', the web server will attempt to login with7 }7 r7 m3 R) ~& Y; f/ X" h, R4 t
every principal specified in the keytab file
( r8 `$ T9 V% E1 ]0 k dfs.web.authentication.kerberos.keytab.
% w" o5 z% @) a! l4 Z</description>$ h6 O0 p. A) b9 T2 K2 A7 P) [
</property>
/ e: v! `( U) i, d<property>
( S9 m6 v5 w y. c' Z <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
; e. i% \6 m0 `- Y <value></value>5 m- m: N+ z+ o$ D1 w& m' r
<description>
+ _# I& q( h/ {. l- E) T5 A- Q The server principal used by the JournalNode HTTP Server for( |3 d1 A8 m6 r
SPNEGO authentication when Kerberos security is enabled. This is
. B( n4 S$ |; a typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal
$ u3 r. w: A7 P9 b3 Y begins with the prefix HTTP/ by convention.! B% J2 `1 y1 ^# N
If the value is '*', the web server will attempt to login with# H" E" j" ~$ s$ S
every principal specified in the keytab file
% g0 A. C& g( u7 W0 r: x dfs.web.authentication.kerberos.keytab.
% g: J5 ]7 k: [ B% X For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}
" J* ]. u/ [% p0 R3 I; Q' t i.e use the value of dfs.web.authentication.kerberos.principal.$ g6 S) e4 b+ k+ H( T5 M5 ?
</description>
9 w( g- l' H& ]0 o</property>* \4 U; F+ |1 S- h# L
<property>
7 \8 v0 E( K1 U- H6 k <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
, ~, Y. s ~! x <value>${dfs.web.authentication.kerberos.principal}</value>
/ o! Q' O; C- s4 ?; \& T7 ^" M <description>
( r# h' Y% |0 s( \( d The server principal used by the Secondary NameNode for web UI SPNEGO
+ F# n: }3 i. V& D authentication when Kerberos security is enabled. Like all other6 Q& m: u- H3 |# a8 @
Secondary NameNode settings, it is ignored in an HA setup.! Q- z6 Q5 v; C) I2 n" f# M' [
If the value is '*', the web server will attempt to login with% f* C7 K. R4 X; T+ h7 z
every principal specified in the keytab file
& _1 l6 F4 Y* T: p c0 ~9 P dfs.web.authentication.kerberos.keytab." Q( ^% I5 o0 o1 j
</description>" E/ b0 G1 h# G& C4 `# F( D- L% M
</property>
6 Z9 F2 R: s4 M! w* X. t<property>; D1 m9 i0 u4 G; w8 p! p1 i9 w
<name>dfs.web.authentication.kerberos.principal</name>
, B. v& m4 \: U2 t <value></value>
D1 B) {6 ^! f5 ]. S <description>
% U9 }2 [ M) k" \- z. M The server principal used by the NameNode for WebHDFS SPNEGO
d; z: N6 _! O" U% m8 l. u authentication.8 E9 S2 }( _' M+ U# R
Required when WebHDFS and security are enabled. In most secure clusters this
6 f- \& E1 F- O8 R8 E setting is also used to specify the values for
0 B2 n: w5 U9 P5 s$ S- i3 t# m* c6 r dfs.namenode.kerberos.internal.spnego.principal and
6 l! O0 @; X0 _( d+ _' n6 \5 Y dfs.journalnode.kerberos.internal.spnego.principal.
$ c3 d4 q7 m/ w </description>
% I3 _% u2 H+ c3 v$ g! A% M: V</property>$ C# _% `0 S7 [# J7 ]" J
<property>
: J) v4 i8 z* N% N) { <name>dfs.web.authentication.kerberos.keytab</name>- h; Y6 b+ d( B# c" u
<value></value>
4 x3 ]) P7 s5 l' k; R4 ^' v5 q <description>* O" }' X3 I6 j9 d3 y* h
The keytab file for the principal corresponding to* q/ ?0 r" [1 u G: M4 L
dfs.web.authentication.kerberos.principal.
d$ R$ v2 x% J7 ]4 [ </description>
+ n, s6 Q8 Y3 s7 ~: |0 }- y3 O</property>
: |9 R' Y' K9 u<property>
4 B" I3 b; @0 }+ J3 l <name>dfs.namenode.kerberos.principal.pattern</name>
& j6 d3 f/ X- @# }3 w7 X <value>*</value>6 `6 ?2 _# @7 `! Y4 E5 E1 J G
<description>) I: N2 F# r# S2 C8 C- u9 I
A client-side RegEx that can be configured to control! N5 ~9 U! S; v7 _
allowed realms to authenticate with (useful in cross-realm env.)
# B6 U, I6 s4 D% b# a </description>- u/ ?6 f }: X2 [' D
</property>7 P( [8 M' C% u+ f7 m) _8 P
<property>
0 G) R9 u- B" A& h9 ?' Q <name>dfs.namenode.avoid.read.stale.datanode</name>
3 W7 H, T# O y- @ <value>false</value>5 a3 _" F, j' d
<description>+ ^/ ~, r2 n5 ~
Indicate whether or not to avoid reading from "stale" datanodes whose
' _' D; [4 @/ A! w heartbeat messages have not been received by the namenode 4 y" ~. M, z- D( p7 R
for more than a specified time interval. Stale datanodes will be
! i& G5 ^5 \# D: p7 T moved to the end of the node list returned for reading. See% g' a, d* m; G$ n n" P
dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
/ g/ g @2 C/ e C </description>
5 {3 ?( ^ h3 z2 }& m# \) T) P/ E L</property>
+ q( E) e& _" o$ {2 v$ X4 n' [<property>
' v* `* Z& Q/ q3 y0 U9 n0 l <name>dfs.namenode.avoid.write.stale.datanode</name>
% `7 N& \# ]4 Z P' T# z( E4 T <value>false</value>
. X& H) Q P1 |; E7 h <description>1 A# W+ y8 i* {! i" d i
Indicate whether or not to avoid writing to "stale" datanodes whose
P: U8 B- }1 p, Q* q heartbeat messages have not been received by the namenode 3 n$ H. G' i/ T- x/ j
for more than a specified time interval. Writes will avoid using
1 u2 t1 {" X, P( k( A stale datanodes unless more than a configured ratio 7 O( w% y8 J% I
(dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as + X* \* p/ B1 d V; l
stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting! y+ A' c& Q2 D* F6 D
for reads.) O8 [: S: B8 d, X4 h# }/ R
</description># h; ~ S5 |( a* G
</property>: r6 W. U, E: A) d$ E0 ?9 ~, R
<property>3 q- C9 \0 U7 T# N6 B
<name>dfs.namenode.stale.datanode.interval</name>
5 Y. X/ h8 z. x! @9 b. d _ <value>30000</value>; F9 L1 e8 `+ N% X3 ` a
<description>
; P5 S. Y- m. b$ Z0 Q$ A Default time interval in milliseconds for marking a datanode as "stale",# [% D2 o5 `7 q
i.e., if the namenode has not received heartbeat msg from a datanode for
4 e- ^, n% s# `( t more than this time interval, the datanode will be marked and treated
! V" u% z. \; Y+ z+ X% X as "stale" by default. The stale interval cannot be too small since 6 d5 `- X+ |2 Z5 W! m
otherwise this may cause too frequent change of stale states. 0 g& x3 _5 ~3 S: P) S
We thus set a minimum stale interval value (the default value is 3 times
5 w; Q7 r; z. u7 R: d of heartbeat interval) and guarantee that the stale interval cannot be less7 }: F& k8 j2 T( r( A6 p# h# A
than the minimum value. A stale data node is avoided during lease/block8 i' Q" E. R, \0 p; }
recovery. It can be conditionally avoided for reads (see& R$ Z `; Q" N$ e* C
dfs.namenode.avoid.read.stale.datanode) and for writes (see* M% ~2 o3 d8 `0 A0 |
dfs.namenode.avoid.write.stale.datanode).5 u$ M3 l% |5 A2 L' y6 A
</description>$ a, W0 `% |( Y$ c
</property> O V" W5 C* _( i% P8 C {
<property>0 h# W7 B3 z: l% J/ A1 P
<name>dfs.namenode.write.stale.datanode.ratio</name>* `$ u6 f# ^+ p* U, p
<value>0.5f</value>
" q8 }, j3 E+ V8 W2 a <description>
* ]" |3 N$ H$ }$ U7 k, M a! [ When the ratio of number stale datanodes to total datanodes marked R0 t; O; |' L* J6 N3 v" P
is greater than this ratio, stop avoiding writing to stale nodes so2 Z) }5 y- P! L% `; f- A' Z
as to prevent causing hotspots.# t8 d* ?0 u) S4 [. }# L2 x
</description>
& D# m3 ]4 f( s5 L" R</property>0 H0 o7 ]$ `; T6 C u
<property>
: Q5 y. }: o9 ^: t0 x <name>dfs.namenode.invalidate.work.pct.per.iteration</name>
8 q8 E- ~/ K, [: f4 C <value>0.32f</value>
4 X" F# W/ g: |7 J5 B$ P1 y <description>
( @2 |& G* Z) ]# Q2 |7 [* H- v *Note*: Advanced property. Change with caution.2 d6 S9 J! o) w$ M; b
This determines the percentage amount of block
* {, ~: D0 Z/ T- r3 N( O4 c5 k invalidations (deletes) to do over a single DN heartbeat/ r s! Y5 I } l' L2 R2 k7 N
deletion command. The final deletion count is determined by applying this# W# B6 k, s- K
percentage to the number of live nodes in the system.
4 v; t: g4 E5 m- L k2 i. @ The resultant number is the number of blocks from the deletion list
2 v$ _3 f/ W: R% Z% @6 \, U E chosen for proper invalidation over a single heartbeat of a single DN.
2 x3 W V- u8 P4 D& {( ~ Value should be a positive, non-zero percentage in float notation (X.Yf),3 S! e \- r+ K
with 1.0f meaning 100%.2 e0 k0 L; h/ a7 P* Y& B% @- S+ N; }
</description>
; P+ V3 \- r# L. {</property>7 e" D* r. X) Z6 W O/ Q& W6 S
<property>
! u1 o* v3 p- D/ z2 ]5 S! S <name>dfs.namenode.replication.work.multiplier.per.iteration</name>4 @, e$ |: w9 r* m$ X0 O" M9 l$ K
<value>2</value>9 n# Y% \+ V' I9 |9 i( e
<description>
C) b; c8 W. J* n9 a2 ^. e *Note*: Advanced property. Change with caution.
1 K$ }# O( `+ q( d# p This determines the total amount of block transfers to begin in
5 j* q1 V4 ` C' G- L. x7 t6 u parallel at a DN, for replication, when such a command list is being6 [( U5 P* u# F( A7 p
sent over a DN heartbeat by the NN. The actual number is obtained by/ r6 w, f+ q8 Q& q
multiplying this multiplier with the total number of live nodes in the
2 G5 C. j8 Q; U# x) ` cluster. The result number is the number of blocks to begin transfers# S+ e ]. x5 v
immediately for, per DN heartbeat. This number can be any positive,
: `, b2 e8 m; h- ?: i: v5 Q) c h2 w5 Z non-zero integer.
( F6 Q; M0 h: L- Q9 o6 _) h </description>
7 `) }% U5 V1 K8 I t</property>4 E# X1 i# G# v+ Z2 y X5 U
<property>$ t' Q+ a' s) _1 z
<name>nfs.server.port</name>
2 a( ~, I: K9 e: {0 c2 f0 Q <value>2049</value>: M5 Q7 t: R1 i% w, @0 W) h$ F; |
<description>7 z5 j6 L: r' b/ X, s
Specify the port number used by Hadoop NFS.+ N9 z( }" E3 w5 l& \' W; G
</description>
) W6 c9 ]. L# O; A$ G8 E</property>
& G9 Z0 v" T t: K& w<property>
. ~$ F/ f$ L5 E; x8 Z h/ ^9 U <name>nfs.mountd.port</name>
8 w8 g, U" t1 } _4 F7 j <value>4242</value>) A2 n. J& z: q0 k; p, l! D
<description>
9 E4 o$ o0 m5 T g+ [) c2 Q Specify the port number used by Hadoop mount daemon.
5 H& p9 {4 N v% I: f! B5 H </description>7 h2 p% R2 O' z/ m
</property>
% d; d7 C+ c3 H( W% U- V$ P4 |<property> 7 e: B* I: \5 m- O: P! s( K, U1 L: ~8 k
<name>nfs.dump.dir</name>
7 d5 r6 G, G, Q) |+ e' v <value>/tmp/.hdfs-nfs</value>
9 c7 A" U$ w; e3 }: n! O <description>5 h4 X" a- p* {2 y* F6 ?
This directory is used to temporarily save out-of-order writes before
; Y W- o; Z' J( \. K: u writing to HDFS. For each file, the out-of-order writes are dumped after, W2 P; U$ `1 {
they are accumulated to exceed certain threshold (e.g., 1MB) in memory.
! j: c q6 L6 F v One needs to make sure the directory has enough space.
' a: C# o8 ]9 W( z2 x2 Y# e) j1 w; l </description>
1 r7 ^3 `8 \- \3 ]6 ?1 U% D, x, g</property>+ X2 ~6 b, v5 p0 t# r) H0 e
<property># b! G$ j' P. c6 b' |; }
<name>nfs.rtmax</name>" A6 N; @" r% J5 D! k1 Y8 g
<value>1048576</value>
4 X5 C9 M5 R- g$ D( n' K. P <description>This is the maximum size in bytes of a READ request
& D) _. u# |6 X5 Y- }# m supported by the NFS gateway. If you change this, make sure you
3 ^- D" B& q1 ^( `2 j- M5 {: n" R/ \ also update the nfs mount's rsize(add rsize= # of bytes to the ! y0 \; A6 i# N( a! _6 P3 H: Q3 ~" N
mount directive).
2 Y1 t$ s8 s& f( w </description>
; @! G5 g. `# w/ Y</property>
# w& Q, \& X6 }- \4 |<property>; z# M9 g, I# N. q2 G/ t
<name>nfs.wtmax</name>
4 ^. h5 J" P3 c <value>1048576</value>
- J+ `, j( C8 M U$ _5 A <description>This is the maximum size in bytes of a WRITE request! d3 P2 P/ j! O' E8 D
supported by the NFS gateway. If you change this, make sure you0 X- _7 p* u# D& m) g# I8 k2 k3 B
also update the nfs mount's wsize(add wsize= # of bytes to the ' b9 q+ {( x* X3 p( t
mount directive).0 y) z3 F" A( d' L4 Y0 w$ X
</description>
" D- D" w$ ~2 k- d. B& k" V* L' [</property>. O9 D' w& b9 W7 ?8 R( Q" {
<property>
6 _0 x# a5 Z* l <name>nfs.keytab.file</name>% T$ J5 z t" R
<value></value>3 t, N4 L) w7 ]) e1 U) C) l
<description>/ f. l7 Z; r0 Y" H- p/ z
*Note*: Advanced property. Change with caution.$ k3 R# O3 n' ]* b1 i4 B
This is the path to the keytab file for the hdfs-nfs gateway.' Q& b+ T: ] v, W0 M4 i
This is required when the cluster is kerberized.8 A2 |( A" n3 n7 d" T/ |* f+ A
</description>9 {* f1 @; ]+ K% n4 i6 G
</property>
# @: V! b9 H ]4 J! u, [* l<property>
/ B2 |) B7 |% ]$ [# m- \ <name>nfs.kerberos.principal</name>
) t4 @1 F6 j$ u8 S" u <value></value>& E$ A2 G# l+ Y
<description>
r/ I5 M+ Y$ C *Note*: Advanced property. Change with caution.3 T% I! I, l, v' w5 k
This is the name of the kerberos principal. This is required when
- {- o( T) Z8 G- v$ m. O the cluster is kerberized.It must be of this format:
& p" P* i9 g e/ s nfs-gateway-user/nfs-gateway-host@kerberos-realm
# O- Z0 J5 \3 l P0 ?0 t7 K </description>" S5 u/ }: ?4 Y* b3 C4 m6 ]
</property>
) ~1 p9 V/ l6 b% d; h3 n# j<property>6 }% N" S' C# S3 |9 a3 k
<name>nfs.allow.insecure.ports</name>
8 `9 G8 W$ y" {8 p( { <value>true</value>5 w+ U, Z5 v* {+ h6 R& `
<description>
" l' T4 ^- |+ L& X6 V# x3 }" ` When set to false, client connections originating from unprivileged ports
6 i. s% s' N% \! b. S- ~ h (those above 1023) will be rejected. This is to ensure that clients) l6 ~3 E) M# o
connecting to this NFS Gateway must have had root privilege on the machine
9 m% n( K$ O2 D" B: W where they're connecting from.
9 J/ z3 V7 N- U2 e5 |% g </description>8 K }8 f/ T6 W% V* j
</property>
+ N1 L* a7 i* y2 i, a a<property>* t r$ `/ _) Y: H0 U
<name>hadoop.fuse.connection.timeout</name>
) e0 B: g3 g; z" n <value>300</value>/ k( d' a) f7 I! C3 s+ _# v
<description>
) S" {6 A0 ^# ]9 c- M The minimum number of seconds that we'll cache libhdfs connection objects9 P+ `$ a) s1 Y# V
in fuse_dfs. Lower values will result in lower memory consumption; higher
) Q; X& T# e: l$ c$ I values may speed up access by avoiding the overhead of creating new. a# X0 v' U v. z& D
connection objects. k$ I) X4 c4 \6 U7 Q* ^3 a
</description>
/ `. O% w; F; I; y! e</property>
( Y( a4 v* o% t) ~* I) N U2 t; G<property>
# O. q8 x% O( H5 ~# \) i! t <name>hadoop.fuse.timer.period</name>" p4 e" t$ b" A% X/ T1 _* d
<value>5</value>. Q% X) H& Q* z$ |) }. ~$ k) N7 ~. x
<description>6 a% W+ G7 t4 v# O
The number of seconds between cache expiry checks in fuse_dfs. Lower values# ?3 w/ R5 f3 B5 H: c+ x6 a
will result in fuse_dfs noticing changes to Kerberos ticket caches more! s! R. _: K4 x o9 Y
quickly.) `+ J" d; o( B
</description>
, W8 y' p4 x- p0 h4 ?- M9 z5 m</property>
* u5 L f+ e( e; Q<property>* P6 c3 j" g' \6 b" ?2 r; n2 \! G$ ~
<name>dfs.namenode.metrics.logger.period.seconds</name>
; ]' ^5 f. l* H: U) t <value>600</value>
6 T K, b1 v2 Z5 C8 \; @" A" V% X <description>
' m6 x5 ^- s. F9 {$ n7 g This setting controls how frequently the NameNode logs its metrics. The
& |* l) _9 m8 O6 S logging configuration must also define one or more appenders for, o4 U( {' m* L+ M' F$ ?( L( | X
NameNodeMetricsLog for the metrics to be logged.7 E8 \4 S$ \ h( v3 [/ |
NameNode metrics logging is disabled if this value is set to zero or
# r0 n4 \. H- R8 i. s0 V& r less than zero.
) H. l: A( ^9 _( a6 a' x! Z( G4 { </description>2 `5 |# I$ T* P- j2 O% T* I
</property>
# N1 }( L3 ^! g4 [6 d6 `) y<property>4 A& C7 C0 R, H' \6 D) S
<name>dfs.datanode.metrics.logger.period.seconds</name>" ~" w2 G. \4 D2 a# ~& `# Q
<value>600</value>
; l2 e e( ]3 E" e0 _! E <description>
1 K+ c( l' B* `. u' I" T% | This setting controls how frequently the DataNode logs its metrics. The$ L# D: f; b$ i- g! K
logging configuration must also define one or more appenders for' _: C7 e! x" R
DataNodeMetricsLog for the metrics to be logged.
/ \) ^1 m4 Y1 m: m DataNode metrics logging is disabled if this value is set to zero or
- R% ^9 T/ \5 |1 g M7 H less than zero.
+ r. E( `3 `% x2 a* y( | </description>
) v' @" ^7 J ` t) q2 Q</property>
& M3 A+ E! S" ~9 O4 b" B<property>" }6 P3 [0 b1 D1 F! g
<name>dfs.metrics.percentiles.intervals</name>) s; l+ I- k2 O6 j8 @$ ^
<value></value>2 {! Y9 d8 v' p# Y
<description>2 N- {; N4 v2 W7 f
Comma-delimited set of integers denoting the desired rollover intervals 1 k; ~: J1 @3 v8 {0 Q
(in seconds) for percentile latency metrics on the Namenode and Datanode.
4 d5 w$ n9 C4 y7 t By default, percentile latency metrics are disabled.. p4 c( g% W4 R- e+ `9 p |
</description>. Z- ?5 A9 p4 E7 @& f8 C/ p
</property>8 M4 \7 s5 Q) q# d. q
<property>
, n: Q8 g2 C3 X' E' ]% X <name>dfs.datanode.peer.stats.enabled</name>
6 H, r4 O8 x. c6 q0 @4 X% x <value>false</value>
8 \! z; V& P7 L% l) G8 r; V+ Y <description>" v/ o, K7 w$ z6 D& q' l$ J
A switch to turn on/off tracking DataNode peer statistics.
. p3 K: _0 @6 s6 S6 J2 D5 j* { </description>: c* _ B- h0 }0 S7 `
</property>
% C' \8 d8 m: _ J( s& L$ _<property>
7 s- Q5 x# L9 g& s' C <name>dfs.datanode.outliers.report.interval</name>
8 M2 I$ n1 m9 D( M; R& c5 C# `, ~ <value>30m</value>
% T& ]" P, e; g; A+ a <description>
' b7 c; s; u" ^; E% t This setting controls how frequently DataNodes will report their peer0 `4 _5 h2 s7 A, R. N) p
latencies to the NameNode via heartbeats. This setting supports
) [5 x( W' }6 _# j* N) v% T& [ multiple time unit suffixes as described in dfs.heartbeat.interval.4 C6 L1 g9 O$ v& f9 l1 u
If no suffix is specified then milliseconds is assumed.* l* z R, P. _0 E7 @7 ]" x
It is ignored if dfs.datanode.peer.stats.enabled is false.( n2 L R6 K" i/ B% V/ P% }& \: I
</description>
$ l$ P( l# ?% j' S! {. g</property>
: r# G8 p( Z# ], R$ W<property>
3 {6 A" g' Z5 h y6 m <name>dfs.datanode.fileio.profiling.sampling.percentage</name>
1 [. m9 \+ B u1 _; \9 b0 K2 H <value>0</value>
4 V" c/ t! Z- n! T <description>5 h9 r. q0 U# Q& w; i# ~' s
This setting controls the percentage of file I/O events which will be
3 H' {7 o F; l$ g- [% n profiled for DataNode disk statistics. The default value of 0 disables
3 {- L0 U0 G8 w7 ]% B disk statistics. Set to an integer value between 1 and 100 to enable disk
3 c3 c1 s; o3 N statistics.
8 i) f$ b* n: j9 I% i </description>- d9 v( J8 _: |8 O% X1 G6 G
</property>
. c K% L1 I1 I<property>
* A/ _! B: T, Q+ \$ a- {0 l# g <name>hadoop.user.group.metrics.percentiles.intervals</name>
9 E$ i5 M* B9 [5 w7 W <value></value>. n" t; l) Q$ _3 u% @
<description>9 v5 g3 H9 @0 F4 U9 }
A comma-separated list of the granularity in seconds for the metrics! j7 A) ]) s) t; J* w% l* ]
which describe the 50/75/90/95/99th percentile latency for group resolution5 _5 i+ l: T' x# q
in milliseconds.1 j& A$ @! B/ l7 Y5 I
By default, percentile latency metrics are disabled.! M/ D! @/ _& E0 e' c/ P" n+ q8 E
</description>
7 b3 o$ O1 B$ Q$ R</property>
% S" z- k, j$ X) r2 Z0 I1 y<property>3 j' m( J$ u* y8 v1 l
<name>dfs.encrypt.data.transfer</name>; D; i( A4 M$ @ Y$ ^
<value>false</value>
/ \( }/ ]0 x4 E/ }2 \0 G <description>" r4 y$ B. i2 Z, ?
Whether or not actual block data that is read/written from/to HDFS should# S4 O: B4 B, a3 S. [/ C0 S7 s
be encrypted on the wire. This only needs to be set on the NN and DNs,- c& R* _- V1 k3 L( j
clients will deduce this automatically. It is possible to override this setting : p9 E. z, i' W5 O7 r
per connection by specifying custom logic via dfs.trustedchannel.resolver.class.
7 z1 P7 [) X4 i </description>+ _7 Z6 G; p& q) |. k: U) x
</property>- k/ x- i: q: r+ \0 ^9 l) ~' x: i
<property>
& [) b6 Q& ?$ N <name>dfs.encrypt.data.transfer.algorithm</name>; O e f$ |) a) O o( z0 E. K7 i E
<value></value>: Q& L7 T$ G$ Z7 f5 [$ o) A
<description>! F$ `$ i) e) E& z# K
This value may be set to either "3des" or "rc4". If nothing is set, then
( z2 e. P/ F& e9 G3 n the configured JCE default on the system is used (usually 3DES.) It is4 _& p: ~6 o2 q4 ~
widely believed that 3DES is more cryptographically secure, but RC4 is4 x: \1 Q" l+ g' \% d/ D
substantially faster.
5 ?$ N5 P5 R% Z! Y/ r Note that if AES is supported by both the client and server then this
# w3 @1 _7 I# {9 v3 Z- w# _ encryption algorithm will only be used to initially transfer keys for AES.
! j2 M$ ~; u9 `( ?9 m (See dfs.encrypt.data.transfer.cipher.suites.)
$ L* d% s1 q: K- U </description>( U$ p% G1 Y2 R' k8 t# |7 f
</property>3 ?. T# }$ x' b* _7 E3 v
<property>
% N: n' h3 ]+ v% U( O$ b* T <name>dfs.encrypt.data.transfer.cipher.suites</name>5 y* u1 Q, U. Y; d5 K
<value></value>
# a0 i# X0 a8 d <description>
( G1 g( C3 t8 L0 ?& X$ \; ` This value may be either undefined or AES/CTR/NoPadding. If defined, then( J# J$ A' h- L' p! Q- h, Q
dfs.encrypt.data.transfer uses the specified cipher suite for data% ~! k8 b: }# `1 O+ X G
encryption. If not defined, then only the algorithm specified in) u' W3 F8 b# T" X( w/ w( @
dfs.encrypt.data.transfer.algorithm is used. By default, the property is
' m L t8 g* Y" N' v, { not defined.. b5 l' q( {% t% w% G" y
</description>
( q: v" g: |! _: x1 g</property>) x8 p6 D. f: p& u& f7 l
<property>% j! w Z. M% q" g: j$ D* l
<name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>+ w3 }( K2 o \/ b, U
<value>128</value>
) D7 u, S# q" ?! N0 E <description>
0 I1 E7 E. r( S) ^9 w8 `1 I The key bitlength negotiated by dfsclient and datanode for encryption.
; c, W1 c' g. M* K3 I/ F, z This value may be set to either 128, 192 or 256.4 i4 ^" F& ~4 M, j+ r
</description>. H; t' [& c% \ P% `5 S
</property>; ~+ l% R# W' ~
<property>1 I+ O" k$ U( T8 d: f% z% L' y
<name>dfs.trustedchannel.resolver.class</name>
! y6 [9 V4 ]% w2 u% A& T <value></value>2 Y+ d- q( C' s. w
<description>$ Y5 U* g$ U3 I( n- i7 a
TrustedChannelResolver is used to determine whether a channel + v5 m/ p, t/ o$ o
is trusted for plain data transfer. The TrustedChannelResolver is
2 {1 r& Z z2 O! j; v' A; b invoked on both client and server side. If the resolver indicates 4 j2 M G0 ], S4 g' f+ y8 p
that the channel is trusted, then the data transfer will not be 3 d9 Z1 A* C5 z! I
encrypted even if dfs.encrypt.data.transfer is set to true. The
G8 L! w$ E: q E default implementation returns false indicating that the channel
( E0 `) e# e9 @- v- o is not trusted.
+ H+ e. b1 u( @; u/ w' [8 E; A </description>7 r9 ~0 y* q# w# Z- X3 H* L# e
</property>* C$ ]9 @0 O0 ]4 P7 u8 p
<property>3 r8 D2 q; e# C$ p9 j$ X0 ]
<name>dfs.data.transfer.protection</name>+ b7 _) T6 @5 Q1 F9 B
<value></value>1 T2 R9 b, M* q5 N) ]
<description>
' O# J/ X4 Q% g% i4 t, t A comma-separated list of SASL protection values used for secured# T1 }5 x7 m# M. a% C% I" R
connections to the DataNode when reading or writing block data. Possible2 G- i6 [( w) u4 n
values are authentication, integrity and privacy. authentication means
0 O5 K; m' D; u8 K2 |; g% i, A authentication only and no integrity or privacy; integrity implies; L% B4 ^0 V6 [2 X8 H @& h
authentication and integrity are enabled; and privacy implies all of
( X, d0 Y- |* g( @5 M5 A) f1 W authentication, integrity and privacy are enabled. If
/ G+ F( H: r' `# i3 V8 H+ d7 q+ h dfs.encrypt.data.transfer is set to true, then it supersedes the setting for
2 ^5 |4 R" p( \3 A dfs.data.transfer.protection and enforces that all connections must use a! a, A8 C* p& Y2 j" f
specialized encrypted SASL handshake. This property is ignored for& A5 c9 O6 _1 ^7 h) g5 G
connections to a DataNode listening on a privileged port. In this case, it
* b/ c4 u1 \( G2 Z! x is assumed that the use of a privileged port establishes sufficient trust.
+ u: Q* J7 I- H, ~) J+ C </description>6 T& h# S- J) S Q
</property>
8 T1 `4 B2 w; C+ V9 l3 U<property>5 e, D0 t4 j5 A. D- R+ f
<name>dfs.data.transfer.saslproperties.resolver.class</name>
& m; n2 H' f# k <value></value>
4 ]$ n. G) r4 }9 W <description>
. N9 r" b( {1 N! S% s+ E SaslPropertiesResolver used to resolve the QOP used for a connection to the T& w3 c7 k0 q+ ]) h' T: g+ { G) X
DataNode when reading or writing block data. If not specified, the value of& q" h# A9 J/ K( c/ y5 a9 F
hadoop.security.saslproperties.resolver.class is used as the default value.
% _# r* |' e# ~; h# z. @6 F% ` </description>, u" I. R/ E5 j: C; i
</property> ^8 C* l; U2 G9 b, i. C$ H* r+ L
<property>6 h( W& [3 a. `( [# h0 F& U
<name>dfs.journalnode.rpc-address</name>3 L5 `7 m& s: @. T) a9 p1 D! t
<value>0.0.0.0:8485</value>
6 R( c. {5 M1 \9 K4 P9 n. | <description># e1 T# |1 ~1 v$ ^( c1 I
The JournalNode RPC server address and port.
0 T2 p0 _( c6 F6 y& f </description>; \: j( ?# n- _5 r! S+ H9 t
</property>- A. s% t( F, L# j2 r$ C
<property>
7 {) [) Q& ~5 Q3 b1 o( @ <name>dfs.journalnode.rpc-bind-host</name>3 d- Y6 ?' N/ l. p2 o1 S
<value></value>
. W! A% J8 j. i <description>2 L2 r L8 M1 E8 ]1 {
The actual address the RPC server will bind to. If this optional address is/ V9 x7 H2 V' d0 t" G5 H
set, it overrides only the hostname portion of dfs.journalnode.rpc-address.7 D, o2 D& H3 v0 p) L
This is useful for making the JournalNode listen on all interfaces by: {; I4 F( }) A% x( X/ q w* R
setting it to 0.0.0.0.' k+ Q; X" i( E& I' F6 C, P
</description>
& Y& N* K2 R+ P$ r</property>9 x9 V4 x8 V5 y5 u# S
<property> X8 H9 V% Z: ^, [5 _
<name>dfs.journalnode.http-address</name>
2 o, c1 _1 d0 c <value>0.0.0.0:8480</value>
3 Y" ?) o5 p! P9 A5 A* G: z0 ? <description>% \* ] `! H# k; n# q4 V% G' p0 i
The address and port the JournalNode HTTP server listens on.- I, j9 y) x ^7 n, w9 V, T
If the port is 0 then the server will start on a free port.
9 k* b8 ]2 ` j+ [ </description>
* K# O! p. s( F6 H, A' t</property>' T! ~% d' |6 f) A4 t& B
<property>6 S3 Y2 a' E N: c
<name>dfs.journalnode.http-bind-host</name>! i' y9 Z* t4 K) l5 m3 Q; j
<value></value>1 {. \! u- a( u4 m' g
<description>' F: u/ E6 { U. Z* C) @4 m
The actual address the HTTP server will bind to. If this optional address
0 F' N" S" d" } is set, it overrides only the hostname portion of
2 H& ~" l! b3 X8 j dfs.journalnode.http-address. This is useful for making the JournalNode# r( t3 u0 z+ `; n4 g# c% n# w
HTTP server listen on allinterfaces by setting it to 0.0.0.0.
, M! @1 ^! O L4 c4 n Z# l3 d/ n </description>
: r: K6 p" y; k' w6 Z# F! B) R</property>
$ D- u$ e9 c1 \<property>
. h) h! ]. x4 L5 X <name>dfs.journalnode.https-address</name>8 ~; F2 E! i' {. O0 A
<value>0.0.0.0:8481</value>. I' [, X9 r; t% @5 u/ h
<description>
3 ?2 K7 T- `6 @4 e9 R6 V The address and port the JournalNode HTTPS server listens on.
# j, i. k% z/ x* i% k( p If the port is 0 then the server will start on a free port.' _+ W6 n( z7 _: v+ L. R$ N
</description>& r) z! Y7 {1 q
</property>+ V6 X1 D0 @" ~& N8 X
<property>
3 @# U/ z l+ x8 F' W n' n <name>dfs.journalnode.https-bind-host</name>" Y) ? @# b% E8 N% O1 u
<value></value>
' b5 M) N3 T; c5 A- m1 m2 k <description>9 U, G& l/ e) z. e& y* X
The actual address the HTTP server will bind to. If this optional address
# w T; f8 T% f7 z$ a7 u is set, it overrides only the hostname portion of3 ^' r& m+ \. Q8 s8 D; a$ W% Y
dfs.journalnode.https-address. This is useful for making the JournalNode
$ i- Y! q9 ~6 p" @$ `. E HTTP server listen on all interfaces by setting it to 0.0.0.0.
' y4 A- Y& }+ O& H* z9 ` </description>
" A- y; T+ J( ]4 |( k4 _</property>+ V/ v y& z8 _( f! _! U
<property>
& h0 M. L' t* Q$ r `2 j <name>dfs.namenode.audit.loggers</name>
9 Q: E' w" R3 \0 f7 b1 D' ~ <value>default</value>
; V% U, P! K. {/ } <description>" w p9 g/ |7 y( X+ f2 t( a; F; x
List of classes implementing audit loggers that will receive audit events.
# @+ t8 Q; E3 ~) |8 ~ These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger./ \" x; H8 Z. r% Z7 d( h& `1 F
The special value "default" can be used to reference the default audit
/ q1 L( T8 v- a; Y) _ logger, which uses the configured log system. Installing custom audit loggers$ S: a0 B' h; J4 ?% ]! e
may affect the performance and stability of the NameNode. Refer to the custom
( b( k6 a0 l7 B0 s$ N% F6 ` logger's documentation for more details.
$ ]2 t3 E4 |( ^2 `3 b) p8 G </description>: t+ ~) A& X' Z7 N; L* s1 D
</property>2 W- b% Y6 W+ b* A$ [, d
<property>
* a. ^3 V/ k( q6 T <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name>7 Z' [( y& T: ]2 C
<value>10737418240</value> <!-- 10 GB -->
* V$ \1 N# F" Y3 J! o <description>
/ @* e K. e! M% ?6 k. d' h Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
5 N; B/ J! A2 G% c+ }6 y org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.4 b9 i+ _6 A6 s" k: R- E) h; M
This setting controls how much DN volumes are allowed to differ in terms of% M; ~. k' W, X! m9 l( g# _/ ?
bytes of free disk space before they are considered imbalanced. If the free' V9 N0 }" E7 Z) J0 g
space of all the volumes are within this range of each other, the volumes
0 D" V a. L, f' g+ _ will be considered balanced and block assignments will be done on a pure8 W5 d6 `0 p0 p# J2 p
round robin basis.9 D, r4 ~: Y: b) w8 l
</description>
7 `8 p" _6 R- q+ h8 G! L</property>
0 e8 |+ J' R* z9 t<property>* A2 v2 f+ P) `
<name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>
( }7 e7 z# {# D. Q+ d1 K <value>0.75f</value>: p y" O, A. h$ \1 D; B6 \4 j
<description>" c; D7 z: {9 h* X4 \
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
& M7 I) [& Z: |. w org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
4 a2 m6 k, Z) U$ p8 A @ This setting controls what percentage of new block allocations will be sent( T. B7 P4 ?+ ]3 k3 e' Z" m% Q
to volumes with more available disk space than others. This setting should1 w @9 }% y& b# ^
be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should& [5 X( I8 Y; j4 q& Y% ?1 d% z; m
be no reason to prefer that volumes with less available disk space receive5 }7 M# W9 e' g7 e6 |. ?
more block allocations.! [! f$ C- O$ w c0 H6 z' |. [: l1 O
</description>% H8 P# Z. r% Q+ U* h9 r
</property>; G+ r y& f/ t: M/ B8 D6 @1 c( O
<property>
$ z6 [1 f* ]; m' ~$ t <name>dfs.namenode.edits.noeditlogchannelflush</name>( b- }* x8 J# `1 w* I; F
<value>false</value>. Y' v. D9 Z3 t" [9 j Z, z
<description>, n! }# l/ l; k, n# F; e, D9 K; \" R
Specifies whether to flush edit log file channel. When set, expensive
+ ~8 C, V" ~1 Q; r2 p# N) s" | FileChannel#force calls are skipped and synchronous disk writes are
1 Y Q g1 ^- H7 U/ l! n$ ]9 e( C. h enabled instead by opening the edit log file with RandomAccessFile("rws")3 f, f' W' E" ^- d
flags. This can significantly improve the performance of edit log writes' O' q e: i$ `6 Q$ D
on the Windows platform.- a7 L+ u& n6 D' p1 v" v
Note that the behavior of the "rws" flags is platform and hardware specific
9 \) g/ u8 ~3 @0 s& ` and might not provide the same level of guarantees as FileChannel#force.
$ h$ R: f$ c, P! F# q For example, the write will skip the disk-cache on SAS and SCSI devices
2 z! R! U B0 }- y* B1 {& i+ T while it might not on SATA devices. This is an expert level setting,
3 U* v/ J$ c5 W! E- R change with caution.6 D- o- f# F, U& j( u; [8 D
</description>
2 S/ M" t2 O; Q8 ~</property>( M8 B2 B5 Y! n5 v- J7 ^; |
<property>
5 |% _/ `* U" e& K5 x$ K* D <name>dfs.client.cache.drop.behind.writes</name>
/ Z8 l' H6 @) i+ o# y W/ t <value></value>( Z. }8 Z: ^( Y( C+ k9 K' w
<description>
: Z7 O3 Y* z) P5 O Just like dfs.datanode.drop.cache.behind.writes, this setting causes the
! y7 b) W5 i8 y. i: ? page cache to be dropped behind HDFS writes, potentially freeing up more
. M" i( }- C. N0 c' L! e memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this
' ?& |* K. A9 ]1 I# `& k is a client-side setting rather than a setting for the entire datanode.
' {9 B: D- i7 `3 M0 K6 g- q If present, this setting will override the DataNode default.
% S* u E) F. U; J+ K' w1 h1 G( X9 h+ c If the native libraries are not available to the DataNode, this8 l0 k; N2 e4 u; O" [
configuration has no effect.
8 P. T# l5 `- q" Q </description>/ Z. {# v+ }4 m0 J' Y! r: i1 r
</property>+ \/ v# H' e" l" h
<property>/ O) R ^% m p% A3 l( }6 A; K, W C
<name>dfs.client.cache.drop.behind.reads</name>
# _ F3 t6 ^/ z% b3 m <value></value>
* F$ Z; Q, Z C( B1 o <description> ~8 ?( @' z$ w U
Just like dfs.datanode.drop.cache.behind.reads, this setting causes the$ R; A) W1 S! N s/ r% z& K K( `( P
page cache to be dropped behind HDFS reads, potentially freeing up more$ j; Z/ @7 s; b" T) p, S! D Z
memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this2 I' n" H, ?3 R7 c8 _: U& v
is a client-side setting rather than a setting for the entire datanode. If
- @4 D. F' q/ s" I present, this setting will override the DataNode default." E s0 z7 E2 |( |; T
If the native libraries are not available to the DataNode, this8 B9 R3 S8 @# q. a4 r8 n0 O F
configuration has no effect.* E9 v7 Q5 E. ?$ f9 A0 m
</description>1 ?# I% _ J% l
</property>
+ k# i3 U% x3 E- a<property>
9 B% G1 K4 q) [2 K7 ]0 M <name>dfs.client.cache.readahead</name>
. O4 Y* j L8 J+ a <value></value>
3 c: R7 F D. L. b: ? <description>- [0 ^5 t( _0 r. L E7 N* w
When using remote reads, this setting causes the datanode to
* T2 C, M0 A8 V g; I5 h2 D read ahead in the block file using posix_fadvise, potentially decreasing
5 }: R/ R8 O( e/ y, w) I- @0 A; [ I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side
2 G* }9 @, u' c2 P1 w setting rather than a setting for the entire datanode. If present, this
& _' T; @+ |: y/ s setting will override the DataNode default.
( A- `5 o7 w4 u+ [ When using local reads, this setting determines how much readahead we do in
5 @6 ]' C( n5 |: x& B0 s BlockReaderLocal.3 ?( |: C( a3 N1 x# x
If the native libraries are not available to the DataNode, this
/ Y" `8 f0 \* v configuration has no effect.# E) G/ V2 R6 T
</description>
+ h% c4 `0 D3 N6 V+ J</property>1 ?$ h6 V) h U( U- P
<property>6 m0 M8 |! A0 S" y7 L6 Z
<name>dfs.client.server-defaults.validity.period.ms</name>& ?$ Z' H2 M- q/ j" U
<value>3600000</value>5 o' L% U5 f# X5 ^0 y# {! Z
<description># |, w6 o" f' E( E8 M; s! y
The amount of milliseconds after which cached server defaults are updated.. |, r4 w0 l; |' G
By default this parameter is set to 1 hour.3 l4 N/ f. w, G
</description>
7 p: u I+ h5 Z$ r</property>
) v2 t% }; Y# L* E0 ~, z<property>6 {6 p+ E% A6 z: }. b7 |
<name>dfs.namenode.enable.retrycache</name>
' j& _+ w. Z7 e2 E4 j: } <value>true</value>
% z& e' D$ K, |) p: ` <description>! ^5 \7 T, v) F
This enables the retry cache on the namenode. Namenode tracks for
e r5 u$ ^# e3 U1 J4 D' B non-idempotent requests the corresponding response. If a client retries the: C( ]2 S6 m9 d4 W
request, the response from the retry cache is sent. Such operations- ~9 ?; u. _& R& F8 @; h8 m) R+ @
are tagged with annotation @AtMostOnce in namenode protocols. It is# N# M' t: d: f( L
recommended that this flag be set to true. Setting it to false, will result
7 M$ z4 o ]+ ^9 [1 P( f in clients getting failure responses to retried request. This flag must
! Q# d% x3 }2 f1 t be enabled in HA setup for transparent fail-overs.
3 C+ c9 Y6 [5 P& n- x! Z The entries in the cache have expiration time configurable
: A: ^& @. W- z* S& d2 T& _ using dfs.namenode.retrycache.expirytime.millis.3 a; v$ W1 _$ ^# G5 J8 O
</description>3 a ^9 r0 S6 ]% b C& s8 g, w4 o
</property>
$ E8 L) \- s1 E6 }4 D<property>
% ]$ h! V0 Q# y" F0 G( @ <name>dfs.namenode.retrycache.expirytime.millis</name>% K, Z+ D K$ Z' h% {9 b
<value>600000</value>
1 V m0 _) B- E- b <description>
! a+ U8 C I- U The time for which retry cache entries are retained.& |! ~; s- V) B9 }9 r- D" ]
</description>6 j6 `) Q/ D+ s
</property>
" g8 h! t. E* i. D4 u5 r! i<property>
4 F, y! ~. B. E6 V3 M4 y' d <name>dfs.namenode.retrycache.heap.percent</name>
2 v5 u" J. ~ I5 }* z& J <value>0.03f</value>
* r- U; V0 p8 P$ h# b. U8 Q. J <description>0 N" X0 O/ [- y
This parameter configures the heap size allocated for retry cache
8 ^3 B+ o" S& T8 G7 V8 l b+ B, C) u (excluding the response cached). This corresponds to approximately$ _# J% S% L7 s) L' ^" F4 r+ ]
4096 entries for every 64MB of namenode process java heap size.2 ?# c& X& K% w+ A$ e
Assuming retry cache entry expiration time (configured using% H6 `! t) h m& f
dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this
: L2 ~+ h* D, w/ x1 @1 G: D9 h0 g4 ? enables retry cache to support 7 operations per second sustained: z; ?& G2 F* ~
for 10 minutes. As the heap size is increased, the operation rate
6 G4 {$ v$ X* \- u H7 p% g( ^ linearly increases.
, M9 T# P& ]/ J! S </description>
9 l# M! x% J2 F+ l</property>
' ~) Q c0 k% b/ a, R<property>. U* u# a' M2 _+ r6 p! m% @; u
<name>dfs.client.mmap.enabled</name>
0 @ h8 H! ~( G4 |4 A8 Z A <value>true</value>. \" ~ s9 ?- [: }3 z
<description>
9 x* e7 T. Q* h1 F; R6 B" p If this is set to false, the client won't attempt to perform memory-mapped reads.3 ^2 c4 p0 U# r. {' v( P
</description>
; q) A" U( T3 s. D3 V1 r</property>7 n: I0 Y" O* Y: q
<property>
5 h) g. m' H: P$ d <name>dfs.client.mmap.cache.size</name>
+ ]" D4 }" J. I1 v1 @& }$ Q <value>256</value>
7 \' m% {* m9 p$ R <description>
6 N5 M' E" c* [. D6 l& Q! L6 t6 d& p When zero-copy reads are used, the DFSClient keeps a cache of recently used5 C+ U. ^. c" n U& G
memory mapped regions. This parameter controls the maximum number of% W! ~; w4 r/ ]6 S: J4 q7 s
entries that we will keep in that cache.. `8 @2 m1 d/ x! S" n# V" T
The larger this number is, the more file descriptors we will potentially8 P; ^) `: |# @4 W9 C
use for memory-mapped files. mmaped files also use virtual address space.
0 _ V3 R3 y. |, \5 p- q You may need to increase your ulimit virtual address space limits before
/ @. _9 X' r- b: F increasing the client mmap cache size.
" B' [( H* D+ K Note that you can still do zero-copy reads when this size is set to 0.
' _, Y5 H" c) y' c0 i% t </description>, ]; r$ v/ {, \$ d- _
</property>7 _8 I, R2 S: `9 V! E
<property>
! P3 F- E5 J7 u+ ^& ] <name>dfs.client.mmap.cache.timeout.ms</name>. ^2 p/ ~% E1 R+ w5 E4 _& m
<value>3600000</value>
# [' h7 q' y: G <description>
1 p0 X8 U+ u1 e3 R3 ? The minimum length of time that we will keep an mmap entry in the cache2 m: @0 L8 u6 o; ?! G3 t9 ~. U
between uses. If an entry is in the cache longer than this, and nobody
( i6 v! d* i5 ~. r& B uses it, it will be removed by a background thread.2 E m' U* h/ R3 h
</description>
% o* a c% a0 M: y* Z, `+ B</property>! [. H5 u: p7 T, h% G
<property>
4 A1 X7 W) R0 K5 @) o <name>dfs.client.mmap.retry.timeout.ms</name>
$ I' l6 M- `3 k7 H; o- m0 C <value>300000</value>. F! D- F% K0 Z- i( U$ V8 n8 f
<description>+ g0 q0 Q. _1 b4 J# Y! T" N3 R
The minimum amount of time that we will wait before retrying a failed mmap
2 }0 d x+ G9 Q6 ?; I operation.
/ k* J' U0 v: j" s, ~6 a </description>
9 T( }& R) ?5 M! ^3 X4 x! I</property>
" V+ I7 i+ B6 J" y<property>
( Y- t" `: i/ g( i0 P2 ? <name>dfs.client.short.circuit.replica.stale.threshold.ms</name>9 x0 J+ V4 V' Y7 I9 t+ _+ v
<value>1800000</value> a4 Q3 z8 L: Q4 N- d$ G
<description>5 m4 P. {0 Z; ?9 w
The maximum amount of time that we will consider a short-circuit replica to2 I8 {3 O: \5 P4 A
be valid, if there is no communication from the DataNode. After this time
/ r# \0 U( A5 X8 p1 J$ _6 t9 Q6 { has elapsed, we will re-fetch the short-circuit replica even if it is in9 r% @. p, s% W4 o% U \5 D
the cache.
) l' `5 N: b, P </description>. I8 G; ~$ B! u/ Z9 J4 Y5 \6 K
</property>/ \# e. m6 T% n/ l. g1 f
<property>8 O- _5 J# z; _5 D( h' i- `/ j" h
<name>dfs.namenode.path.based.cache.block.map.allocation.percent</name>' C* @; ^( B& Q6 n% ]0 R
<value>0.25</value>5 W: `7 Z7 @. G8 T+ O4 i% ^4 U
<description>
( t# f7 X/ R) | i The percentage of the Java heap which we will allocate to the cached blocks
$ O* {% ?- ]6 a9 `% \$ Q map. The cached blocks map is a hash map which uses chained hashing. v: l6 _0 b" i( r4 N! l
Smaller maps may be accessed more slowly if the number of cached blocks is7 P5 V, {! K+ Y) v m* o
large; larger maps will consume more memory.
4 v' ~9 r# k. I& B </description>1 w9 Y& D) |, d
</property>
/ }* A& T, |; f8 b5 x V3 g1 b<property>3 Q8 j: p0 f: |6 c# s5 F' S
<name>dfs.datanode.max.locked.memory</name>
" j1 G o" _; J( u+ t <value>0</value>
6 n a4 N" A; v' o <description>
7 d3 f( ^) t2 { The amount of memory in bytes to use for caching of block replicas in
1 O' q# K, @$ r" v memory on the datanode. The datanode's maximum locked memory soft ulimit
& l# e; y" S7 P) b7 V (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode
. z0 I0 r, O. ~ u2 z will abort on startup.
. z0 T# E% U$ Q+ o$ N' X$ d By default, this parameter is set to 0, which disables in-memory caching.: W" _1 J3 o" ?: p
If the native libraries are not available to the DataNode, this
, J$ [% [( i( u) p; J- p M configuration has no effect.1 Z4 w: j/ l: |9 }3 @% F0 R* v
</description>
* T# e& g' ]0 D# p8 m</property>
* f( ?7 g t) b8 K<property>
- u% x. ?. [# d9 s+ w/ O, w <name>dfs.namenode.list.cache.directives.num.responses</name>
. E4 j9 i" W7 \2 L; ~6 V <value>100</value>! ]3 ^) k8 T1 y/ j
<description>
4 j$ W- v$ |% y0 J4 K4 v This value controls the number of cache directives that the NameNode will$ h* p: Y- K ]4 s# v, W( C
send over the wire in response to a listDirectives RPC." C+ }. v8 E( [! @( u. v0 }
</description>
( Q1 o* f, t. H! p$ b</property>
$ q4 y) Z( A5 }<property>* J1 c/ L. R/ w3 e2 \1 T, ~0 ?
<name>dfs.namenode.list.cache.pools.num.responses</name>
/ A& X0 k0 y; L$ f" G) L <value>100</value>% }, f' M, W0 B# r
<description>2 s1 f- G$ g3 n3 Q
This value controls the number of cache pools that the NameNode will4 H+ `. `. X: Z4 i# L8 k
send over the wire in response to a listPools RPC.& t( ?. g0 S) }, D# t; h; v9 K; U
</description>
8 S* C/ M3 e$ h8 P2 [# W" `. h/ I T</property>
' H* X$ D8 Z. q$ ?<property>, i& x1 P3 ^2 a) ^0 u& H8 x7 ?
<name>dfs.namenode.path.based.cache.refresh.interval.ms</name># e* Z4 b) `8 ^9 G6 ^
<value>30000</value>
A# [' U8 F j( A9 g <description>, X0 o( J& {4 u+ M" o; d- r% T9 U
The amount of milliseconds between subsequent path cache rescans. Path# ?1 v. h( T2 L& {
cache rescans are when we calculate which blocks should be cached, and on
' b5 g5 ?' W" A5 ?4 I0 L what datanodes.
4 e' M+ P) m6 y' G6 _- k# S! T" D3 y4 K+ G By default, this parameter is set to 30 seconds.4 e. g7 K/ w' t9 M
</description>
* s9 B$ {, s6 c6 Q* x; t5 p- K( i* \</property>
! B' l7 y3 p, {+ H<property>
! T' d" z, g1 M" w) S <name>dfs.namenode.path.based.cache.retry.interval.ms</name>( L" `' ?- _$ j
<value>30000</value>
$ t% A3 Y3 R6 ^& ] <description>/ _6 T2 @, U8 g {* x. W. ?) y; L
When the NameNode needs to uncache something that is cached, or cache
2 \- t: v7 @3 X6 T* U" r something that is not cached, it must direct the DataNodes to do so by
! Z1 N5 x0 z6 L3 @- L# n. n( o sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode! M0 ?& e" `4 X9 M6 k
heartbeat. This parameter controls how frequently the NameNode will
( O- f9 u' n$ T' p" d% }& W. n( x: s7 t" p0 y resend these commands.- W1 ^' B4 l& y1 E! c" b. g
</description>6 [ Y: h6 n" @) a% `. z d
</property>
7 J T% d* G2 @1 N5 g! n- m<property>6 w# X0 |/ [- \1 x" e
<name>dfs.datanode.fsdatasetcache.max.threads.per.volume</name>5 ^6 ?9 A! H- P- l6 b
<value>4</value>7 |; F( v8 ]& g
<description>3 m' I) b* i* }$ \4 a( P, b( Z
The maximum number of threads per volume to use for caching new data& N* c8 x% y( o0 _2 A k
on the datanode. These threads consume both I/O and CPU. This can affect. G0 o b [/ ?4 z& {
normal datanode operations.& A W) d5 H0 h! j8 m
</description>% N4 A8 b; Y9 z/ c% a
</property>4 g! ~1 ]$ X/ }- ^; k
<property>6 ]% k6 L% @# E0 e9 R8 k1 |
<name>dfs.cachereport.intervalMsec</name>
. O) Y; |2 F4 d9 L$ c <value>10000</value>7 h: c: i3 [4 x7 c' F _& w
<description>" ]- U; O* o; q0 Y0 i
Determines cache reporting interval in milliseconds. After this amount of
% P1 D0 Z6 w) A+ q time, the DataNode sends a full report of its cache state to the NameNode.
& |1 s1 k, q7 [. ~; v8 ^9 W The NameNode uses the cache report to update its map of cached blocks to% \, B# n# R8 S2 w+ d2 w. ?' s
DataNode locations.* K, i/ ^) w; w! u
This configuration has no effect if in-memory caching has been disabled by
; x" f4 r/ U2 a) n& R3 S* N setting dfs.datanode.max.locked.memory to 0 (which is the default).3 o, k$ f8 o- X) ]- s
If the native libraries are not available to the DataNode, this
) f, }& M2 G: Q; j* P configuration has no effect.
) J6 _2 ?6 J9 I9 ?* c! f5 ~! { </description>% ?" K4 K5 d( I, Q5 e& ~% v. ~: p. X% D
</property>, L& r1 U, Y6 } g
<property>
( \. V0 k% _5 _ s& f! F1 N <name>dfs.namenode.edit.log.autoroll.multiplier.threshold</name>
) l$ W. ?8 w; s4 _& q' U# H" r <value>2.0</value>
, v; D* `: Q- f$ U* R* r% B <description>
* Q4 v O4 R6 w; D8 Y Determines when an active namenode will roll its own edit log.2 N) U( D0 b5 |# S0 |
The actual threshold (in number of edits) is determined by multiplying7 d) m; \6 T$ Y9 e
this value by dfs.namenode.checkpoint.txns.
# k: p( |) c5 o, |- m This prevents extremely large edit files from accumulating on the active
A) Q$ u( Z3 o. n) |. F+ V namenode, which can cause timeouts during namenode startup and pose an
H( e8 u5 w- P5 { administrative hassle. This behavior is intended as a failsafe for when
3 M5 c' ]! ^# m' d2 m5 N1 ?$ S3 ? the standby or secondary namenode fail to roll the edit log by the normal
1 L# t# ^/ l! ^, }3 O q checkpoint threshold.' m! }7 T9 d: I. [$ B4 z, s
</description>+ I$ @% _5 |6 i( A$ q. U
</property>
, V. x9 |1 f/ ]$ f4 z% M<property>
, |+ \$ d1 j' }+ O <name>dfs.namenode.edit.log.autoroll.check.interval.ms</name>+ f* I: }* b0 p( L+ f
<value>300000</value>5 S' e5 Z( x' H- L% d1 E
<description>
( U( j# ?' N1 d$ K+ P How often an active namenode will check if it needs to roll its edit log,
# V; v+ I8 s: p! D" W in milliseconds.& q; `6 D& r2 f2 d' u
</description>2 {- ^/ N' E4 i" t8 ?" h
</property>" y j) Q9 k* e
<property>
4 x+ T% i0 k, O4 p8 W <name>dfs.webhdfs.user.provider.user.pattern</name>. r$ Y: c% i6 p, k
<value>^[A-Za-z_][A-Za-z0-9._-]*[$]?$</value>. ]6 i$ ?2 z8 m `$ E" d4 B4 \
<description>
. }4 q! h) l' k3 n! L Valid pattern for user and group names for webhdfs, it must be a valid java regex.
7 Z, f0 q8 l% v5 I+ y" r </description>
* y) D k. v7 S( r. U</property>
% `- P' u: X. G1 V6 H, f% w/ W<property>
# _- P2 {8 ?7 e8 {- a6 z* W3 a <name>dfs.webhdfs.acl.provider.permission.pattern</name>
$ \ K* ]2 Z& k- q! Q <value>^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$</value> G; ]; E/ ]7 P8 w, F
<description>: K* }! g. I) w. |3 i
Valid pattern for user and group names in webhdfs acl operations, it must be a valid java regex.
, W! t6 a) ]. R8 i3 x0 ^, Y </description>/ p! l9 ^! e' M& j& J( R' m0 g
</property>
- l# M P1 I" r/ a<property>
1 s1 T L9 P! Z! L1 v; u2 ?: A3 V <name>dfs.webhdfs.socket.connect-timeout</name>) X$ b) X3 J6 _4 M9 Z3 m
<value>60s</value>6 ^; j/ Q6 ~$ M, c
<description>
; X) p/ @5 K2 r/ K' J& ? Socket timeout for connecting to WebHDFS servers. This prevents a
# x3 N4 {9 U) m! n$ k WebHDFS client from hanging if the server hostname is
8 K: h/ T9 `6 F% S0 {, v misconfigured, or the server does not response before the timeout
/ F5 |. W2 H' Y expires. Value is followed by a unit specifier: ns, us, ms, s, m,/ _9 f" L e* K$ C% P
h, d for nanoseconds, microseconds, milliseconds, seconds,
& e5 L, W4 p' [+ Z. I minutes, hours, days respectively. Values should provide units, H8 a8 ?/ w6 {( l; U/ h8 y6 g: O" a
but milliseconds are assumed.3 o: x% t4 H/ D
</description>
3 F- W5 g/ J8 M) p: \</property>: T' h& u* x$ L, w
<property>
! M4 @( _9 x* k5 |# W+ I <name>dfs.webhdfs.socket.read-timeout</name>
+ m4 ^' b2 b/ {0 b6 s9 a* h <value>60s</value>% K( h! [; H- Y1 v4 p" @4 C- Y
<description>
. X0 Y0 j, n3 K" [- i/ F8 E8 t Socket timeout for reading data from WebHDFS servers. This
" m9 o9 M* {0 \ prevents a WebHDFS client from hanging if the server stops sending
/ [/ y& D; b8 r, y w data. Value is followed by a unit specifier: ns, us, ms, s, m, h,$ t' q7 H% K0 t' K) p$ L
d for nanoseconds, microseconds, milliseconds, seconds, minutes,
$ w0 m6 Q, Y# [' X9 [ hours, days respectively. Values should provide units,5 y: g/ l$ W" @* e; n1 E
but milliseconds are assumed.
0 F7 [7 [4 m" Y </description>
6 m' ]2 X2 U! R% l8 B</property>/ V$ U5 u# R$ h7 [# p8 u# Y' g
<property>& O1 I- T X) o4 W# E
<name>dfs.client.context</name>
W" Z0 r5 @* Y S0 D <value>default</value>' K6 [; J% h/ y; P( G6 k$ Z
<description>. Z8 r6 U D4 d! A+ Q
The name of the DFSClient context that we should use. Clients that share
8 a7 }7 A; z# G2 V: J a context share a socket cache and short-circuit cache, among other things.
$ N+ A& g% P/ ^4 r& H& A9 {$ W/ ^ You should only change this if you don't want to share with another set of
5 l* `* r8 Z+ B* Q threads.
" f& ]+ ~! M8 E7 d$ w" r </description>' h4 N$ i7 Y8 \7 K9 ]9 T. e7 H
</property>
v# w/ |: |: ?3 B# H<property>. E' `; C) p) |" N6 j
<name>dfs.client.read.shortcircuit</name>5 l5 h3 L, k+ u
<value>false</value>
7 W: F1 g3 C, C+ r4 s <description>2 W( _& ?+ w0 v3 n0 G7 r- R
This configuration parameter turns on short-circuit local reads.
, l4 w. X. X. b9 B( { </description>1 [. Z. g# C3 Y. K$ X9 L
</property>
~ v2 Q; i8 s$ b8 z<property>
& Y2 ^3 V9 B1 _0 |! d <name>dfs.client.socket.send.buffer.size</name>* x# G& F1 k5 p( P4 P, n3 V: E
<value>0</value>6 l9 S& l) ~/ w0 s/ L
<description>" h/ R, L# Q! @2 y" U. q
Socket send buffer size for a write pipeline in DFSClient side.
, q1 {& P/ }3 ~7 [6 a This may affect TCP connection throughput.8 \! I, K5 }: B8 Q' i
If it is set to zero or negative value,
% N. a* G4 m$ ~' J no buffer size will be set explicitly,
7 B; Z: a6 a1 R+ m, K! t4 H1 O thus enable tcp auto-tuning on some system.( [) _4 I9 k$ H, r/ f
The default value is 0.
, C( b- o W3 U& a </description>& V( i5 ]) V4 @
</property>
1 c4 L3 B7 n2 q# y& ^# |/ C<property>
f* @" T4 O& S4 D3 o0 S& F <name>dfs.domain.socket.path</name>; Y( [* S7 N. e2 m% y
<value></value>* C: o' k# c. t. Z/ S3 J Z
<description> _ b$ v% [$ y* Q8 z. u. E1 A
Optional. This is a path to a UNIX domain socket that will be used for
% Z+ Q+ U# _9 M4 S5 v/ v. I8 ` communication between the DataNode and local HDFS clients.
4 o9 F5 Z* F4 ?! r If the string "_PORT" is present in this path, it will be replaced by the6 G, Y* O! @. R: c) H3 r( J
TCP port of the DataNode.
9 l1 K' ?; @- v! p </description>- K% a% \# \, j1 b6 d$ ]
</property>
5 z3 [& ~& @( P$ e! Q. d<property>
+ \+ L& S+ h; ?2 m, I0 w <name>dfs.domain.socket.disable.interval.seconds</name>* m2 }2 ?" |& V9 x# ?
<value>600</value>
' F% y) I2 v6 Y* c8 G, z <description>1 @( i, b! e4 q( S% z* g
The interval that a DataNode is disabled for future Short-Circuit Reads,
' i( W# \, p/ A after an error happens during a Short-Circuit Read. Setting this to 0 will
* L; d6 X5 B7 h: K0 X not disable Short-Circuit Reads at all after errors happen. Negative values
! t9 G* O' l( V$ s$ Y# u are invalid.
; w0 A" k1 s: ` </description>
4 @2 E0 M$ S7 y* x' P</property>
: R; ]5 l) ?. P* S# X<property>7 V/ _8 q% F% ]9 P
<name>dfs.client.read.shortcircuit.skip.checksum</name>0 K R& J. v0 u: k: j
<value>false</value>
! C) J! T4 G& y5 C: X <description>* c$ K4 e7 c5 f
If this configuration parameter is set,
+ }* o7 r/ C. h6 }. Y short-circuit local reads will skip checksums.2 V; K4 T8 i( q
This is normally not recommended,3 U, d$ @- Z$ x1 s( c/ z5 R
but it may be useful for special setups.) S5 |. Y+ ?% d* u. n0 _
You might consider using this4 m, G# I& O0 w$ J: _
if you are doing your own checksumming outside of HDFS.
' g L4 [6 n; m7 h7 |+ e </description>. O6 E4 h# E5 }* f
</property>
; m' j$ n! |; j X% a<property>' b; ?( D( k! h$ y
<name>dfs.client.read.shortcircuit.streams.cache.size</name>
- |; `/ X0 [& q7 \ l' a7 a( r <value>256</value>, Z1 u+ i3 B. J4 T
<description>
b# a t8 N, s/ R The DFSClient maintains a cache of recently opened file descriptors.( a9 X5 T5 y0 n- ]$ o
This parameter controls the maximum number of file descriptors in the cache.
; h9 n* A5 W& S, n Setting this higher will use more file descriptors,4 u/ O% ?( N$ X6 D& R Z+ }8 P
but potentially provide better performance on workloads- M5 y4 g, [& L3 w
involving lots of seeks.$ d- l$ q8 m6 H
</description> {/ Z! {" T! C, W, z) O. \8 p% H
</property>
" b" N7 y+ j6 v ?* x3 t<property>
7 D% P; A* L0 A/ l <name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>9 c. I5 o1 f9 Y' w2 q
<value>300000</value>5 p' L6 ^ D: `
<description>
, Y1 B( d7 O7 l0 A This controls the minimum amount of time
& p! m% T8 F) T# O+ ~# q$ q file descriptors need to sit in the client cache context
( U" Q% i& H) |$ ?$ e0 n before they can be closed for being inactive for too long.8 ?1 h& O1 W* [& G& f0 ?/ T
</description>" l2 m9 [/ @ ~# _
</property>
" q# Z/ H: _5 T$ c$ s<property>5 n% \# C4 Z, P, h" ]/ ?
<name>dfs.datanode.shared.file.descriptor.paths</name>( C f& O, ~! v t7 d
<value>/dev/shm,/tmp</value>0 K: {; K' [% Z! X! z0 c/ E) L
<description>
& l9 `" b8 Y/ {. P- {& H Comma separated paths to the directory on which& G2 U( G9 P v: K6 C/ J6 U
shared memory segments are created.4 i0 R8 s- f% R
The client and the DataNode exchange information via6 E7 S: h' N; ~9 ~. p
this shared memory segment.3 C. x" [9 d% E" R* ?( A
It tries paths in order until creation of shared memory segment succeeds." n' q, J+ v; P$ H# ?* a4 H# L
</description>
x: O& f7 c- S5 T. t( g* L</property>
# F8 o. O6 {& }<property>9 F7 |! ?6 C8 K; n& K, m
<name>dfs.namenode.audit.log.debug.cmdlist</name>
# Y, g$ \' [ x1 z$ g- O# d <value></value>
1 I I7 i2 M5 \9 e7 O, H <description># ]8 v5 d" S5 |
A comma separated list of NameNode commands that are written to the HDFS
! p1 r+ t, I, J7 U: F& n namenode audit log only if the audit log level is debug.5 J- j6 Y& M2 @! j m0 Z6 _
</description>
" h) k3 y0 X' d7 g$ C/ {</property>% A* x" Q. }$ M1 Q
<property>
% z& K( r. H. M3 o" z H5 A9 e <name>dfs.client.use.legacy.blockreader.local</name>; _6 F! e# B4 \9 |% J' o# j& ]1 r
<value>false</value>
. D, @ n( v6 h- q* v <description>
m5 y8 h: k$ T8 _7 H Legacy short-circuit reader implementation based on HDFS-2246 is used/ s8 Z q0 P9 k- h9 o- X
if this configuration parameter is true.
& z2 }* @& ?4 w+ ^ This is for the platforms other than Linux" X. W7 }( z/ B( N$ {- }& C
where the new implementation based on HDFS-347 is not available. j: i" B: Z8 X; i! A6 t6 |3 R
</description>* S5 C0 E4 d: U
</property>
+ b, k' E/ M! q% w& I<property> e$ U6 R7 _) I s) u3 k0 g9 v
<name>dfs.block.local-path-access.user</name>6 e% I- m# ~6 B( F$ }
<value></value>
% t. B, [( W0 S& d4 Z <description>) o* Q$ B* A, f+ _+ g
Comma separated list of the users allowed to open block files4 m. h; S. u. D6 S
on legacy short-circuit local read.
3 c0 \6 h/ L* I </description>9 e+ `" G; R; X; M( U9 V
</property>
# x8 U( k2 f, @" V<property>
# ?: m# B8 F" ~: W0 C: E% w <name>dfs.client.domain.socket.data.traffic</name>
9 N9 M4 `8 m5 Z/ q <value>false</value>( \; e j! m* V- W
<description>
. \5 x" L6 K; d; G" b This control whether we will try to pass normal data traffic
0 C5 b* B) H% u4 h0 ^7 a over UNIX domain socket rather than over TCP socket
/ i$ o7 j& L) T8 t& [2 C4 M& h on node-local data transfer.
& Z; a$ \. a( p" ^" u This is currently experimental and turned off by default.
* k( B/ b |! l7 u. O </description>( W& F) | b" c! h: N
</property>
+ w" ? B R+ N<property>
0 k/ _! r8 m1 i, X6 w" \$ Q. [ <name>dfs.namenode.reject-unresolved-dn-topology-mapping</name>1 M- A5 D9 z2 f- _' m( a
<value>false</value>
7 z8 O2 T8 b X7 b <description>. f" `! Q. Q7 w) l. @
If the value is set to true, then namenode will reject datanode ! l8 N7 K9 o, c: J5 Y
registration if the topology mapping for a datanode is not resolved and " E. v) p0 F* d2 O9 ^5 s
NULL is returned (script defined by net.topology.script.file.name fails
! @/ x( B& I' G' h to execute). Otherwise, datanode will be registered and the default rack & _% F: Y) L6 w2 o" K( K! N
will be assigned as the topology path. Topology paths are important for
4 \$ S) ~2 U% l" |3 i5 A5 g data resiliency, since they define fault domains. Thus it may be unwanted
E) O C7 }, M# x behavior to allow datanode registration with the default rack if the , W9 Y" ~- ?/ F: V3 S
resolving topology failed.
; }0 r: k7 I# N; O$ F+ L- ^ </description>3 G# j/ a4 H( \
</property>
l0 ?. `- _6 z+ g<property>
9 J5 ?# L' u7 T7 j) s) N <name>dfs.namenode.xattrs.enabled</name>
$ a5 T- l0 ], W+ ] <value>true</value>- p, z4 r1 u Q+ ]2 \
<description>
* C- n# r! E# }7 \3 ~- X Whether support for extended attributes is enabled on the NameNode.
6 c D/ k3 Q3 D) ~7 D# o. G </description>
) O8 c0 J) u3 p! d, ]. s, j</property>
; r. e: w$ [5 j; n+ Q3 d<property>7 m; A; U. L! C: X: {. B5 D
<name>dfs.namenode.fs-limits.max-xattrs-per-inode</name>
$ Z, W( | u, U- Y Q$ _ <value>32</value>
: U6 v/ j H. i! H <description>
9 J! C) b* q$ s2 d0 O* w6 O1 z+ i Maximum number of extended attributes per inode.
/ p7 F% k( }* b8 y( X8 j </description># T B' U4 f/ q0 a: [' s# g
</property>
; N2 K" w6 ^7 B9 z& F* v! _' x! L3 s<property>$ J; Z8 p$ C: w- Z2 h
<name>dfs.namenode.fs-limits.max-xattr-size</name>1 u' @/ P2 @5 B% v; _, g2 e
<value>16384</value>
3 C5 B3 a: b' x" i H7 P <description>& ~/ B& l+ o( [' y9 Z/ e
The maximum combined size of the name and value of an extended attribute
4 q& c0 v# E8 Y- P in bytes. It should be larger than 0, and less than or equal to maximum$ F% t" g2 T' u+ k0 y
size hard limit which is 32768.) F7 N$ e2 |7 L. c3 T' x9 b
</description>
8 J$ I$ i3 m$ I</property>9 d! f9 p3 ?- y+ Y
<property>
. N9 G3 v6 ^5 h, C& j+ I$ R <name>dfs.client.slow.io.warning.threshold.ms</name>6 \& _: O8 G$ p; W6 o
<value>30000</value>
6 {; m' o3 `# L3 n8 v1 h <description>The threshold in milliseconds at which we will log a slow
2 d5 E5 r+ S) h1 k" Q) F6 ^ io warning in a dfsclient. By default, this parameter is set to 30000
) H- _5 a3 b8 U, ^9 v/ ^+ @# d9 ? milliseconds (30 seconds).9 K) A# C; h5 H( G
</description>
; `4 p1 R) H/ [. U0 V/ p</property>
1 B& C8 {; c) U" }/ k1 v% G<property>
; C$ K* e& _9 C! [; y <name>dfs.datanode.slow.io.warning.threshold.ms</name>
b Z5 o% H: A. V2 Z! G <value>300</value>6 w& H" ?) @& F
<description>The threshold in milliseconds at which we will log a slow
' r3 Y; q, F2 z- H2 p, B io warning in a datanode. By default, this parameter is set to 300
( W8 d) R3 t; \ milliseconds.
( J/ K' H9 B; I9 Y7 K </description>( j! f0 K- h% J8 M$ o# l
</property> C9 J% R+ C$ ?8 ?' t
<property>
7 \0 c& w/ @1 @, K1 v, g: q5 M <name>dfs.namenode.lease-recheck-interval-ms</name>* Y5 `2 `. A9 x* k" a( S
<value>2000</value>. B2 p9 w* [$ A- h/ U4 D# }( l
<description>During the release of lease a lock is hold that make any& S8 J: @6 ~0 A& K# ?
operations on the namenode stuck. In order to not block them during
0 {% S/ D# I. n; s a too long duration we stop releasing lease after this max lock limit.! h/ x$ q0 m# U
</description>
- b5 S" { S+ \3 P</property>" w6 c; D4 t; A
<property>( S$ z; T. p2 X: H4 J8 ^
<name>dfs.namenode.max-lock-hold-to-release-lease-ms</name>
& w2 r u. }+ t <value>25</value>+ e1 e( d5 S* w4 d5 Q
<description>During the release of lease a lock is hold that make any6 k6 v n: C& D3 q7 l0 K" m5 w% T# }
operations on the namenode stuck. In order to not block them during! `9 }6 X0 q% L: L z$ W) J0 {
a too long duration we stop releasing lease after this max lock limit.3 N5 W( b1 z! [; j
</description>' |3 P+ P8 @ g/ F3 R( s
</property>
3 a1 Y3 m! K) _: U<property>0 w( ?- B* f9 n9 q
<name>dfs.namenode.write-lock-reporting-threshold-ms</name>! E+ w F6 h0 x5 m
<value>5000</value>2 r! n& B) I; x# m4 T
<description>When a write lock is held on the namenode for a long time,
% b. S: R% s9 s8 U) Q. ] this will be logged as the lock is released. This sets how long the Z( r3 I4 Q. u, C& ?: E& c& h
lock must be held for logging to occur.2 g) i) e9 I# s$ @/ U4 C+ ^
</description>
7 L- C" P$ g" c3 q I3 i9 H! \</property>/ p6 b5 k' s1 x0 \7 w6 ^" S. R
<property>3 m. l5 s: j1 O% {" m3 q
<name>dfs.namenode.read-lock-reporting-threshold-ms</name>
+ e5 C p+ i4 o: H, ]* d <value>5000</value>
- i* a5 q( R" F8 U <description>When a read lock is held on the namenode for a long time,
& K4 q$ r* y2 D( x" a0 S+ G" x; X this will be logged as the lock is released. This sets how long the! f$ |7 L9 q& |5 W+ j1 l6 |
lock must be held for logging to occur.$ I0 D( B& D( f! c/ A& | X- J
</description>5 u n5 t6 z7 m6 @
</property># M, _. `: a0 S/ v7 \" \
<property>
- p0 D5 p8 N- S! f! U: p5 D <name>dfs.namenode.lock.detailed-metrics.enabled</name>
3 n! i6 K( d7 ]) w4 F <value>false</value># B: ~ O8 ]) ~2 M+ |
<description>If true, the namenode will keep track of how long various
# y5 C+ m, X/ l: ~6 d# N operations hold the Namesystem lock for and emit this as metrics. These, d8 V: @, W! @( E0 O I$ ?, |
metrics have names of the form FSN(Read|Write)LockNanosOperationName,, g, e6 Y( }' \2 U1 a
where OperationName denotes the name of the operation that initiated the
6 Z, d' F2 F, F. D: @ lock hold (this will be OTHER for certain uncategorized operations) and8 C* u o. v4 s7 @/ [5 J& G
they export the hold time values in nanoseconds.$ Y3 F3 A( s: ^* A+ ^' s
</description>
3 P6 c$ X; w: O# t8 z: w</property>
1 b8 P$ m Q& C' z" s5 g4 w<property> ?5 i/ {. `8 U9 G4 }& ^2 y
<name>dfs.namenode.fslock.fair</name>9 @$ N$ Z* u0 j! b) V, L
<value>true</value>
+ I: V! {5 o( k0 G9 v. R- t) L4 { <description>If this is true, the FS Namesystem lock will be used in Fair mode,7 `' }7 o/ Z& {) w
which will help to prevent writer threads from being starved, but can provide
) v+ {* y# X* q lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock
" I7 n+ r Z! ~7 Y for more information on fair/non-fair locks.
5 S* I* \2 Y( U1 }& }, ? x: \ </description>
$ ^$ o: c3 Z- ~$ E7 H- e) Y</property>& k4 t& V; V- ?3 J. `, A
<property>) D2 Z* }7 @! z6 E& r5 X1 S
<name>dfs.namenode.startup.delay.block.deletion.sec</name>9 D" d' p, G+ }' |
<value>0</value># ~* }, X+ L7 j( q6 {5 i5 J" c
<description>The delay in seconds at which we will pause the blocks deletion
, d, Q2 P8 D2 n after Namenode startup. By default it's disabled.
z z* {% [3 l* v In the case a directory has large number of directories and files are, K& F9 t6 j5 ?, |9 q |1 M% V2 c
deleted, suggested delay is one hour to give the administrator enough time
/ e+ j' u- M, E( k7 o$ t& k7 x7 Z+ V to notice large number of pending deletion blocks and take corrective
3 n4 k. B( I4 ^/ N action.% O9 U) r# `( h: A, H% N
</description>. S) S7 @6 w9 s
</property>' E4 G$ x( J1 }
<property>7 p8 ~+ h4 d, o' [/ X' k9 E
<name>dfs.datanode.block.id.layout.upgrade.threads</name>
9 n) `1 Z ?9 P5 a2 _ ^. v8 u <value>12</value>1 p; S( S$ d: q/ i. u4 [' f
<description>The number of threads to use when creating hard links from: V$ d/ V* i0 W" p& a) p7 H
current to previous blocks during upgrade of a DataNode to block ID-based! K7 l8 K$ y" h" k5 W
block layout (see HDFS-6482 for details on the layout).</description>
$ I+ T; k& b ~3 i* `( c" ~</property>
, m8 |& D- Z* G3 c<property>
, R' T0 l) \5 X( u3 ~ <name>dfs.namenode.list.encryption.zones.num.responses</name>$ S7 T: n( \6 ?0 G. O
<value>100</value>( d3 N# }* @; ?, K
<description>When listing encryption zones, the maximum number of zones9 g% j j! z7 O' }& R
that will be returned in a batch. Fetching the list incrementally in2 y; U+ d) N k' _1 H7 G' `
batches improves namenode performance.* i; L4 m e( P# o! ]% M
</description>" D" q( u6 a" B+ L1 U# i
</property>( `! d& x! ~4 f6 J5 j
<property>4 \* x1 w' J3 l X, k7 v
<name>dfs.namenode.list.reencryption.status.num.responses</name>% w) P+ q4 R/ F5 [" z7 t
<value>100</value>
4 g) F7 H( B/ |' K, E! F8 P <description>When listing re-encryption status, the maximum number of zones0 ~) W. A" I- U( O8 ~
that will be returned in a batch. Fetching the list incrementally in/ r& V9 I8 u4 s5 g1 H; h! K
batches improves namenode performance.$ g' B0 [( h; v A
</description>6 A. g! @- D! d/ J* x4 n
</property>" ~( d+ k2 `0 @1 \* _$ d- L
<property>
: K- e4 f& E$ a( w% l <name>dfs.namenode.list.openfiles.num.responses</name>! q5 z( S- b1 A. C8 Y0 X
<value>1000</value>- d# o/ ~; O+ x: Z2 j
<description>
" T ?7 z; p6 T When listing open files, the maximum number of open files that will be5 ^+ j% ^" {, V0 X
returned in a single batch. Fetching the list incrementally in batches
" d! G: Y3 N" N3 q; J1 p improves namenode performance.! |' b6 K4 W& O7 ]
</description>% h( k7 }4 t, x c$ p
</property>
; W* I* |( a& n0 K. T$ M# L# N& S |<property>4 T1 i/ j0 x4 R8 S
<name>dfs.namenode.edekcacheloader.interval.ms</name> X# Z* B- f/ ~9 F1 |6 w7 Y+ m! I
<value>1000</value>6 _: v3 L1 a7 N; ^4 I
<description>When KeyProvider is configured, the interval time of warming0 G( ~4 S- f. [! |' O" s* J. w8 d6 \
up edek cache on NN starts up / becomes active. All edeks will be loaded1 Z4 Y8 @6 L* w( L$ v a
from KMS into provider cache. The edek cache loader will try to warm up the5 H f8 `1 d; w) R$ D n$ l3 u% e
cache until succeed or NN leaves active state.$ b1 s- T& p' e
</description>
4 t6 H) b- A3 ^. | ]+ F/ x</property>
* g" M# \$ v. H<property>8 O$ s$ }" _- ]% K3 _1 t
<name>dfs.namenode.edekcacheloader.initial.delay.ms</name>4 F3 Z7 V$ ?& `
<value>3000</value>8 c. m7 x7 M8 n( T
<description>When KeyProvider is configured, the time delayed until the first2 }- Y! F! {& V4 B/ m B! Z
attempt to warm up edek cache on NN start up / become active.( T0 E$ h+ d1 f9 t% e: j$ d: X* I( K
</description>
' A) x& _: Y: `1 Z K% }" V</property>
; Z! |! q# k8 L<property>
% D) Z/ S1 {; f; U. B) n; p: Q <name>dfs.namenode.reencrypt.sleep.interval</name>
+ x6 G9 k5 ~7 L4 \ <value>1m</value>
# m3 u( |" X0 N& M4 N& l <description>Interval the re-encrypt EDEK thread sleeps in the main loop. The+ H2 S& O- E( r6 A/ P* N- y
interval accepts units. If none given, millisecond is assumed.
: M+ U3 c6 r3 A% N2 A: Z </description>. |, R2 @! K+ J
</property>
2 Y2 }9 R0 L( Z/ R E n<property> `2 L( Q3 Q- s3 @, a% ~
<name>dfs.namenode.reencrypt.batch.size</name>/ N$ p, b/ C' H, n; C- W. ?
<value>1000</value>
! |0 F+ o+ f' I% x) A; U1 l <description>How many EDEKs should the re-encrypt thread process in one batch.
0 s) d+ n9 A3 |7 r0 g( x </description>
& z: n# i9 T# a" v8 I8 s1 [* M</property># |$ P4 W! F _% j; \: j
<property>; h) E V X2 f) k$ s* |* y
<name>dfs.namenode.reencrypt.throttle.limit.handler.ratio</name>! _2 I' X1 S2 Q6 _1 U- e0 C
<value>1.0</value> `( C9 ^ N9 c
<description>Throttling ratio for the re-encryption, indicating what fraction
1 m$ h/ W- p3 e" M, a- L of time should the re-encrypt handler thread work under NN read lock.
6 E6 s0 I; }5 S r: [6 r; V Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are% b+ z8 C$ X( ?. t" j' j8 D* V: t; C
invalid values and will fail NN startup.
: X9 X9 X) U3 `. l </description>
2 a% B. B* ?- X4 I6 f. Z% q1 I</property>
0 ^0 W5 n1 T4 F<property>
. x A4 ^+ [5 Z/ F" m <name>dfs.namenode.reencrypt.throttle.limit.updater.ratio</name>
& `, y6 L |* }+ m <value>1.0</value>
8 i; w4 M; B. p) e' P <description>Throttling ratio for the re-encryption, indicating what fraction
z/ g; M/ l4 n/ }$ o+ n- J1 {- y of time should the re-encrypt updater thread work under NN write lock.; ]8 b# D4 J2 q7 f* X8 W. @
Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are
% Q! Z- _: L7 J/ k1 U; |; \; j invalid values and will fail NN startup.! X" P8 v4 S7 C( {! w6 i
</description>, n5 f; D/ ]. H
</property>6 O% |+ ~1 r; u1 k3 X+ N& W9 O
<property>. K. o9 u6 U _- B( X! f% t
<name>dfs.namenode.reencrypt.edek.threads</name>7 z# `. O) Y- n$ w
<value>10</value>
/ W/ n- m6 M# ^0 } <description>Maximum number of re-encrypt threads to contact the KMS
+ Z7 `, e2 Y& G and re-encrypt the edeks.
' Z6 G6 a- W( U# K </description>3 B5 u3 w/ a# D
</property>3 F! r4 d+ h9 S8 }1 z
<property>
" B3 |6 j3 i$ [* l <name>dfs.namenode.inotify.max.events.per.rpc</name>; J2 i+ H6 c/ b9 ]7 |
<value>1000</value>
C) B8 C4 `2 z2 l$ ~ <description>Maximum number of events that will be sent to an inotify client
9 T' w4 _9 j1 c) d p in a single RPC response. The default value attempts to amortize away
% U! g9 |9 _5 n8 i* c9 {! d: w% H9 o the overhead for this RPC while avoiding huge memory requirements for the
, I4 r! g9 r" a% C; X9 }: A: | client and NameNode (1000 events should consume no more than 1 MB.)6 U" ^! m5 r" U! ^
</description>
' {: v# S0 i: ~</property>, ~9 W$ v* Y9 \% c# a9 p
<property>& Z+ x9 e- H. E1 P
<name>dfs.user.home.dir.prefix</name>
* {! k0 N8 k: t3 }4 e& E8 j! D <value>/user</value>! ^+ F2 _! z+ O6 T9 n- b# S& `4 }0 T
<description>The directory to prepend to user name to get the user's$ t+ j' ^; c8 G6 a: \3 q. v, c& J
home direcotry.) W' f; p0 z" q
</description>
: a9 {0 q4 g8 A+ n</property>8 N3 z2 O) B- M) c/ v4 x+ }/ P9 s
<property>
, m, G9 Z0 n0 f- K. l& [( O9 b <name>dfs.datanode.cache.revocation.timeout.ms</name>
7 x& }! Z, N3 k <value>900000</value>2 d& y4 J" O. t3 `$ k: f
<description>When the DFSClient reads from a block file which the DataNode is
4 {* ~! Y: v. k5 u5 J caching, the DFSClient can skip verifying checksums. The DataNode will8 f* ~* M B( A2 \
keep the block file in cache until the client is done. If the client takes+ a9 L" J: r* N3 J0 [- E9 F
an unusually long time, though, the DataNode may need to evict the block
& j% P9 j e4 u; p: J& i3 P1 M file from the cache anyway. This value controls how long the DataNode will
% z. m( [4 D5 j2 d9 e, S wait for the client to release a replica that it is reading without; a) b# H3 _: h& Z0 i% y% F# g
checksums.
7 l, g6 W% t$ e( m </description>
9 c2 o- j4 x# K* q</property>0 s2 ^% O7 P, Y8 h1 S. S
<property>
H% e( u! |# t4 G* S" F9 g <name>dfs.datanode.cache.revocation.polling.ms</name>& X, t6 x7 X2 |9 b7 @/ ~
<value>500</value>- g7 u" w- N; H8 p. ^, C, U8 j
<description>How often the DataNode should poll to see if the clients have/ ~6 x9 s3 i" G" q* q. G
stopped using a replica that the DataNode wants to uncache.8 N3 c5 ]$ z+ Q# I3 G
</description>7 B, Q& W, a8 z* d$ h, p
</property>: E( T3 n! u |$ d
<property>
. o* J( v/ `: {1 } <name>dfs.storage.policy.enabled</name>
4 x5 j3 S% B2 g5 p4 i5 b* l <value>true</value>; n7 M4 M" L9 i
<description>
`% k3 w! n) N3 P2 E Allow users to change the storage policy on files and directories.! B W. E# Y4 |9 { w, i
</description>7 U( Q9 q9 O. d9 B) d% {, R
</property>* X I8 y1 k* X/ [' T" e, J
<property>$ Y1 e9 v" J4 U8 u u# b
<name>dfs.namenode.legacy-oiv-image.dir</name>$ Y7 s8 S& X# T- Q8 g$ B
<value></value>
$ }* G( [* p' J. y4 t v <description>Determines where to save the namespace in the old fsimage format- Q/ P/ Q, W$ o2 P# S: F: }0 V
during checkpointing by standby NameNode or SecondaryNameNode. Users can
- ]! k# }/ c1 S! d4 ~ dump the contents of the old format fsimage by oiv_legacy command. If8 \, r( j8 Q0 ]; k! o! ^
the value is not specified, old format fsimage will not be saved in
$ v. X4 R# t" _8 [( D. t checkpoint.
, ]4 l! a' H ^! }: @ </description> O* k% I6 X1 _. y
</property>
; e$ l7 R8 `3 F# |8 u3 E, x<property>
1 V" ]/ a1 e6 g/ A, a <name>dfs.namenode.top.enabled</name>
, U% v/ j5 o7 y! x; j" Y& c# O' H* h3 X <value>true</value>
" i' j/ z. l( W <description>Enable nntop: reporting top users on namenode
2 j% v) ]0 e7 m9 J6 p </description>
% V7 @7 W+ O" T. @2 e</property>& t- |8 g2 p1 Z/ P' U: U/ P3 ]8 `
<property>% I" c7 H2 z- X+ `
<name>dfs.namenode.top.window.num.buckets</name>: ^3 A# f" `8 H
<value>10</value>$ c' \" B) K+ Y. m8 |9 C ^
<description>Number of buckets in the rolling window implementation of nntop) ]0 I# q7 h$ I9 _1 }* _
</description>: z* E0 z9 P$ A! _& v7 m& b' U
</property>
) C- U K7 C& c! H% c/ I<property>" r; }$ i( J6 @
<name>dfs.namenode.top.num.users</name>
9 g9 W( ]$ j" R5 e. i5 S' x <value>10</value># N) P* _0 d0 T- f! a
<description>Number of top users returned by the top tool3 z- r# ?# {+ C& f# ^! G
</description>; ?% D, \1 z9 i+ s! \
</property>
4 |& u, Z2 Y" z m<property>
) M, U; N3 W- W <name>dfs.namenode.top.windows.minutes</name>" I' e% m$ z0 q) K8 H" l4 d
<value>1,5,25</value>
: w" H. Q" k" y: `/ G <description>comma separated list of nntop reporting periods in minutes; J: U9 S9 h; f; N6 F" \3 U
</description>
" `( g4 I3 M* L8 z+ D& [0 V</property>0 V2 U3 H" U5 v0 A& y" n) H
<property>, N V) S0 }3 b, |
<name>dfs.webhdfs.ugi.expire.after.access</name>+ l! U; N8 }9 W) c& T
<value>600000</value>
. n9 q3 @3 l$ B2 v) ~8 _$ I2 D( Y <description>How long in milliseconds after the last access; R% z7 o" ]7 B6 _* R7 \
the cached UGI will expire. With 0, never expire., k1 L$ t2 K. I# y7 ^& N9 o1 v
</description>6 A0 D/ U9 ]$ C/ }% k$ i z' L: L. `
</property>
Y0 Z; i+ s6 T: q7 A- m2 l; u/ o) G5 W<property>
- k" k$ _( q; J0 y4 V/ @) Q <name>dfs.namenode.blocks.per.postponedblocks.rescan</name>
# s2 g. n# e! [! x <value>10000</value>. `$ _. l" e' ^: g" }' o
<description>Number of blocks to rescan for each iteration of
; t1 p% ?! B1 Y3 @* l- t$ j) z postponedMisreplicatedBlocks.
( h: c. H+ C- O% j' ]4 J" r5 R </description>1 x$ y; G, k _9 P& o* ?9 k
</property>
3 C) O* m+ A$ j8 c9 w; ~0 c<property>8 h4 s; Z. Z- L4 `8 o0 k
<name>dfs.datanode.block-pinning.enabled</name>0 ?! t9 n: U& x; R& `; T1 H
<value>false</value>4 {5 x/ u& z/ k4 H
<description>Whether pin blocks on favored DataNode.</description>$ }) K# L2 ~3 l% ]
</property>4 @7 ]* l( i1 H# @- K0 _
<property>) [8 P" I D, Z- z! @
<name>dfs.client.block.write.locateFollowingBlock.initial.delay.ms</name>1 {9 O+ F$ d! Z" Z# a1 _9 k0 }
<value>400</value>/ c& U8 O4 @' g' a* u8 W+ G
<description>The initial delay (unit is ms) for locateFollowingBlock," h& R/ O) F$ m$ M! A
the delay time will increase exponentially(double) for each retry.' v+ E* Q; {( a: H: k, o
</description>
/ R* A3 t+ {2 Z6 o, d</property>
. i8 p6 |$ n+ m( x! R& G2 r+ B; e" k<property>
* u S0 C, R. i1 O1 h& L! N" ~6 d& B, q <name>dfs.ha.zkfc.nn.http.timeout.ms</name>. N0 M0 ~0 @/ I1 V6 y. N' ^! B
<value>20000</value>6 }2 L! P8 M$ l; ?
<description>7 |6 p# P" a' j
The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC7 \- X; V, T0 [- B
tries to get local NN thread dump after local NN becomes6 g8 ?# a7 m1 S* v3 U1 h2 s
SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY.
2 ~* R5 S3 ~3 ^* m& C. Z If it is set to zero, DFS ZKFC won't get local NN thread dump.2 y3 W1 M3 i) U4 C# b4 Y1 n
</description>0 L' _7 Z$ K- |. ^. a, ^
</property>9 g) F$ Y( U# Z
<property>* ~% r/ C4 S: }$ |. o1 L' y" V
<name>dfs.ha.tail-edits.in-progress</name>
0 g: a) P; l; Y- | <value>false</value>6 Z' N3 K: T* Q$ u/ N/ G( W( K7 U
<description>8 B0 }+ K) R+ j# x k# A
Whether enable standby namenode to tail in-progress edit logs.
3 ~9 V' f: t* x, y Clients might want to turn it on when they want Standby NN to have
0 L" V6 j; ]) q' @, o more up-to-date data.* M6 a$ J0 k1 f* ?$ ?8 I* P& ~" \
</description>7 r8 |3 {, g' O+ p) u
</property>
( T" Z* s6 V2 Y) h0 m0 A' b<property>/ s" z. o) @2 c* i z) k
<name>dfs.namenode.ec.system.default.policy</name>
6 N8 U) Q( E" H% } <value>RS-6-3-1024k</value>
, \) I. d$ V. g v' `! G0 p" G <description>The default erasure coding policy name will be used9 A/ C! Q5 _& |* Q+ ]1 z
on the path if no policy name is passed.3 ~6 {3 p, v5 ~0 m. x
</description> n, |2 H+ K; y% h: X
</property>
! Y! K y" Y' W& F$ o<property>
9 j+ h: z& Q/ X& q% d2 i <name>dfs.namenode.ec.policies.max.cellsize</name>
+ [; T9 P! o" s& Y3 s <value>4194304</value>& \$ F4 I' o/ U# ^) x$ i: V
<description>The maximum cell size of erasure coding policy. Default is 4MB.
7 O: |/ n) o5 r. g </description>
+ z5 }1 k* Z# I* E5 N5 \</property>
+ n) }/ J- M& D8 j0 ~<property>
( d1 T4 A. M9 c4 x- b$ U <name>dfs.datanode.ec.reconstruction.stripedread.timeout.millis</name>
" g, D# F$ ~! Z- ^4 T/ G7 c, z <value>5000</value>' D6 h3 F) E% u! f
<description>Datanode striped read timeout in milliseconds.
7 M. U, H9 u3 g8 {4 A# h3 x </description>
8 O* y, ^) n3 ]6 r</property>
( _# r+ I4 k/ c- ^6 y<property>
* X+ f" W0 s0 A' I+ y" R) o <name>dfs.datanode.ec.reconstruction.stripedread.buffer.size</name>8 s# k) T1 ?& ^/ Q! l8 J
<value>65536</value>
- @ M q, m$ J2 `4 n" f8 p <description>Datanode striped read buffer size.
# o+ Q+ `0 a2 \8 q </description>2 W- P9 |: ]; X: h
</property>
; O" ^) ^" L% N7 _+ N6 `<property>
3 v# ~8 X2 `# M* `6 |- j) S9 z' ^ <name>dfs.datanode.ec.reconstruction.threads</name>
* r# j( L+ g B$ b: Z0 O$ C- @ <value>8</value>. Y! m/ N% C8 h! |+ X4 @) E9 m
<description>1 V8 V: S5 {( Y) o9 G. r; ^7 n
Number of threads used by the Datanode for background1 N; Q/ Q& s* I
reconstruction work.( ?2 u9 o/ `7 m7 ~0 R( V* F
</description>/ ^( j4 ?2 D4 R
</property>
( u5 R4 H" j; P7 g3 D: E<property>
2 D( ?2 ?+ I$ O+ r' @5 d <name>dfs.datanode.ec.reconstruction.xmits.weight</name>5 M0 o$ Y- h% ?
<value>0.5</value>
8 ~, L0 i# K. h2 v( Z Z9 r <description>& b3 o/ B1 y$ x: U% I: G) w# k
Datanode uses xmits weight to calculate the relative cost of EC recovery8 L5 X1 j3 v. W* f6 }! @1 [
tasks comparing to replicated block recovery, of which xmits is always 1.
) ~+ p& O* ~) F5 ~/ z Namenode then uses xmits reported from datanode to throttle recovery tasks3 e7 W; Q9 K! e
for EC and replicated blocks.4 }' q7 ~% z' x# g B
The xmits of an erasure coding recovery task is calculated as the maximum" a% d x( _" h# K* L* u! |
value between the number of read streams and the number of write streams.
# y5 c5 w o m7 Y: B0 f3 Z! c) L! S </description>
# o& e% s2 T- A+ k' a) {; c& J</property>
- q5 ]# ^8 F5 n1 x' [; N+ K<property>
3 j1 g+ v! M3 V% o! J <name>dfs.namenode.quota.init-threads</name>
w8 N8 w0 o/ @: R* L; O+ C$ @ <value>4</value>
, Y) w9 _' V! O$ Z5 @1 p+ h/ C7 S <description>
2 l Z! p ]- |( c: o The number of concurrent threads to be used in quota initialization. The
+ g' R' Q6 u9 z1 a9 e2 f2 J speed of quota initialization also affects the namenode fail-over latency.1 z: K& X! _2 m1 j# u+ s B# v. I
If the size of name space is big, try increasing this.
$ Z5 Z% m; z0 g, ]! C" L6 I& {/ k, z </description>% j2 ~8 Q* [' [; h/ J( A
</property>; T% ^4 F5 n8 ^3 M4 {7 Y! `
<property>
1 B; ~( ?) M0 M7 I# j <name>dfs.datanode.transfer.socket.send.buffer.size</name>
5 X6 T* g7 B% w6 d2 h <value>0</value>
6 ~" q- `6 H* f4 Z2 U# e4 N) @! g <description>- b3 x8 j! Z' g" [8 g9 p
Socket send buffer size for DataXceiver (mirroring packets to downstream
! F |& y. k2 {1 z5 ? in pipeline). This may affect TCP connection throughput.5 b+ b1 E9 g' b/ I0 L4 _7 d" O4 @
If it is set to zero or negative value, no buffer size will be set
8 ~) @0 D+ w+ f& I8 d explicitly, thus enable tcp auto-tuning on some system./ r7 @+ E! p9 l) W, O0 v2 U
The default value is 0.+ G: U2 _( D- v$ J
</description>
+ Y7 r! R: c) S; z9 h; l</property>
. u+ x ^, p( z v+ {; r<property>" e! \ z. G. n/ i1 H
<name>dfs.datanode.transfer.socket.recv.buffer.size</name>
+ w/ A- P* U$ {# H <value>0</value>0 D! o; h, k3 d. Z% [
<description>
9 X1 f* C) s' v Socket receive buffer size for DataXceiver (receiving packets from client
+ w1 ?# F U5 L) S+ e" \ during block writing). This may affect TCP connection throughput.. R) o7 B; f5 h6 u4 e: a
If it is set to zero or negative value, no buffer size will be set
) ]4 Q* f1 i* W B- O! H& K! U explicitly, thus enable tcp auto-tuning on some system.0 R* W$ I% N8 q; A0 l
The default value is 0.; F; h) `6 c3 b$ `" z6 i
</description>+ j; | O3 S0 C, {( l4 m4 e
</property>- Z: x/ g* m6 ~7 U
<property>" M1 z/ b- `! m5 }( y0 k( ^0 i
<name>dfs.namenode.upgrade.domain.factor</name>
" E% l" A0 r8 Z- H' q5 K* o9 a <value>${dfs.replication}</value>
! i) v' U( V D <description>1 e e3 B+ _ {2 @4 u: p2 a
This is valid only when block placement policy is set to
. z- g9 z0 p% w7 u4 z BlockPlacementPolicyWithUpgradeDomain. It defines the number of
8 O& s! z |; ^2 z/ V5 P$ Q unique upgrade domains any block's replicas should have.% n) F2 x2 s$ {$ _$ t
When the number of replicas is less or equal to this value, the policy, W! E6 S2 ?9 @6 C+ u* J6 J( D9 ` u
ensures each replica has an unique upgrade domain. When the number of
- G, a2 ?- X0 \" W0 A- p( W' ~ replicas is greater than this value, the policy ensures the number of
+ T, M7 u1 Q7 B0 ^ unique domains is at least this value.% F5 n% z$ ^0 W# O
</description>0 u; V3 c4 t. K r6 y5 r0 e- f* B' O& R
</property>; I' r4 G) M i% s A: |
<property>
' \( m8 E% L1 a' a/ t <name>dfs.ha.zkfc.port</name>( [% i! `! Z6 m% |, J% b! q
<value>8019</value>
: O; y: T7 }) r4 P, @; T' b, f <description>% u* W8 u( @( _* P5 f
RPC port for Zookeeper Failover Controller.
" u5 |7 x) j! ~, |9 K9 h </description>
& M: p1 y9 h5 a' T2 ?# a</property>$ s" s% n/ w$ S: n3 `5 }# z2 {% W* L
<property>
- M$ m6 K1 b& W- v0 C& F <name>dfs.datanode.bp-ready.timeout</name>
: k5 f% ~. {' j. Y9 x! w <value>20s</value>( W) q( R- |7 n8 u# R
<description>
# k+ G% Z' S5 F5 k( X The maximum wait time for datanode to be ready before failing the
+ [; w, N4 a1 } received request. Setting this to 0 fails requests right away if the# _* i+ m& d) z: {
datanode is not yet registered with the namenode. This wait time
; f4 t( M0 h a1 f1 s! i reduces initial request failures after datanode restart. S9 K! R' p; G. C0 i8 x
Support multiple time unit suffix(case insensitive), as described
% m8 ^9 m# N: E2 { in dfs.heartbeat.interval./ C" `, l& m9 a$ |2 w5 B) J) t
</description>! ] }7 _( C% ~5 G; `( |: E
</property>
) ]! J. v. g: h+ i' w* h( C1 b<property>
6 h( u6 [0 G- O |# j ]: i8 z <name>dfs.datanode.cached-dfsused.check.interval.ms</name>
' [$ i2 Q8 V* W" a- u. f% s8 a <value>600000</value>, W! |# c+ u, s: e6 x
<description>
% J+ ?: c5 l# U. k9 i0 S8 T9 t The interval check time of loading DU_CACHE_FILE in each volume.
{2 g) C) k) D( _ When the cluster doing the rolling upgrade operations, it will% d8 m1 {" }8 i* N+ o- c
usually lead dfsUsed cache file of each volume expired and redo the0 ?) m4 t0 H' G8 ]
du operations in datanode and that makes datanode start slowly. Adjust
3 e4 g# \0 y% X: ~ j( p9 b% m: N4 ] u this property can make cache file be available for the time as you want.( A; B4 A2 h9 T' t/ y$ K: h
</description>
: R s* ]! {* `, w; R</property>
9 D' V- L) ` W5 H& C$ j# N$ G<property>9 \4 l8 b+ H B& t9 W2 R
<name>dfs.webhdfs.rest-csrf.enabled</name>/ ?, T0 l) h0 ^0 F1 |
<value>false</value>) W: r3 r' {' d, u- W: d
<description>
& ~2 h+ L* N/ N/ u. Y5 m* \ If true, then enables WebHDFS protection against cross-site request forgery" r6 x g' q, n
(CSRF). The WebHDFS client also uses this property to determine whether or
9 C+ x$ p6 Z( ~# d not it needs to send the custom CSRF prevention header in its HTTP requests., o$ w" P$ l0 g7 `
</description>
X" ?2 [" U/ e" d</property>( `4 R* N0 ?% F$ Q }" N
<property>
0 g3 V' P$ k) p <name>dfs.webhdfs.rest-csrf.custom-header</name> y( m2 @5 |- U- w
<value>X-XSRF-HEADER</value>0 f8 M3 t3 R! R7 ^$ y
<description>
4 ~3 T* ]8 E7 \( H- g' j: O& ` The name of a custom header that HTTP requests must send when protection
* N- W- ~' m! Q& \4 @% p. z0 D6 o against cross-site request forgery (CSRF) is enabled for WebHDFS by setting2 x. \4 |5 D( t% |1 v7 q/ ~
dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this m/ }0 Y/ [* k
property to determine whether or not it needs to send the custom CSRF/ v6 ]8 a" [9 B. N* j) g
prevention header in its HTTP requests.$ g2 w( S, U" ]# v2 n* l8 S
</description>
* F# D& {/ X2 Z</property>! V8 V3 `+ D" L1 N
<property>
4 s% D( e& e7 M+ q4 @ <name>dfs.webhdfs.rest-csrf.methods-to-ignore</name>
) @5 u6 x* t; D <value>GET,OPTIONS,HEAD,TRACE</value>
/ R$ u+ y6 I: n _# N& V. Z5 r <description>
$ q# S+ F/ E2 Q/ Y, c A comma-separated list of HTTP methods that do not require HTTP requests to/ i# z/ Q, o" a5 D
include a custom header when protection against cross-site request forgery& `! q+ u$ a1 @* i2 X- k, |
(CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to
( k) q% u% o" {. s4 P true. The WebHDFS client also uses this property to determine whether or
; F+ a, z7 P1 t not it needs to send the custom CSRF prevention header in its HTTP requests.! ]5 l& Z+ E, i6 ?5 J' [ l
</description>" J2 g" _5 H6 G) N
</property>
, B1 A0 h2 u6 N0 ^1 y2 N<property>3 { q( [4 L2 Z
<name>dfs.webhdfs.rest-csrf.browser-useragents-regex</name>$ J# N2 z6 u9 l$ ^0 \
<value>^Mozilla.*,^Opera.*</value>
% |: G3 N: B9 D6 ^' r' o; M <description>; u6 b. b& |# h# b' N
A comma-separated list of regular expressions used to match against an HTTP
& l& O x8 g* w% {4 Y0 X# I" g4 A/ X request's User-Agent header when protection against cross-site request3 I# C& C& i% _
forgery (CSRF) is enabled for WebHDFS by setting, @! ^' Z3 A( l: O7 K' r+ N
dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches# d( Q4 s9 H X
any of these regular expressions, then the request is considered to be sent
" n/ l# a( a$ F2 M by a browser, and therefore CSRF prevention is enforced. If the request's
0 O2 h6 u& i. F0 p$ R6 \ User-Agent does not match any of these regular expressions, then the request
5 `% d7 n+ a- G" z- m# A' t is considered to be sent by something other than a browser, such as scripted9 J. b4 Q5 m7 j4 y
automation. In this case, CSRF is not a potential attack vector, so
; w+ F g4 R* d6 t' |# W0 ]1 k the prevention is not enforced. This helps achieve backwards-compatibility
" P* T' Q! ] f t with existing automation that has not been updated to send the CSRF
- q3 I& ~$ m$ e2 V prevention header.: p2 G1 K' K) z! |# v
</description>
" `1 k5 P! T! S& s' k3 @+ c( c. r0 z</property>+ e1 c- v( }$ L+ _) P# E
<property>
1 [0 m$ Y$ {0 n* R/ l/ B( f <name>dfs.xframe.enabled</name>+ q2 W0 n3 ?6 i. u' ^
<value>true</value>
3 k: {# K5 g7 @% q# k <description>4 b3 D0 U: @7 T0 z8 W
If true, then enables protection against clickjacking by returning
8 ^, F$ z" y5 [1 P& f/ j X_FRAME_OPTIONS header value set to SAMEORIGIN.
2 T% f( ]) U! I3 R# e! @* D Clickjacking protection prevents an attacker from using transparent or, {" a* D4 s" ]' X- Q
opaque layers to trick a user into clicking on a button
" V: [$ v, R6 ?# E/ x3 I: E or link on another page.1 A6 y* _6 O/ Q2 N" a, j
</description>' W0 C* u) T$ N' H' _* K5 l4 T
</property>
4 l) W/ c& c$ g. Q, p+ R <property>
# n/ F+ h% W! G. Z1 d) }% O! u <name>dfs.xframe.value</name>( g/ n+ c: J3 y
<value>SAMEORIGIN</value>- R7 x/ I8 `5 h
<description>
& `. p5 j o8 ~- j This configration value allows user to specify the value for the
+ T; g3 u( o3 \6 R4 M9 }0 W X-FRAME-OPTIONS. The possible values for this field are. h; j) S: X6 V$ D: t
DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an! F8 W: c: \ e3 G4 [" t
exception when namenode and datanodes are starting up.5 `' b- }1 O; X7 b& |
</description>* A3 H$ s3 t6 J4 [/ X
</property>. ~7 P( ^: a/ n" D$ D
<property>
6 l' G- }& q( W# r <name>dfs.balancer.keytab.enabled</name>$ q# Q3 s) [) l& O
<value>false</value>* ?2 o, b/ [9 n& I
<description>0 b& @# d+ b' C+ B: v& G
Set to true to enable login using a keytab for Kerberized Hadoop.
0 e4 C, G. C: G. E4 G$ W </description>
% `+ U/ g9 k5 ]+ m/ ]</property>7 _+ N: g$ d, r: ?& Q
<property>. K# M0 P/ c; f( g( z
<name>dfs.balancer.address</name>
, F# Q; `" q' R% G <value>0.0.0.0:0</value>
: A8 i9 J4 t+ a& |& o <description>1 A6 q1 s$ u7 E7 X& @
The hostname used for a keytab based Kerberos login. Keytab based login6 {6 Y! w: i2 G6 F
can be enabled with dfs.balancer.keytab.enabled., X, n4 W% b8 K& l; }
</description>7 S4 f0 c0 w$ Z
</property>
# f1 T) q2 m9 U<property>" K' E" _9 z1 j7 ]4 L7 b# b& l
<name>dfs.balancer.keytab.file</name>. V+ {, B5 p5 E, }3 k* g, e$ b4 c& v
<value></value>- Z) M. P* ~" |" n$ M: R
<description>
+ B S7 P6 ~! o4 i The keytab file used by the Balancer to login as its8 d# n' i$ D9 B6 E
service principal. The principal name is configured with1 r3 n% s( w) K$ ]) `
dfs.balancer.kerberos.principal. Keytab based login can be
% I5 L! d; U( K" \ D H enabled with dfs.balancer.keytab.enabled.% z4 T# n0 k ~' u9 C0 C
</description>
3 t/ v; J: j2 a2 x/ m! a( f" p</property>
$ H6 x+ x' D# O$ C<property>0 R |+ L! [' I9 p0 {% e
<name>dfs.balancer.kerberos.principal</name>
; `: p+ `3 a Q- ~ <value></value>
# v! _ h% Z( @0 C <description>/ }- Y4 `0 Y. x* ~8 Y7 O
The Balancer principal. This is typically set to
! }2 O8 S) w0 T* U8 Y balancer/_HOST@REALM.TLD. The Balancer will substitute _HOST with its8 T* ^% [* `/ @2 {! y2 Y+ Q# L8 v; J! U
own fully qualified hostname at startup. The _HOST placeholder& g5 i" m9 V- w0 {& f
allows using the same configuration setting on different servers.. y/ a6 B3 o. E6 m
Keytab based login can be enabled with dfs.balancer.keytab.enabled.& k! r9 L4 _# T* B2 `0 K
</description>
: x, F0 O4 }+ v</property>0 C1 s. q3 j3 }9 Q# e2 c
<property>: ]; y4 K/ T8 h3 ]6 _7 ~0 ^4 X0 {
<name>dfs.http.client.retry.policy.enabled</name>
" `4 k) G3 V* B. B <value>false</value>1 R% M5 n0 p/ m* m
<description>
6 N. O% d" v# T If "true", enable the retry policy of WebHDFS client.9 o3 a3 ]$ |0 m
If "false", retry policy is turned off.
. q3 c; A9 H. {+ X( R- j' l" ]: t# M Enabling the retry policy can be quite useful while using WebHDFS to8 t( b$ M7 M! a! j4 [7 D- n6 D
copy large files between clusters that could timeout, or
4 m0 @3 W5 V) w copy files between HA clusters that could failover during the copy.! n# l% R3 R. J! |% Z
</description>
( n; V6 Y4 ^4 U7 p9 V9 N1 o5 E</property>) s8 R( [6 d+ T/ A
<property>6 c$ {+ i6 F3 y
<name>dfs.http.client.retry.policy.spec</name>
# ~7 x2 ?+ O0 p, F1 | <value>10000,6,60000,10</value>, |: U& v! G1 J! y5 j; e* {
<description>: f+ N, A/ N h; ^$ {4 k& `
Specify a policy of multiple linear random retry for WebHDFS client,
; ^* L% @- u0 E) U$ E$ G) [% U e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),2 x+ K0 s" d, p) ^7 l2 K5 W
..., the first n0 retries sleep t0 milliseconds on average,
9 q; \2 F+ ?4 v* @) D3 E. u the following n1 retries sleep t1 milliseconds on average, and so on.
# i5 B8 |" R9 l3 Y$ O </description>
2 t4 j9 F$ Z" I: V4 F7 [( q</property>* F7 ]0 R0 R1 p/ b) B# `2 p
<property>6 d" \6 J6 I$ r) ^) `. }6 K5 a. E
<name>dfs.http.client.failover.max.attempts</name>
6 V q# M, o; ] <value>15</value>- m, X" U! M! U# F8 C
<description>
: B% |; q3 ~% _) ? Specify the max number of failover attempts for WebHDFS client/ v$ G$ Q7 H" q9 X
in case of network exception.! |; B: m7 K9 |9 Y# g' r
</description>
8 s8 }! O K4 u5 ~) i9 V. l6 @</property> ]. h! _! [* d
<property>
9 T& |, E/ U+ V! c: J ~& G <name>dfs.http.client.retry.max.attempts</name>
* O" }+ b" ]7 a* w$ v. ^. _6 d <value>10</value>
- a% Y' `1 A0 f7 ]) I! m7 U <description>
8 q W6 y3 L- T) @- p2 _( A% _+ \, M Specify the max number of retry attempts for WebHDFS client,0 Q: ^. ~, n5 Y; p* G. \5 J
if the difference between retried attempts and failovered attempts is
9 c8 g, g/ Z, g; V. Z' I larger than the max number of retry attempts, there will be no more% |$ i z$ S+ l9 j" g
retries.2 c: r0 _% |, r$ p1 s0 @
</description>: g0 T3 I: m8 B1 F2 g1 E
</property>
+ C& o% j6 i! [' x<property>1 u. H+ M% f6 |& T8 n! @* {
<name>dfs.http.client.failover.sleep.base.millis</name>5 {, o6 @, V( q7 N p
<value>500</value>
* ]* g/ F: }7 H6 h" y <description>1 x# j9 d" T# F! X
Specify the base amount of time in milliseconds upon which the
2 H. Y3 R" D) V3 K; I exponentially increased sleep time between retries or failovers
9 [* Y4 ?, q4 u& b% J is calculated for WebHDFS client.$ G' q* t) O2 ]" F' C- i5 Q
</description>' x/ F5 T3 ]+ ~5 O a4 c
</property>$ g3 k6 _2 J4 D5 S h6 C
<property>
' {! g+ h. ]* Z) s8 P- g <name>dfs.http.client.failover.sleep.max.millis</name>! l8 P9 Z) I, [* L
<value>15000</value>
( G% m3 x( z' ^0 }, J8 \ <description>
9 f" e' q/ _; Q Specify the upper bound of sleep time in milliseconds between5 a& u, f6 y: Q3 O+ o6 {
retries or failovers for WebHDFS client.' u/ l) M' `7 k+ n! \; \
</description>2 h* r* D7 L6 @( I
</property> b/ r9 W: O% f. H- A. o& s
<property>9 V5 y# B3 s$ k3 K
<name>dfs.namenode.hosts.provider.classname</name>0 C/ V) j, P# h9 @% `
<value>org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager</value>
, U* O% [' a" g) I <description>
' F. k) i H9 ~ The class that provides access for host files.
9 A! c0 g {" c" ? org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used
1 h; E, k! @3 v4 m" t' B; x- ? by default which loads files specified by dfs.hosts and dfs.hosts.exclude." N% I- q! m! f5 I. `
If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is
[9 m' b( P9 ~3 { used, it will load the JSON file defined in dfs.hosts." G8 r; p: ?) r, M% \' i5 L
To change class name, nn restart is required. "dfsadmin -refreshNodes" only4 F. S: \/ j; w. t. B7 p0 q
refreshes the configuration files used by the class.* ?' ]% w8 p& l
</description>
0 {6 H! \/ d) N+ x</property>
1 d4 P; {6 ?$ i# t. k9 H, [ t<property>; X; X( y/ G$ l5 r
<name>datanode.https.port</name>
( E% I7 a3 J) a <value>50475</value>
2 s2 R& S; K: d7 W; q <description>/ }+ Z. A2 G, O# F4 g
HTTPS port for DataNode.8 ~& @# A* `# E0 Y9 M, u+ e
</description>
, f$ c9 j4 Y' S' |4 y</property>& ]# |1 z' p0 R) C# \+ S* K6 ?
<property>
" l( n! }: Y% l1 u/ g! e3 L% I <name>dfs.balancer.dispatcherThreads</name>5 J$ J; n- A/ l+ ^
<value>200</value>% r* t" [6 d# O2 G9 x# {
<description>& v' P% @- o. K9 U1 G* A. w' |
Size of the thread pool for the HDFS balancer block mover.
& ?* e/ G5 f2 N1 a3 j. O4 s9 E dispatchExecutor
s9 T8 E$ j; O0 m+ C# g. o </description>
1 J4 r7 D0 X, h% y, v" k' V/ W</property>
; h4 W* @# M3 U0 Q- W* i) x! u9 L<property>8 Y- Y7 ]3 t1 ]5 I
<name>dfs.balancer.movedWinWidth</name>
' Q1 C! t v7 P8 @9 Y4 ` <value>5400000</value>
* S( Z& X6 n3 u& N+ H <description>' \) K! X/ S3 v& @# b2 Y0 B; j
Window of time in ms for the HDFS balancer tracking blocks and its
: J- s2 s1 ]; ?1 C5 t; @* d$ l locations.
' ^* V: t5 |8 d* r& \ </description>. o3 q* v" Y0 {0 T. d
</property>+ y3 S/ i0 S6 F9 u ` Z# c$ Z2 e
<property>
9 I, P6 A- i2 d* \4 S <name>dfs.balancer.moverThreads</name>
/ W' m: ^/ T4 ?" e+ E <value>1000</value>* `, {: J' J j- F% ?
<description>
' q! Q3 d. ~' I8 F6 V, d5 ?, H Thread pool size for executing block moves.9 u. n' V h# c4 q9 K0 b( s) t: T
moverThreadAllocator+ ?9 { ? W# N
</description>
( {+ |# |- p& n</property>
1 x3 v2 G* z( g- Y$ O' r<property>9 Q3 i; C+ I, |1 ^7 ^ `
<name>dfs.balancer.max-size-to-move</name>7 |2 F; i4 f$ n$ |8 b1 B% V- h
<value>10737418240</value>- Z7 V5 l6 Z* J$ R# Y0 D8 }7 j4 y
<description>
; M* Y" y* s6 ?: d. ?4 h v Maximum number of bytes that can be moved by the balancer in a single
5 w* B" }( n/ F( ]# t thread.( S& X' N$ k1 `
</description>
0 V8 N4 }1 Y+ e" F5 u</property>& l* [$ s6 K9 K9 u% e+ m
<property>; s( {1 n% n8 {4 `# V
<name>dfs.balancer.getBlocks.min-block-size</name>1 T0 U5 n1 X% z h$ y" h
<value>10485760</value>
8 ]) n. z1 g' C9 F <description>
% E" F, [* B& n$ S: a Minimum block threshold size in bytes to ignore when fetching a source's
8 F, P: V* M+ C! e, n- R block list." n% R: z/ D3 w4 _1 Z
</description>
a2 [* V. g3 k9 \* \5 Q</property>
; i `. T1 H- L8 F/ L<property>
# l+ [3 r% I0 ~( S2 f6 x <name>dfs.balancer.getBlocks.size</name>! e. ^9 }1 z5 F3 w
<value>2147483648</value>4 g% n2 y2 `8 c
<description>* b- L+ f# X) I" R7 c2 ?8 b
Total size in bytes of Datanode blocks to get when fetching a source's
9 z( L% Q( \+ Z( N2 V block list.# b, e4 \( I! A$ F4 ]# W
</description>
, _9 Y# ^/ K! o</property>
" v6 e* Q7 {1 x% h3 a<property>; y$ x- P/ \1 J5 i' D
<name>dfs.balancer.block-move.timeout</name> K# F- Q! f! T% K) h+ M
<value>0</value>
/ `6 c2 t" Q8 F: W/ l <description>
Z* x+ H6 e) [2 n4 } Maximum amount of time in milliseconds for a block to move. If this is set
5 S9 G4 U, {/ A; @* x+ K" Q! R greater than 0, Balancer will stop waiting for a block move completion8 N( Z, R/ r) d
after this time. In typical clusters, a 3 to 5 minute timeout is reasonable.
1 `) i6 o [7 j If timeout happens to a large proportion of block moves, this needs to be
" h' T0 o1 I! @* v9 S, c increased. It could also be that too much work is dispatched and many nodes3 w8 [+ ?" k" s
are constantly exceeding the bandwidth limit as a result. In that case,0 F q3 f( _. [* _2 r/ @0 ]) |. r
other balancer parameters might need to be adjusted.
& J' [& s1 F- I; F$ ] It is disabled (0) by default.
4 H7 ?1 p0 i& s </description>! }% a" b! h& {! i* s
</property>, b- |6 M ]2 t. x( Y
<property>2 a* g+ }/ W; D- K/ X
<name>dfs.balancer.max-no-move-interval</name>, R6 u% A2 t& c8 o# }" R
<value>60000</value># {) [ l% ~1 r
<description>
1 S+ _7 p3 q( Q3 [ If this specified amount of time has elapsed and no block has been moved
( O- O4 z0 o3 o( C out of a source DataNode, on more effort will be made to move blocks out of- W% l" h4 G {
this DataNode in the current Balancer iteration.) o: t) i* f7 l3 r. ?
</description>/ D2 `/ O/ K6 v o* i8 |- l6 z
</property>4 f) J2 {/ b5 ~) A+ V7 _
<property>
. C m" ?6 a$ P! \ <name>dfs.balancer.max-iteration-time</name>
4 h! t0 p$ k% B( I$ W! m% p <value>1200000</value>
$ c- Z/ G7 h: z/ `# H3 W. o$ F <description>
( K. W, A% \9 E3 W& g) A! K& W! n$ S Maximum amount of time while an iteration can be run by the Balancer. After, z; a( t2 q0 p0 F! X6 Y
this time the Balancer will stop the iteration, and reevaluate the work
1 p }# P- N4 ]6 @& w; O needs to be done to Balance the cluster. The default value is 20 minutes./ @& i' `# A+ ^
</description>4 D# Q2 Z3 T2 l( D9 U+ [
</property>
. w) a! c2 m3 G<property>
6 V% ^; k) C. j" _ <name>dfs.block.invalidate.limit</name>
" ?3 I* ?6 z4 d2 s: u% U <value>1000</value>
3 [# X/ m, \) H0 w1 U9 W <description>
1 s6 K; c1 x( [, D8 N2 H8 \6 F The maximum number of invalidate blocks sent by namenode to a datanode- V1 {$ h: o5 d% x6 v' t
per heartbeat deletion command. This property works with, d+ B- w( a+ u% ]# A7 c: a j
"dfs.namenode.invalidate.work.pct.per.iteration" to throttle block, A7 a9 V; M2 ^, K/ ]' w6 q0 |
deletions.! [% Y m# n7 D; B' c
</description>& k- x- R- d8 c" ~0 I3 O! x
</property>
. r* M1 J/ J1 c" F' [/ H4 T<property>
8 P5 t/ [/ E4 C" U* ~' n1 i <name>dfs.block.misreplication.processing.limit</name>
/ t/ Z1 C/ D9 H <value>10000</value>
, ^% d, p% U9 U, I, R <description>
0 b. l0 X6 g1 p9 `0 \, n- Z Maximum number of blocks to process for initializing replication queues.- N1 U ~4 P: p5 Y4 Y
</description>; [( _* b0 I& j- M; R) F* F& h: F' ~
</property>
( _5 W9 I( Y: V! T4 Y' R<property>) s+ l" ?" \( D, U4 a: D& a
<name>dfs.block.placement.ec.classname</name>' {+ H: u' S" \. I7 j
<value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant</value>) j1 k/ n: U, O' H! |2 N, j" t0 R
<description>
: y9 g% }! q3 e+ }/ F& m Placement policy class for striped files.
8 L% o `1 ^# I9 ` Defaults to BlockPlacementPolicyRackFaultTolerant.class" i: }- T) ?. v( I" Y
</description>
1 b8 E+ t) A2 G$ X</property>) p1 a$ N L7 X: z6 ^: P8 b
<property>
1 s M i" V' q' i; J% N& a, N0 z <name>dfs.block.replicator.classname</name>
" x6 L! v9 ]8 n* F <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault</value>( Y6 W* K& e3 b; h: D
<description>! L, Z! p3 j8 l5 L1 m9 g
Class representing block placement policy for non-striped files.6 @5 u) L+ ^0 }3 w
There are four block placement policies currently being supported:
; K" A6 B- \5 m4 M4 q BlockPlacementPolicyDefault, BlockPlacementPolicyWithNodeGroup,( K, z& G2 p6 a4 s: C8 E( C$ h6 \
BlockPlacementPolicyRackFaultTolerant and BlockPlacementPolicyWithUpgradeDomain.$ ]0 S1 M- A2 o: s$ Z/ |
BlockPlacementPolicyDefault chooses the desired number of targets$ V8 E6 D" s" V) b; S9 G5 O% J
for placing block replicas in a default way. BlockPlacementPolicyWithNodeGroup
. T8 B! |, W6 ?" \ N( j( o) x6 ^ places block replicas on environment with node-group layer. BlockPlacementPolicyRackFaultTolerant
) s# q7 ]% h8 Q+ M places the replicas to more racks.
# D( R# B# N& D% Q* H5 T* Y3 { BlockPlacementPolicyWithUpgradeDomain places block replicas that honors upgrade domain policy., l% \8 {& w( |; B; `1 A$ E c' j
The details of placing replicas are documented in the javadoc of the corresponding policy classes.: t: |6 l% d2 L0 A& Q$ q% L
The default policy is BlockPlacementPolicyDefault, and the corresponding class is
# I7 [9 i5 V8 N3 i; J [% Z* N org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
1 w% W- C; \5 ` </description> Q' y( f ^" X' s8 X6 Z- ^6 h
</property>
; M1 Y8 `! C2 p3 f0 U% F j$ i) e' S( G<property>
7 L& x1 z* a/ Y3 w+ ] <name>dfs.blockreport.incremental.intervalMsec</name>1 O- c" P+ J9 V6 a! u
<value>0</value>) h8 l/ w6 \8 v4 x% k7 [# u
<description>' e( G% N/ ]' N
If set to a positive integer, the value in ms to wait between sending+ P, r1 B- R9 O# f' F- j0 E
incremental block reports from the Datanode to the Namenode.' v& c) T9 u0 \; i8 v
</description>6 ?% y, E6 i8 S/ K3 k. z8 b! F
</property>
1 ^ @" L* x, @ z J; w9 A<property>
9 @5 B4 i$ C3 S1 z: ?6 v <name>dfs.checksum.type</name>1 D5 \( |5 g+ ^. p4 T
<value>CRC32C</value>
- X0 K5 q! Z1 ^/ }1 e' Z1 @. j1 @ <description>
" U# u% D8 h! g. ^- b8 s Checksum type
0 N7 A$ y5 P8 ~! f9 l$ ^ @1 \2 f </description>
. j$ N9 M2 s; e: u! |</property>; Y5 A2 J* F) S2 x5 T
<property>) P( [' g. c( y4 y0 l# K( r, l5 h
<name>dfs.checksum.combine.mode</name>( [2 A# {; \* I' I' s1 a& J* a. M
<value>MD5MD5CRC</value>4 Y# ~: o. r" f5 g$ [
<description>: W. [* H: R4 Z# \
Defines how lower-level chunk/block checksums are combined into file-level
$ G1 ?( q, K+ `/ o5 W) Z checksums; the original MD5MD5CRC mode is not comparable between files3 G, L+ b( \4 a0 q$ B. p5 o2 V
with different block layouts, while modes like COMPOSITE_CRC are
3 D; k [* |% {* X' P1 z comparable independently of block layout.0 Q& ~0 |# l4 ~% U
</description>6 K ~# H8 @+ j( w5 J/ X* y, a
</property>
+ E2 ~* K! P0 j3 t! v<property>% c ?6 n1 ]: j, b1 o& B4 M
<name>dfs.client.block.write.locateFollowingBlock.retries</name>( \& E# |6 }. v% J# y/ Q: c
<value>5</value>% b8 w0 w/ i# \
<description>
( P1 ^- N' D! ^" M$ j! { Number of retries to use when finding the next block during HDFS writes.
* l4 C) M7 f# k; e; H7 b </description>* n) u8 G7 d, S4 ~
</property># g8 C' x- S$ U
<property>* y) Q, e+ c0 Z/ b& r
<name>dfs.client.failover.proxy.provider</name>
9 x+ n) S) s i9 E1 a* M% g <value></value>
. u( T! q; I1 U <description>5 q. X5 G, ~/ g7 v% T5 l' `& p3 f
The prefix (plus a required nameservice ID) for the class name of the0 S2 Y6 r5 w, m [: l' d4 f0 L% Y
configured Failover proxy provider for the host. For more detailed: O0 n! \7 X' i9 [ b; J5 j
information, please consult the "Configuration Details" section of
. Z/ F6 T, ^ O7 r# ?6 O the HDFS High Availability documentation. R( |# T8 z( n
</description>1 r5 Y) U- o- e8 x( Z
</property>) w5 E8 E7 j% {2 ]' S
<property> e4 ?% a; i7 ]3 ^. a n
<name>dfs.client.failover.random.order</name>& _+ `/ n2 d! T) g, J
<value>false</value>
! C/ h( L4 t6 m. h% J <description> Q7 s+ I9 ^7 b1 v/ C. ~
Determines if the failover proxies are picked in random order instead of the: I' m8 P6 `) }. c
configured order. The prefix can be used with an optional nameservice ID
0 b4 I* q" [9 B Z$ f7 w (of form dfs.client.failover.random.order[.nameservice]) in case multiple. y6 e" d6 P0 e2 a" C: k
nameservices exist and random order should be enabled for specific4 w7 v0 A% {; ?1 I
nameservices.: d5 I; O3 G4 i2 L: U
</description>' c1 J2 e( e0 c) z
</property>
! g7 S8 k5 G8 `2 U! U* P4 p& _<property>8 u1 G/ j: o/ Y+ E* L
<name>dfs.client.key.provider.cache.expiry</name>4 s$ P2 z8 i/ x' h
<value>864000000</value>; a& M6 _9 |0 Z6 P
<description>
( c; c# I, [" x* E DFS client security key cache expiration in milliseconds.8 [! d) U& u' L+ D2 t- n% n% ]
</description>
9 Y5 \8 z. {' e$ }! K0 g</property>
J: u4 a% p! j8 ~6 v7 F<property>; ^2 F- Z1 l7 _6 G& ?1 Q
<name>dfs.client.max.block.acquire.failures</name>9 y" c. ?% Y R, q
<value>3</value>- M$ j1 j. P. ~3 U* Z W
<description>
8 G7 D" {5 S: f' g Maximum failures allowed when trying to get block information from a specific datanode.
# B+ y2 J" [: P9 S0 E8 w: S' z </description>
" I5 U1 |( I1 w* g' \% f/ K, |</property>- J% t7 V$ a2 s0 a. I7 P9 E
<property>
4 u1 }, ]' ?, A <name>dfs.client.read.prefetch.size</name>
' S: f( F& Z6 k3 [ <value></value>
, _+ _0 v. @% y <description>
* v! ]$ M2 p, u0 O# R, Q" U The number of bytes for the DFSClient will fetch from the Namenode6 u8 Z; Z3 }9 ~. T1 s1 F% _( [
during a read operation. Defaults to 10 * ${dfs.blocksize}.
: K1 K7 K; f2 h7 N1 s/ ~3 q </description>/ @- L/ F: V' C3 k
</property>' _ w. }3 G; _& a9 v E2 j' N$ r
<property>
% l& ?8 D. q3 D <name>dfs.client.read.short.circuit.replica.stale.threshold.ms</name>0 m/ m$ ^& D# s+ n1 U, f2 ^0 D
<value>1800000</value>' g4 H O& n* V0 U3 E
<description>
K3 h+ Q6 f4 N1 b# t Threshold in milliseconds for read entries during short-circuit local reads.$ J0 L1 G# i) K, V8 a
</description>& u' A% r: F6 j8 v1 j
</property>
: m: o+ }. U" Y' D n! b5 N: I<property>9 n- n. X7 E6 S+ K7 U
<name>dfs.client.read.shortcircuit.buffer.size</name>+ `/ n! u' L* Q. n. A" n6 |
<value>1048576</value>; A* n. j0 i% [( `3 o
<description>
W l W6 A( Q4 e Buffer size in bytes for short-circuit local reads.' u: q3 S- f6 w+ j# X* N7 s+ r
</description>4 d! Q1 `% i8 d# @0 N( C* j
</property>
. b) { t/ `2 g( J2 g<property>
9 ~6 c0 d. n* ]& W" C! U# L <name>dfs.client.read.striped.threadpool.size</name>8 R n& p! X+ i9 W5 W- W' H& {) C5 M
<value>18</value>
8 x3 ]6 n% E: [+ G. a7 ~ <description>( k) p8 Y$ F9 p4 K# }& k
The maximum number of threads used for parallel reading9 O, `5 e( _' N6 m; Y/ l' v
in striped layout.& ~8 J9 X# m# D V! h
</description>
; R2 T7 I% j+ o' R9 c6 \2 v* M</property>4 J3 f9 M7 \+ V+ m
<property>
! ^! l0 l) O( B <name>dfs.client.replica.accessor.builder.classes</name>
; q1 d8 T. ~$ d6 H, M <value></value>' b/ N6 ?9 I0 K5 m
<description>
- a* }* m$ E0 U Comma-separated classes for building ReplicaAccessor. If the classes
: s5 I4 g# h4 K are specified, client will use external BlockReader that uses the/ x% s$ h; D' i: c
ReplicaAccessor built by the builder." }) ~% [+ z/ i* i9 w4 Q% ^' R
</description>
r- {, P: y. n. t( j</property>8 j" J4 m" y3 y; }
<property>( |+ j; L/ x' d5 _* y
<name>dfs.client.retry.interval-ms.get-last-block-length</name>0 H) L: }! B8 n
<value>4000</value>
+ V, d8 ~; F6 u4 `3 { <description>- q' S6 e! b+ M' p8 Z8 w
Retry interval in milliseconds to wait between retries in getting- }* w2 j2 o. D- [( j7 {# r+ H
block lengths from the datanodes.
2 d" B$ l- d3 T' r* S </description>
% _2 ~2 N' V$ ]5 z1 A1 v0 c</property>+ Y3 U4 S5 m# X% U
<property>
, y/ a' h# ]0 v( b4 t$ K: c$ B4 O <name>dfs.client.retry.max.attempts</name>6 ]5 F2 C! V& I) z
<value>10</value>) o' ^5 G3 e( b- h0 Z1 s9 J% _
<description>4 [( H4 n1 \, W
Max retry attempts for DFSClient talking to namenodes.
" `7 `! K0 q( {6 v) f& ^ </description>8 b2 x7 E7 {9 D, h+ o" h9 g: F
</property>" n3 s W+ T% i! O+ d1 d$ M6 {7 }7 V
<property>7 ?+ _6 I% P: j, g/ `
<name>dfs.client.retry.policy.enabled</name>/ ?- [/ f2 ?9 T$ \; F1 v5 n
<value>false</value>
& e: P6 d1 l/ |5 h+ V; ~# h) I- F! c3 E <description>
# Q% y% {4 g: } c K$ V If true, turns on DFSClient retry policy.
6 @) r+ `9 n! C# @: Q) {1 E </description> I* L: E0 K6 w
</property>; I$ N9 y1 v1 }
<property>
1 A2 C7 g4 W& V+ U/ O <name>dfs.client.retry.policy.spec</name>7 w6 o8 w& m; t. c- ]
<value>10000,6,60000,10</value>
0 h$ y4 ^, v7 e1 L0 W& K <description>- [9 F' O) e& ^( r
Set to pairs of timeouts and retries for DFSClient.7 X) H9 W! T2 e
</description>* V2 c- _! O2 }4 g% x7 [
</property>* v/ d) w" C! _/ D" Q2 ^, t: E5 z
<property>( A( m1 s* u! s; F" n. U! v5 G
<name>dfs.client.retry.times.get-last-block-length</name>
) ?# h1 j2 P! D <value>3</value>: Z7 h- r9 W! |' A C) Q
<description>9 Z. d# | } |& }0 I
Number of retries for calls to fetchLocatedBlocksAndGetLastBlockLength().
$ @: f9 @# ]' K* I/ o4 z$ Y3 _ </description>! k% ]7 L* B, M* ]
</property>
/ J p/ K1 q6 m! s) H; d' F<property>) b. ~) N- a4 _9 l7 t( r* w
<name>dfs.client.retry.window.base</name>
+ v) u# j" X& o6 s& Q$ R+ u <value>3000</value>7 C3 A2 J" _9 N" Y( S8 T7 }
<description>
& K# @( p" ]# S5 N5 f Base time window in ms for DFSClient retries. For each retry attempt,
% S: T% C9 I2 S0 p$ W! S( M this value is extended linearly (e.g. 3000 ms for first attempt and
3 C- h; _' }% U$ u! R. m first retry, 6000 ms for second retry, 9000 ms for third retry, etc.).. W6 Y L1 Q1 J7 T) l W" a
</description>
; S0 w. [' k F; N</property>
' ]. j( |; q" Z9 p0 }<property>
+ a7 E( [5 Q9 G7 X <name>dfs.client.socket-timeout</name>
# R0 K$ x5 l: ]0 u <value>60000</value>
% [5 Q# ]3 O# }2 ^, ?, S0 u <description>
9 \( c8 R7 k- P J Default timeout value in milliseconds for all sockets.1 \& u6 X9 M% a. E! f
</description>- Y1 j# x7 W' z" R' t
</property>( w4 f, ~- P5 }" J6 e/ v
<property>
2 K; o& r1 t3 A: R( }% b* Q <name>dfs.client.socketcache.capacity</name>6 N( R9 E4 X6 T2 b4 G
<value>16</value>
4 u$ b. G" |. p$ i7 C$ _* o% M- g4 Y <description>6 a& {+ n, Y" N* q( `! G! ?
Socket cache capacity (in entries) for short-circuit reads.
; P6 P" s0 }9 i, h4 t </description>) n$ {; i0 D% i( x/ C
</property>9 E: J+ i* \7 `9 d5 v( t
<property>! a# e5 {; j6 @% Y
<name>dfs.client.socketcache.expiryMsec</name>' ^% D8 x! R* |; s. d9 @! t
<value>3000</value>
: m! z( r8 m7 D% F- g$ a <description>
+ G" Q' @7 g: s Socket cache expiration for short-circuit reads in msec.
* }7 B* P6 v, P+ X# W1 @& ^ </description>2 e4 z# y6 J4 L# W1 D9 W
</property>& t1 L# D4 l9 V
<property>9 n1 m# [! }( s; k
<name>dfs.client.test.drop.namenode.response.number</name>3 h" ^$ j9 h' O, y3 s
<value>0</value>
, m' y4 M4 ~4 w* b& S4 w <description>
0 y; t8 z* }% C9 Z The number of Namenode responses dropped by DFSClient for each RPC call. Used
) W3 g$ f$ P4 h# m" I6 ~2 Z5 a5 ` for testing the NN retry cache.
0 m" I- k0 O0 E% u# E </description>
) T/ ?; T* a' M- F</property>3 y" B8 M4 r5 f x, L6 q6 N
<property>% x6 V+ F3 `( X
<name>dfs.client.hedged.read.threadpool.size</name>2 i f/ b7 U+ h0 n
<value>0</value>7 w0 s6 D+ @/ ~) l4 i# Z# z
<description>' Y/ D3 E5 y, o, a+ c6 f3 `
Support 'hedged' reads in DFSClient. To enable this feature, set the parameter
4 w0 \/ @% r( D) ]$ X% Q to a positive number. The threadpool size is how many threads to dedicate( D3 Z& s5 u, a# m, d+ Y6 z/ i
to the running of these 'hedged', concurrent reads in your client.# X* H/ n2 {( i) U) s! {
</description>6 S& S, G7 Q9 h6 `# }8 r
</property>8 F: U4 r2 J/ p! G% H, [' Q4 [8 N
<property>
: z7 v. c/ a6 t% H3 ]( |) x <name>dfs.client.hedged.read.threshold.millis</name># x" e. o Z* Z( u |4 q& W
<value>500</value>
3 R$ t- i% [! e+ Z( M$ q1 g; a <description>
, q/ R: I) M7 \+ h Configure 'hedged' reads in DFSClient. This is the number of milliseconds
1 W! D- ?2 r( z3 J& `: d6 ~& o to wait before starting up a 'hedged' read.: P: K( C- D* X' N l
</description>
! l J6 Q& w' Z" N</property>
5 J) A/ r' V- o7 H<property>
! X, X4 n% x$ C <name>dfs.client.write.byte-array-manager.count-limit</name>* U6 |* M I+ }! p% E: B9 S- N
<value>2048</value>. _# y8 }8 U" C$ t
<description>
3 c& f* [) E0 A S4 H" x) N The maximum number of arrays allowed for each array length.: ~0 ?8 l. u; }% o. d, r, Y
</description>: z7 Y" S& d/ {/ d4 b/ S# `
</property>' T3 x" Q: j! V# g; C3 z s
<property>" O+ {( ^3 h) `' d' ?
<name>dfs.client.write.byte-array-manager.count-reset-time-period-ms</name>% z& `' u4 G& @( u" s0 u1 C' e
<value>10000</value>4 M% O5 E0 P. U! I) Q# O, m
<description>* _# {5 y1 F# I- I
The time period in milliseconds that the allocation count for each array length is6 \/ \1 w5 v" k. L
reset to zero if there is no increment.+ l# _1 {+ G" J+ a$ a7 X! |9 C
</description>
6 X2 h4 f- K- J$ ~</property>
" M% ^( F; s) }* y<property>& j2 J0 S0 ?" G+ }. P: U
<name>dfs.client.write.byte-array-manager.count-threshold</name>
8 C0 ` P9 Q3 W$ F# n' `0 F$ T <value>128</value>
& N8 Q' }" O2 T( m. l4 H <description>, V, ]. c9 q8 s5 A L3 R& F- T4 M2 r
The count threshold for each array length so that a manager is created only after the
; N; |! \( y) ~& j allocation count exceeds the threshold. In other words, the particular array length
2 M0 @$ |2 O8 z% ~* j is not managed until the allocation count exceeds the threshold.) c; M; V! K3 y0 S
</description>
1 Z: ?" F9 k* L- V</property>
7 Z7 ?- ]( V/ r6 i, i0 d' S<property>" i8 s5 g9 T; Z( k
<name>dfs.client.write.byte-array-manager.enabled</name>
p, G4 g. U9 z$ } <value>false</value>$ M* G, Z! B4 S. M5 e
<description>" }: W( G: V8 m8 u/ f6 {. u
If true, enables byte array manager used by DFSOutputStream.
2 j7 m; i5 g" c6 l# h </description>( T9 h* P h2 g. n$ Y/ p
</property>) Y6 ^: f1 g8 A( b
<property>
) I1 W/ p3 ]& `# X/ E/ @" R- r# _ <name>dfs.client.write.max-packets-in-flight</name>
! c& S& W$ B% b# J" Q1 F <value>80</value>; G$ f2 X2 t- [; q% V
<description>
/ Z- Z j& V/ ]+ c# L The maximum number of DFSPackets allowed in flight.
! z; g/ h' W1 D2 B5 h i </description>$ m$ X: f0 N6 a) O6 o
</property>
& H8 j- [, Y i1 s6 u4 f: ^<property>
- R% T" p4 \" }) ~/ ? <name>dfs.content-summary.limit</name>
8 b$ k+ ^" N, v4 K6 e6 p <value>5000</value>
& v. F) V, R* \- _3 ]; [4 ] <description>
: o1 l3 Y; j; U The maximum content summary counts allowed in one locking period. 0 or a negative number
$ r* @. D; }6 G* D8 I3 |: ^7 m7 d means no limit (i.e. no yielding).! k! g6 O) U6 V O* X
</description>$ V3 W+ j4 z! L3 s3 s) i C
</property> l& x1 q0 \% \! l: R
<property>
) M. T: J5 V3 F, O <name>dfs.content-summary.sleep-microsec</name>
4 Q# T) T* |$ q; C# S' r <value>500</value>" _/ c% _# n' g
<description>3 s- B" C, z. k- @
The length of time in microseconds to put the thread to sleep, between reaquiring the locks8 l- v7 S# B. F/ X& \, G5 ?
in content summary computation.
* \2 s# i) w" Y& E </description>) F8 u0 n3 M! q& ~
</property>5 I8 ?4 Z) R" U, f" {
<property>$ R4 N2 w5 G* m. D
<name>dfs.data.transfer.client.tcpnodelay</name>& P+ u L3 `# h5 M$ N i
<value>true</value>( K; T1 f7 g( W- ^9 I& I( s
<description>
& f2 |; p, s+ s9 D9 G5 w If true, set TCP_NODELAY to sockets for transferring data from DFS client." d7 v' O& X# v' M! i8 M x
</description>0 b' `# O# `# E: u3 h
</property>2 d7 O) @8 i. J4 s5 T7 W1 [) t
<property>2 J) b2 g* E0 z! E. P
<name>dfs.data.transfer.server.tcpnodelay</name>/ p9 b* f- P7 Q8 y* \+ K! {
<value>true</value>% [2 K1 }7 q$ v* A
<description># y& A/ h" E" t d) Q+ B4 C
If true, set TCP_NODELAY to sockets for transferring data between Datanodes.
W0 W: l4 e8 ^9 H' h& p5 h. Y </description>$ B# M: o0 r# h1 q5 ~" n3 s
</property>
3 K9 W; X, ?+ a; \" }<property>/ b3 @6 n* Q P+ ?" t4 h3 W3 i: P
<name>dfs.datanode.balance.max.concurrent.moves</name>- {) J. W" w; U$ Y( e1 Y
<value>50</value> H0 d+ @" O; n
<description>* S9 ?" ]2 A9 v
Maximum number of threads for Datanode balancer pending moves. This
: M& ?! t* m; \/ J: C) ^ value is reconfigurable via the "dfsadmin -reconfig" command.& O- O5 ]& H! I
</description>
% @) R5 A& ^+ A3 a. `: C</property>- M$ q+ t/ v1 a! C- G. H
<property>% ^* p* B" \6 @" w+ Z) ~
<name>dfs.datanode.fsdataset.factory</name>" W/ _6 s2 R1 [
<value></value>
- C$ @! L" u. G$ K, C <description>
" \6 W8 ]6 `: M# g* }; c The class name for the underlying storage that stores replicas for a
% v2 Y" u0 J# A8 q& c$ \1 S Datanode. Defaults to9 g; G: Y) W. `0 X U8 L4 {" d
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.
4 V" M3 v! G( [2 ^+ u; N* U </description>9 ^$ B' n/ W! G$ a* X5 ~! w0 z
</property>, M' j+ `7 [; u4 Z0 n
<property># v9 n+ W2 N, X* l6 s
<name>dfs.datanode.fsdataset.volume.choosing.policy</name>4 |5 y6 h/ l# G9 m
<value></value>
q8 s( F: h# e$ D" @4 @ <description>) D9 N" o$ Q3 Y M
The class name of the policy for choosing volumes in the list of5 p' w1 r% C* s! t
directories. Defaults to) s' G2 I' _. i7 W
org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.
* t# ]# J, B @) h4 }4 c! E If you would like to take into account available disk space, set the/ q6 P3 u6 ^3 V. X& r
value to
& p4 X. y! k& x7 | "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
6 u) y0 i7 D% R. d% z </description>& g* u b( P t4 Q
</property>
" q" j( K, e m" M<property>
9 ^$ x. J( b. D$ P, S- b3 i% Q <name>dfs.datanode.hostname</name>6 m+ `9 X: @/ P" F& e9 T
<value></value>
9 G4 A) O" K2 V$ V <description>; K: p0 }( A* s6 K5 k
Optional. The hostname for the Datanode containing this
4 {1 _8 }) k" _0 o8 B configuration file. Will be different for each machine.1 }% k7 a$ `" p( D; d) o
Defaults to current hostname.2 J! H/ `$ D" _0 Z) p
</description>
# p5 v/ @2 X, ?# c</property>
3 E* }9 @ m9 \<property>
2 p5 x, _; C+ r% b- K7 D; ^0 s& p <name>dfs.datanode.lazywriter.interval.sec</name>
- j% N' B1 V4 I! L3 ? <value>60</value>
) H# w' Z0 [ [& ^/ n! D4 |9 T9 C7 j3 e <description>: p! s- ]/ h1 g2 s8 j# k
Interval in seconds for Datanodes for lazy persist writes.; j. B8 z3 S1 ~4 z. W8 U
</description>1 X0 G! U2 a) m: P! f
</property>. S8 N1 @5 Z5 [
<property>
. ?0 V7 G* F! h+ S, F- K <name>dfs.datanode.network.counts.cache.max.size</name>( L8 e& v/ Q9 ]. i: B1 s% D
<value>2147483647</value>) l* s( R- ^- L9 }* y3 H5 b) p( M7 x
<description>( l+ b% ?( I7 K' T t
The maximum number of entries the datanode per-host network error
' P: h$ @6 m# G4 ]% C) _/ t0 e count cache may contain.
4 q& v$ N- I1 Q0 z </description># f+ t a! s" l' M' }( E
</property>
; y! C9 Y7 s4 ?6 F<property>
; s8 S/ ^3 \, K9 z <name>dfs.datanode.oob.timeout-ms</name>
" F8 ~# Q% U6 v$ y/ l D# c. P) X7 ` <value>1500,0,0,0</value>
. ?; B( }) [* e; u5 ? <description>
: G" |* |0 [3 p$ U1 p X) X3 u Timeout value when sending OOB response for each OOB type, which are! C9 S; ]- t! V6 y; d
OOB_RESTART, OOB_RESERVED1, OOB_RESERVED2, and OOB_RESERVED3,6 s. e9 L8 v+ i2 O0 E1 m
respectively. Currently, only OOB_RESTART is used.
9 L- G5 j8 b# v- F </description>
; P9 Y3 _, w8 W* C, F$ F$ Z. Z) Z</property>
0 O8 R5 f& k4 f: H. W$ B# `<property>
8 p; Z4 k4 X/ u; {" X6 y$ H6 O0 g <name>dfs.datanode.parallel.volumes.load.threads.num</name>5 d) y4 E# U+ ~: v
<value></value>* M( i' [( N6 D& y; V) Z2 Y
<description>
' ?# }, D- X7 a% C+ e' g, I! n Maximum number of threads to use for upgrading data directories.
6 u! h: t; v8 c The default value is the number of storage directories in the
3 |, {* ~3 U( v DataNode.8 W- ^" e* i4 c6 l
</description>, `. h G g$ o. l+ k% T
</property>
4 h8 R7 o J* Q; B3 ]4 {2 `3 f4 ~: |2 S<property>
/ M2 j7 L; b h! B- r0 g <name>dfs.datanode.ram.disk.replica.tracker</name>
& D( M: C. w& `( P# D <value></value>
8 N; L A! s1 Q, Y <description>; M$ n- H6 \% {. i) u' I
Name of the class implementing the RamDiskReplicaTracker interface.# o% N( J2 y! V3 M, c% ]& v
Defaults to) I7 M, o, ~) t) E
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker.
( S$ _, W l6 `$ j3 z: E0 E" D </description>$ P, h8 X. ?/ @3 Z) P5 m
</property>
- @5 Q0 V" @2 ] Z5 I2 i<property>: [# w, G6 C4 r9 r* _5 A
<name>dfs.datanode.restart.replica.expiration</name>
5 I2 A! T$ p; F' k <value>50</value> x7 w" p0 C/ M, }
<description>. s Q% K& y/ R1 d$ c
During shutdown for restart, the amount of time in seconds budgeted for! C. J0 p! D! s. j
datanode restart.8 g% l& q4 o: E! p" Z! b
</description>
; Y* Y0 L; T) E+ E</property>
, A( H( U" K8 R6 f1 L0 E<property>8 c0 z3 T1 H4 Q% ?2 j
<name>dfs.datanode.socket.reuse.keepalive</name>
- m3 O& D% I3 v7 x. D5 `. D* b9 x <value>4000</value>
, J. s% E* `9 `: l <description>8 ?9 s0 k" p7 E/ D3 F( h0 E! c- @
The window of time in ms before the DataXceiver closes a socket for a% L2 H$ e2 E0 f- C& u
single request. If a second request occurs within that window, the
* t' ^; P9 |2 }7 _ socket can be reused.8 h* L) X- H) o2 t/ T a
</description>
; c3 w3 C8 A( Y% f* x2 U</property>
1 E& [- b z: U2 A5 @<property>* U g+ q- ^! D ]
<name>dfs.datanode.socket.write.timeout</name>
# q; K2 y2 f! w( ^4 { <value>480000</value>& i# K' m. S8 x) K, f) I7 R5 Y
<description>9 m; _/ _. Y% ~4 L8 G* }5 P" S
Timeout in ms for clients socket writes to DataNodes.5 k7 U( [5 f7 r) ^3 o
</description>/ w6 s$ e: M/ i2 u" B8 Q) R- G
</property>
a3 |; ?4 E# ^: W" }$ I<property>
3 A4 o: X3 s! \5 l7 u: E <name>dfs.datanode.sync.behind.writes.in.background</name>
7 M8 K" y3 J' [ <value>false</value>! u6 Y3 {6 m3 c) s1 x) B
<description>( S6 k* u1 h, g- t( N, e+ [% ?
If set to true, then sync_file_range() system call will occur6 l$ H" I4 J- `, j
asynchronously. This property is only valid when the property
0 z- f% D7 y6 o2 a9 W6 j x8 w! P dfs.datanode.sync.behind.writes is true.
7 v3 J; \# i& A# Q( B </description>' g& F) F( c( q3 D+ T$ v. o+ W, a- n
</property>' M0 j3 [2 ^( S! V9 l
<property>0 u+ ^/ ]$ m- o3 B8 B. `
<name>dfs.datanode.transferTo.allowed</name>' O; v& S0 B, r0 |4 I
<value>true</value>
. t( ~' Y6 V1 w1 J+ |: o$ ~ <description>
1 y7 l( w' M0 V% u If false, break block transfers on 32-bit machines greater than' I1 I |; k- B. S9 l/ \ B& g
or equal to 2GB into smaller chunks.
' T( S3 x, J4 K# q3 \ M </description>) `9 _8 q. @, h4 Y
</property>6 K. l$ o0 ^/ U* O
<property> ~% R' E7 A; f/ r2 Q2 w
<name>dfs.ha.fencing.methods</name>' p7 u2 h' |7 i( Q& l& }) s
<value></value>' k! z! r7 h6 x
<description>
2 n% R8 R. y% |) T3 \ A list of scripts or Java classes which will be used to fence
& V5 h. D7 g& ]- ~2 Y the Active NameNode during a failover. See the HDFS High P/ s( V) c5 b- g5 D
Availability documentation for details on automatic HA' g3 M8 f/ P8 D6 L- L$ ]# M" H
configuration.
% V$ s- M/ o: `' d9 ^0 m </description>3 G# J+ Z! n7 ]& k9 V( K
</property>* m. G4 \4 q( w" N9 |6 p; D
<property>
. w8 u2 \; @5 f7 L. e# ~: F <name>dfs.ha.standby.checkpoints</name>0 ]9 X7 f* E" @" E
<value>true</value># P3 T f6 {. `/ t1 |
<description>
: j5 h7 o$ U7 t4 K If true, a NameNode in Standby state periodically takes a checkpoint
, F& J' F2 X3 p6 T8 B- k8 l( D of the namespace, saves it to its local storage and then upload to! X2 L. M2 A# l( Q$ ?
the remote NameNode.
2 U/ Y! K" F! O' N6 W </description>
3 _, Q$ s3 |/ d/ ^</property>
1 G. U9 j: j- r# G! ]# w( n$ L3 M<property>9 V# j# ]6 T/ W6 d* ~
<name>dfs.ha.zkfc.port</name>
: c! i; F. [& O' p# | <value>8019</value>6 D9 G0 M% f& Y" w
<description>
- ], `8 x: [" x; Z8 p, n- ]# W The port number that the zookeeper failover controller RPC
( z$ q/ b3 x* D8 [# A7 w server binds to.9 P5 s5 _* n& p* d' b5 `
</description>; }! X# H5 S! a1 k: w
</property>
X% {, p6 c, {% w<property>0 r4 G5 o0 w& j) l, z; O
<name>dfs.journalnode.edits.dir</name>* R/ s2 [" j4 j+ f
<value>/tmp/hadoop/dfs/journalnode/</value>. m& W* D9 q" E' ]
<description>
, [6 N" I% B% n3 t O The directory where the journal edit files are stored.( e# N% x( [! K( O2 {
</description>
( Y/ C9 R) w% l9 B4 a</property>
3 s; B8 C1 z1 L- {, i% r& i% C<property>! A ]( P$ I6 X0 v9 i
<name>dfs.journalnode.enable.sync</name>" h: L! C y1 v! J0 H* n0 `
<value>true</value>
+ m# f6 |* Y' Z S8 C9 @0 O0 c/ K <description>5 l: q5 N5 F6 Y1 y0 V
If true, the journal nodes wil sync with each other. The journal nodes/ p4 U2 b% k, B0 r+ V/ V
will periodically gossip with other journal nodes to compare edit log' T- C: b% c( o2 V
manifests and if they detect any missing log segment, they will download" H. o# q) E( J
it from the other journal nodes.
' V% C- @6 R0 p. l </description>; }% T9 N; _ |- n3 J3 H' }
</property>3 f T; A. D _' y' q! G C
<property>3 W6 B8 w Z$ R4 c% M, I
<name>dfs.journalnode.sync.interval</name>
+ W7 ]' d7 B4 @9 p# f- Q& r <value>120000</value>) y. M3 ^4 }8 `4 S. U" r
<description>9 H" J$ J; E: b0 U- k6 P
Time interval, in milliseconds, between two Journal Node syncs.1 G6 T& x6 J: e# h
This configuration takes effect only if the journalnode sync is enabled8 \1 ]! y% j! N
by setting the configuration parameter dfs.journalnode.enable.sync to true.6 ?% y( t. `9 M
</description>
8 R. w+ ?% {7 b: y</property>
2 W+ `, @, z0 b0 u; M<property>7 D0 K/ O2 o8 d* ]# N* ?5 K4 {" Z
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>+ _0 _3 j" w# B# G
<value></value>
) b( ^' j/ N6 `/ x! l! w: Z <description>5 Z3 J w; T' o7 }# X4 j
Kerberos SPNEGO principal name used by the journal node." c" f' ]' d& y
</description>+ Y- ~7 `) ]6 y5 a7 X- P p5 n
</property>, ^8 s/ J" \1 s8 w5 K6 C: G
<property>
0 n+ G' \) c: U1 C k <name>dfs.journalnode.kerberos.principal</name>
5 ] `3 y) p z0 N <value></value>
% q) }- M9 L: J4 W <description>
t: f( D/ x: n8 o/ u Kerberos principal name for the journal node.
5 U" k. `" v: J. o7 r* M& z3 j( V. o </description>, t3 g5 C% S2 a7 A" M3 e
</property>7 z3 J# q+ Y% c
<property>
3 z `. d6 Z: D6 S <name>dfs.journalnode.keytab.file</name>) B' H* X8 ^9 I
<value></value>
7 j% r+ Q: d" u) n' I <description>, J8 F/ c1 l) a/ G9 ]
Kerberos keytab file for the journal node.
& o! K# f( } @' q5 h' q </description>
; g& o# [ a7 ~' i# a4 c</property>
/ r0 j0 {3 d3 E& w<property>
6 G# d0 V& P7 l0 j4 p <name>dfs.ls.limit</name>
7 w5 k; R# W" a <value>1000</value>
) @9 L0 _. F6 E1 e- @0 ? <description>$ h0 s( e) U w4 ]: q' B
Limit the number of files printed by ls. If less or equal to' M: G- b! G) Q" z- V/ E+ w: |( l
zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed.
/ J5 j' \0 e5 a2 t( l4 @1 R) H </description>
8 Y* u/ H6 s1 t. p' X' Z</property># t2 q+ I0 W# R" l3 L3 d
<property>
" y/ `5 D0 E5 {6 c/ l5 B2 l <name>dfs.mover.movedWinWidth</name>
: `, l* o: |' e1 i0 D& M3 P. G/ b- D- z <value>5400000</value>
$ I" v0 N. |$ c' B; { <description>
' I- e0 }$ C1 |4 a) }" Z* Q) j The minimum time interval, in milliseconds, that a block can be, G8 U: E! X& N- M/ X& y/ ?9 J# p
moved to another location again.* F! ~) L# g/ F- g, x% f
</description>8 l7 Y r' j* k! R: T* ]6 f" t
</property>
9 _' {7 q9 M3 _3 j$ Q<property>0 S- \8 i+ f3 q
<name>dfs.mover.moverThreads</name>
- O- I" w, o6 K& y8 l4 x$ C <value>1000</value>
: o% a. v9 P; A, |& m <description>
# `0 X6 [, ~) _( j6 Y5 x* T5 s Configure the balancer's mover thread pool size.
6 }+ R- S/ j% r8 l8 o6 w </description>1 \8 E* G# {0 Q+ [2 c0 A
</property>- r9 ~9 k0 n$ [: @3 ?6 ]5 e
<property>" V5 k; ?% s, B3 k- |
<name>dfs.mover.retry.max.attempts</name>
|4 ~# R( ?" p& \5 O4 H <value>10</value>
- Z4 M5 C% b9 M ?, I <description>" o5 b4 E4 ?" Y9 X( b k
The maximum number of retries before the mover consider the" R( d+ M" `; l: Z5 N2 t+ d. C7 P
move failed. B' c+ \1 w. e/ A0 R# \7 G) |" M
</description>
7 c. D, b+ F6 w& ]</property>6 m; [& |" k# j# y. O
<property>' `* L$ F/ @/ i6 x
<name>dfs.mover.keytab.enabled</name>7 A: p- _; U1 i6 x" y, Q* C" w% K
<value>false</value>
0 {4 @8 `, ~' v. O; E# I <description>& E' m0 Q* p1 H6 W5 I- \. N
Set to true to enable login using a keytab for Kerberized Hadoop.
6 t7 R0 j) p* @$ T; n8 `) n </description>
9 T r9 h8 ~( Q5 Q+ t7 L, X</property>
$ x; s1 a# c# \! b0 c8 Y, V& I<property>
, \! Q* c2 t& g: t. y$ G9 \ <name>dfs.mover.address</name>
% z, {6 w" s- u8 [/ i <value>0.0.0.0:0</value>/ D; X+ F# ^3 b2 F' @) D' Z" H2 Y
<description>
; [0 _' H) i7 l1 a* a& O$ r The hostname used for a keytab based Kerberos login. Keytab based login+ P. C( q* ]4 Z& L
can be enabled with dfs.mover.keytab.enabled.
/ N- U" R9 N$ H+ x </description>* }1 a- C) n* `8 `, T
</property>
0 C* Z% [+ w5 z5 s6 {& |* `# E<property>
+ T+ Q- n8 ^+ Q, X' U <name>dfs.mover.keytab.file</name>
; B. Q0 [9 }; u' r) e' u, q" x <value></value>
3 L4 a& w5 P0 s. b P <description>6 F2 ?. o! _; W9 P! V+ _
The keytab file used by the Mover to login as its) \3 G6 ^5 x. f( Z5 v( B& H
service principal. The principal name is configured with
1 o% n/ B2 b! {6 }" f dfs.mover.kerberos.principal. Keytab based login can be2 [1 W! f" k$ f+ l, V( G
enabled with dfs.mover.keytab.enabled.
$ A+ Z1 }1 m, S' C* R9 R5 l$ H" Y </description>
3 q! p% w8 i2 P- e! h</property>
3 x$ D6 h( K2 u' Q3 z7 `7 |<property>5 v$ j# ]4 N% ~' _) K
<name>dfs.mover.kerberos.principal</name>7 X* |1 c5 Q l' u
<value></value>
- O5 |% E' s0 e/ W <description>
* W9 F7 E' B6 t$ ^; e" `1 T The Mover principal. This is typically set to# b5 e* P6 @8 a$ s6 i! b! U) @
mover/_HOST@REALM.TLD. The Mover will substitute _HOST with its) y8 w* N% U8 d/ j, ?; l2 c
own fully qualified hostname at startup. The _HOST placeholder) e- o! M8 C) H- K
allows using the same configuration setting on different servers.
( k& e, z$ e5 O Keytab based login can be enabled with dfs.mover.keytab.enabled.
+ A2 E3 w: z& O# c6 _( W </description> N* o7 X5 M0 t
</property>: B8 c& X( z3 e9 ]0 L$ }( z! _
<property>
5 z0 X1 D, i" U0 N ` <name>dfs.mover.max-no-move-interval</name>+ F! N8 Y' @/ c8 y) x; U0 c7 o
<value>60000</value>' }# X3 t* ^; a! v; Z
<description>
0 [8 E! V8 I N! h9 R9 ^2 b2 @. { If this specified amount of time has elapsed and no block has been moved/ x' f# d. W/ v5 N
out of a source DataNode, on more effort will be made to move blocks out of
; e& {9 N4 h Z; e. o this DataNode in the current Mover iteration.& h; g/ d9 ]9 t) a8 G2 L2 O
</description> G- o9 Y3 O6 B2 {7 _# Y
</property> o4 r7 N& Y( Y" l
<property>7 b5 ~# c- p) i8 m1 f
<name>dfs.namenode.audit.log.async</name>
& m, q) o j9 T <value>false</value>0 }+ P( k. P0 M. O2 P
<description>
* }. a% i% T8 H6 L! k! F7 b If true, enables asynchronous audit log.
' G9 @" W, Y/ T T </description>
c/ U0 l3 d, o P</property>/ l1 v* E1 x" @) w. y) e! f
<property>
; p; q5 ?8 u( B- P8 |6 t( z <name>dfs.namenode.audit.log.token.tracking.id</name>
) m1 Y( f/ | N. I <value>false</value>
3 R% ~7 H- Z+ w9 A. n$ { <description>; ^, b4 P3 r) b1 K7 N
If true, adds a tracking ID for all audit log events.
; _* z9 v, q( k% F </description>* s+ T% {# v# W" ?! P7 d
</property>, @( v; o( i" `! s0 h( Y
<property>+ ^' B7 [; _% o6 s5 Y: U+ ^0 \ R% \2 }
<name>dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction</name>3 I1 H) v$ x4 j) i7 N% k
<value>0.6</value>$ @) B; Z- h4 D+ V
<description>& Y2 z' }2 @* l" j) X4 V
Only used when the dfs.block.replicator.classname is set to5 s7 `: x# V, _% i$ |
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.: Q; L0 A& Q6 O4 v2 J4 ^, t3 U
Special value between 0 and 1, noninclusive. Increases chance of! D2 ~ S2 c2 W
placing blocks on Datanodes with less disk space used.
4 k# ]0 |$ \3 u5 S* y- H </description>; x; {) f1 g4 }# } W. `& Y- z
</property>
3 L% V4 P* f& @" U<property>2 K1 d# C; C+ b4 |* w" s1 `3 c
<name>dfs.namenode.backup.dnrpc-address</name>5 y3 l& I5 D' h ^
<value></value>
1 a: [8 u: |( w+ G <description>
+ J7 F/ z3 `) R% K Service RPC address for the backup Namenode.% j! X: y0 m/ b4 {
</description>
: K# X/ G2 U4 g* Q$ L& \</property>
9 @" s$ w, ^) Z9 y$ u+ @+ \<property>
; F3 i4 b# @" r8 g& \' J, ? <name>dfs.namenode.delegation.token.always-use</name>
0 `' k2 n6 i5 R+ {5 b <value>false</value>/ H" K! _7 [% x' K$ l
<description>
; F0 n& _0 ^# C For testing. Setting to true always allows the DT secret manager4 ]0 ~; }9 S0 X" a
to be used, even if security is disabled.
' o Q7 s! H3 ~& B8 {" n: t- m </description>
( g: @' K4 g4 r; l/ d</property>
; X% I0 G- E! A' K# m- k' Z, D% K<property>
( H5 m8 D" S' T6 o <name>dfs.namenode.edits.asynclogging</name>
% j" h$ q1 V( {) a3 J <value>true</value>
0 m4 y, ~! L3 z# p, W <description>4 q! W }3 F* z( b5 ?
If set to true, enables asynchronous edit logs in the Namenode. If set1 h7 @3 `# b& X9 H; U* P: S
to false, the Namenode uses the traditional synchronous edit logs.
0 l4 p) h- y: K) s/ V </description>* D2 x u9 ~. }( e: j
</property>% w" k! ^, g, Z0 |
<property>6 V" P4 N9 ~' Z- M5 P! s
<name>dfs.namenode.edits.dir.minimum</name> v: Q+ y% z$ V7 D% G {1 b
<value>1</value>* @1 u( q# C" Y( K
<description>7 t$ u0 f7 q" {. r9 {2 \
dfs.namenode.edits.dir includes both required directories
" S& E: Q( W( [) v. t; g (specified by dfs.namenode.edits.dir.required) and optional directories.
* p w) N9 r' e; I C! C7 e3 S The number of usable optional directories must be greater than or equal
7 T c% y1 }" W! _ b8 y to this property. If the number of usable optional directories falls$ F- v5 O% Y$ W/ m, G
below dfs.namenode.edits.dir.minimum, HDFS will issue an error.# G2 Z$ @3 [/ q. j+ g$ N" ~' \
This property defaults to 1.2 P# r, o/ Z( j$ o# G& F7 W" K" S( @
</description>
& w+ W; W1 C8 N3 P, Q- r( ]</property>
% n' g/ I; Z1 B4 ~4 {. d+ h<property>* {- Q2 X6 S I; T& Q( e: P5 \
<name>dfs.namenode.edits.journal-plugin</name>
1 [, W8 v! `$ Z' Z/ r5 U7 _ <value></value>. z+ F: m$ x7 Q' d x
<description>
- G# T& [' v2 g: v% i) i When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir,1 R r% R1 z f2 F% U) L7 P
and it encounters a URI with a schema different to "file" it loads the- B& k4 k( {9 E/ M
name of the implementing class from
9 l; C9 r8 Q! e$ J) X& Z4 } "dfs.namenode.edits.journal-plugin.[schema]". This class must implement a8 i$ H& l/ a' @9 |8 e* Q
JournalManager and have a constructor which takes (Configuration, URI).6 {; ?! ~$ J; \6 _% N9 K1 i! q
</description>, E1 d6 y: q* v( S& O6 o# b) C
</property>
. I# G \6 x2 b# j P+ P- n1 @2 i<property>
4 i% R5 T! G a <name>dfs.namenode.file.close.num-committed-allowed</name>
+ q, M- [) J5 t$ L% y <value>0</value>
0 ?: K" R. c0 Z- f4 Z2 a3 y8 T# E <description>
: R( x8 F9 k, Y, g Normally a file can only be closed with all its blocks are committed.: w) {5 J0 n" I. Q/ C
When this value is set to a positive integer N, a file can be closed9 x9 f+ O% H6 l% B# e: @
when N blocks are committed and the rest complete.
4 K! N. D5 c7 c3 Y) z$ V </description>
- [- L8 m! x" c& K# V; W1 q% j</property>+ t5 o! r! Q- d
<property>
) A9 k( l6 u. r' [2 x! v; S3 S6 O <name>dfs.namenode.inode.attributes.provider.class</name>4 B/ n: E! ]* e) Q, t6 Q
<value></value>
! M" K/ g) K R( r& h# y- n5 ? <description>
' s/ K( n! c1 I/ x4 x1 x1 z. N" a R Name of class to use for delegating HDFS authorization. ?& u& W$ p; d7 E6 i# f/ r
</description>
( R/ |; [" f+ G1 D. x+ K</property>+ F4 D- B: r- h( q$ H4 m' C
<property>" T* `( S1 n+ h: W3 B Q
<name>dfs.namenode.inode.attributes.provider.bypass.users</name>
. G: E' z2 y" ]- V& K# c <value></value>* R+ A- d2 x& r; r4 {
<description>+ ~; M2 V. s/ `( X8 k! Q+ v
A list of user principals (in secure cluster) or user names (in insecure
. B' ~" g( F, G6 L' L8 h cluster) for whom the external attributes provider will be bypassed for all
7 L( w8 v" K6 f9 T operations. This means file attributes stored in HDFS instead of the
/ J, @/ r! ^1 m+ V) J" K external provider will be used for permission checking and be returned when' O/ x/ y4 E m ]* T! X, t8 E9 U
requested.. `8 |3 G3 T3 I0 M
</description>
2 r9 Z& a$ ^+ `/ a( `1 n</property>
; m' X- M2 Q( p" _ Q; W<property>
* a% H/ w; K& |6 Y <name>dfs.namenode.max-num-blocks-to-log</name>
% P& m; w0 n2 } <value>1000</value>
# F, D& z) T6 x$ L <description>
f8 b w8 \7 ^ Puts a limit on the number of blocks printed to the log by the Namenode
9 O9 [. _$ `4 n: B after a block report.
5 H$ P) @5 c1 d- H$ t) n Q1 c </description>( u0 \5 }8 I( d) [
</property>) D1 p2 V1 L- t$ R3 k
<property>
K& h# v: X+ j9 j& v <name>dfs.namenode.max.op.size</name>- S2 s5 B3 s: \+ K. H, e
<value>52428800</value>, ]" Z# U' r+ l' C+ ]
<description>
, H$ H' ~+ K+ G8 f/ [8 Q Maximum opcode size in bytes.
. _6 r# L6 \- q u3 I' ]* a& q$ d8 t </description>
) Z; u; S$ l$ @) p l% W</property>
. ?$ U0 S# ?! Q<property>8 h/ i; ?* ]8 @! q' N0 x
<name>dfs.namenode.missing.checkpoint.periods.before.shutdown</name>) @, g: r2 G* v0 A- b
<value>3</value>
; ]2 X2 k! @1 k: ^& ~ <description>6 b6 U8 K% \/ x0 j7 N& A
The number of checkpoint period windows (as defined by the property7 ~, D; b* A$ `6 i
dfs.namenode.checkpoint.period) allowed by the Namenode to perform
* j+ S9 s7 k0 `! B saving the namespace before shutdown.
9 N. O4 x- a* O& L; }2 V, F </description>4 c# N8 ?% Q1 e7 V+ v
</property>2 q" Y- a$ a7 c& p4 u/ Z9 x
<property>6 G7 T* Q E; U
<name>dfs.namenode.name.cache.threshold</name> b0 F1 {4 S) f$ \
<value>10</value>
+ S, C( H0 }# V3 o/ W ] <description>! r8 B" e7 u/ K' f! _7 a$ i
Frequently accessed files that are accessed more times than this
/ i3 `0 `# j5 z% q3 u/ L+ b# ` threshold are cached in the FSDirectory nameCache.8 @) h) ]5 o: A; o4 {: ]/ b
</description>1 z9 K9 E" J- R2 h4 D
</property>
+ P' w6 c4 D9 j5 t# S+ \<property>
% [7 d8 T* l. c% [ <name>dfs.namenode.replication.max-streams</name>* o" Q- l. g! x4 H% D
<value>2</value> W( F% Q M" E8 p
<description>1 F8 s& P; D9 z7 C
Hard limit for the number of highest-priority replication streams.
# E Z' j; B" N. d </description>
0 r1 h. X. q# V4 j' j# j6 z</property>* N8 t# Y# z! ~' T# P0 h- z& K
<property>* v8 v, g) C7 a9 l. t1 V" f
<name>dfs.namenode.replication.max-streams-hard-limit</name>
* M. [# }7 {; y: U3 N4 L+ t2 V: i <value>4</value>
* s( `* W; a! Q# f3 S <description>
/ l) ^5 r+ V6 f Hard limit for all replication streams.
4 V7 o$ E; C. u1 O, x </description>2 _* T4 @: T6 j. \/ F. z4 j
</property>6 C/ {7 w1 E h' W. M, n4 n: v+ U
<property>
/ P0 I" i2 Z+ c( Y# N <name>dfs.namenode.reconstruction.pending.timeout-sec</name>
/ ^2 h- i0 d: M1 D <value>300</value>( ?- a/ i# ~" j& A
<description>
3 x8 I$ b: e, w, [ Timeout in seconds for block reconstruction. If this value is 0 or less,. Z% f$ ~% i! |
then it will default to 5 minutes.( r5 D$ z- s V+ o# l
</description>- E8 A2 ~8 O/ O9 k
</property>
2 x4 [% g% `9 I( T, b! K$ x# K<property>
- [1 r& a T" I9 \& m; _- U <name>dfs.namenode.stale.datanode.minimum.interval</name>
& d$ C% W3 h* b# k1 P- p1 q1 m7 Y <value>3</value>
- F: j: w+ h1 o9 i: \ <description>
{6 Q* s5 J% \& ` Minimum number of missed heartbeats intervals for a datanode to6 u& M+ u. a3 Q5 F# Q. r
be marked stale by the Namenode. The actual interval is calculated as ~$ f% I# C9 k; T+ ]7 p; n
(dfs.namenode.stale.datanode.minimum.interval * dfs.heartbeat.interval)$ g0 @4 ^: H. H! v' v& p
in seconds. If this value is greater than the property/ Z1 C1 `7 d9 E
dfs.namenode.stale.datanode.interval, then the calculated value above
3 Z1 o/ \( D9 k% h is used.7 _# y' t1 d9 G, V) }4 D4 E
</description>- }; O9 k2 z* c5 `
</property>
; K% |+ }; k# C9 e<property>$ \$ C6 @. h: ~3 h" Q U, ] \% d
<name>dfs.namenode.storageinfo.defragment.timeout.ms</name>' ]4 s6 d+ f/ v- R8 C
<value>4</value>
2 L, e5 Y! O2 I1 x; i' \8 u9 |& J <description>* E7 A* c" j; Q$ w, i( ]7 X, i
Timeout value in ms for the StorageInfo compaction run.
- |8 q7 I' O: W- z$ m </description>
. Z' z ~ j% X</property>! `, R0 c" n/ V2 m
<property>
/ D* R! y4 z- E8 G. M <name>dfs.namenode.storageinfo.defragment.interval.ms</name>( m b! j& X' D" {8 D. D! |
<value>600000</value>
2 D$ p5 ^: f; H) g0 A7 { <description>
Y) q) ]/ S a. }: m, @, t The thread for checking the StorageInfo for defragmentation will
3 K3 e" C6 v: v' l run periodically. The time between runs is determined by this
# w5 h! u' G/ e2 Y: K+ c: u# s7 e property.1 L- j4 M- p3 P. N$ ?2 N
</description>
4 h/ J/ ^, T1 q! {</property>5 [0 ^* {2 z( D* p5 x b. t
<property>3 _4 T3 X" F* u
<name>dfs.namenode.storageinfo.defragment.ratio</name>' L# e6 `( R; ^* v
<value>0.75</value>6 t4 v T3 ?( G @ Q. y
<description>/ Y! t. ]& {( p
The defragmentation threshold for the StorageInfo.9 H# H/ j) T& `
</description>
9 s& y: J/ D1 R</property>
( @5 N2 x' l; x: g- j<property>
$ \. m3 \; U1 `0 J <name>dfs.namenode.snapshot.capture.openfiles</name>
) k X7 Z4 U$ U! r$ Z <value>false</value>4 V D; {" J# M k
<description>
. v8 w' T- [& Q9 G8 z7 H; ~ If true, snapshots taken will have an immutable shared copy of' h' ]$ g" u5 W2 [6 \4 G2 {# S
the open files that have valid leases. Even after the open files
8 }7 c* R' s7 e# g' j grow or shrink in size, snapshot will always have the previous/ T' @2 e) w" l2 @( D
point-in-time version of the open files, just like all other5 l' b) s$ z k3 {% B
closed files. Default is false., ~9 Q) h2 }% i' F
Note: The file length captured for open files in snapshot is# _; C6 r+ ]* T+ u8 b1 u
whats recorded in NameNode at the time of snapshot and it may
# u9 F/ C" N5 \3 Y2 f be shorter than what the client has written till then. In order+ O, M( {! D2 ^8 w3 n+ H4 l. C
to capture the latest length, the client can call hflush/hsync g' O+ T5 W5 k8 I) t* U l6 g9 P
with the flag SyncFlag.UPDATE_LENGTH on the open files handles.
L5 M, h" r. i1 \. [8 B </description># ^/ m% h! p$ \' I* I
</property>& o! P9 I8 P( _9 G- C5 k* @2 ~& ?
<property>
F: o, g. X+ U <name>dfs.namenode.snapshot.skip.capture.accesstime-only-change</name>
9 M% D6 |7 D2 P% _ <value>false</value>
, r2 b' t, s8 K8 e: u2 u! ]* k# A <description>4 W& ~$ @3 v( r8 B8 K/ d5 l" b* i8 D
If accessTime of a file/directory changed but there is no other5 m; O/ `' X* d( k5 W( p
modification made to the file/directory, the changed accesstime will" m4 m0 ]2 M, ?$ @% R
not be captured in next snapshot. However, if there is other modification
% L' s! Q4 }$ C" e# P4 ] made to the file/directory, the latest access time will be captured( x1 o4 y D' R& T1 Q
together with the modification in next snapshot.4 R+ i% p A5 [$ [! t) e
</description>4 c3 n8 }8 B0 [: B8 d7 E( Z$ P
</property>/ }' [9 h: H8 y3 R$ J
<property>
' r2 q; q( R# o! r0 H' P( ?3 ?$ M8 C <name>dfs.namenode.snapshotdiff.allow.snap-root-descendant</name>- g7 X3 L0 H y" R% m% Z3 f
<value>true</value>
# T+ |8 i0 t3 a/ [$ y <description>
& e, {( @8 U" Y- r If enabled, snapshotDiff command can be run for any descendant directory
. @5 y6 U8 C2 v4 v9 M: ? under a snapshot root directory and the diff calculation will be scoped
& W$ c% r5 A1 x6 D6 Q to the given descendant directory. Otherwise, snapshot diff command can% M" @1 X- Q2 @1 n* m/ Y3 T
only be run for a snapshot root directory./ k- I6 N, d+ F
</description>
2 R1 h: j4 N8 y8 t y</property>5 Z: J J3 t* n5 X3 B; j
<property># ^3 ?- b5 l' I- Q4 S: Z
<name>dfs.namenode.snapshotdiff.listing.limit</name>& ~* X: P! d/ g( ^7 @9 E; S# t1 N- Z s
<value>1000</value>' I8 |1 S& n y$ E. Z% j
<description>
' u6 e/ F$ x/ t: ~- B) k Limit the number of entries generated by getSnapshotDiffReportListing within3 z, D# ^& H I F- D
one rpc call to the namenode.If less or equal to zero, at most4 L/ d) r( m+ V5 r/ P
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT (= 1000) will be sent
8 s2 D( t* k Q& D' O across to the client within one rpc call.
' a; {4 H) z: L, p+ x" ? </description>
2 g2 S r' `3 ?' T* {</property>
9 L S4 j; x; ^<property>
0 s B; p9 q& |! O <name>dfs.namenode.snapshot.max.limit</name>
+ c. B; q! i6 p% R1 k" p" T" [ <value>65536</value>
{; m' ^$ m0 L; B7 h: N2 o <description>' f0 h$ ]" o) a( X4 H
Limits the maximum number of snapshots allowed per snapshottable
$ Z! M m" [7 R" m$ B$ S/ A directory.If the configuration is not set, the default limit
' c- X- r0 s# u0 C: j2 b for maximum no of snapshots allowed is 65536.
6 l3 W0 D4 l% g. g# G* w X </description>
* B1 J8 t) j$ _3 @5 s) e( ]6 ?</property>
) l: v, M. l8 r! u2 g( P& o<property># b7 ?- \8 A. G& l0 b# C8 g
<name>dfs.namenode.snapshot.skiplist.max.levels</name>* r- R# J. M3 x; h, L8 B. _! F
<value>0</value>
$ k5 E( u# M# E- D9 \$ r. F1 A3 R <description>
D8 h5 o/ {$ K. n& s Maximum no of the skip levels to be maintained in the skip list for
8 E/ P) `/ V8 P storing directory snapshot diffs. By default, it is set to 0 and a linear
2 O' j, O% _* F5 G4 }3 x$ v. n6 V list will be used to store the directory snapshot diffs.
9 Z% r, Q" O( ~# z0 ]5 o3 e( |$ A </description>: M/ N8 v! \) ^" y1 t! F7 [
</property>
2 P/ E. D+ c8 G<property>
3 W- E1 I; g' X3 }- C1 W <name>dfs.namenode.snapshot.skiplist.interval</name># @0 k" d* U) X2 v8 Y: `) H( A, b
<value>10</value>& R8 y# I/ u" q
<description>
4 v4 p- M+ d% p% l7 ~+ g I2 N! l The interval after which the skip levels will be formed in the skip list9 x6 o& b F8 s2 J- Z
for storing directory snapshot diffs. By default, value is set to 10. q. _' l, t w* b5 b
</description>
% V% _! C3 a" R; n: c9 p</property>
8 w/ p3 b( E0 F! ~- d, ]3 N" S<property>/ E. X" ?1 G# y2 C* n
<name>dfs.pipeline.ecn</name>% c, l8 N) l: _/ j& c) L) r+ G
<value>false</value>) x3 a# c0 W! t, X4 ]
<description>7 y! a" z* a; |" e: X
If true, allows ECN (explicit congestion notification) from the
* _$ I: C, e# C0 T! K Datanode.. Q6 ]) b% ~6 l/ y
</description>" t$ ~4 q" u4 s2 l8 D
</property>" z/ h6 O. D5 C* x3 G6 i3 n& B
<property>& Z5 U' M& i/ p' F
<name>dfs.qjournal.accept-recovery.timeout.ms</name>, y( }6 [0 p( I, }% b0 N
<value>120000</value>
5 g9 y: x& q3 C" p5 S <description>7 K# ]# J. R% M, m3 c8 O; x
Quorum timeout in milliseconds during accept phase of
* A. a' \7 H- j; e0 T" E8 R recovery/synchronization for a specific segment., t7 M* l. r: [+ X4 F, k3 ?) \# X
</description>
" K7 ^, ^0 m! m% ?% H</property>$ h& f0 N: Z9 n: h! o p$ t
<property>
% d1 V" D/ s$ l. Z; A( P. h <name>dfs.qjournal.finalize-segment.timeout.ms</name># K7 ]6 [8 x, H4 x- l9 a% g* ]
<value>120000</value>& Q/ O3 t ]8 g+ W' }! H
<description># D$ W/ k2 b! l5 b4 V
Quorum timeout in milliseconds during finalizing for a specific
% A. d8 j. L$ X+ I: b8 M( C5 d segment.
2 b8 m$ [# ^/ i: s+ q0 { </description>
7 ]+ q/ [% |3 V! _5 k3 j; j( W</property>/ {8 E3 ]! n1 ~0 _
<property>
! c' o+ ?- C5 L" k; p7 K0 C <name>dfs.qjournal.get-journal-state.timeout.ms</name>+ @. N# u1 @) V( |' P8 K! H9 U
<value>120000</value>( D' i" h" Q% a+ m3 H8 f& Q* b
<description>
$ S/ ^" P0 m6 x& T* f. S. X Timeout in milliseconds when calling getJournalState()., T1 Z* N1 t. Q' K
JournalNodes.) h! F4 Z% `" Q9 p
</description>
3 h7 H/ \1 q+ l2 J& o3 b</property>6 F5 _# [4 N% f
<property>% Z# j; @4 e" b7 B _0 U' X
<name>dfs.qjournal.new-epoch.timeout.ms</name>
# d0 i6 [# E* ^1 U7 _9 R <value>120000</value>5 u* ^- I* H7 s2 P8 T( a! B
<description>
2 K7 m: h: |0 X( b `4 p! | Timeout in milliseconds when getting an epoch number for write7 x/ ?) C% ` A! i: S
access to JournalNodes.
! r3 d4 W# K+ y/ I% r </description>
( N8 `' `+ h: J4 b+ p1 k) E% O* Q& u, L</property>& M1 y6 y P$ Y/ p) B8 ~; G' e
<property>
0 i O m' [/ W. c' W0 c0 s <name>dfs.qjournal.prepare-recovery.timeout.ms</name>0 Z$ q0 J) @1 h3 y
<value>120000</value>
' ^" ]0 D% ]: D% g0 N5 E <description>
2 D: x% I7 e1 H Quorum timeout in milliseconds during preparation phase of
. s" Y9 {7 S' Y- L: ? recovery/synchronization for a specific segment.
: p3 ^& M7 _4 d* Z0 W6 K$ a7 P9 i </description>
9 K5 V( q, B4 g</property>
`& K ^- Y9 C<property>0 b% u6 Z6 p* X4 r# b
<name>dfs.qjournal.queued-edits.limit.mb</name>
[& w# N0 H, R <value>10</value>8 O2 J" f; Y F
<description>
1 ^, y4 N/ R8 ?5 c& z. N% \6 { Queue size in MB for quorum journal edits.
& R* R+ R( W$ q0 t0 B </description>& R9 M; n8 c6 t! Q, X# v
</property>, e7 ?) [8 i: [& O4 F+ y
<property>: Y- s" Q7 R+ E+ O3 O1 k- l
<name>dfs.qjournal.select-input-streams.timeout.ms</name>. f$ N! K% @ C$ J* ^ s
<value>20000</value>
8 Y' A; G, ]6 n% _" W <description>! D% h* t# f/ N! ]$ x6 v
Timeout in milliseconds for accepting streams from JournalManagers.
7 h3 o% _) D- q1 ]4 w/ N( [ </description>
2 d g/ d. e" k- _+ x0 h, r</property>
3 S7 y- U5 g# Q7 X' ?4 B! n5 R* F- X<property>% L7 m u! ^) O
<name>dfs.qjournal.start-segment.timeout.ms</name>
" [' P- x9 ~- C <value>20000</value>
' p: V9 U0 R' {2 Y- {) h <description>
# E1 a+ L0 r: r1 z/ v Quorum timeout in milliseconds for starting a log segment.4 i4 f. Z, u: @4 C! ?# n2 b
</description>
! u- J; P' D2 v% r1 z</property>
/ O2 m8 R8 o: j- ~/ a& \; r2 Q T<property>
* e+ o# o% v3 M) {+ ^% _! H. C <name>dfs.qjournal.write-txns.timeout.ms</name>9 H; }% B) Z& I4 ~* {- a9 K9 f9 N0 Y
<value>20000</value>
' L9 H' z& V: T2 M, T( z <description>
7 N4 V5 e' x1 u' a- m Write timeout in milliseconds when writing to a quorum of remote
% N9 |) T" x2 Q4 } journals.
6 z6 Q- k, L8 [ </description>. l* ]& g( L- p6 W% s" p
</property>: s( }& j% ]* N; [4 R# I: Z7 a6 R
<property> o+ [& A8 s6 x; }: A+ e
<name>dfs.quota.by.storage.type.enabled</name>$ {8 ?/ g4 T; F4 s2 G( A4 Y3 B4 Q
<value>true</value>
( T8 D4 u+ Q% ?# F9 m5 {/ S <description>
$ L; _9 }% D8 L/ Z9 ^ If true, enables quotas based on storage type.2 {, X! Q8 P, I1 U8 W- m8 d; g5 o
</description>- z. j2 E# h# D5 Y4 K, n9 A" m
</property>
4 Z/ i: Z" v( T. _0 |<property>0 k6 A0 @3 n* W' y0 }. _: J
<name>dfs.secondary.namenode.kerberos.principal</name>
+ v% F" u8 Y+ Z <value></value>
1 O+ G1 J, i; b- M. H <description>
/ S1 j: A: C/ s! U/ j Kerberos principal name for the Secondary NameNode.. ~5 L* a" v1 A( K9 o" C/ k8 T2 X' i
</description>! b2 a& X" k& k& z) a" ]2 D+ S
</property>( u5 T! K& c8 j; l1 L6 v' K' H
<property>1 L s# \) f! u% ~2 M
<name>dfs.secondary.namenode.keytab.file</name>
8 l \% E4 `% v; B* L% v, N <value></value>
; h A) P1 `& m0 D! A8 N <description>2 b5 n6 [4 _5 O. C
Kerberos keytab file for the Secondary NameNode.
2 u) g: x; x5 z) C/ `! R </description>$ Q9 `* ?! n4 a p
</property>' W, L- N& m1 L2 `- D
<property>- h8 Z& X; ^/ s: y6 {
<name>dfs.web.authentication.filter</name>
5 z1 ^/ U" U4 y/ `- G- Q <value>org.apache.hadoop.hdfs.web.AuthFilter</value>+ W3 {2 W: C7 z- ^. Z3 J5 n3 ^
<description>8 R9 k8 k* E2 A% n+ Z4 o
Authentication filter class used for WebHDFS.- [/ f2 \% K5 Q" q
</description>3 o# ^6 s \) T/ w0 O
</property>
1 M7 W; l- @' u# q% d6 J2 f<property>
( @& A$ a' H, j8 {2 B1 f/ a <name>dfs.web.authentication.simple.anonymous.allowed</name>+ f: J* H4 v" p% Z3 X1 t
<value></value>4 }8 t8 C7 @: z
<description>3 q2 v& F; u- U4 i9 {& [' \
If true, allow anonymous user to access WebHDFS. Set to5 j: w& c( j" x& a [
false to disable anonymous authentication.- _" X# }( O9 C* x9 }7 o8 t: L! ]
</description>' O: `# {; E& i$ J
</property>8 R; V0 @1 t0 x
<property>" t$ O9 ?% |* s0 I) k/ j
<name>dfs.web.ugi</name>. @+ H( b8 |* ]; \4 Z
<value></value>" [3 Q, l2 f; Z; M- m
<description>3 ~; Z7 k. J* T4 K
dfs.web.ugi is deprecated. Use hadoop.http.staticuser.user instead.
( g: V) ^3 [& I, Q </description>
! H+ J; O5 F+ j0 c+ G' C/ [</property>
- ~6 B& X: o7 R; g<property>
* z1 @# G: x1 ? <name>dfs.webhdfs.netty.high.watermark</name>
0 ^- t! L6 d! | <value>65535</value>
2 D; i" z$ b3 O7 h6 q" H. ?$ E <description>
7 S3 P* z/ @2 S3 L$ L High watermark configuration to Netty for Datanode WebHdfs.3 ]& x( m( |% c- S- o( c( R# p3 G
</description>
e \; u7 {& F/ Q9 Y! ]3 _</property>0 s( a& e, ]" R+ g% N( M
<property>/ u5 W$ r4 O: L9 [# M
<name>dfs.webhdfs.netty.low.watermark</name>
4 \# i6 d5 L& ] <value>32768</value>
8 V$ R7 ~7 X% c) D3 l <description>
/ f9 G7 F6 U6 X6 Q. I9 f' j _) t Low watermark configuration to Netty for Datanode WebHdfs.# X; I: Z1 \4 S; L
</description>
. R. M( X' N7 L& F</property>3 T# H' Z+ ]3 [! d; K
<property>7 q3 P9 v, {& H" s. e5 M* _2 X/ P; `* K1 |
<name>dfs.webhdfs.oauth2.access.token.provider</name>- Z- g4 H6 Q; j0 q- O
<value></value>8 S( `; ?% F* f9 K, m
<description>
; ?& A" Z5 h6 l) Q8 q Access token provider class for WebHDFS using OAuth2.
4 b) C5 U9 y% M" u9 t$ E0 s; J% I" F Defaults to org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider.
- F& \* g) \' ^& k H: B </description>
5 u3 t& l/ d3 c$ ~& V</property>3 B; w; j! f3 m+ U
<property>$ [. ?2 S- s0 l( x& |7 D" N
<name>dfs.webhdfs.oauth2.client.id</name>
6 P& s8 n( D4 t/ [ <value></value>
' c0 h3 L+ ?, E1 l d$ o+ t3 @8 ^; A <description>8 [/ N- a" q/ k- k1 S# \8 _
Client id used to obtain access token with either credential or5 k- A# [. \7 Z7 A2 [, }6 c" l
refresh token. |+ X7 R$ I# t2 L$ C/ D6 ^# g' j
</description>
J, N) N" J: y0 V4 [" F+ \! j- H8 J</property>
. [: j+ _7 Z0 D: u0 M<property>
S- c/ P8 B7 S- d" |* z$ ` <name>dfs.webhdfs.oauth2.enabled</name>5 {) |! L, Q6 n$ ?1 X
<value>false</value>1 t, U" d8 N+ N+ T
<description>- X5 P6 f7 x5 K+ L9 F
If true, enables OAuth2 in WebHDFS
0 Z: g& O' V! z6 T </description>4 i( I" }" A6 ~3 W) Z& Q1 g
</property>
% G% |" G5 u' T: C# e' I9 O4 c<property>
- e. t. S. d+ t2 X <name>dfs.webhdfs.oauth2.refresh.url</name>
) Y# ?' W% S# c' J <value></value>9 \+ c6 h( j# }; x' ]
<description>! F/ U9 a- y+ [ x& G1 n
URL against which to post for obtaining bearer token with/ |: P/ m( @8 m( M
either credential or refresh token./ R8 L \+ ~6 q
</description>
0 x$ d3 s6 N c3 g</property>: o: r# F0 N- S3 K2 N& ]
<property>
' J7 S) W# [) |" L" Y! e( r <name>ssl.server.keystore.keypassword</name>7 p! h; t: m+ Y( w( N; R- Q
<value></value>% }$ e% H. S# F% H+ Q
<description>
+ x# S. Y# Y8 d' X Keystore key password for HTTPS SSL configuration
8 W6 G) [, _. t. i7 _! g5 M( Y# f! z </description>
& O2 `* B* d1 S( R+ m, ^. P</property>
3 Z3 y! r+ R8 R2 {<property>( W( r. y& F5 l4 i/ i" k" o0 j; L
<name>ssl.server.keystore.location</name>
9 u# q# e, F% w+ l <value></value>! F& w/ i# d. M$ ^
<description>& a9 k: Y5 ^* Y H T+ A
Keystore location for HTTPS SSL configuration
f6 }. N( X3 F4 M- v </description>
& J8 j3 }" p/ C/ c</property>4 U* [3 [ ?6 o" j
<property>- u5 w. ?6 x( N9 n3 \1 o
<name>ssl.server.keystore.password</name>
* ? O# R& y# C+ x <value></value>
8 `9 h1 G% v7 y! @ <description>
3 \: Q2 c/ K6 F4 I# ~ Keystore password for HTTPS SSL configuration7 U2 ` }* I2 A$ q* E) i7 j
</description>. H/ p/ J8 e$ N3 k$ W* x
</property>
/ h T6 z6 i- X4 w6 g; s( X$ p9 P<property>0 C! {; D+ G6 b3 W- T# E) P
<name>ssl.server.truststore.location</name>
0 p# M+ b. ]6 u <value></value>
/ c3 B' t6 Q6 v. _3 M <description>
6 o6 ~; B0 a8 Y; P7 {. s8 F Truststore location for HTTPS SSL configuration* L3 S! T& B7 i) d0 Y
</description># u X. ~; c3 ]3 t2 |' q$ @; e
</property>
+ d3 G$ |& j3 v3 y<property>3 \2 g- s% v; @) t D1 P
<name>ssl.server.truststore.password</name>. e: t) ?5 i5 }3 o4 z% q7 J
<value></value>
, t% U+ ]7 ?/ ?$ ]8 R% { <description>
$ U4 c% _3 {1 K. p# L Truststore password for HTTPS SSL configuration$ H" ?! Q4 x5 X4 v ^
</description># c" B6 F7 O) A! e a% Q7 l
</property>% S" o3 M/ s j# F7 m
<!--Disk baalncer properties-->
, P! N M% v( X* y- O <property>. Y, c! F; W J. d
<name>dfs.disk.balancer.max.disk.throughputInMBperSec</name>- s, Q" Q6 m' V" v" A
<value>10</value>
6 @8 }( z) Z, F# D+ V <description>Maximum disk bandwidth used by diskbalancer
2 x1 N; g0 C& R5 z9 _! E: s during read from a source disk. The unit is MB/sec.
" K9 Q% y0 G# ]; F9 f4 e, X2 E G </description>: L) c. l& Q0 e; w. c; p
</property>
" ?. [- C7 t8 P# P$ l <property>1 d- D* Q. ], Q' ~# g
<name>dfs.disk.balancer.block.tolerance.percent</name>' w. |$ ~+ U! Y
<value>10</value>
, t* C9 p$ r) o8 }& }2 i <description># }9 ^# i- Y5 U
When a disk balancer copy operation is proceeding, the datanode is still. Q: N; E# C! g D1 v
active. So it might not be possible to move the exactly specified8 z' Y" L$ b. ~% F; J+ w
amount of data. So tolerance allows us to define a percentage which
1 E1 P. M' H6 S# h defines a good enough move.
# A. L v/ z4 |' D </description>' A1 s- B4 U; ~0 J& n: o
</property>
; l3 C9 B O6 |$ C5 T! j7 z* i <property>
/ e4 Q- M1 d7 v; o1 C& ` <name>dfs.disk.balancer.max.disk.errors</name>9 n* U8 m% o; J+ R
<value>5</value>
# D1 B0 L. s! t7 Q1 U5 O2 y1 a! G' N' ~ <description>4 M; c; q. i/ S- e
During a block move from a source to destination disk, we might# q6 Q5 \3 l5 G& B! P5 G4 @
encounter various errors. This defines how many errors we can tolerate
* c. b' O( m: C before we declare a move between 2 disks (or a step) has failed. N& ^* g8 a: r
</description>3 }1 Q# n1 Z2 d0 g" J! _# i
</property>
- K0 t! R+ y' p2 H" _; l" O6 N <property>% O% ~0 y4 ~9 b/ D
<name>dfs.disk.balancer.plan.valid.interval</name>
- t" D U) ]3 {# f <value>1d</value>/ J, Y3 P/ [) z# p! @, r
<description>
/ z( B" h9 o* P9 B3 r X Maximum amount of time disk balancer plan is valid. This setting
' z5 k( Y9 [' i5 k, o+ {) ? supports multiple time unit suffixes as described in& Q( L! P+ ^; X0 y$ l! x$ `
dfs.heartbeat.interval. If no suffix is specified then milliseconds
1 z3 W. H0 N" L9 w4 q, f' g is assumed.
+ F+ ]5 ?* g% B. J </description>4 t8 f& Q2 `. S: ^% l) }1 M
</property>
! d2 _1 n6 g; {! M$ F: t <property>9 ]: L. e. f; w' ?6 N
<name>dfs.disk.balancer.enabled</name>4 O+ ]% n6 B; l, y) s: Y5 A" A! B
<value>true</value>: Z8 @( v, h) P1 L# b
<description>
- `- V7 l0 Q, x1 l1 N9 H0 w This enables the diskbalancer feature on a cluster. By default, disk- Z( U: V5 E; M/ _( J, R
balancer is enabled.
3 d6 x; B, g; B o# |! K7 ] </description>
: V( d. {& T, ]- C# @0 }; W </property>: J& u+ J$ |8 ^9 ` W- @) j/ ?
<property>
; X" k: H& h/ u9 j <name>dfs.disk.balancer.plan.threshold.percent</name>
0 V* E& s9 f' G" H <value>10</value>
) H' A+ b/ t K( } d" U. M0 v <description>$ K8 Q1 o9 V. Q u! ]$ o; \" \
The percentage threshold value for volume Data Density in a plan.
, N% K/ x# m% J2 W2 s* [ If the absolute value of volume Data Density which is out of: A; ^) \3 z6 {, ^5 O/ m8 K
threshold value in a node, it means that the volumes corresponding to
* [8 Y5 O- V1 i6 W, B the disks should do the balancing in the plan. The default value is 10./ [ n2 P7 y& Q+ Z: u6 w
</description>* ?! u) i- I; _# q! u2 A2 N% I
</property>
1 H8 W+ O: ~1 P <property>
! Q% Y! E5 l3 T0 j3 P <name>dfs.namenode.provided.enabled</name>2 c2 l' J' E# |* B+ Q5 F
<value>false</value>/ v' O. G6 ^# m
<description>
# s, z8 U- S. i8 H4 @ Enables the Namenode to handle provided storages.5 r6 i9 B" t U8 ?3 V3 u
</description>8 [$ d' z. e; O' w* d
</property>
( E, s& j. H# k+ q <property>1 \ \/ F m. C
<name>dfs.provided.storage.id</name>
% {5 Y6 f2 K5 A Q# l: R& C+ Z <value>DS-PROVIDED</value>/ q' C: ^, y: v( U. b" q. V
<description>* w$ x+ i+ E6 M" R& G4 I8 O+ f, w
The storage ID used for provided stores.
, x) Q6 t" k4 Q7 z </description>& z( Y! c' m1 D
</property>2 l1 m: n) }3 {' u8 g/ G
<property>
; _0 B( f5 H) V# {5 q <name>dfs.provided.aliasmap.class</name>
6 l7 Q) E2 v; C <value>org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap</value>
\2 ^. f$ ^: _; @ V+ y7 S <description>0 a, q3 t/ ^- ?' _8 G
The class that is used to specify the input format of the blocks on
$ U/ {- @" s3 r8 i& }/ o provided storages. The default is
0 Q5 f* ~( R3 r# V org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap which uses
( S: \) ~% E3 m5 C" f$ q0 c/ q file regions to describe blocks. The file regions are specified as a
" j( I" X4 Y2 q- U/ S( g$ Z delimited text file. Each file region is a 6-tuple containing the
$ |) \: {% O4 h' ~6 S7 F block id, remote file path, offset into file, length of block, the
, Y% E: s* |4 A# T4 X block pool id containing the block, and the generation stamp of the
( R3 w& I5 o2 t" ] block.* a: a3 c) K$ F- ?: X
</description>) c+ _' o L+ M0 O
</property>" m' G3 s. _$ b) a3 a# M7 z
<property>
. A$ u. E# |, ^& v s; g3 [ <name>dfs.provided.aliasmap.inmemory.batch-size</name>% y1 l) `6 M& I. s" O3 n
<value>500</value>
4 h+ Y I! O* G1 \ <description>
' i) A( l- f0 S The batch size when iterating over the database backing the aliasmap
( \0 n8 S: k4 ]) U" {# U- w9 R9 t </description>- \& \+ q/ L+ W g6 `
</property>
7 R( X {% ?/ T0 V; D( o/ F, y <property>! E5 y# L7 [. H! j9 V2 L2 \
<name>dfs.provided.aliasmap.inmemory.dnrpc-address</name>& A5 B! c( @9 ^: q( r
<value>0.0.0.0:50200</value>
. Y! s. T0 i0 s8 \ <description>
; ]0 F+ B+ G. l) w The address where the aliasmap server will be running0 v4 o. N2 K! e' S5 q9 y
</description>) ^# }# K {5 m; @" O& {
</property>
: ]6 a, g/ x3 J <property>
5 O& m! T' e2 }$ o& _, J <name>dfs.provided.aliasmap.inmemory.leveldb.dir</name>
* Y' V/ Y9 Y& h( E3 V9 {- Z <value>/tmp</value>
, \- O+ `3 |1 a9 y( n# P/ ` <description>
/ q% [3 c& Y# ]: o4 Z/ @ The directory where the leveldb files will be kept
; ^2 j. t+ F& V3 i$ V2 Q </description>. r7 ^3 L5 p% O& \$ x$ ]& g- S
</property>0 x, V1 [& z/ D1 h' {/ x: H6 X
<property>. b! @8 `9 z9 w( f1 a
<name>dfs.provided.aliasmap.inmemory.enabled</name>
5 Z& H8 D9 ] e1 z <value>false</value>
4 Z; C& t% X x$ l, z! o/ c <description>
$ b$ _( p# n% O/ C Don't use the aliasmap by default. Some tests will fail* N1 j6 y$ w$ h5 G1 M! A8 }+ A# a
because they try to start the namenode twice with the
3 x9 C; O! y0 c& R$ u same parameters if you turn it on.
l( t5 |6 b1 G0 [. j+ }) c( Q </description>
& F& i" @; W& u, i </property>: O6 n2 {' @1 q5 S1 N3 m7 @& n
<property>
6 Z2 K% v) `' u- T& ]: Y <name>dfs.provided.aliasmap.text.delimiter</name>
- Y+ T2 A6 n% L& }0 S& t <value>,</value>
" e) A, D; F/ a3 Q$ K s+ b <description># B4 f4 r2 ^' l: z5 r+ t
The delimiter used when the provided block map is specified as
, N" s! C& {6 a5 Z" `+ T a text file.
; R' a R! J6 @) r. Q/ z* Q </description>' E" ^3 l' L% P
</property>: M2 k& z9 S0 C& t7 G/ @
<property>* h3 ^0 @8 x* m# K, B1 C
<name>dfs.provided.aliasmap.text.read.file</name># B, I* m4 _* x+ V: J5 Z
<value></value>
" s: U0 t9 B0 F; e4 e& b: r <description>
. M* n/ ?: D. k- I4 o The path specifying the provided block map as a text file, specified as
$ H' y. N- ?6 |. ]' b a URI.* ]+ A$ R2 _3 D2 l, N( R
</description>
- t2 _0 l2 g2 ^3 Z( m) G2 X </property>
X5 l" k2 x* l2 W1 S+ m5 W! l7 W <property>9 @% T# Y' H8 J
<name>dfs.provided.aliasmap.text.codec</name>9 o6 a, Z) A4 }! l4 a; u( D
<value></value>
+ Z5 G4 k6 u; e6 m' X2 i2 r <description>
5 H# q/ `0 p+ m. y) w8 U( A The codec used to de-compress the provided block map.8 f/ B8 q; n. E
</description>
9 t, M* Q( B; H' b </property>
, K& b: ]* \- P# u <property>. L8 a( P* Q6 r
<name>dfs.provided.aliasmap.text.write.dir</name>
+ v3 c F' z# s1 y- t# Z5 f <value></value>
( i# k6 p$ C7 L. a" P, `' P <description>
# ~% ?6 ^1 D" ~. \6 j- X The path to which the provided block map should be written as a text$ E) u' D6 x& g. O0 E9 i* [4 t2 ~
file, specified as a URI.2 o5 ~& z. A, i; p2 Z5 m
</description>
0 ? v3 O. U$ \9 b$ T" u% p </property>
! d& t( W8 R0 g <property>6 V) q/ r4 c9 T$ u$ Y1 T
<name>dfs.provided.aliasmap.leveldb.path</name>8 Z- b! h3 u6 A+ [3 e
<value></value>
- `) j' H' r( ?; ]" R& b- \ <description>
9 [, W, ]" A; ^ The read/write path for the leveldb-based alias map% y# `2 ?8 i) @; `3 |- W5 g. N
(org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap).
& o+ A& d7 `% a n+ P# A The path has to be explicitly configured when this alias map is used.
2 C* C- m0 @# N! d9 F. m8 r. o# [ u </description>2 g' M/ b' Q6 N9 F1 G
</property>
* B; T- q3 V- k; [- h* d3 G& { O <property>( N& F# K8 K! Z. p" @' Z- T+ ?
<name>dfs.provided.aliasmap.load.retries</name>+ ?; U2 I: J w
<value>0</value>8 \1 s) }% c0 b: V
<description>
) }3 f: }7 F3 N# m& @ The number of retries on the Datanode to load the provided aliasmap;
. i' G) Z4 V1 u& S5 ^ c4 \ defaults to 0., `; ^' F' V4 E; u" d5 j0 k1 P
</description>. G: P. [$ B% S* C n% f9 u; A
</property>
3 c% U& `8 J+ P( [0 ]2 E3 O1 [ <property>5 [, G! Z! b- d4 `$ S
<name>dfs.lock.suppress.warning.interval</name>
' X6 K3 q p, t <value>10s</value>$ F( {# `% j" e- K) K6 y1 d
<description>Instrumentation reporting long critical sections will suppress, w3 A) r; ^6 F. a; A
consecutive warnings within this interval.</description>+ h& n. M+ n4 Q& B+ r/ n
</property>' H5 u# r* ?" V7 D5 j8 w( y
<property>. L4 u0 [( s3 b& g! l6 q. ^$ ]
<name>httpfs.buffer.size</name>- [6 q5 ~7 }7 S9 `
<value>4096</value>
# J7 X; S2 t7 }0 J <description>
7 f& x9 X: \& }7 z2 X. X' z0 l The size buffer to be used when creating or opening httpfs filesystem IO stream.- x- O9 ?# F" p4 R
</description>
, X. h( K) I2 e; U, f# |( F </property>+ `1 W9 e; E% y: ?* X
<property>4 n: f6 a$ o& C9 y0 P0 r
<name>dfs.webhdfs.use.ipc.callq</name>6 Z2 W; }2 i+ R4 s
<value>true</value>1 {" ^; r7 \ s5 r
<description>Enables routing of webhdfs calls through rpc
# b7 v) p! f8 v call queue</description>
' h, U/ N- z3 [6 b </property>" t |; {$ H2 V3 e8 ]- {
<property>
7 [! _1 _; V" k! Y9 @ <name>dfs.datanode.disk.check.min.gap</name>' m" h: Y. o& |% _8 O/ E: s; H
<value>15m</value>) m0 t2 }5 G7 |8 d
<description>' z X. O `7 j4 Z. h. E, w7 _: D
The minimum gap between two successive checks of the same DataNode$ a+ p6 c) `" k
volume. This setting supports multiple time unit suffixes as described
9 o& N1 v( e8 g, g6 z& |2 `% k3 B in dfs.heartbeat.interval. If no suffix is specified then milliseconds. F' `8 b1 `( l/ J
is assumed.5 T% b! b9 \' h( L
</description>' ~% v W1 ]0 {- e
</property>: y* x4 P9 l/ x2 \5 S
<property>
/ Y. H+ `/ ]) j/ z7 b) _) r, x3 h1 H" C <name>dfs.datanode.disk.check.timeout</name>) Y+ r% n4 ?) \) d3 c
<value>10m</value>
/ l6 f- |3 ^; D <description>& A5 ~) A- l) @) I
Maximum allowed time for a disk check to complete during DataNode
% u6 D1 m0 C' a8 {$ U, b" b: P startup. If the check does not complete within this time interval* _' t5 V0 J, k, ^* b& [# T
then the disk is declared as failed. This setting supports- V. C8 u8 V: c& t0 R @+ ^; S; J
multiple time unit suffixes as described in dfs.heartbeat.interval.& t" j& x7 [0 Y+ w
If no suffix is specified then milliseconds is assumed.
+ o1 \5 h) X2 `% Z2 J0 D </description>5 Q2 G0 z! W2 u" S* O% v* ]' p
</property>
: U+ } L$ C% C2 c2 h, q <property>6 ?' m2 J5 O$ O7 E: ?
<name>dfs.use.dfs.network.topology</name>
3 Q: b( j+ Z/ p" L! e <value>true</value>! {% I5 r# u& w5 `+ F. q) f
<description>) Q% b( G3 G$ I+ X1 O$ Y$ f
Enables DFSNetworkTopology to choose nodes for placing replicas.8 V/ U; C( U3 S& a
When enabled, NetworkTopology will be instantiated as class defined in
9 ~. A2 c0 b8 ?! i+ a property dfs.net.topology.impl, otherwise NetworkTopology will be
9 o4 W* l! e* A* r/ a5 ?5 }3 I instantiated as class defined in property net.topology.impl.
4 Z; g3 c" _& v- s$ G </description>
& ]$ w4 Q- ^9 [: \! @$ i V </property>1 p# R4 ]/ ]- E4 p# {* C8 ?
<property>$ |3 R& O1 Q* y8 q
<name>dfs.net.topology.impl</name>
5 V7 _1 `( F {% ]* j- c <value>org.apache.hadoop.hdfs.net.DFSNetworkTopology</value>2 a& |4 ^: B: [) i3 H
<description>' a& V) F X0 D( e. {2 S) ]
The implementation class of NetworkTopology used in HDFS. By default,
& d# R( z2 a5 e( B, ~% u1 D) C the class org.apache.hadoop.hdfs.net.DFSNetworkTopology is specified and
7 Q1 P& C" W% c) j+ s used in block placement.
1 G: y% c- h t. \, g; y$ I This property only works when dfs.use.dfs.network.topology is true.
0 a- T0 V! m z </description>
1 B8 t# p* y/ N6 @5 X* _ </property>
' z, g7 @8 t* t <property>) |9 T! a6 I" D: U8 Q
<name>dfs.qjm.operations.timeout</name>
, \* s6 W$ c7 ]$ p <value>60s</value>
; f, ]) t1 b8 x1 J x7 ] <description>
# b0 J+ a2 z4 {& A( l- ] Common key to set timeout for related operations in
# a; D' D; z4 }) ?* i* E QuorumJournalManager. This setting supports multiple time unit suffixes
3 \- u8 Z6 Q5 b% L W- D9 ]' ] as described in dfs.heartbeat.interval.
: A: ^* a) p* {. W( f! w q If no suffix is specified then milliseconds is assumed.
3 D( D( S) q% n4 h- M9 R </description>
% e: h0 s4 w' F3 l </property>
( K5 q0 G, u B- f# N <property>
0 r2 H$ u/ E: W8 w1 |% a <name>dfs.reformat.disabled</name>
( ?; [) q% S- D. Q E, n& w <value>false</value>& _; {4 u- y! X7 d2 S# \
<description>8 C2 C% q1 `* J( F9 O6 C- L Y
Disable reformat of NameNode. If it's value is set to "true"
3 ~& U6 ]! ~3 z) M$ V and metadata directories already exist then attempt to format NameNode
! u- ^" U7 B- D: O! ? will throw NameNodeFormatException.: u: N2 p0 M: @* C j5 r0 D5 \
</description>
: W/ U$ r/ P4 H q& o </property>1 Q8 K/ V% o, z5 g/ x8 z
</configuration>
0 F0 a) n: r) ^+ z$ a( e9 s3.mapred-default.xml
9 x, g3 I* S( N! Y: g" f8 M, i0 _' c9 y
<?xml version="1.0"?>5 l0 a, k2 m+ f* Z: W2 U
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>: d5 X% J' h4 J% f1 v% E6 Z
<!--
& b' k$ [* r6 A$ H) ?' K Licensed to the Apache Software Foundation (ASF) under one or more
- D" g! E+ o; j& `% ^" J contributor license agreements. See the NOTICE file distributed with; y+ i' V; f+ H6 S3 Y- o% c
this work for additional information regarding copyright ownership./ T, n$ {3 V3 u v1 o
The ASF licenses this file to You under the Apache License, Version 2.0
) `* p' G' D- _! z9 l+ I2 r' n (the "License"); you may not use this file except in compliance with5 W8 ^( W" J/ X; h" }
the License. You may obtain a copy of the License at
* L. T, a$ ] j http://www.apache.org/licenses/LICENSE-2.0* }/ [# J; d* `6 y: o
Unless required by applicable law or agreed to in writing, software6 a: o2 P5 f: x! I: s
distributed under the License is distributed on an "AS IS" BASIS,
( A. t) a) R5 b" g2 t0 H% o WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.% j) m7 E% J" p% t' e5 J
See the License for the specific language governing permissions and g2 L& _/ c: Z$ X" p! p& t1 j
limitations under the License./ j4 i% a" i2 N& _+ ^0 d- [( q8 @
-->
' R$ \. W( E g6 W$ P4 J" a<!-- Do not modify this file directly. Instead, copy entries that you -->
- p9 {4 q. {+ c) W<!-- wish to modify from this file into mapred-site.xml and change them -->
4 J2 Y; c7 Y7 J- ]2 e9 U<!-- there. If mapred-site.xml does not already exist, create it. -->+ j# r/ \& E6 |, ~9 ~+ q# a
<configuration>
* ~6 a d, \5 J+ c) ? |<property>
\3 i* ], N3 ~+ \( A( q2 u <name>mapreduce.job.hdfs-servers</name>7 O: W" G' I( z+ m( ~6 p
<value>${fs.defaultFS}</value>
* o2 b+ } _. I3 P</property>- Q- f2 L3 Q. u' g+ k) I
<property>
8 W( j* p+ M1 y# h2 C# @ <name>mapreduce.job.committer.setup.cleanup.needed</name>3 r7 d! Y2 a# G! Z+ H1 }
<value>true</value>+ U* b; I: a K5 W
<description> true, if job needs job-setup and job-cleanup.8 L: T" y' S9 V' m6 Z6 [$ E
false, otherwise
2 K9 y3 t, I" K \& w' N2 ] </description>
! T2 Q. @ J' j. Z</property>
( d- X' K$ G& h+ R0 P, V# T: e3 ^, c<!-- i/o properties -->* k$ [$ c: f8 \9 x/ `0 @& y
<property>
* `# ?+ K7 @& Y5 X! n B1 D& z4 _8 \ <name>mapreduce.task.io.sort.factor</name>
) d3 T' r3 [# |1 \, v- i( T; H$ C: _4 A$ `& p <value>10</value># _0 M4 h$ W. |: W6 o& u3 \7 v1 H
<description>The number of streams to merge at once while sorting
+ e4 }: A5 x% h% L9 d files. This determines the number of open file handles.</description>$ {# X2 L8 Q9 ~) j. \7 J9 H( ]
</property>4 H8 i( t; p! \8 {3 F5 `7 Q
<property>& J& [) I# [0 N2 N' b$ p! E
<name>mapreduce.task.io.sort.mb</name>8 s) {1 K# j9 H' }8 j
<value>100</value>; p0 f2 U; }6 C7 r/ {- a
<description>The total amount of buffer memory to use while sorting' F+ W; y p3 a% w O4 T
files, in megabytes. By default, gives each merge stream 1MB, which
% d; ~; k/ x( p. B should minimize seeks.</description>- ]4 ]4 f) [2 Q* }
</property>
, K4 Y" d: Z( t# N<property>& e: ?6 i5 [+ b$ U) |2 i" O ^
<name>mapreduce.map.sort.spill.percent</name>7 g3 Z! |& f& ]! A" g% k
<value>0.80</value>1 [8 V: |0 D* N8 K# c. A; j
<description>The soft limit in the serialization buffer. Once reached, a
5 X+ u- m. m6 W. p( [! Q thread will begin to spill the contents to disk in the background. Note that% r$ c/ a: @# v
collection will not block if this threshold is exceeded while a spill is
: e" p+ Q# M# Y8 G3 i! I already in progress, so spills may be larger than this threshold when it is, ^: X. r$ }; N2 H9 H% ^5 p
set to less than .5</description>
7 M3 b0 w: D; W+ [' B/ E$ M) k$ j8 {</property>
6 I7 l4 _! k3 R$ |9 t$ i6 @<property>
% l- m& ?/ N" Q: w0 I+ q6 M <name>mapreduce.job.local-fs.single-disk-limit.bytes</name>" @: l, r' T4 p2 s$ U8 `; G
<value>-1</value>' E3 M9 E! r) e9 @7 O8 b$ [! w
<description>Enable an in task monitor thread to watch for single disk
?5 W3 v- z, W consumption by jobs. By setting this to x nr of bytes, the task will fast( |# b8 T% {7 |4 B
fail in case it is reached. This is a per disk configuration.</description>
8 F( H1 G7 K, u/ H9 j# v</property>4 a- {" R( Y9 W( K; n' `
<property>
1 O# C8 g2 ]4 C5 G- l3 ?2 ]" U <name>mapreduce.job.local-fs.single-disk-limit.check.interval-ms</name>
8 R. m Z' Z: ~ k# B G/ ^- F: h <value>5000</value>! V+ \9 u! E" _
<description>Interval of disk limit check to run in ms.</description>
% j# b' _# |# K% _</property>- T4 v& W2 M, x) f
<property>
2 ~2 L6 O3 C: R6 R* u* a <name>mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed</name>
+ j: {; H$ @+ B* k' I7 U2 V) T <value>true</value>8 D& p3 I: T9 T9 k7 _1 {4 F/ }7 z
<description>If mapreduce.job.local-fs.single-disk-limit.bytes is triggered m; \1 H8 O! R1 q
should the task be killed or logged. If false the intent to kill the task/ J, C5 @. H: s
is only logged in the container logs.</description>- u8 [6 D+ G& [% n& q
</property>3 f6 y3 r0 `, p' W6 I& G
<property>6 G) ?/ e* B; }# H% J# h% p6 r$ J
<name>mapreduce.job.maps</name>
9 X1 J# V \4 |$ L6 | <value>2</value>( C- C( M8 z9 [8 d
<description>The default number of map tasks per job.
2 [' ]* ^! A" E% x% ~, x( S Ignored when mapreduce.framework.name is "local".4 ~! q9 U2 i+ p
</description>$ Q' p" v) U: M4 c: h
</property>
& n! e+ l5 K4 i/ t) a4 q<property>
$ a% @. y/ C7 C% z, Y2 D% a: F <name>mapreduce.job.reduces</name>0 q+ _9 m7 j7 N! t
<value>1</value>/ `: e+ G3 w* P% N4 q( v
<description>The default number of reduce tasks per job. Typically set to 99%
$ S" S% P% `( l) \. n of the cluster's reduce capacity, so that if a node fails the reduces can
# G+ n$ l& t2 m1 P8 b, Z; n" ] still be executed in a single wave.
; C5 P0 r% u5 D8 ~ Ignored when mapreduce.framework.name is "local".
% F( h9 W9 P/ S </description>3 {! ]3 }* {3 A s: ]1 Y% ^0 b
</property>5 r, M7 T* d& Z& ]7 F
<property> n& v, w/ k: I& Y" A4 @
<name>mapreduce.job.running.map.limit</name>) h+ m$ D7 y _2 u
<value>0</value>2 F! D6 V9 s. \/ b3 t6 K0 n! c
<description>The maximum number of simultaneous map tasks per job.# p1 S v0 H6 z9 T7 t. w& d& ~4 {
There is no limit if this value is 0 or negative.9 A& n- b; Z) w; O7 a3 n
</description>1 o5 q7 @& C; x1 O8 Q) R. f
</property>
* x7 ~" v% Q0 g& v/ r9 ~ p. V<property>! _9 ^5 p: E/ b' f/ U+ @
<name>mapreduce.job.running.reduce.limit</name>
8 |* t& ]8 s( ^0 m <value>0</value>( M6 ~+ g" J9 w" e$ h) u
<description>The maximum number of simultaneous reduce tasks per job.
( s1 N. T; {5 u7 s There is no limit if this value is 0 or negative.* `) _/ O) p2 J* N6 S. p$ _
</description>) r8 |+ k/ v* d e3 V
</property>$ \& I, h w/ ^1 Z' E2 x
<property>: w9 e$ v8 E! O1 d1 k
<name>mapreduce.job.max.map</name>
1 K! D3 p) t7 ^% ~, R7 S9 ] <value>-1</value>0 {+ w8 \0 l2 w& m1 c- h
<description>Limit on the number of map tasks allowed per job.
0 t$ f* k( B( t4 z) V* N There is no limit if this value is negative.
$ b; ^8 M4 I; N </description>
6 H- v6 c% I9 h- Z% P9 ~ m# S</property>
0 _7 ^# x4 H' D7 B7 w <property>
( W" y+ }+ Z2 @; H: o# |0 ` <name>mapreduce.job.reducer.preempt.delay.sec</name>3 g' Z) X/ H3 M3 P" R I
<value>0</value>
) P5 E2 l; X" b0 M! [+ v <description>The threshold (in seconds) after which an unsatisfied
$ e5 b( o+ B% Z( o mapper request triggers reducer preemption when there is no anticipated2 ^ m! U8 \) l8 s) ^2 F
headroom. If set to 0 or a negative value, the reducer is preempted as
8 H4 r$ I& f# Z: [9 w8 p$ h soon as lack of headroom is detected. Default is 0.
& C% p! u+ \; D </description>; l( I( z8 |* m% y
</property>
' t# D: C* a& y& a5 g <property>
' r' i4 F5 }8 b6 n <name>mapreduce.job.reducer.unconditional-preempt.delay.sec</name>: Q3 k" t9 t7 P8 ^ ]% J" l) ~$ p
<value>300</value>
% w5 Y/ T4 F2 {0 r" w( G <description>The threshold (in seconds) after which an unsatisfied6 N# {/ A% Z0 ?/ u/ u) a
mapper request triggers a forced reducer preemption irrespective of the2 ]! k2 _. y# X1 D0 w& T6 t, q" M% X
anticipated headroom. By default, it is set to 5 mins. Setting it to 00 k1 {' U. }, n3 |
leads to immediate reducer preemption. Setting to -1 disables this
9 f, Y8 W& b" n* I+ y preemption altogether.0 B7 \, b7 T! j k
</description>
; _' b4 U- Y3 g </property>
4 Q* }9 A) T1 G3 K <property>
- A2 E" k, z# v4 K: r r <name>mapreduce.job.max.split.locations</name>
# o. I4 b; U4 f" i7 R <value>10</value>
$ E; V# l( A1 i n' _, G <description>The max number of block locations to store for each split for9 x+ `# ^+ e: O8 L o9 J6 s
locality calculation.* u' }6 ^. F' B# k+ E
</description>
' C+ i# C) F" {2 }7 C- x</property>5 c0 V3 `% E7 j
<property>0 ?% l( y; a4 i% _" Y! C7 l9 O
<name>mapreduce.job.split.metainfo.maxsize</name>/ O* c. ]8 p1 x: h1 A1 n% e
<value>10000000</value>( \9 n+ ? C% u L2 ]) ~& v
<description>The maximum permissible size of the split metainfo file.
8 X# G0 I5 V2 r3 `2 e+ ? The MapReduce ApplicationMaster won't attempt to read submitted split metainfo
+ R3 B& ?: x8 N% v( a) M files bigger than this configured value.6 c+ @/ r/ j2 B3 A- i9 |
No limits if set to -1." C5 Y8 L# O3 |8 Z* R, o5 r" p
</description>5 U7 M1 o, D" q) T/ ~! w, c
</property>7 h i9 g, Y6 k# W
<property>5 R+ Z2 D- U3 b4 U" N) A& F
<name>mapreduce.map.maxattempts</name>
6 v+ G K# w* p <value>4</value>$ M7 a, g5 u4 J+ p* g9 J0 ~
<description>Expert: The maximum number of attempts per map task.& n6 q+ z3 G- `! D' B! N' d) k
In other words, framework will try to execute a map task these many number' l/ M: Y+ F* g- n. r- r5 @
of times before giving up on it.
1 F% l3 v) o6 M, @ </description>$ }0 Q+ ]6 s* h# ]4 e4 W" f
</property>
4 {; a. P# c9 L<property>
- L6 ?5 G/ D# E2 x& d <name>mapreduce.reduce.maxattempts</name>+ ^" U! o- v) w
<value>4</value>! L8 I, D5 U; R
<description>Expert: The maximum number of attempts per reduce task." W. @1 ?. g9 X" ^3 A; x2 Y
In other words, framework will try to execute a reduce task these many number% r8 n, a9 a6 f8 `, B& T
of times before giving up on it.
) E: Y5 A; f" n4 C. B; e' _/ t; y </description>
5 Z" J0 ?8 ^+ M0 W0 k0 _</property>
* k" o, c* V/ T( [<property>
. D/ l: S7 I5 F/ G" { <name>mapreduce.reduce.shuffle.fetch.retry.enabled</name>
2 n2 l- y4 Z; R) a ?1 T' i) K2 W, d8 v# f <value>${yarn.nodemanager.recovery.enabled}</value> O# y+ V5 w3 w: X; y( J& S
<description>Set to enable fetch retry during host restart.</description>! x3 M" E) Y( ^' P
</property>
; ]0 ]& }2 @) z+ u# B& M b<property>, j( x1 t+ H6 }
<name>mapreduce.reduce.shuffle.fetch.retry.interval-ms</name>
- q: W G2 A3 u, p; m <value>1000</value>7 F1 I- v! S% V7 K, |$ z0 s
<description>Time of interval that fetcher retry to fetch again when some( m$ M( g; c! Y$ n
non-fatal failure happens because of some events like NM restart.& W& k w! m+ K! m4 E7 @) [. e
</description>3 R8 d( L% f, X- x% W
</property>
! [3 a% |, j! v4 h c! _% c<property>
1 v# h: P0 W3 r, l- T! K7 [8 P% C <name>mapreduce.reduce.shuffle.fetch.retry.timeout-ms</name>
. b9 f3 o. b6 ?# y2 s6 W) F2 K <value>30000</value>
' E- M5 i5 h! j) p9 M) _/ | <description>Timeout value for fetcher to retry to fetch again when some
+ y4 t0 g. y% K& v+ F F/ z) N non-fatal failure happens because of some events like NM restart.</description>
( a6 F+ h5 n& |- ] i+ \8 Y</property>
; W& f6 I! Y; [" ]<property>
0 m& M% x- p$ Z% v <name>mapreduce.reduce.shuffle.retry-delay.max.ms</name>
9 n% v \2 Z4 q8 z- H <value>60000</value>& x3 z. K9 j5 V" B! J! [
<description>The maximum number of ms the reducer will delay before retrying, v" b* Q, I- F
to download map data.! J4 ] c3 X6 e0 h/ P4 t
</description>
% E/ ]5 S# g" c( T" s. b</property>* `# i3 s J, V) A) k, @6 ]4 K, I
<property>' l# r7 m2 Y+ l m# j9 r7 V8 F. Z
<name>mapreduce.reduce.shuffle.parallelcopies</name>9 _' a8 P: F+ `- U7 y# c# @
<value>5</value>" E1 J& X" F& q5 r- @
<description>The default number of parallel transfers run by reduce
5 S: ?( k1 K; X: Y+ g6 m! Q7 r during the copy(shuffle) phase.
* ^& x! o3 I% E( u2 G* B& [2 J </description>8 [7 h p: g$ U/ H9 }
</property>
7 [3 o6 b' j* P: j1 u<property>' y2 z" M: Q# Y1 j. ?( M
<name>mapreduce.reduce.shuffle.connect.timeout</name>$ i; q/ t. ?8 E: o6 W
<value>180000</value>
5 k( M9 t; b6 P <description>Expert: The maximum amount of time (in milli seconds) reduce
, ~0 I7 i7 @ C2 i2 K J task spends in trying to connect to a remote node for getting map output.0 v* j8 ~7 C# O" _
</description>% {. e$ T% `9 G; i$ |# d
</property>
" H. d* H4 A0 Y& Q) P7 g S) K<property>
* m# w* `+ v. {! |8 Q6 v" ~ <name>mapreduce.reduce.shuffle.read.timeout</name>3 ]' u' ~# R6 c4 H0 d
<value>180000</value>
$ g" @ C# J8 t <description>Expert: The maximum amount of time (in milli seconds) reduce7 N( [! e3 `9 k, \; N' _5 v. \
task waits for map output data to be available for reading after obtaining
: k$ ^% J4 D1 n* y2 L( S9 r0 S. ? connection.
7 }' N6 h3 z+ |- A0 O </description>5 d# [: g- h) j: j' [+ I# j
</property>- t. M& I2 a7 n$ y
<property>6 B1 `% r1 J( m) i |0 d& Z& C
<name>mapreduce.shuffle.listen.queue.size</name>
f6 ]. A% k6 S' @ <value>128</value>
; W* `# \- z% K N5 S <description>The length of the shuffle server listen queue.</description>
) S9 A$ [# O0 ^0 d+ n</property>& T4 S7 @" ? |; u2 y7 O; L) P7 s
<property>
1 G7 V3 F# r% B; Q$ D0 X( ?7 E7 H( _ <name>mapreduce.shuffle.connection-keep-alive.enable</name>5 M3 B/ k5 ]! I: g$ X
<value>false</value>9 r! b- _$ ^/ m& i! [" W( w- V3 A9 s E$ M
<description>set to true to support keep-alive connections.</description>
+ }. _) r# W1 V6 c& Z6 `( Q</property>- g' A6 x. B+ N& X, x! |
<property>' {) Q9 w/ ]9 q7 L0 Z
<name>mapreduce.shuffle.connection-keep-alive.timeout</name>
0 h' V4 Y- _# ^( n( x <value>5</value>, f' j7 D' |) P, P6 u
<description>The number of seconds a shuffle client attempts to retain
, s, |" U$ d: E/ ], c& x3 ? http connection. Refer "Keep-Alive: timeout=" header in
6 X z% X& U. M, A( I5 x. _ Http specification
# I% c0 z" G$ b e) x+ l </description>& Z+ u( X: [) R, o0 V' v7 }
</property>0 w6 |$ L* Q* f( K9 c0 u# \
<property>
( o5 e) o) @- O# A) U y <name>mapreduce.task.timeout</name>
$ u9 n" c% h- q; F S& g/ } <value>600000</value>
K( d+ D" P5 z9 P( b6 y <description>The number of milliseconds before a task will be
0 o8 p7 k% r- K& u1 R$ f terminated if it neither reads an input, writes an output, nor
( ~* j, Y1 |0 E updates its status string. A value of 0 disables the timeout.+ e! a" b( K! @( \( k( ^, H0 ]) s
</description>) B& v4 M- ]6 P! {
</property>
6 s* f+ a& y9 }. H<property>. _' w; x; l& I% l" e
<name>mapreduce.map.memory.mb</name>1 T" y# w1 ~3 p2 t! ^
<value>-1</value>0 @( b8 [$ e( \! R X- N
<description>The amount of memory to request from the scheduler for each. r& ~* K* W+ g3 ]
map task. If this is not specified or is non-positive, it is inferred from0 ^: R5 w W% o
mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio. o3 ^+ @1 v- ] S- z2 h+ u3 e0 w
If java-opts are also not specified, we set it to 1024.8 l: E7 _8 @- }3 V& Y U
</description>
+ R) ^8 Y3 q% M+ J/ E: N6 v. C</property>
b1 k. E9 C% G* F# b2 P& w% T<property>
" D! h6 {" t6 A# L% B/ _2 X4 x <name>mapreduce.map.cpu.vcores</name>! ]8 l' `4 ^$ }2 Y% ~ S, M C' ?; l
<value>1</value>" o9 e5 V* J* |4 L g
<description>The number of virtual cores to request from the scheduler for+ A% z l+ p0 g
each map task. q2 V' K! Z: J9 P& b; H9 B4 y) r4 F
</description>
' K* G4 A' R' [0 }+ M</property>
1 f! m( d, W" j, W( }" ~. J<property>
0 g1 I+ |! _* g( T1 |2 X <name>mapreduce.reduce.memory.mb</name>
- i$ Y9 B2 ?) w0 f <value>-1</value>
, N1 v' b8 ]* o$ Q6 g% {% j <description>The amount of memory to request from the scheduler for each$ s/ Y! {% C4 Z0 \" n! T- E$ h
reduce task. If this is not specified or is non-positive, it is inferred
6 N; i L, m: H3 n/ V1 } from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
6 V; f2 R( C8 i. T' w' p; R2 v If java-opts are also not specified, we set it to 1024.: b# d3 n" w3 x
</description>
+ Y1 c/ j' | y. e8 M- m</property>7 i8 _* o' U1 y$ z7 D3 l
<property>; J) z. ^( |& j6 M. I# | m
<name>mapreduce.reduce.cpu.vcores</name>
6 ^ E$ L. x+ L: y/ S- k# ^) y <value>1</value>
9 D: {8 m0 B/ C/ m2 a9 X2 T <description>The number of virtual cores to request from the scheduler for
% F* }! E* k3 A8 G. R& i each reduce task.6 V2 q `, y7 d# n5 z
</description>) f" I5 |& _% g( l w% N
</property>
% r" B# B4 x( U! `<property>- ]2 @, a" X6 \) B
<name>mapred.child.java.opts</name>5 m# y& r" j& J' n$ q/ ^$ @
<value></value># }0 H& `6 }! ?( r3 D' c0 @
<description>Java opts for the task processes.
6 p6 s* G0 j4 L The following symbol, if present, will be interpolated: @taskid@ is replaced9 W4 {/ {' |+ s2 \9 y* g
by current TaskID. Any other occurrences of '@' will go unchanged.
' G# ]1 L1 }8 \( k, O: N For example, to enable verbose gc logging to a file named for the taskid in# l# _3 D; _# }. g- Y8 u
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:0 V* P. F" l. U p/ b" ?
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc0 `" z- p0 | n2 n* u
Usage of -Djava.library.path can cause programs to no longer function if3 H- G" {+ R. h4 u2 Q$ {
hadoop native libraries are used. These values should instead be set as part0 l% p9 i( n- V; J$ H% M* p
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
" U% N: T9 I R& z* ` mapreduce.reduce.env config settings.
0 J/ l+ C1 `5 x( K" v If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and; f* |1 R# T$ `" O
mapreduce.job.heap.memory-mb.ratio.' A1 [& p5 v9 ~8 Y( O1 ^2 w/ ~
</description>/ F% F4 _4 O* A! \
</property>
% a& Z/ g' o5 p$ O: Y8 k<!-- This is commented out so that it won't override mapred.child.java.opts.
' [. h- i9 e* X$ `- f<property>7 [( X4 m) V/ A; f+ o3 ^% a4 V
<name>mapreduce.map.java.opts</name>8 j) s5 }4 J" c- T _( b( F! i
<value></value>9 ^: W/ v2 {) D3 a! [6 h
<description>Java opts only for the child processes that are maps. If set,$ s6 H. e2 x; H1 f V+ Q
this will be used instead of mapred.child.java.opts. If -Xmx is not set,
, B1 @; F" F* C2 z+ B- w) S" i# w, L. a it is inferred from mapreduce.map.memory.mb and. m6 p7 m9 q5 e; q7 K/ o
mapreduce.job.heap.memory-mb.ratio.9 Y1 [0 P! y% g
</description>
6 N9 [* e4 `% E7 t</property>' v: g; F5 M: R3 T- I3 v
-->
" K _4 j" l5 D$ A. m/ c<!-- This is commented out so that it won't override mapred.child.java.opts.4 r6 u( C# x0 N' N0 B8 b7 [
<property>) R! |8 K& E) d2 ~: J# L
<name>mapreduce.reduce.java.opts</name>! L) t3 L; m |8 m
<value></value>2 Z6 |# S, m' j
<description>Java opts only for the child processes that are reduces. If set,
6 w! ]7 D& s7 Q this will be used instead of mapred.child.java.opts. If -Xmx is not set,) T( M$ N- u, i4 m0 K
it is inferred from mapreduce.reduce.memory.mb and
5 p. |" Q0 z/ a7 d mapreduce.job.heap.memory-mb.ratio.! i7 P) X3 A# ^
</description>
) Y4 e; ?8 F5 [9 l8 K+ D& N</property>! f0 E( I, z9 ^0 Q: u! W) w
-->
# m! |- `! l$ u% |3 u9 ]) d<property>( k; |5 `' b# P% n8 Y+ [
<name>mapred.child.env</name>
+ W- [, I# y5 U* f7 Y <value></value>0 _1 a* E, |( j6 a
<description>User added environment variables for the task processes.) t0 t% u; l! x4 l2 I/ P
Example :
9 _' O6 M$ {" e( }( T6 q 1) A=foo This will set the env variable A to foo
& p( |3 O4 a. s; U' O) g 2) B=$B:c This is inherit nodemanager's B env variable on Unix.- g+ `9 a; t0 [6 G
3) B=%B%;c This is inherit nodemanager's B env variable on Windows.
5 i) C4 z& B! g6 z2 c </description>; A" \* Y7 ^+ Q" B6 {# F7 |" m# _
</property>! N9 a( q5 R8 c
<!-- This is commented out so that it won't override mapred.child.env.0 r. ?$ J" E. N
<property>
4 k) F4 g9 O+ G/ G <name>mapreduce.map.env</name>/ N: i4 w; e- k: P+ g: T6 {/ ]- z
<value></value>
3 j+ v6 ?* p( n4 R3 \- j) t+ @/ J <description>User added environment variables for the map task processes." t4 d' e# F/ D
</description>: ~* k4 g7 V0 k
</property>
/ d, _, j5 Z; I-->; N0 D( O) ~3 x$ i( n8 m
<!-- This is commented out so that it won't override mapred.child.env.+ I0 X* w% }4 R. J6 o; n
<property>1 j O' {/ q- w% R
<name>mapreduce.reduce.env</name>. |! G5 z' X" `: j
<value></value>
S+ D; U9 b4 @# X; | <description>User added environment variables for the reduce task processes.
0 B7 j; \) }0 C* e5 P" n i* ^4 w </description>4 F; S: h& ?+ d* e5 Q& W
</property>6 U1 R a. X% P3 Z+ T8 R* r/ _; Z
-->
! O U/ d* X+ X) X: I% i% ?$ C' T<property>
9 B4 t. g- T+ ~+ Q <name>mapreduce.admin.user.env</name>
) J; ^( ]/ s% T* a <value></value>
7 c, S5 P/ t" A- n2 C <description>8 I" ^% P5 b- G8 ?* a+ k2 Y9 h+ o
Expert: Additional execution environment entries for( ~/ }3 R' q2 F# Q" ]
map and reduce task processes. This is not an additive property.
! n7 ?" c( K, ?) N1 b1 Z You must preserve the original value if you want your map and
- n, ?4 d- y: n reduce tasks to have access to native libraries (compression, etc).
% y8 @# C% U/ R! D8 _$ R! y When this value is empty, the command to set execution* K8 O# q0 c( [( y
envrionment will be OS dependent:) T: E2 |; i6 A; Q3 X$ f# y
For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.8 t8 s9 r/ u( g
For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.+ s/ i& ]4 P" I
</description>' }& f" B1 W# n& i( S* z
</property>
+ B& c4 ~9 n2 v<property>
% @4 ~$ i) k6 ]0 ?/ H# M3 X <name>yarn.app.mapreduce.am.log.level</name>
, G8 M1 h* } `5 V& j <value>INFO</value>
V7 A: W! u# n" |$ W* a& D* u <description>The logging level for the MR ApplicationMaster. The allowed5 s! n0 f; ?2 [! e3 T- V+ n& ~
levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.: N: [8 E( L5 c/ `
The setting here could be overriden if "mapreduce.job.log4j-properties-file"6 P/ \: k; {- N2 y3 k
is set.. M2 N; v$ K7 \6 O
</description>: S; }" ]. M+ K3 G& T
</property>; H& s g; a& p7 {! W. P! N0 q% b3 O
<property>; q( {2 c2 | l2 J1 c
<name>mapreduce.map.log.level</name>- l' S. w: }2 P5 m8 f
<value>INFO</value>
, s: ~9 M+ d# |( P q <description>The logging level for the map task. The allowed levels are:
7 w$ @: m3 Q1 N8 V) N OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.4 r4 N, N8 ^: M* U! |
The setting here could be overridden if "mapreduce.job.log4j-properties-file"2 t% e2 H+ X. X0 z) ~
is set.' Z8 }" h% i/ ] s& Q5 O
</description>- |! Q% S7 o2 l
</property>
7 B0 j4 h, n. w) G5 [: j% {<property>1 c- l& j% V! O$ s
<name>mapreduce.reduce.log.level</name>5 D1 [6 p' }. Q7 d/ I
<value>INFO</value>3 {" j; d+ I5 h) W2 ~
<description>The logging level for the reduce task. The allowed levels are:" ?3 o1 G% d5 n, Y4 e: o' f
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.: @ [! q0 _% i* s& P: n9 E
The setting here could be overridden if "mapreduce.job.log4j-properties-file"% s7 V# O2 \5 T5 V- P
is set.3 X; s$ G% ~: ~8 F% i# t+ y
</description>
% a6 z0 M* U; h2 T& G8 K# w; ?" ~</property>
3 H* ^# [' g" T$ x& j0 I! L4 n/ t<property>' X! |7 A/ @* I! f$ P2 v6 t
<name>mapreduce.reduce.merge.inmem.threshold</name>5 B+ q" R9 x2 z' `, M
<value>1000</value>+ p2 W/ N6 o+ }/ M0 X: L8 j& F
<description>The threshold, in terms of the number of files
[1 T. q1 q" ~! z* @ for the in-memory merge process. When we accumulate threshold number of files7 `; \1 y* @ K' W1 V
we initiate the in-memory merge and spill to disk. A value of 0 or less than
8 ^! r5 [' {4 {$ f1 O; J6 p 0 indicates we want to DON'T have any threshold and instead depend only on. R. T5 }! l! L& o
the ramfs's memory consumption to trigger the merge.
) M7 l) @7 d" S0 a h4 ^/ W: V </description>
) P( p' J' ]0 _: k</property>
) O6 s- |. M" N9 P4 H. @! S<property>
2 s: q# j" C+ _; g <name>mapreduce.reduce.shuffle.merge.percent</name>
; N$ {' i4 [; O& ^) ~! y; I <value>0.66</value>! ?: G# C. }5 P7 {
<description>The usage threshold at which an in-memory merge will be
2 `! i( N& H+ V initiated, expressed as a percentage of the total memory allocated to
0 o1 T; r1 U3 u0 c+ A; s4 R3 Y, x storing in-memory map outputs, as defined by1 j5 r& t, c3 u3 M" F3 V
mapreduce.reduce.shuffle.input.buffer.percent.
) a D% ^% ^2 s: @7 I% g </description>0 {0 W# a* p8 H& V% d6 {
</property>. _" n& F1 s& h0 Y3 D, S1 t
<property>
5 S# s6 }% U, h/ | `: T# H6 G" ^ <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
2 w) ]3 j% w- M5 u <value>0.70</value>5 R: E! P! u. s' f* S
<description>The percentage of memory to be allocated from the maximum heap
, @$ \4 i# F/ x& R size to storing map outputs during the shuffle.1 Y" |+ _( _: b& X ^+ ~! Q
</description>( |7 K" g9 K8 q) |5 t# _
</property># k. l3 b8 ?' p1 S8 t
<property>. b8 x" d, G4 w {, e0 L! F
<name>mapreduce.reduce.input.buffer.percent</name>
# J& G6 G) _* s4 F4 `: l" U <value>0.0</value>
: D6 v, Q m3 T9 L <description>The percentage of memory- relative to the maximum heap size- to- {3 E6 Q) O# ?; J
retain map outputs during the reduce. When the shuffle is concluded, any4 U- b$ E& R: ?( i4 D0 K) O( T
remaining map outputs in memory must consume less than this threshold before
" @; \( L: P D& r0 b8 }$ v, L the reduce can begin.
( |3 i$ } w5 _: v% F </description>
5 L& N" Z+ I; s0 r: H3 Y K: `</property>
8 ?8 F7 T( ~5 N3 M" y: a2 N<property>7 f7 {4 J8 z+ d' J4 k$ O
<name>mapreduce.reduce.shuffle.memory.limit.percent</name>8 y1 {* B5 V3 E
<value>0.25</value>
% U/ J5 O: ^: u+ A M* E% Y7 q <description>Expert: Maximum percentage of the in-memory limit that a. n6 l8 D% x4 Z g- i+ ~
single shuffle can consume. Range of valid values is [0.0, 1.0]. If the value, Y7 f2 f* C: W9 e
is 0.0 map outputs are shuffled directly to disk.</description>5 B3 e" q2 D7 m) q5 c& z, p6 c( R
</property>
; F" M. v4 }2 p<property>! @' k* u, z; D' `$ I
<name>mapreduce.shuffle.ssl.enabled</name>4 z% w$ @2 h" [1 h/ n" t
<value>false</value>9 P- Y: @" U/ b5 }+ r! y" ?: s
<description>% Y% r+ i0 T* p- Z' q/ @
Whether to use SSL for for the Shuffle HTTP endpoints.
% E! X- \$ ?+ ?& M, H$ O6 R( s; F </description>
+ e* i+ l/ o; b: v; {7 x. C8 u4 G</property>+ i* u6 s2 z/ N
<property>" [$ F3 D/ \6 C0 k
<name>mapreduce.shuffle.ssl.file.buffer.size</name>9 q6 ~( [% v4 ?9 s7 l* o4 N% v0 k
<value>65536</value>
C5 W [2 `, h2 K3 G: x <description>Buffer size for reading spills from file when using SSL.
1 O2 K; h6 a8 d9 `. j </description>7 p! s- C# _# I! Z# s; Y" y
</property>5 ?; f4 N& l0 d) O, t
<property>+ ?/ a8 p7 R% p1 e: W& i" t
<name>mapreduce.shuffle.max.connections</name>" e5 @% p- c8 G7 [+ c. ]0 Z$ j" p
<value>0</value>
; R$ q6 J8 G# s$ s; [' G% V- ^ <description>Max allowed connections for the shuffle. Set to 0 (zero): y& R# |2 @4 l+ q% \3 @% d
to indicate no limit on the number of connections.+ Y( k2 q! f& N/ S5 l/ V3 _8 ^
</description>; K. R X# F: A' @* d
</property>
6 Y8 e" }: M0 q# C1 l# K<property>
0 l- W L' X" R <name>mapreduce.shuffle.max.threads</name>
/ Z6 P0 ^. K" w1 i <value>0</value>) f7 L* W( a$ z- ~
<description>Max allowed threads for serving shuffle connections. Set to zero
2 ~, O& a/ T* S" l' C5 B9 k, Q to indicate the default of 2 times the number of available4 m) ^4 F* W0 s P& C
processors (as reported by Runtime.availableProcessors()). Netty is used to
; g5 C; M8 t0 A. W; d/ ` serve requests, so a thread is not needed for each connection.- G6 ^7 i% k+ J, R
</description>
' H2 G8 z0 s. H$ M. n- z; l2 ]</property>
O2 I( Q6 F. |, q; G<property>
: j9 O1 M4 L# l3 K& L ? <name>mapreduce.shuffle.transferTo.allowed</name>, d3 I8 ^) O6 p( m
<value></value>8 c! x' X& O+ P p9 C
<description>This option can enable/disable using nio transferTo method in
' ?4 D5 ^7 q9 U the shuffle phase. NIO transferTo does not perform well on windows in the$ s; `# E# S0 u- s) H! L
shuffle phase. Thus, with this configuration property it is possible to3 n1 L5 F; e: g
disable it, in which case custom transfer method will be used. Recommended
" z3 X' B/ ^( X value is false when running Hadoop on Windows. For Linux, it is recommended
6 f; h/ S& ~3 l5 |7 v3 }9 p to set it to true. If nothing is set then the default value is false for
# S( ?6 w7 f: s! b1 a( | Windows, and true for Linux.
/ }5 v$ z, I( g* S( o7 _! n1 Q </description>
' c7 T1 W9 h% n; p4 n. ^ e</property>
7 |5 ?6 R Y& d1 z. W$ X1 U0 {<property>
2 `) t& \. ~% y+ u <name>mapreduce.shuffle.transfer.buffer.size</name>7 y) X: R9 ?( C8 |
<value>131072</value>
2 v$ K5 t4 U+ F* P* _0 U- l <description>This property is used only if- V. y# b: [+ E" U: `- ?
mapreduce.shuffle.transferTo.allowed is set to false. In that case,
2 z S1 W+ G% h+ n this property defines the size of the buffer used in the buffer copy code: ?, F" I/ D3 q" X6 U4 }
for the shuffle phase. The size of this buffer determines the size of the IO
! N$ f V# m9 i requests.* ~9 I, r' N9 G0 R7 ?2 u# @) B X/ i
</description>
. {; t& j q' i. }: l</property>
# v' j8 D& s1 r# o; ^) U+ k1 V<property>; c/ f6 d! W( k
<name>mapreduce.reduce.markreset.buffer.percent</name>
0 c1 ?2 J0 D* c <value>0.0</value>, _ P, G3 k' H3 K) _+ w
<description>The percentage of memory -relative to the maximum heap size- to$ T3 d& f7 _8 H/ Q
be used for caching values when using the mark-reset functionality.0 [% s0 A8 e2 f9 W+ Y6 T) }+ w
</description>; r* R6 M& Q6 u7 S* U9 d
</property>- @, ^ S* }+ V8 g/ Y* G1 t6 A
<property> }4 r, B% F: C9 i( m. g7 g' c0 Y
<name>mapreduce.map.speculative</name>' N; j1 n, s$ e8 M, V7 C
<value>true</value>6 M1 W* D$ x) E
<description>If true, then multiple instances of some map tasks/ C& M& u0 U N
may be executed in parallel.</description>$ D- Y5 y& K$ O$ f
</property>$ T* S1 F, P' ]. q
<property>6 `0 N: f$ |8 S9 _6 g
<name>mapreduce.reduce.speculative</name>/ B! _! L- Z; q5 u
<value>true</value>
, \* _4 H& P$ O& u9 ?3 ?9 G. L <description>If true, then multiple instances of some reduce tasks% \& F3 Q7 i: i# J9 n
may be executed in parallel.</description>0 D m9 d( V; x% o' \
</property>7 T0 q( d w5 o- g6 }
<property>3 M8 h, h: i+ l c" n5 ?4 e( N9 ]
<name>mapreduce.job.speculative.speculative-cap-running-tasks</name> I5 t: H3 T6 E! Q+ Z; m
<value>0.1</value>; N3 J; L9 a0 E4 O
<description>The max percent (0-1) of running tasks that6 n' t; g) @3 X) v w
can be speculatively re-executed at any time.</description>
! u" q( q7 Y* ?( W: N% ]</property>
+ o& b/ G; h& V: i0 P" e<property>% K. _' e# o: B
<name>mapreduce.job.speculative.speculative-cap-total-tasks</name>1 T# x3 v4 t" b( ~& W& V: {: _
<value>0.01</value>4 |0 z+ U( h4 P8 E8 g
<description>The max percent (0-1) of all tasks that
2 x) G; E/ ]# Y4 a- g+ r- s& s can be speculatively re-executed at any time.</description>/ `8 W4 p( o4 o- ]( U- N+ W
</property>
% m2 R' [" s/ c! s<property>
& t6 l% t/ G7 o) D! \8 \ q <name>mapreduce.job.speculative.minimum-allowed-tasks</name>
8 ~. d' q: a, g- g4 W <value>10</value>
8 g( Z' X. p+ `. W4 f6 ^% K <description>The minimum allowed tasks that2 m+ W, \0 d' w* G0 N8 p7 a
can be speculatively re-executed at any time.</description>
) {! Z# `; ]7 u' z</property>
7 J* S( q: K6 Y<property>& g" [2 G" g$ [% X& V3 q
<name>mapreduce.job.speculative.retry-after-no-speculate</name>* R7 m$ Z; D# P. a; t. H
<value>1000</value>
8 x' M+ w2 d: G <description>The waiting time(ms) to do next round of speculation1 T; D0 k4 g4 i1 |" M3 w H
if there is no task speculated in this round.</description>
! z1 `; Y, V; J& t& O, s: l( c</property>1 o, N. K6 Y- K" G! t* ^
<property>
, _! r" w: C5 f, g: B <name>mapreduce.job.speculative.retry-after-speculate</name>+ i% s1 A* {& u" j4 }
<value>15000</value>
# t$ X/ p2 N- b7 d/ ^ w <description>The waiting time(ms) to do next round of speculation7 `6 R9 i0 c f1 C2 E; n
if there are tasks speculated in this round.</description>$ S% t8 `; L4 x5 P: k
</property>
' N* x3 ?. U( ~! q<property>" a3 Z9 I: X5 D7 p1 B
<name>mapreduce.job.map.output.collector.class</name>
4 G5 T4 h8 W; R4 D. t% C <value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value>
) w( v) J1 M4 u& I: Q4 r <description>9 k$ V, [1 Q4 Q5 H
The MapOutputCollector implementation(s) to use. This may be a comma-separated
$ H+ Z/ _( u. n* Z2 r0 A" O1 x list of class names, in which case the map task will try to initialize each* K2 Y8 j4 Q. C' T4 J4 Z6 i
of the collectors in turn. The first to successfully initialize will be used.( X0 ^. r1 F0 j% p! ^
</description>$ m* u y% Z8 Z( W6 ~7 |9 q% M
</property>
# T6 ]8 o2 x6 z/ T5 O; C2 T<property>
' L5 F+ e2 l' F: Z <name>mapreduce.job.speculative.slowtaskthreshold</name>2 k: L! K' g/ w) v( i. K" `9 B, S
<value>1.0</value>
8 M X: w8 {' v- o) q. y1 \ <description>The number of standard deviations by which a task's
. d2 E8 | v4 c* G' T1 Q# f ave progress-rates must be lower than the average of all running tasks'
5 ?" Y8 t: j' k' r for the task to be considered too slow.! b# o2 k3 ~% p& z; V6 w
</description>/ W/ L9 J9 S) \% \. t
</property>" T" ?3 K# d" g. [$ @
<property>0 V2 {3 S& {& n$ R3 x9 M+ Q
<name>mapreduce.job.ubertask.enable</name>
7 A* r3 W* V8 d; g <value>false</value>% B: \8 j- S" j! {1 }+ J
<description>Whether to enable the small-jobs "ubertask" optimization,
) u+ o1 G# ]7 V7 u) ]' j which runs "sufficiently small" jobs sequentially within a single JVM.4 E, J0 Q% `4 O
"Small" is defined by the following maxmaps, maxreduces, and maxbytes
) @! P, w8 w! R6 `: F! t' E8 b settings. Note that configurations for application masters also affect0 v2 T# u# V/ @
the "Small" definition - yarn.app.mapreduce.am.resource.mb must be5 h# x# |% O( e! d: M3 F
larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb,7 Y+ B* S |! Q
and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than2 _! ?" I5 H' P2 w6 v
both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable9 V' n. E. H1 K1 X' I1 W
ubertask. Users may override this value.
1 ~' }' L q; m </description>
+ v- q' p, d6 b( D0 l4 k</property>
. X( \% n S5 R9 p: F<property># C1 k/ G& T* W; Q
<name>mapreduce.job.ubertask.maxmaps</name>
- r) n" b5 S7 l- c <value>9</value>
+ T( g0 F+ j7 M' B( r <description>Threshold for number of maps, beyond which job is considered3 Y' R# o; J" d+ T
too big for the ubertasking optimization. Users may override this value,1 a1 v( X) D( b& c- L/ {+ d
but only downward., q9 J& L; I* ]2 U8 Z
</description>
; w6 U2 B/ p$ `0 J1 B1 Z! V9 C</property>
2 H- f& r: [) F9 j: R4 h<property>4 a7 B9 [& I6 ?& N: H
<name>mapreduce.job.ubertask.maxreduces</name>& q: t) c; q0 @
<value>1</value>9 ?* k+ o* w" |# T
<description>Threshold for number of reduces, beyond which job is considered4 H4 x0 A# }( c" x9 F
too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT% m3 z" W1 \- r8 `$ B% Q8 k
MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max,
" l- R# o7 m( K5 l' |( ~ however.) Users may override this value, but only downward.
7 |, v Q2 V" W/ \1 L+ I" N8 |5 g* c4 C </description>
. f& X% u1 W" t0 y5 [) t</property>
! P+ n" }, N: S0 c0 |<property>
5 K2 ^0 C" f& ?. i X& ^ <name>mapreduce.job.ubertask.maxbytes</name>- M7 i! E' G" }) ]1 T
<value></value>+ I+ W& A8 ~$ B- u# j
<description>Threshold for number of input bytes, beyond which job is
9 a$ |0 \8 B3 a/ S considered too big for the ubertasking optimization. If no value is! M7 @) g$ z: ?- S/ C6 w2 t* k
specified, dfs.block.size is used as a default. Be sure to specify a
+ \8 j& ^# C2 b0 w8 a( o3 H default value in mapred-site.xml if the underlying filesystem is not HDFS.# f- n' P; t! x7 o
Users may override this value, but only downward.7 v5 R$ Y2 q# p/ D( R8 Z+ D
</description>
. v/ @- s7 k- o</property>
2 ~% c6 [0 Z" s0 y& B4 r<property>
! s$ f' C! P& r1 U# a <name>mapreduce.job.emit-timeline-data</name>
" \: ?+ q7 |$ v( F. r <value>false</value>
* P$ _8 C [% v8 y4 G. i: a <description>Specifies if the Application Master should emit timeline data. Q& L2 c9 ~6 s% X* k& u
to the timeline server. Individual jobs can override this value.
! n# J2 _" `+ M/ j; t$ s$ M </description>
* s% u9 n8 R: C0 a</property>& a0 f- D C+ \$ i/ n# ^0 l% g
<property>
; h' ~4 W1 Z& H <name>mapreduce.job.sharedcache.mode</name>5 y2 S* S# ?; r/ y4 q
<value>disabled</value>
) m- e& X4 j; m( Z <description>
' e I$ }. C5 b3 ? p0 s8 h A comma delimited list of resource categories to submit to the shared cache.1 B8 j. ]* l g, ]% j8 W
The valid categories are: jobjar, libjars, files, archives.
" a% K# a6 S2 a) Y! j If "disabled" is specified then the job submission code will not use8 b' E% K: w" l/ E/ l) a% \+ V
the shared cache.
* ^9 S. @4 j" h3 Y+ b9 R4 I9 s: E </description>
2 s; ~0 R. M. @' t9 j0 q0 h+ M+ L</property>0 n9 ?1 D& v* Y2 H; ~5 v5 D9 o6 G
<property>
' U8 U5 ?* K1 f3 P <name>mapreduce.input.fileinputformat.split.minsize</name>
* w5 A4 k% \. N/ P. S9 k4 X <value>0</value>
; g: l4 E1 ^3 X/ w <description>The minimum size chunk that map input should be split
$ f4 c7 I& R3 F2 X# m$ ? into. Note that some file formats may have minimum split sizes that( h7 |% w( { G8 {0 Q) M/ J
take priority over this setting.</description>
& D& M+ o" k$ p# C) l* [1 M' V</property>
# D4 Y' p& K' H5 v) ]1 L6 i<property>2 r8 W6 {# x* ^4 A0 \
<name>mapreduce.input.fileinputformat.list-status.num-threads</name>; S, l8 S8 W4 i' w( J
<value>1</value># `8 W7 C5 `- ?
<description>The number of threads to use to list and fetch block locations; o3 W6 h5 z% s7 J
for the specified input paths. Note: multiple threads should not be used
5 O7 E! l9 T6 x( f3 F& S if a custom non thread-safe path filter is used.
2 e* S, _# L/ e1 B" O </description>
0 R6 d) o. L% \% P</property>/ Z. j0 [ f9 H+ A
<property>
/ p+ a$ u4 R$ l8 c. e% u <name>mapreduce.input.lineinputformat.linespermap</name>5 P" r, w3 P+ E& P1 @, X# F
<value>1</value>
@7 |6 @1 j( ^: _9 U( s <description>When using NLineInputFormat, the number of lines of input data) y$ S: N/ c7 y. H2 y7 f
to include in each split.</description>
" u; K, }7 Q! V% c1 s</property>
/ O" z# V; v+ j ~3 Y<property>
6 ]4 \( z/ R$ j, L; F% J <name>mapreduce.client.submit.file.replication</name>; R( h6 C" D9 p8 V
<value>10</value>
* w7 H! B3 z5 I% W6 @+ H <description>The replication level for submitted job files. This
1 V& ]( y6 J- Y7 n3 |7 f7 R& ^! C% }: e should be around the square root of the number of nodes.
" v2 _ W: T! f6 Q/ R </description>& J! z4 i5 R2 z* [: D# E' ~1 s. _8 @
</property>
' v, `# S5 U: c& [, x- S+ P3 X<property>
0 y1 @) b; T: x1 Y! N4 P5 D <name>mapreduce.task.files.preserve.failedtasks</name>
+ b5 k3 f$ N( B <value>false</value>
- \0 R G g7 z) F" ] <description>Should the files for failed tasks be kept. This should only be1 n! X9 k; n. s# }3 H* b
used on jobs that are failing, because the storage is never7 o- I! ?% @9 S4 |- L9 M
reclaimed. It also prevents the map outputs from being erased
' j2 s( c* s! S4 P3 z from the reduce directory as they are consumed.</description>" w6 R- \! z' B/ p; k
</property>
, v ]) B! R; c" P9 ?. \3 m<!--/ R* M8 }6 X% t0 W0 T- u
<property>
8 B1 X3 {0 W( O <name>mapreduce.task.files.preserve.filepattern</name>2 M5 Q: z+ d2 h' I* Y: U) w3 k
<value>.*_m_123456_0</value>- b* `1 [' ~6 M1 K: @
<description>Keep all files from tasks whose task names match the given0 U- z% b' R+ V6 o( o' ?
regular expression. Defaults to none.</description>
/ P- h7 Q( [5 v/ [6 A& n </property>& v1 c; c {8 W/ t& Z& V
-->( ?6 j; k/ c* I5 k
<property>4 ?0 s" Q- d* q! x& S# W* R
<name>mapreduce.output.fileoutputformat.compress</name>0 K& T; T- I, k
<value>false</value>6 m8 w0 \( u0 n/ z' a4 j5 E. @
<description>Should the job outputs be compressed?4 Q0 ~0 O4 r/ a" @
</description>
7 I. U' Z- a2 p- u, L</property>
% m( x7 o5 r) ~<property># D4 L' ^# T) b' q7 u
<name>mapreduce.output.fileoutputformat.compress.type</name>3 \5 `* u8 K: U6 U
<value>RECORD</value>
, y: S- M& Z1 d! g' y5 @ <description>If the job outputs are to compressed as SequenceFiles, how should
4 F; I1 o/ ^) ?* E( n! e they be compressed? Should be one of NONE, RECORD or BLOCK.# s7 L& E; @! h- F8 h: X" ?
</description>
0 b3 _/ b4 A( n! k</property>5 X: {4 ?9 o5 e8 L
<property>
1 {$ C$ d2 v3 t" L2 T |8 }" I/ C <name>mapreduce.output.fileoutputformat.compress.codec</name>
$ N; t/ v+ }2 V% b/ |6 `4 I- E <value>org.apache.hadoop.io.compress.DefaultCodec</value>
# b, F0 y& o0 l <description>If the job outputs are compressed, how should they be compressed?6 w9 \+ D* I, d3 ?
</description>! f7 D y' q* a9 P: x8 B
</property>
! U. K( i& ]* G+ O. \6 q$ j<property>) u/ i2 D8 G0 W6 Z* ]$ X3 H
<name>mapreduce.map.output.compress</name>
. S' h' ?% p% F7 z7 S <value>false</value>( ~/ r, D6 \' S e# K: C
<description>Should the outputs of the maps be compressed before being0 E' B+ @* S8 D( y. T% K
sent across the network. Uses SequenceFile compression.
0 f6 I$ g; \" y! c. f, _- m </description>
$ B! d6 C! l" Q3 U</property>+ x& I+ s1 }; R' [) g9 c% W
<property>+ R4 a- Q5 B! l+ e, [5 m
<name>mapreduce.map.output.compress.codec</name> t$ I$ m( L" n9 Q+ V' e; x
<value>org.apache.hadoop.io.compress.DefaultCodec</value>
4 J# n# T5 Q2 I6 d9 x <description>If the map outputs are compressed, how should they be
$ D! d* F0 z' {, {0 d5 e compressed?
' f' K, n z7 o </description>
* D$ w0 A" B$ a+ t</property>
4 X8 }2 t) J4 z<property>
3 ~7 A+ d: H. [" e <name>map.sort.class</name>" C# G1 ]! F0 P2 _3 M7 W1 D
<value>org.apache.hadoop.util.QuickSort</value>
' } u* H3 H+ E <description>The default sort class for sorting keys.9 w+ J. K, z2 a5 d0 A6 l# W2 c/ L
</description>
& v& G* u! c0 _4 i! n R& h6 Y</property>0 p$ ^7 s( b ~. y( G) m; R# Z2 p- R
<property>5 Y, [" c0 ^; e- W) I8 `
<name>mapreduce.task.userlog.limit.kb</name>
+ H+ a( H3 e; x9 p( P5 g* w1 P <value>0</value>
; M2 H" U% G( X# [2 B <description>The maximum size of user-logs of each task in KB. 0 disables the cap.8 \/ y9 w$ P' h8 l
</description>
8 g" b) ?6 u7 B& j% |2 \</property>: D. V. m( \1 v1 g" V
<property>
$ [4 b" ]. R3 G- X' i$ f <name>yarn.app.mapreduce.am.container.log.limit.kb</name>. q* Z0 T& l5 |& G
<value>0</value>
& [2 \) B m5 b <description>The maximum size of the MRAppMaster attempt container logs in KB.
/ n; t6 y. t w" | 0 disables the cap.
3 [% f7 o# {4 G. n </description>
2 a7 s( K+ o/ J1 L# a, b* d</property>
7 ?7 k+ P3 R. t8 d; f9 A9 h& E<property>
- Z- l4 v8 P/ v; w. q <name>yarn.app.mapreduce.task.container.log.backups</name>
1 Z/ @8 Z% p U$ y5 E/ ~) h. M <value>0</value>
( R6 Y$ p; Q$ l& k5 @" D% q <description>Number of backup files for task logs when using& _1 }0 M& S$ o$ {
ContainerRollingLogAppender (CRLA). See
* }! e- W U' G3 {* S' }8 R; j org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,
- }8 k+ ]$ n6 I h; @/ X% J( Y! U ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA
+ l- |% o$ [; Z4 W3 ]( A is enabled for tasks when both mapreduce.task.userlog.limit.kb and
7 z3 L- m+ U$ X5 t; Z* I2 v yarn.app.mapreduce.task.container.log.backups are greater than zero./ v. m, y: z% O
</description>
2 R( }5 B% C0 q9 ~3 q. @</property>& M6 @3 ]. \* v
<property>
5 S5 z% O6 ^! J9 V" W <name>yarn.app.mapreduce.am.container.log.backups</name>
2 S2 R3 K( i$ ~& p4 ^/ y6 }- } <value>0</value>
# ^9 {' i3 ]6 l+ r <description>Number of backup files for the ApplicationMaster logs when using0 B" u8 V# ^8 q' k. Y
ContainerRollingLogAppender (CRLA). See" D* C* g# K/ V( a2 a
org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,
6 ^/ V% R$ f, C9 k3 i" R ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA
& T5 A* @5 L e9 V+ I. W is enabled for the ApplicationMaster when both
0 {4 i# r. h% y1 C5 Y yarn.app.mapreduce.am.container.log.limit.kb and
9 O$ C# @% v3 g r8 G* j- z' Z yarn.app.mapreduce.am.container.log.backups are greater than zero.- s$ B, V/ a6 Z" P% y
</description>
/ b* Y8 y3 _6 y C0 a. \/ T# z) v5 _1 W</property>3 v! H9 U- B1 s- ^2 x @, d
<property>; g+ M2 z3 c2 P3 p
<name>yarn.app.mapreduce.shuffle.log.separate</name>, F4 Y% s* i% m x* x- F
<value>true</value>8 ^/ i* j' G: a% I6 Z3 s; J
<description>If enabled ('true') logging generated by the client-side shuffle
6 v/ {8 ?4 x! H classes in a reducer will be written in a dedicated log file' d' p- W' u" g0 i+ }7 }( W
'syslog.shuffle' instead of 'syslog'.
4 l5 M9 A& t' K" |9 b/ } </description>3 T' d2 V" O$ k" n
</property>" D" N# |. x. \8 g. ]5 Y- b
<property>
! m: ]" w# @6 \3 B4 R7 [9 r! u <name>yarn.app.mapreduce.shuffle.log.limit.kb</name>% i" W! ?$ u& l6 |
<value>0</value>
7 s4 T* I" d, d( }% R <description>Maximum size of the syslog.shuffle file in kilobytes/ x4 V; W* g+ ?3 j
(0 for no limit)." r( ^2 _. S( H
</description>
& M- V- W; v- X8 F* h' J/ b: ?</property>
* S' S: g2 T5 ]7 T _3 A- Q<property>5 I' T, k: j9 ^7 q( Y9 _
<name>yarn.app.mapreduce.shuffle.log.backups</name>
2 W Q/ T) H1 ?: P <value>0</value>
0 {; S: `7 Z `+ R' D1 [2 z <description>If yarn.app.mapreduce.shuffle.log.limit.kb and
- Z- O9 _* b7 G) v' i yarn.app.mapreduce.shuffle.log.backups are greater than zero
( B7 m9 Z! u+ d! M0 B then a ContainerRollngLogAppender is used instead of ContainerLogAppender
, @2 ], \+ ~% ]8 v1 o2 D; W. S: S for syslog.shuffle. See, T. A) r, H9 A2 U ~
org.apache.log4j.RollingFileAppender.maxBackupIndex4 A8 \, y$ D G, c
</description>
+ u- ]: S/ c5 [( q2 O6 o, U& v2 r) f3 v</property>
' \6 P. {+ j2 ^2 V/ E7 E" h, j<property>
4 G! E' a$ g7 } <name>mapreduce.job.maxtaskfailures.per.tracker</name>
2 f8 D6 S$ y; o7 U4 k/ ` <value>3</value>; P" _/ {+ K( S; |0 }
<description>The number of task-failures on a node manager of a given job* c3 |, E0 ~6 }/ R: h2 x4 e
after which new tasks of that job aren't assigned to it. It
d2 [. p3 _7 C) R; E. l) k MUST be less than mapreduce.map.maxattempts and3 E, J- d2 \1 J7 u- ]
mapreduce.reduce.maxattempts otherwise the failed task will
7 \- h, }6 z, E/ n2 U2 B never be tried on a different node.7 J( O/ J0 A9 n1 Q' c/ F* A
</description>; U3 \6 n5 @6 m
</property>7 h) m! F" ~" I. K7 j
<property>6 X7 g) j6 v9 A, a. j
<name>mapreduce.client.output.filter</name>
0 n9 a7 }( a) M! D3 A+ D; T* b <value>FAILED</value>$ d% r: N4 R1 ^. s/ E+ p
<description>The filter for controlling the output of the task's userlogs sent
; \1 V4 I$ S# ?- W to the console of the JobClient.! ~( x+ ~+ P |$ }! S! m/ E; [0 ?
The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and
2 m) D3 M2 @ Q+ \/ @: m1 X ALL.4 W F+ b/ W% G% H$ J
</description>( D: T" z$ C$ v) M
</property>
* z4 i" j$ ^5 B <property>
( j8 h% F& e" `5 M/ B( ` <name>mapreduce.client.completion.pollinterval</name>
; V' a p5 I' b1 o$ X5 G1 \ <value>5000</value>
3 R7 d7 b( U' t s' c- l T <description>The interval (in milliseconds) between which the JobClient
* g [- t2 _3 ?, S& P) B8 ]/ @ polls the MapReduce ApplicationMaster for updates about job status. You may want to3 B+ i% M$ z% ~% n3 R
set this to a lower value to make tests run faster on a single node system. Adjusting
$ o$ L/ m( ^6 ^4 a d this value in production may lead to unwanted client-server traffic.
. Z% ~6 f" I9 s. }! a8 y </description>1 A q' Q: ^* K$ F' K M0 B! z0 v5 s
</property>
- k! S4 f7 R& M, y <property>
6 S" G' `3 A% j2 f+ P1 k <name>mapreduce.client.progressmonitor.pollinterval</name>3 C$ C% q4 O; ?
<value>1000</value>
" O" q/ G3 A1 H9 O) b J <description>The interval (in milliseconds) between which the JobClient5 t+ z8 L" N5 H" ^( R% A
reports status to the console and checks for job completion. You may want to set this
6 l9 N+ v) V( D6 v3 @& G9 B4 { to a lower value to make tests run faster on a single node system. Adjusting
% {. o2 g) B, w3 I' ~ this value in production may lead to unwanted client-server traffic.3 U* U% o( _2 G( Z; X
</description>, E: C! s( j( N1 e& z2 g; V+ y5 C- ?
</property>& E2 e x0 q; M7 }
<property>) i9 X. ~6 S2 D& _. H
<name>mapreduce.client.libjars.wildcard</name>) ?0 \+ }: A; T; E
<value>true</value>' j0 Z9 P0 M5 l( X( r) V6 f4 v: }, X) r
<description>: B# \7 N2 O" p6 ]% G$ n
Whether the libjars cache files should be localized using7 j* G9 Y6 ?' h& H
a wildcarded directory instead of naming each archive independently.+ a5 u, F! T/ X) C) V8 q
Using wildcards reduces the space needed for storing the job' T. ]7 f) x6 I. `' ~/ T7 b
information in the case of a highly available resource manager
; | a3 K7 C$ Q/ Q1 j; g" y6 O+ I# C configuration.' w1 P( {0 X ]$ B5 X9 q/ I
This propery should only be set to false for specific! D: q$ d9 p, ?. p, R
jobs which are highly sensitive to the details of the archive% R8 ~7 O' l# T9 x" b" X
localization. Having this property set to true will cause the archives
4 r \" L' t. p; C7 E* F to all be localized to the same local cache location. If false, each, h" x+ F% W- {. W4 t z# S
archive will be localized to its own local cache location. In both
y4 L/ v, I! R. c0 Y4 p cases a symbolic link will be created to every archive from the job's
+ X/ h% H4 N5 A$ B$ ~' L8 B6 A working directory.3 t# N( w4 |3 W5 l
</description>" l6 T3 [% ^1 z
</property>
9 J& ^+ w9 |6 {( Q <property>
9 y" x5 f3 q0 ` g1 u <name>mapreduce.task.profile</name>
1 V/ l- a6 z) |* K) Z. s: j <value>false</value>/ V9 C+ s7 V. V) J+ m/ Y2 v
<description>To set whether the system should collect profiler$ S5 ?' S8 B# \" m- b% p3 F
information for some of the tasks in this job? The information is stored/ s2 y6 d0 ~+ `/ h5 ?+ u( X" x
in the user log directory. The value is "true" if task profiling* _5 t7 c2 ?4 S" h$ x
is enabled.</description>
/ @- }$ s A0 l </property>
- K! D3 j9 E& t( ]: c8 j <property>
$ v1 R; Q+ K8 X3 P7 \ <name>mapreduce.task.profile.maps</name>
3 G! x2 J' |. q) D X <value>0-2</value>
. u3 c3 F0 L( L+ V, k <description> To set the ranges of map tasks to profile.( K, k& D4 @+ p0 t! a
mapreduce.task.profile has to be set to true for the value to be accounted.
5 k' Z6 e6 `; ? </description>- K# u9 {9 A# g, B& ^) f
</property>
0 ~5 f4 O" w! i4 |. A1 E <property>. D" Y( Q' w) a$ ` R
<name>mapreduce.task.profile.reduces</name>
) m6 \9 @$ s- E: c/ p$ _" j <value>0-2</value>
' p+ D8 |$ P6 J) c0 {+ P <description> To set the ranges of reduce tasks to profile.
3 j* e, ^: {7 { mapreduce.task.profile has to be set to true for the value to be accounted.
3 Y( `5 J# i" L3 _& o' h, P/ N </description>
: V3 J2 q+ q& s9 ? </property>! {# C0 {) \0 ~4 M, C2 c
<property>& j( ?' |) l( U3 M% l/ B
<name>mapreduce.task.profile.params</name>9 @$ Y) V% i; M; n
<value>-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s</value>0 X/ o2 L; T, E5 ]5 ~
<description>JVM profiler parameters used to profile map and reduce task0 x3 z. L4 P2 F8 q4 H
attempts. This string may contain a single format specifier %s that will x1 T7 y P7 I! l$ V
be replaced by the path to profile.out in the task attempt log directory.0 Q! ]1 O: l( p! r" a9 M; R% P" d
To specify different profiling options for map tasks and reduce tasks,
Z* H4 p7 D+ n' C9 V B more specific parameters mapreduce.task.profile.map.params and
5 ~/ h3 s, _) x# X! j2 j4 r) s/ v mapreduce.task.profile.reduce.params should be used.</description>5 G# z r, X: ]
</property>
# G( D* f* C+ [: U. R3 x' d( ~ <property>, r$ t/ ~; s! S4 R* h, W
<name>mapreduce.task.profile.map.params</name>
2 v- ]8 J1 _1 Q3 W' H% C N <value>${mapreduce.task.profile.params}</value>% L0 v& H! q& l+ v* a
<description>Map-task-specific JVM profiler parameters. See
5 s- O2 E; E" F W5 ?9 ? mapreduce.task.profile.params</description>7 c" W* ?; L4 r) V& U# z7 f; f y
</property>$ i" l$ o. { e/ P; P
<property>1 H. a4 ^( h# B% J3 T0 O7 F( |
<name>mapreduce.task.profile.reduce.params</name>6 i4 o5 `; {3 u9 @. R# I
<value>${mapreduce.task.profile.params}</value>! q* P0 I: f& o
<description>Reduce-task-specific JVM profiler parameters. See
% T4 n2 S8 ^1 _% Z7 ?! ^5 Y( h mapreduce.task.profile.params</description>
; V H9 Q4 ?% \& X; c </property>
2 b* i% s0 P% B/ q: A! [3 J <property>
r$ `: H- s+ {+ Z3 Z0 e <name>mapreduce.task.skip.start.attempts</name>
$ O3 t; t7 g$ y7 e4 V <value>2</value>
- t6 S9 a3 W3 l I l7 A <description> The number of Task attempts AFTER which skip mode" L# p, q2 T6 v& d9 p
will be kicked off. When skip mode is kicked off, the
3 Q: R2 X6 X5 ~. m; L4 U tasks reports the range of records which it will process
2 a$ w9 B! H' O% ?9 i6 o( D8 r next, to the MR ApplicationMaster. So that on failures, the MR AM
- q# G! ~4 Y5 s+ U! C knows which ones are possibly the bad records. On further executions,2 K0 f0 n7 j3 H3 ^+ f. z
those are skipped.7 K V9 S& ]+ C* g$ Z: C& |
</description>1 G0 D; y6 M) j, h( }; e
</property>
1 N" X% O c% Q* O <property>* R. J; c7 K+ X, m: f
<name>mapreduce.job.skip.outdir</name>3 A. J2 s% I3 `# b% S p0 F8 `
<value></value>' d& @! ]; ?3 M1 a, ^7 c& e
<description> If no value is specified here, the skipped records are
' o+ ~; @; p7 k- V' O( S4 t8 z written to the output directory at _logs/skip.
6 L! O9 p( [: S, Y( Z7 | User can stop writing skipped records by giving the value "none".4 F; U# s& Z5 G% v! Y! l
</description>
2 Q& H/ h5 d2 h" G7 J3 F </property>: B* n* o0 ~( r& T
<property>
2 t9 J2 p7 _; i! @+ _ <name>mapreduce.map.skip.maxrecords</name>
, A3 j; Y) @! @* ~( }2 y' k <value>0</value>
6 ^! K$ e6 g# \% }* A <description> The number of acceptable skip records surrounding the bad
. H: ~ u1 \) f, z3 M0 H1 y* v record PER bad record in mapper. The number includes the bad record as well.
0 B6 [! O6 B9 j1 e To turn the feature of detection/skipping of bad records off, set the$ l9 ~9 C1 O' w7 R; \+ G
value to 0.
" S* n2 c* s, j: B6 H1 s2 d The framework tries to narrow down the skipped range by retrying
+ ~& V T/ m$ ]- Q& s8 j until this threshold is met OR all attempts get exhausted for this task.
& @$ d$ e% g9 [ Set the value to Long.MAX_VALUE to indicate that framework need not try to
% c0 G/ n2 k% b/ q narrow down. Whatever records(depends on application) get skipped are* d+ K. x7 e {" x
acceptable.
* w8 z6 F5 L- n( \$ l9 u) P </description>
% l" b: `' A6 U7 G) q5 k </property>
( m. C$ w7 y( c" z <property>: P- N1 L6 m3 q! X7 q- a
<name>mapreduce.map.skip.proc-count.auto-incr</name>
1 I& D8 I9 V' n( d; ^1 i <value>true</value>8 a9 X1 ~; b6 c9 i/ r: J- S( [
<description>The flag which if set to true,
9 }7 Q# e/ Z/ P3 @( k1 W y/ z SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by
. W( O8 F1 |5 o5 D4 y7 L MapRunner after invoking the map function. This value must be set! L/ Z" ?5 y1 Q% k
to false for applications which process the records asynchronously% M5 A; X4 Z) C( V( @) F2 O$ ^
or buffer the input records. For example streaming. In such cases5 q: }) j. U+ D
applications should increment this counter on their own.
4 E4 ~1 _5 T0 Z </description>! B5 \. z3 c2 @+ q+ S2 K
</property>
. y2 K: W9 E9 X0 _( y- k4 q <property>2 t9 ^; p( ~6 u1 n
<name>mapreduce.reduce.skip.maxgroups</name> o# y7 v) T7 T( f7 `, V
<value>0</value>
7 e& o* [9 h0 {0 P% k2 m- Q <description> The number of acceptable skip groups surrounding the bad
& e4 ^. `4 r( ~$ z group PER bad group in reducer. The number includes the bad group as well.+ g: H" ^: }: G1 l5 C h, v. i- c
To turn the feature of detection/skipping of bad groups off, set the
$ A5 V% `6 Y' s value to 0.3 h1 H& E. q+ t$ v) E2 b
The framework tries to narrow down the skipped range by retrying
9 j, J& L5 j& O& j j5 G @ until this threshold is met OR all attempts get exhausted for this task.
3 g. e! G8 N9 R+ C& F1 P& y$ h; e) Z Set the value to Long.MAX_VALUE to indicate that framework need not try to
. D, L/ s% X$ L% I4 Y narrow down. Whatever groups(depends on application) get skipped are
, J/ k2 f" M8 M3 ^ acceptable.
- Q% p# U* H- c* e, w7 Y </description>
7 {+ c4 a/ ~) }; { </property>2 h! }' l7 x( Z7 j2 f
<property># x9 v+ X j$ b) i
<name>mapreduce.reduce.skip.proc-count.auto-incr</name>
$ U( J, |/ t! e/ [ Z5 L: M <value>true</value>
/ }- f0 z4 {* j. M# ] <description>The flag which if set to true.3 H8 S- M! k$ M9 J
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework
" q0 ^) ] P# D: D( w- N. _( a9 V after invoking the reduce function. This value must be set to false for
: D& R# X- Y: a& X6 c% i( R applications which process the records asynchronously or buffer the input
6 y1 ?- a5 `/ \! ` records. For example streaming. In such cases applications should increment
5 h6 T$ t0 }$ x" V( }# d this counter on their own./ R* W3 n6 z( J- s0 a2 m5 I9 ~1 V
</description>
7 K" x7 h: L3 |& M2 { K </property>
# `3 j/ H# p5 S! s& O: N5 Z; p5 v <property>
$ ^3 x6 [+ O6 Q$ Q" l ` <name>mapreduce.ifile.readahead</name>
. }) G! I3 X C/ O- ~ <value>true</value>( t0 D# y0 g0 q# g
<description>Configuration key to enable/disable IFile readahead.
. q9 m' i1 z! w9 k" G </description>
- B/ z" w; V x$ O6 `& q </property>
~% [7 j( k" y6 f) @, x <property>8 a7 X1 n4 Y6 c. h6 ^ N8 t4 O
<name>mapreduce.ifile.readahead.bytes</name>
( X6 C( q; z/ U9 k <value>4194304</value>
6 g' s# b7 J- c( X8 Z0 }" ] <description>Configuration key to set the IFile readahead length in bytes.9 V; R1 q- @4 B! v* ~" w
</description>( K' f+ a. T% o" a: {$ d* f
</property>% W5 h0 D. C" C5 J7 O' F
<property>
& {0 r( U1 i6 Y! h( Z/ a <name>mapreduce.job.queuename</name>
( ~. X4 _' e5 V( a( ~5 Q! a; w% r7 p <value>default</value>1 } U' D3 l9 n; q
<description> Queue to which a job is submitted. This must match one of the, I+ U8 O/ g g0 }% h2 T# v# C
queues defined in mapred-queues.xml for the system. Also, the ACL setup6 w( V) p% a4 R+ D
for the queue must allow the current user to submit a job to the queue.
* s+ A- `7 Y( T1 N Before specifying a queue, ensure that the system is configured with
: N3 k7 n/ @" P# O9 s1 r Q the queue, and access is allowed for submitting jobs to the queue.
& J/ e# A" Q, f# z </description>
! C. Q( D: d" x7 u! U</property>
# ?8 R2 ]1 y3 e' B" `& n7 O <property>/ N* L' z2 U/ i& V( Q
<name>mapreduce.job.tags</name>
* f+ p( a# _5 L- T <value></value>* x+ {% z/ h6 l% s6 H0 p) w. K
<description> Tags for the job that will be passed to YARN at submission
4 J+ p! p( w- u' @+ J time. Queries to YARN for applications can filter on these tags.2 Z6 P/ ~0 v/ C9 \) |4 S
If these tags are intended to be used with The YARN Timeline Service v.2,# L$ `4 U" w3 K' g* n" v1 p' a
prefix them with the appropriate tag names for flow name, flow version and
& }) [% Y3 i/ \ flow run id. Example:
% ?( H. Q7 N* p, `; f p5 b timeline_flow_name_tag:foo,
. [# U" y+ T2 X0 R) l timeline_flow_version_tag:3df8b0d6100530080d2e0decf9e528e57c42a90a,5 x( D: i7 @: |- M8 p
timeline_flow_run_id_tag:1465246348599
5 {0 Y+ f+ f) Q) }1 K; ` </description>
& ~9 A- x; |# m% B+ F4 u </property>
0 |( w! J0 _# W% H<property>
) t7 a `; z9 @8 n <name>mapreduce.cluster.local.dir</name>
- r: A, X- x0 ] <value>${hadoop.tmp.dir}/mapred/local</value>) _! ?" M1 m$ S# ?& z9 Z
<description>
P1 v$ j; i5 F( n The local directory where MapReduce stores intermediate* x S6 c- d8 u. t1 k: P7 G
data files. May be a comma-separated list of
/ m& O3 ]4 g% c- Y" s! l+ F directories on different devices in order to spread disk i/o.- u ^7 Q& e4 Q; r7 ^& d
Directories that do not exist are ignored.
% O, p, d, x/ v7 B( Q </description>6 o8 h1 Q) Q! e! p2 D3 d8 G1 G
</property>+ @# d* k. V2 r9 q7 m
<property>
! w* I& ]' b6 u' X; m4 d4 [ <name>mapreduce.cluster.acls.enabled</name>
4 l/ l* o4 g& t% K& u+ M8 Z <value>false</value>
* y7 v* K5 s. P6 Q <description> Specifies whether ACLs should be checked! l5 o& m+ [ T% ?
for authorization of users for doing various queue and job level operations.; B6 ^- A A& k% \, Q/ P0 P0 B
ACLs are disabled by default. If enabled, access control checks are made by
, x# N% v$ z" Y; V! T MapReduce ApplicationMaster when requests are made by users for queue
9 m R2 p# t6 q+ l$ b* d: z operations like submit job to a queue and kill a job in the queue and job
' Q# e1 e- u0 r6 o# M operations like viewing the job-details (See mapreduce.job.acl-view-job)
( R* ^. p% Y u8 u3 \; Z or for modifying the job (See mapreduce.job.acl-modify-job) using
& p) `2 |! p* f7 m" t6 { Map/Reduce APIs, RPCs or via the console and web user interfaces.! T/ e& i, l1 @6 J C2 b
For enabling this flag, set to true in mapred-site.xml file of all
0 v/ q! g, f& Q4 v MapReduce clients (MR job submitting nodes).# e2 m( z" c2 l; x9 {# P- |
</description>
# D; W+ `# o0 c</property>8 G4 W) R! p! O0 _; s9 n# o
<property>6 @. }1 p6 L) L* w1 o+ q( @' x
<name>mapreduce.job.acl-modify-job</name>
4 C# [: {" l% B( N& R; t <value> </value>
0 J& g% |8 A- W! J" U) ` <description> Job specific access-control list for 'modifying' the job. It) W' M- E- l8 q0 G
is only used if authorization is enabled in Map/Reduce by setting the4 C* U3 k' T; \3 E5 Q
configuration property mapreduce.cluster.acls.enabled to true.
* Z, Q; h2 X0 P" {7 @8 M" i This specifies the list of users and/or groups who can do modification+ Z. s. K( p" B" s4 e* Y
operations on the job. For specifying a list of users and groups the7 f3 U* {& `$ t, {: q+ ^' i
format to use is "user1,user2 group1,group". If set to '*', it allows all
; l: R4 {5 j1 v* r. a6 G users/groups to modify this job. If set to ' '(i.e. space), it allows
0 l4 k v1 U7 { U! V none. This configuration is used to guard all the modifications with respect
/ ^7 R i. A7 _" v to this job and takes care of all the following operations:% R! x& f4 Y" i% R- b
o killing this job
1 b5 |+ ?1 H) P' ^1 Q# I. f9 ] o killing a task of this job, failing a task of this job
7 h1 ?- W- f# C. Y8 j o setting the priority of this job
! E) W+ }/ n6 X+ A Each of these operations are also protected by the per-queue level ACL% E- D1 F- u! B; J+ g
"acl-administer-jobs" configured via mapred-queues.xml. So a caller should
4 Q; o! C0 x2 w have the authorization to satisfy either the queue-level ACL or the
9 e0 y7 d7 J/ z$ X job-level ACL.' W! E7 h3 S O4 \ A
Irrespective of this ACL configuration, (a) job-owner, (b) the user who
; d7 @: l! T3 q% W- ^3 r started the cluster, (c) members of an admin configured supergroup' e3 L/ O: C2 o7 v: D( u
configured via mapreduce.cluster.permissions.supergroup and (d) queue- |. s$ Q7 F8 a" W8 u i( O( O
administrators of the queue to which this job was submitted to configured' P( t$ o$ |! k
via acl-administer-jobs for the specific queue in mapred-queues.xml can
3 c, b0 s7 t4 W8 G1 ?5 ~7 b% P' F do all the modification operations on a job.3 G d) ~. h5 _, T8 B$ ?
By default, nobody else besides job-owner, the user who started the cluster,
P3 w$ ~, U( @6 e `# M. _' `+ k members of supergroup and queue administrators can perform modification9 b5 O% R, ~; i+ f2 a
operations on a job." o" @' M" S, }7 M" Z
</description>
& O3 k3 k- B+ l6 ~; n* y2 o</property>5 i: d/ \" [! g/ m5 W' Z
<property>
( F+ X4 A; c1 h7 d4 w/ z8 X <name>mapreduce.job.acl-view-job</name>
2 g4 ]4 G0 n# o <value> </value>0 S8 m* N Y) I. Z1 _
<description> Job specific access-control list for 'viewing' the job. It is: h0 n7 l1 l K: Q9 B7 a
only used if authorization is enabled in Map/Reduce by setting the& ^. v3 a1 n U' x+ n; f" I7 D
configuration property mapreduce.cluster.acls.enabled to true.; T: ?+ e; o4 {' ?
This specifies the list of users and/or groups who can view private details
+ z3 H/ @5 Q, @" Q2 G4 @ about the job. For specifying a list of users and groups the3 F& w S1 D+ ~6 G$ n
format to use is "user1,user2 group1,group". If set to '*', it allows all! d8 \ ?) \2 i% I
users/groups to modify this job. If set to ' '(i.e. space), it allows
6 s+ z) X9 k+ t2 K) V. o* H9 r none. This configuration is used to guard some of the job-views and at
, d& O: s! c+ J present only protects APIs that can return possibly sensitive information
; F7 O1 o: H$ A- W; f7 r. ` of the job-owner like1 N8 n7 \. e X4 @( {0 G' j9 e# f, r
o job-level counters' P4 o5 {+ v/ L/ g6 T
o task-level counters
! W B% x* B& U7 |0 ~) Q1 p4 Y# R o tasks' diagnostic information8 X g2 V0 ~# J: i Z. G2 F+ M/ f
o task-logs displayed on the HistoryServer's web-UI and5 w2 \0 q5 }: a# A
o job.xml showed by the HistoryServer's web-UI- M( K1 x) `; g9 G/ Z0 U' E
Every other piece of information of jobs is still accessible by any other
) f) m+ E$ r" V! [# W user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc.. o3 p a( [& B' I
Irrespective of this ACL configuration, (a) job-owner, (b) the user who1 v3 E* s) I6 J- O/ B7 ^2 z
started the cluster, (c) members of an admin configured supergroup6 S2 b9 B" q9 J/ U
configured via mapreduce.cluster.permissions.supergroup and (d) queue) |- Z- W9 a+ w
administrators of the queue to which this job was submitted to configured
2 M0 j8 h! p# v, h# [* y1 E2 } via acl-administer-jobs for the specific queue in mapred-queues.xml can2 }. c1 k* ?1 F: z6 W
do all the view operations on a job.- l3 d/ n: t5 _$ A$ @- p/ `
By default, nobody else besides job-owner, the user who started the/ p1 e- N! H) V" }/ b+ i" q
cluster, memebers of supergroup and queue administrators can perform
1 F7 \ F; h# B6 _ view operations on a job.8 t) r) l9 l) Y# l
</description>
( S% S9 u! G% Z: t</property>; K$ F$ ]# ]5 S j7 D- z2 [
<property> N$ I/ T/ _) R. {" S
<name>mapreduce.job.finish-when-all-reducers-done</name>8 L1 A, ]2 L4 x% @- J
<value>true</value>: a/ b$ w+ F$ H, D% @9 Z I
<description>Specifies whether the job should complete once all reducers* v! ^& ?6 P( s& j
have finished, regardless of whether there are still running mappers.
! ?& e( G0 q- q* G8 v1 O) b8 d </description>
, h4 v) e" b# q1 p9 Q</property>1 {$ k, T) i) [1 |4 ~# v
<property>
2 w2 E+ ^. N: B7 f3 K; |! M5 { <name>mapreduce.job.token.tracking.ids.enabled</name>
L5 d! I5 x5 X& k <value>false</value>- o, y& b" e/ e& K5 a2 U" G
<description>Whether to write tracking ids of tokens to8 h+ r, F9 `& p/ g( O# a: k
job-conf. When true, the configuration property
) Y3 j0 M* |7 F" U "mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of! I' a4 \- i1 r! q
the job</description>/ |* o9 J1 J4 A/ ]
</property>
2 h9 |. w1 R2 Y3 ~2 l0 n' j" V<property>( r& t7 E, k l2 c8 ?
<name>mapreduce.job.token.tracking.ids</name>
* q+ k' z; b- g# K/ W8 M3 K2 K* M <value></value>
' T0 O- R/ c5 K' M9 n <description>When mapreduce.job.token.tracking.ids.enabled is4 X, O9 n" Y7 W! v$ V: w
set to true, this is set by the framework to the# X E8 l* A3 p0 B+ T( Z7 v' q0 ^
token-tracking-ids used by the job.</description># C! n; p' u3 N! E
</property>
5 L- A* g' M$ c4 f- r+ w9 l<property>
7 [) }3 {! R+ p <name>mapreduce.task.merge.progress.records</name>
! g% U1 K$ t3 z& G# A6 u* Q <value>10000</value>7 q9 {5 x5 M9 v1 K; p
<description> The number of records to process during merge before
9 l# o9 Y4 i: I- T sending a progress notification to the MR ApplicationMaster.+ K7 p( r) f, a4 v
</description>
& R( H1 R+ f% z/ ^1 h</property>" A3 a+ N4 K% T% y) j1 \ J
<property>; s9 p9 a2 [6 }+ W2 L* Q3 P; _" v8 p
<name>mapreduce.task.combine.progress.records</name>2 h) F8 I# u; m- a
<value>10000</value>0 b+ N7 e$ f |" C: Z I: |- i
<description> The number of records to process during combine output collection
: R! s" r& m% e E$ |# R before sending a progress notification.' }/ E( X3 r: {
</description>' ~! l$ d+ X/ {, ^
</property>; f; @3 M7 s" r0 G2 I, d
<property>
2 G/ v0 S( b2 [9 b7 ^& s6 m <name>mapreduce.job.reduce.slowstart.completedmaps</name>
9 B- r3 V, J& y$ a- o6 E <value>0.05</value>; t1 \) q" v# O( R5 i
<description>Fraction of the number of maps in the job which should be, D, h4 |$ A8 p; P
complete before reduces are scheduled for the job.
! v/ G7 I5 @' ]' Z </description>. J$ r0 ^; r0 S1 F
</property>
( m0 @. A/ v5 A1 }* e( U% P<property>
3 u) k* t1 q) g& L<name>mapreduce.job.complete.cancel.delegation.tokens</name># T+ M+ R: r" i: |2 B
<value>true</value>( |1 V4 a; ?' Y' v7 F0 k
<description> if false - do not unregister/cancel delegation tokens from
. b4 E* c( ~* r5 R2 ~* J, [: E7 v renewal, because same tokens may be used by spawned jobs. c9 S' m: W2 W
</description>: m+ z$ G/ T9 K6 R! `
</property>
7 }, |4 ~' c7 I6 Y9 t<property>; K: a* b& g. p H0 e% ^# c, S
<name>mapreduce.shuffle.port</name>
u8 W/ t+ a* ` <value>13562</value>
) |1 t9 w$ g# v3 Q2 ?8 ~ <description>Default port that the ShuffleHandler will run on. ShuffleHandler
& R8 L6 P/ P! v4 h1 _5 t is a service run at the NodeManager to facilitate transfers of intermediate, _- K4 e3 W( u4 d- Z
Map outputs to requesting Reducers.( `+ W7 X4 Q7 I* D( @" G
</description>9 D; A; {, m! y4 |# S# c" W) S1 p
</property>
; t9 x: _1 S7 G, @<property>
3 x; b( |' r n, ~2 X) V- @+ D <name>mapreduce.job.reduce.shuffle.consumer.plugin.class</name>
1 e; t7 V4 p8 A5 ^ <value>org.apache.hadoop.mapreduce.task.reduce.Shuffle</value>
% q7 M+ B7 |, R% [ <description>' X; z" | E3 n6 L- J7 j" _
Name of the class whose instance will be used
. S8 P: m- t$ M: B to send shuffle requests by reducetasks of this job.
: v1 o3 O3 k0 z The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.0 V- q! N0 M: U2 b" j* \
</description>
# d' x# T+ {7 @ b0 s! ] x</property>' U3 G. |, _2 Q% F' o5 J. O+ G
<!-- MR YARN Application properties -->$ k) r! q( g7 F1 E9 N
<property>5 B6 X- ?* ^" g v% j) A) s
<name>mapreduce.job.node-label-expression</name>3 ^( T1 ^- g5 W- ^; t9 a1 t
<description>All the containers of the Map Reduce job will be run with this/ t/ w% G2 P; Q# F% a0 C. ]* S
node label expression. If the node-label-expression for job is not set, then2 a8 K5 n% |( x+ y7 ]: z
it will use queue's default-node-label-expression for all job's containers.
z* {) P' I8 P1 Y# h9 r* W$ I& w </description>
3 D; l: G3 T E9 o% K! N& L</property>5 D v# B& n* Y+ ?4 |! H, {5 @1 ^" J
<property>
& k! z' H+ k( _, o. x <name>mapreduce.job.am.node-label-expression</name>
D: G) x+ k$ _3 D0 m2 Z <description>This is node-label configuration for Map Reduce Application Master
, ^+ e2 e3 }- w: E0 m container. If not configured it will make use of' j, W: p; U- }5 i
mapreduce.job.node-label-expression and if job's node-label expression is not
& F; n3 ~4 h4 m) J7 d/ p configured then it will use queue's default-node-label-expression.
]+ b, R' D. v, y+ U8 `' @ </description>
' K7 z8 c; d! @: u- D/ h</property> Q7 k0 a3 }" P
<property>
' D7 P# m! d# e6 X9 T4 n4 a <name>mapreduce.map.node-label-expression</name>
4 r/ @" M* Z4 E' d k) f; U <description>This is node-label configuration for Map task containers. If not5 N a4 y: C# C9 t4 u! T" k
configured it will use mapreduce.job.node-label-expression and if job's
5 ^. r/ h l3 ~0 ? node-label expression is not configured then it will use queue's+ M+ B& J3 C# x+ i/ N
default-node-label-expression.0 q% x% f" ]; ?- `. }
</description>4 f) a' T4 b+ K+ U
</property>- j: e% p9 p& Q5 O/ i2 z9 f# G
<property>+ q5 k( P( E0 `, ~& x+ ?
<name>mapreduce.reduce.node-label-expression</name>) w( q6 I* J+ ~( o( y/ ~$ e D
<description>This is node-label configuration for Reduce task containers. If* u6 r- ~3 o' V7 g
not configured it will use mapreduce.job.node-label-expression and if job's
% c' @' F8 ?/ N$ _% A2 s, [( r node-label expression is not configured then it will use queue's$ {) {( S+ y C" r" p& w
default-node-label-expression.
: X) j2 v7 h9 d0 E/ |) F( h </description>
& d( q- ^3 h" Z! @</property> z+ n) k: S: \$ i
<property>
2 H. x0 X0 G! J- ]5 b <name>mapreduce.job.counters.limit</name>
( o$ y7 }* m& G, L3 Q: B# \) y <value>120</value>
8 ]0 ?4 j2 ^7 W7 }0 ?5 ?2 w" T <description>Limit on the number of user counters allowed per job.9 w. p4 U* r0 ^" n# S! K2 E1 B2 a
</description>
& ?1 e$ S+ K+ K</property>+ [+ o# q8 N9 q1 }0 Q% w
<property>7 J1 [; t& j& R3 R, g h7 [
<name>mapreduce.framework.name</name>. Q- z( Y' R9 X9 L+ e0 }
<value>local</value>
, ~/ p6 k5 p$ H- w. M <description>The runtime framework for executing MapReduce jobs., K5 s0 B; E6 q
Can be one of local, classic or yarn.
' C" J& f9 B% G" P! l </description>; M0 u- N* z- |; s+ \+ W: c" s, J0 H; z
</property>1 n. b4 L3 r: U( `+ O5 i
<property>
! G+ Z- h* U& H, b* J5 E8 c <name>yarn.app.mapreduce.am.staging-dir</name>
- p9 n( s/ E$ E( |* I, d; h- b8 N <value>/tmp/hadoop-yarn/staging</value>
# E4 X7 Y1 y6 U! l7 U" V3 a <description>The staging dir used while submitting jobs.& Q, E" ?$ ?, w% l1 }# k
</description>
1 @; V3 p* ?3 x! S& k8 E& W( g</property>" x: A; V4 L1 E" E% P4 K6 N' ~' M
<property>1 |# O: r3 [/ W. K* c ]
<name>yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled</name>
; O: B: V {4 J8 ~, w <value>false</value>
' k, y. Y7 h" h, l; P1 P <description>Whether Erasure Coding should be enabled for
( v5 u' X8 N O$ |& V5 h0 o files that are copied to the MR staging area. This is a job-level* `$ {6 m/ X# `7 ]- m
setting.
1 `. S) d! X! [# K7 G </description>
+ `7 r# N% s) y* p9 ~8 \</property>( l( `: @5 E; i
<property>9 B+ p9 U& _% {
<name>mapreduce.am.max-attempts</name>9 @* V H2 d0 x' W" Z
<value>2</value>
- r3 Q, m1 a4 u1 C6 i <description>The maximum number of application attempts. It is a3 w1 J" @' i3 C: ^4 M
application-specific setting. It should not be larger than the global number
w! y ]/ o2 A; ^ set by resourcemanager. Otherwise, it will be override. The default number is
9 M/ y5 k4 q% p( @( x set to 2, to allow at least one retry for AM.</description>
0 O5 @! t6 r, u8 t8 h</property>
* P2 Q7 ~+ |. s* D<!-- Job Notification Configuration -->& V! i5 G p- q* m4 n
<property>
* G1 u) S7 H" @6 a4 ]9 t <name>mapreduce.job.end-notification.url</name>
3 h- a* b! y' _3 B* e# ^ <!--<value>http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus</value>-->
; {6 C# X% x# q <description>Indicates url which will be called on completion of job to inform
! c3 N9 w1 o3 b4 U3 x end status of job.' n+ N3 {7 T- M! P
User can give at most 2 variables with URI : $jobId and $jobStatus.
! n; ?: ], @0 L6 a. b& i If they are present in URI, then they will be replaced by their
: p( u0 a# {3 z& K respective values.6 k }$ `7 C' {% N1 `& d- ~
</description>
- e. ]- g& H( q3 n3 J- c0 ^, K</property>
# ?2 s: B. u) B<property>/ ~+ v* i/ i/ \% ]. ~1 e' ?* K
<name>mapreduce.job.end-notification.retry.attempts</name>! k, J3 p4 ^' T. j
<value>0</value>
2 a3 R+ n s% T" u5 X <description>The number of times the submitter of the job wants to retry job" z. s5 h+ ~# K
end notification if it fails. This is capped by
" U( k3 N: ~- N4 l mapreduce.job.end-notification.max.attempts</description>3 X& K; d. t- b
</property>+ y+ s, l% R4 G% j D2 S; b
<property>- ~5 p$ A1 F- l% x( i6 \9 j
<name>mapreduce.job.end-notification.retry.interval</name>
: q6 }& j& I* s# _) [ <value>1000</value>' R5 s# k. U3 Z* Y
<description>The number of milliseconds the submitter of the job wants to
4 R0 A& S/ v" c Y% L% v" q1 W8 P) ] wait before job end notification is retried if it fails. This is capped by
9 Q1 q- G( N4 G mapreduce.job.end-notification.max.retry.interval</description> V9 q2 O* P& p, N8 W
</property>
9 o3 T. }/ {4 t g$ x<property>
9 ?$ d( x* p; }( p2 r+ @/ W4 G <name>mapreduce.job.end-notification.max.attempts</name>/ I/ h7 s; S, @1 |: ^" {" ~
<value>5</value>7 F/ g; ]9 G A* ]% n4 w$ Q: r% k
<final>true</final>, w/ V8 S) d9 Z1 d
<description>The maximum number of times a URL will be read for providing job
2 ^* Y* W" l; f; L' M: Z5 k end notification. Cluster administrators can set this to limit how long
- @) G- D; k/ q* n after end of a job, the Application Master waits before exiting. Must be
, t; v1 B) n6 l% g0 H: }+ V5 ]& X marked as final to prevent users from overriding this.
# V# w7 N- E$ m </description>
5 a% G7 N) D) l! a+ [0 X</property> ^7 x- M& w( F. a+ h6 p: y7 j
<property>
! h3 @( r {& K2 e2 A6 H' ]! A V <name>mapreduce.job.log4j-properties-file</name>
/ I, Z/ v/ H4 M2 l9 c <value></value>; e4 Y/ Y# u1 ^5 H: ]
<description>Used to override the default settings of log4j in container-log4j.properties
7 N9 L3 E) s+ Q" U" D: D' d% E2 o for NodeManager. Like container-log4j.properties, it requires certain4 j" W2 v7 k3 G
framework appenders properly defined in this overriden file. The file on the
9 C! ~2 \1 i; \4 l1 B1 Q path will be added to distributed cache and classpath. If no-scheme is given
5 N1 }( \3 V6 O# D6 ^" C% a2 }0 d3 E in the path, it defaults to point to a log4j file on the local FS.6 j: e+ c( T$ D: B( K
</description>
( H& k2 p, T+ x. _$ u </property>4 l5 Q0 h! l0 P6 j
<property>
, ~& m$ N9 G7 Y# c& f <name>mapreduce.job.end-notification.max.retry.interval</name>6 m {7 D0 o( v1 e# X
<value>5000</value>
" g# G2 p5 m6 t% E3 x7 ]6 J2 L7 [7 D <final>true</final>$ h4 K1 e" P* e0 [/ C7 @4 v1 o% K
<description>The maximum amount of time (in milliseconds) to wait before$ M1 V9 e5 g, n" q) B' m3 y; N
retrying job end notification. Cluster administrators can set this to6 Q* V8 z ? I2 L& p, ^% c
limit how long the Application Master waits before exiting. Must be marked) \% z' r! g0 E/ |; I
as final to prevent users from overriding this.</description>
3 n+ i/ x. T) Y7 T t</property>+ e1 g+ c% B- }) p$ S/ O
<property>3 O" @: p }, z3 N y4 L7 t
<name>yarn.app.mapreduce.am.env</name>& t3 G0 _" N( W- J: @
<value></value>0 P! a1 A& o1 S' S( [$ |
<description>User added environment variables for the MR App Master* E2 Z8 d* a0 v" B |9 A
processes. Example :
6 c& ]2 B4 b% F. U1 N; L 1) A=foo This will set the env variable A to foo
6 \- R7 o; J0 \9 l 2) B=$B:c This is inherit tasktracker's B env variable.
! P0 @: E+ H: ]- B) S2 p </description> U( Z6 }- ` E4 d/ a
</property>
' \ z6 N6 n( F' P. h' p7 D<property>
8 s8 O* z9 e) @2 v1 f9 S# B <name>yarn.app.mapreduce.am.admin.user.env</name>
5 p* v2 G5 a# H% e: i- y6 H <value></value>: o" \% ^' R6 Y- a" }, A6 ]( s
<description> Environment variables for the MR App Master2 p0 b3 }# h* [! a; B1 V/ t8 x. Z
processes for admin purposes. These values are set first and can be
8 D) E5 [( u3 _) l [ overridden by the user env (yarn.app.mapreduce.am.env) Example :# G( p9 Y7 J* Y& f
1) A=foo This will set the env variable A to foo
( i, k- c% ~( C7 R# O2 s0 k8 M 2) B=$B:c This is inherit app master's B env variable.
' I+ B1 M: ]" l! H% U3 S' h </description>. V# Y' E. o2 g1 D+ r" d/ V
</property>
7 D$ m M0 t0 V# A<property>( ]# A& g' {1 F: H4 f
<name>yarn.app.mapreduce.am.command-opts</name>
2 n# _$ ^5 @2 [3 K1 Z1 H4 Y <value>-Xmx1024m</value>
- H& A7 i! t: i3 C <description>Java opts for the MR App Master processes.1 M- d% T/ W! a4 \' K
The following symbol, if present, will be interpolated: @taskid@ is replaced/ p) q2 M6 C" _* g6 P
by current TaskID. Any other occurrences of '@' will go unchanged.
& p% s- m! \2 ` For example, to enable verbose gc logging to a file named for the taskid in+ `7 Z+ o. {1 P+ W: ]7 A
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
& c4 N% `% s6 K$ |/ Q5 R( k8 e' H) p -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
$ J* K; r! _& |! i8 L( F Usage of -Djava.library.path can cause programs to no longer function if
6 {7 c0 R! c8 Q) m$ T7 g2 {/ V hadoop native libraries are used. These values should instead be set as part
8 }6 Z$ N) i( t& i of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
' E1 C+ C U7 @( R, f# v. } mapreduce.reduce.env config settings.
0 J" {$ `3 J$ `2 q# F% X0 O, \ </description>
7 ?: ?: \) Z$ V/ r* y! i# [2 U</property>
% V Q+ f# z( @5 W: u. }5 ?$ ]<property>
; W; |6 c S7 w2 B <name>yarn.app.mapreduce.am.admin-command-opts</name>( Q# V& J2 _4 k8 V$ B! w: C
<value></value># Y3 t4 K4 _7 P7 @
<description>Java opts for the MR App Master processes for admin purposes.
9 S) Y' _7 U# H+ Z5 U It will appears before the opts set by yarn.app.mapreduce.am.command-opts and
3 I5 x+ [9 Y; M$ _. n! Q% [" {+ J thus its options can be overridden user.
4 l, ?1 K% Z) v4 D. ?* i Usage of -Djava.library.path can cause programs to no longer function if
/ u% y8 @) Y" j _' f' F6 H% F hadoop native libraries are used. These values should instead be set as part- W0 O4 A( Y/ K( |
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and- o# K! [) i F& H9 M# C* ~) d
mapreduce.reduce.env config settings.
p# Z& t$ G$ ^( s2 U </description>
1 U, U2 s" S* j- p5 r. j/ A</property>
9 ?6 b7 ^* Q, K<property>
$ i8 ]: v' X, W7 U3 ?! k: g" m1 L <name>yarn.app.mapreduce.am.job.task.listener.thread-count</name>
$ B! p+ l5 m P& w <value>30</value>
4 i% v5 B& M% x+ h6 ?: m <description>The number of threads used to handle RPC calls in the
: o7 I @0 e' }5 b: j6 ~0 o% d MR AppMaster from remote tasks</description>& S4 \3 M: f" T" Y8 T9 s
</property>9 D0 s" S* h# y5 m/ B
<property>, {8 F( g! ~6 y; M/ E- U2 b
<name>yarn.app.mapreduce.am.job.client.port-range</name>. E3 o0 |: T* t
<value></value>
) E9 {1 n; b7 }* G <description>Range of ports that the MapReduce AM can use when binding.
- g" Y, u$ v4 x; m4 b. d/ h1 O Leave blank if you want all possible ports.6 ]% _# t1 B) Y. J
For example 50000-50050,50100-50200</description>
7 d; T* ^$ o4 d</property>9 \% k4 P* n3 J* S
<property>$ S7 p) _8 C) o# _1 t& s
<name>yarn.app.mapreduce.am.webapp.port-range</name>" V( x/ W5 ]1 R7 u/ N5 @7 N. N
<value></value>" u; [, v' c& Z+ z- p" ?5 \
<description>Range of ports that the MapReduce AM can use for its webapp when binding.0 o: H- b- J) ^% L
Leave blank if you want all possible ports.
1 r0 h% V: O- m* Z4 {) R, a% X For example 50000-50050,50100-50200</description>
. K$ o- @" `& L4 }' @; k& d" Q</property>
: F9 R8 Y+ t* P0 |/ z' n<property>, B& R/ d j# A- L1 {3 N9 t) S
<name>yarn.app.mapreduce.am.job.committer.cancel-timeout</name>
* r$ Q) J; @& n4 h3 l* M <value>60000</value>
) ?& `" ]- O4 O8 |" N% K( j <description>The amount of time in milliseconds to wait for the output+ ]; [- [! u' Z1 q9 a$ g
committer to cancel an operation if the job is killed</description>
, H% l9 j0 H3 @</property>
) D% n+ v# g- u' R<property>
1 t5 h' L6 A F8 [' c <name>yarn.app.mapreduce.am.job.committer.commit-window</name>! a' G" c# v$ a9 V* ~
<value>10000</value>
/ h5 F9 b A7 q& b) G- { <description>Defines a time window in milliseconds for output commit
/ g: n: V5 m) s8 g! d5 V3 D operations. If contact with the RM has occurred within this window then
8 b2 X+ D5 K/ T6 O7 ~ commits are allowed, otherwise the AM will not allow output commits until$ U/ L9 e3 G5 ]2 }/ ^
contact with the RM has been re-established.</description>7 {0 v) ~8 ?3 U8 f7 q, f* }
</property>$ T2 d2 N; ^; n4 n4 \7 \" r( c$ V
<property>, F" _6 ]9 h* ^5 d% d- C! r5 m
<name>mapreduce.fileoutputcommitter.algorithm.version</name>
8 y$ w, U4 q* X- w z <value>2</value>
7 {. R0 X# N; J' l# w <description>The file output committer algorithm version% ?# M0 D1 t4 Y& u
valid algorithm version number: 1 or 2
0 t$ e, _" ~& i default to 2, which is the original algorithm2 G, Z3 x, f" C$ y1 L3 c1 ?. v
In algorithm version 1,% }, h1 b/ t9 k
1. commitTask will rename directory% _$ `; S) B7 l% U
$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
( v! _" I( x" O$ h: M& E( w to
2 u6 u) x, z6 G $joboutput/_temporary/$appAttemptID/$taskID/" ]( [8 f+ z6 r
2. recoverTask will also do a rename
7 q8 |" ^$ a& Q/ M* O $joboutput/_temporary/$appAttemptID/$taskID/
9 G6 Q( X. F5 L. e T to
: F% V' j; u) k5 l $joboutput/_temporary/($appAttemptID + 1)/$taskID/; F8 O* A4 u- m- {1 ^
3. commitJob will merge every task output file in6 X% i9 i7 [4 j* C
$joboutput/_temporary/$appAttemptID/$taskID/
* X" V/ U x9 v6 ^3 I7 d to7 N) s/ u6 }+ f) q* B+ e. x
$joboutput/, then it will delete $joboutput/_temporary/1 l( @1 F. _; t. Y2 T. \
and write $joboutput/_SUCCESS: Q" S3 o* |& B- @: H {6 F9 o
It has a performance regression, which is discussed in MAPREDUCE-4815.
$ e: x5 x* d7 ^$ u5 s/ q* x: S9 v If a job generates many files to commit then the commitJob9 j/ c, C0 ?. @8 F& j
method call at the end of the job can take minutes.
8 l- j! B, w0 |# e the commit is single-threaded and waits until all
( R E* @% G& a tasks have completed before commencing.
% W& E( Z2 }: ?9 b- }9 C! c. F algorithm version 2 will change the behavior of commitTask,' Z, T) k/ t- y9 n. d
recoverTask, and commitJob.
: S$ |' L( ^' Y: _ 1. commitTask will rename all files in
$ m2 D+ a9 D5 N) T+ Y" { $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
( ~8 y9 v( B5 f2 a; J to $joboutput/
8 _" O. Z8 ~& }0 u6 s5 E5 k$ z 2. recoverTask actually doesn't require to do anything, but for) M. ]9 y* Y# E
upgrade from version 1 to version 2 case, it will check if there
1 y0 Z1 q2 U: Y5 A are any files in
% `6 ~5 ?; a; N4 ` b" |& o $joboutput/_temporary/($appAttemptID - 1)/$taskID/
$ O1 M( h" y3 i# ]. X0 ?/ p and rename them to $joboutput/
4 f, E a5 R [0 o& X0 G$ j9 m 3. commitJob can simply delete $joboutput/_temporary and write" p2 x' E b+ \8 m
$joboutput/_SUCCESS; c' V/ @- B# x6 @! W* X) K
This algorithm will reduce the output commit time for
9 k% ?* p; h0 K" ^ large jobs by having the tasks commit directly to the final
) B3 \' s; j3 ~* C: n output directory as they were completing and commitJob had
* ~4 F E; G5 a& u' ]) i; R very little to do.
* V2 @$ a2 c8 s2 L </description>0 U5 P+ |2 o3 }; m
</property>4 |0 d7 Z" u0 H/ e" y5 }
<property>
6 Z) v* i$ G% j* s# @. k0 e <name>mapreduce.fileoutputcommitter.task.cleanup.enabled</name>
6 W3 ~$ M0 c* i <value>false</value>% h) D- ]4 k2 f, j( q# ~- v; e
<description>Whether tasks should delete their task temporary directories. This is purely an! t4 h, _9 p% B
optimization for filesystems without O(1) recursive delete, as commitJob will recursively delete
- {3 x3 [ n- G" W4 ^+ C h the entire job temporary directory. HDFS has O(1) recursive delete, so this parameter is left& k. p% ^& H% Z3 T: z: H
false by default. Users of object stores, for example, may want to set this to true.
/ w n) f3 W, _$ s- ^: M* v+ h) u Note: this is only used if mapreduce.fileoutputcommitter.algorithm.version=2</description>
* E7 i; `/ P3 W5 [, r</property>
& j9 h" J! ]" f/ `* T4 U$ _) [0 o<property>4 |' _+ R( N( u, Y
<name>yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms</name>3 ~5 w% |8 ]. f& F+ {( A6 p
<value>1000</value>0 x3 |! J$ u9 \6 O
<description>The interval in ms at which the MR AppMaster should send
. P) k" o6 G% p8 ` heartbeats to the ResourceManager</description>* b8 v) D5 B$ m* r6 v
</property> d% g7 q0 c( l$ j2 k- g2 ~" h
<property> [+ d9 T. S; Z+ o; G& Z0 m( @0 G
<name>yarn.app.mapreduce.client-am.ipc.max-retries</name>
. r5 X( t% V8 D# L- @1 g <value>3</value>/ S1 y- E' H8 m( G% M
<description>The number of client retries to the AM - before reconnecting$ R7 E% C( T1 N+ c
to the RM to fetch Application Status.</description>) @6 }/ a8 e% N) k) B! v
</property>: a, d$ N: R2 n: W. [8 m
<property>
4 Z" r& ^2 N, J <name>yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts</name>
; V8 L" T: {! p, ] <value>3</value>% E5 j. o0 g* q% j. U2 C% v% {: c
<description>The number of client retries on socket timeouts to the AM - before
6 N+ V! ^; w W( z }" X j4 r reconnecting to the RM to fetch Application Status.</description>. B# o& x: j, W. M' {* Q! p
</property>+ v4 t. ]& k$ I" Z8 A3 i
<property>& X4 _% i' u) A( l! L
<name>yarn.app.mapreduce.client.max-retries</name>3 E& G4 _" P% F" H' K
<value>3</value>0 o% Z% r* V/ m/ U
<description>The number of client retries to the RM/HS before% G6 ^4 r) l" L: |+ }7 _. Z
throwing exception. This is a layer above the ipc.</description>
% Q7 A1 W$ D: S F/ Q</property>8 t) p& M7 q0 o; y y
<property>2 ^0 P6 \9 L' Y9 \9 b
<name>yarn.app.mapreduce.am.resource.mb</name> I0 F$ |% ~" O# U
<value>1536</value>( Q: @! H {0 x
<description>The amount of memory the MR AppMaster needs.</description>, ~6 Y' k; R2 `$ ]8 K! Q, a c T' ~
</property>' ?2 J7 K8 l' \
<property>
3 x1 ?3 C* z+ ? <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>4 v$ |$ C' D* \- K+ E1 h
<value>1</value>+ N4 `4 n- S/ L( {
<description>
( J5 B- x v! I/ C* W# _+ x The number of virtual CPU cores the MR AppMaster needs.
" z. ?; _7 C" g9 s6 x$ n </description>, [8 X9 X r0 G- R
</property>
& r8 M* p* r( `! u: P$ c* G<property># o- [- P+ H& \1 @
<name>yarn.app.mapreduce.am.hard-kill-timeout-ms</name>
8 E$ H7 ^# D h% c1 C <value>10000</value>
: d) s5 F T3 S <description>2 j! v6 }7 c8 \9 L" `1 _. O. y
Number of milliseconds to wait before the job client kills the application.
7 Q" W# g7 x& M) F: v* o2 G6 T </description>4 i4 M3 Z% J+ {1 [
</property> e) S" q" O4 O; }
<property>
& T- V( Q5 W8 S7 y: x <name>yarn.app.mapreduce.client.job.max-retries</name>( |( p/ N, d. T9 n$ Y' c2 {! r
<value>3</value>: P4 J5 [2 q; S; v7 }$ D
<description>The number of retries the client will make for getJob and7 m; N8 Q" a5 u; x
dependent calls.
+ [+ Z w) w" h7 ]' h9 H This is needed for non-HDFS DFS where additional, high level
' I- ^$ ?& W$ {/ j6 I retries are required to avoid spurious failures during the getJob call.
+ }8 ~8 B7 q+ L2 Z( y: w 30 is a good value for WASB</description>1 C: j$ U @# P
</property># D+ p, I4 H+ U
<property>' F4 b* k. F+ o' n7 _
<name>yarn.app.mapreduce.client.job.retry-interval</name>. k3 Y+ r- u( b- K. _8 j
<value>2000</value>+ s7 G G8 u* B5 _& _8 r5 A
<description>The delay between getJob retries in ms for retries configured
/ }1 g" k9 Y2 R: O2 k8 S) f# I: v with yarn.app.mapreduce.client.job.max-retries.</description>- y* p6 [( B& ~7 @1 ?
</property>
( u& s8 S% n% E+ ]<property>" o3 K( y9 q, x
<description>CLASSPATH for MR applications. A comma-separated list0 T* V. J. d- h. O( }4 U) K$ ~% J
of CLASSPATH entries. If mapreduce.application.framework is set then this
5 H" G5 ^3 a2 \* ~' |. q% G must specify the appropriate classpath for that archive, and the name of
- x* w/ ^0 D6 c7 j& g: c8 @6 E the archive must be present in the classpath.3 Y$ E( k) d7 |+ p% A3 g
If mapreduce.app-submission.cross-platform is false, platform-specific8 o7 n- i$ T/ w0 V$ }1 w
environment vairable expansion syntax would be used to construct the default, F6 `* `8 L7 S* _8 P
CLASSPATH entries.9 I: B3 Y8 M1 u
For Linux:5 Q9 d+ |6 x# ~
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
* O) [: B5 \: s6 N $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*.1 q$ o1 |4 k; d7 U. L
For Windows:9 G' G& e2 J1 v
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,: ^) `" B. V% I# x- B- T0 E
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*.& N7 x9 O% T% V2 t! o
If mapreduce.app-submission.cross-platform is true, platform-agnostic default
& k& i' f) g- F$ A* ^" F0 L CLASSPATH for MR applications would be used:- ]$ d- m& x; G. Y4 k+ G
{! u3 M% m( m& P2 o$ v* ^
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,
2 F% Z. e. ^/ Y& j$ I/ M4 v/ ? {
+ k# a) }/ o& y; _+ |! h0 ^ {HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/** J( C S% i% \. c
Parameter expansion marker will be replaced by NodeManager on container
: E R9 C/ m$ Z! C$ `& S% b. ] launch based on the underlying OS accordingly. ], T+ d- ~" d
</description>
& d- n9 A, p1 R1 w& @! [$ Z s& } <name>mapreduce.application.classpath</name>
3 K! T2 P, \0 n/ z <value></value>( c: ]5 E4 e" }8 T7 U9 X
</property>
2 t3 a7 V/ ^( P% I. b# N0 v<property>
/ |; j G5 k4 I2 w0 R <description>If enabled, user can submit an application cross-platform
7 o% h& A3 \- v i.e. submit an application from a Windows client to a Linux/Unix server or5 ^: b# q+ F, i3 Y5 H
vice versa.) I4 }; P% n( j& e. O
</description>, }! Y$ j# [! h7 O* v; s0 P v! l' `
<name>mapreduce.app-submission.cross-platform</name>7 x' Z7 R+ U& e0 D
<value>false</value>
6 G9 m5 m' a& }% U0 L</property>
( ]) A2 s9 j+ ^1 y, E<property>
1 n1 [) i& z( N& b& ]4 z& ` <description>Path to the MapReduce framework archive. If set, the framework
1 ]2 q+ A4 Y" C7 h4 P+ F archive will automatically be distributed along with the job, and this
7 u- Q0 U$ W- N, B, S path would normally reside in a public location in an HDFS filesystem. As2 I' v+ j: _, x) ?2 T+ S
with distributed cache files, this can be a URL with a fragment specifying' n2 y: y4 B/ y* A% L% l( ^, ]
the alias to use for the archive name. For example,2 U! A* f: x, W- O0 L
hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would
: u6 f% b2 V5 G2 Q8 k- c alias the localized archive as "mrframework".
4 c! ^4 O8 S- r/ H. Q Note that mapreduce.application.classpath must include the appropriate" P: Z; h M# M- e. ^! T! a
classpath for the specified framework. The base name of the archive, or
" C; t' q: J9 N$ E alias of the archive if an alias is used, must appear in the specified8 Q8 g" ~; ~. T( ?. e" b
classpath.9 c w1 K! e' r$ L+ S2 x
</description>4 s2 | W3 E: s$ x. J# A7 a4 C
<name>mapreduce.application.framework.path</name>6 J6 ]- q* C% g) @
<value></value>
0 s) E0 m7 M" J T5 k</property>1 B& n& @3 m' V3 c) p! j6 k; k
<property>* l7 a; N6 H- p! x+ Y
<name>mapreduce.job.classloader</name>8 n- r1 m% K# Y* D
<value>false</value>* T, V4 g( S: v9 L
<description>Whether to use a separate (isolated) classloader for
3 R5 P% a" z% h& p2 q9 i- G* h user classes in the task JVM.</description>
0 ~' }! E, b; U</property>
/ e1 ?. B3 _8 ~0 U<property>/ Z8 o6 s4 K, D
<name>mapreduce.job.classloader.system.classes</name>; j: P" W+ C; \( Q" H; m# U# l
<value></value>
+ Z& U5 y# a% R% r& t <description>Used to override the default definition of the system classes for
1 w, R3 [; i0 q2 t5 I* v the job classloader. The system classes are a comma-separated list of* p1 f% O9 g* I/ Y, B
patterns that indicate whether to load a class from the system classpath,3 }: ^* D3 X! t% \; F
instead from the user-supplied JARs, when mapreduce.job.classloader is' h+ R5 q, ]: Z; F: x: X [
enabled.0 N* p3 p4 [' G6 L! K2 I) I
A positive pattern is defined as:6 c* x6 Y( i" [1 \. {
1. A single class name 'C' that matches 'C' and transitively all nested9 v1 |4 C" V9 s" R/ D) H
classes 'C$*' defined in C;
4 l: h b. x' f+ T- X% p 2. A package name ending with a '.' (e.g., "com.example.") that matches
9 U/ n4 z$ M' ^% l: T all classes from that package.; `3 o7 g, a1 M/ G! C
A negative pattern is defined by a '-' in front of a positive pattern
( B) x+ l( l; G! R1 j% X! p (e.g., "-com.example.").
' \- j/ v. n& s& U A class is considered a system class if and only if it matches one of the
# C* S+ q; M* _( v" H positive patterns and none of the negative ones. More formally:; Z- s, r2 ] z; o; ~! W7 A7 C
A class is a member of the inclusion set I if it matches one of the positive+ @' O7 J3 q# r T {
patterns. A class is a member of the exclusion set E if it matches one of
6 L* g6 h/ B( v; B W3 l the negative patterns. The set of system classes S = I \ E.3 e2 u" B$ N7 s1 D0 `% Q& d' x. [1 h
</description>$ Y: j' t& _6 E5 g2 D2 ]
</property> f3 C; @5 r7 P' @2 g; t2 W
<property>3 q7 x1 m) U! F7 { d: _
<name>mapreduce.jvm.system-properties-to-log</name>: ~2 B- ~, c6 G3 [# W; v( Y7 P
<value>os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name</value>2 m5 \5 o5 t) u; x
<description>Comma-delimited list of system properties to log on mapreduce JVM start</description>
. f# ]; T/ |3 \* h( V* x</property>4 V% R. p8 ]( y7 ?0 P
<!-- jobhistory properties -->
- P+ f. }- T }2 j4 [: B. V<property>% R' Y$ X3 @1 e4 c h( E: F& Q
<name>mapreduce.jobhistory.address</name> p- C3 c$ ^, V/ l! n r; _; p
<value>0.0.0.0:10020</value>* J1 u2 O1 }* n9 M
<description>MapReduce JobHistory Server IPC host:port</description>! `# _2 v% A9 r- {
</property>' c2 h6 U' Y. R
<property>8 K3 r( K/ t1 g9 `& j$ b% p* @' z( a
<name>mapreduce.jobhistory.webapp.address</name>5 l+ {3 ?6 ?; v
<value>0.0.0.0:19888</value>8 A* k7 W6 a- T1 {8 ~
<description>MapReduce JobHistory Server Web UI host:port</description>! J5 F* i8 ^/ o& Z3 s1 |) O+ K# y
</property>' `9 a7 p1 ^. X: E! H
<property>( t/ l7 M# F: \4 l& P5 x& i. B
<name>mapreduce.jobhistory.webapp.https.address</name>1 n6 a# `' f+ g+ X8 Z$ F) M
<value>0.0.0.0:19890</value>% T3 s6 k+ C8 ~
<description>6 r, D0 G& E5 G1 R4 ^
The https address the MapReduce JobHistory Server WebApp is on.
. F& t9 n; Q* A2 n/ X. J </description># ~3 Z8 Z0 {, j4 Z, ?& w0 J! v7 u& @+ @
</property>
, J; l5 @1 @' Z# R- W2 l* I<property>
! o& }# ?9 F# a0 I" j% u <name>mapreduce.jobhistory.keytab</name>
, Z( ]3 U! \9 h4 M( D <description>
4 Z" I. ]; u: _+ ~7 m Location of the kerberos keytab file for the MapReduce
1 D! ]* v/ z& y( S JobHistory Server.
" U7 P x1 c6 D' }2 n0 V; t </description>' h/ v- ~( Y* h' G2 q
<value>/etc/security/keytab/jhs.service.keytab</value>9 G6 Z" y; W, b) X! l: g
</property>
3 ~0 P d; e' F- c<property>
" V7 [) q: c6 O8 s <name>mapreduce.jobhistory.principal</name>( P B- X7 Z# s" c: p1 v
<description>
# x }0 ^! K3 H8 z/ e9 z t% g Kerberos principal name for the MapReduce JobHistory Server.
" h/ h8 u4 L2 [( [ </description>
1 p: ~: a, C8 _7 u- S2 w <value>jhs/_HOST@REALM.TLD</value>
0 D: t+ Q7 |$ R</property>6 t7 m1 z' j" x/ f9 Z! j
<property>0 W& p+ M6 K: n) I4 A# t, _
<name>mapreduce.jobhistory.intermediate-done-dir</name>! O& @# j4 k- g& g) O( f$ [ Y
<value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>
' a) H& T. Y+ s& l <description></description>. w0 V" L' e8 o' C2 `* R
</property>" @1 y* v$ _% {- _- q
<property>& z/ }" ]1 E' F
<name>mapreduce.jobhistory.intermediate-user-done-dir.permissions</name>
+ n; m% H1 S9 [ <value>770</value>
: P9 j: T& u$ L" z: [& Z <description>The permissions of the user directories in* n/ D$ }. H$ Q y4 z- c
${mapreduce.jobhistory.intermediate-done-dir}. The user and the group
: t+ K2 c. d$ e0 c) b$ a$ K; H permission must be 7, this is enforced.$ ]+ ]7 _0 }6 _5 X" V8 g! N& ^
</description>7 q, |3 B7 |0 I- b* k1 T& y3 x6 _
</property>+ S" I; }" Y( H$ M2 n
<property>
$ a4 @+ X1 r6 Q' G4 Z& S- g <name>mapreduce.jobhistory.always-scan-user-dir</name>
- B9 B) M7 s6 r! ? v% x- m1 M7 v- [ <value>false</value>- l8 O- W! j$ s* l: u, c" W6 [* @
<description>Some Cloud FileSystems do not currently update the
7 K* _, @6 L3 ~1 v: X: A+ X) J: c modification time of directories. To support these filesystems, this4 K; Q& F3 A' e% j/ U- `" g4 n% r
configuration value should be set to 'true'.
: ]8 _2 H0 L1 [" h' ?. T1 H: A2 g3 j$ k </description>
% `3 Q0 f# a" ?4 c; }3 `3 i$ U0 S</property>1 r2 K6 n2 Q& l$ ], W
<property>+ H# I. ]8 ~7 ^' G
<name>mapreduce.jobhistory.done-dir</name>
! L% P* G6 @+ p# T <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
0 `9 w% Z+ G1 ?: X, y$ k <description></description>, C6 w, {3 m, b9 h
</property>3 b2 v0 L4 s9 ]! Q. L( p
<property>; Q1 {7 ?# W4 x8 g8 |/ L% F% |4 U* r
<name>mapreduce.jobhistory.cleaner.enable</name>
8 {- S7 R L. m, Y& n <value>true</value>
0 w3 \) Z" i4 s4 R: s: [- ?. u <description></description>& X/ z* }- Y n2 ] G5 P
</property>
/ G6 C6 H" _8 P* g K! S+ H7 X<property>2 j1 j: V0 p0 [2 L
<name>mapreduce.jobhistory.cleaner.interval-ms</name>! ?8 ~; {2 p7 J( k
<value>86400000</value>
1 k* V9 L! {; j5 b <description> How often the job history cleaner checks for files to delete,
, Q; U D4 F6 o, O in milliseconds. Defaults to 86400000 (one day). Files are only deleted if
4 p$ k+ Q, i- \: v4 o4 m8 E8 t they are older than mapreduce.jobhistory.max-age-ms.
' }0 U) u& f# U$ \2 r) x, l </description>
9 f$ R9 F9 ^6 F, n6 F</property>7 g! g, p0 @+ L3 J) y
<property>
5 I) \; V4 u1 y% h <name>mapreduce.jobhistory.max-age-ms</name>
% U1 c7 i0 E2 X7 ?6 g! Z3 b <value>604800000</value>- _% y2 a+ [; S5 `) y; j# _% S
<description> Job history files older than this many milliseconds will
" [! q& V7 i8 E be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
, A6 Z* k0 X% ` </description>
+ o/ ?7 E3 \8 Z% D</property>
! n* y7 f8 U g. {$ k2 ^<property>
5 G- ?2 [$ T c4 k4 K: U <name>mapreduce.jobhistory.client.thread-count</name>
9 u+ K& l8 M) T# w0 S; k3 _ <value>10</value>6 b3 \0 A( ]7 s$ E9 V
<description>The number of threads to handle client API requests</description>
6 U. v) e9 m2 o/ ^</property>4 m+ m) `( X; T v: \+ ^6 B4 S9 {, B, M1 i
<property>
! A8 k8 G- o) b" T+ j <name>mapreduce.jobhistory.datestring.cache.size</name>; q6 @! V6 `! A+ n( \9 H9 t4 n3 E* s
<value>200000</value>! i$ N& C2 R: U7 d" b( s1 ]( W, T/ M
<description>Size of the date string cache. Effects the number of directories
+ A: }6 s# L2 Z [8 y1 G which will be scanned to find a job.</description>
7 ] [" c& ^3 _* Q) i</property>6 r+ t( x9 W# a {4 B$ ?1 {
<property>$ `4 U2 [6 ^" }8 |
<name>mapreduce.jobhistory.joblist.cache.size</name>7 H& K0 w' z& V% a0 g7 y$ T- n
<value>20000</value>3 ^& d0 U) L4 I
<description>Size of the job list cache</description>( n/ F/ q0 q1 s" U# |5 [6 A: \
</property>. h9 P6 e9 Y; g3 u0 b z
<property>, T3 r- m* h j, w: ?
<name>mapreduce.jobhistory.loadedjobs.cache.size</name>
0 y5 ~& `, k+ S& U8 L/ H <value>5</value>( d* |- U, {/ a2 L/ \
<description>Size of the loaded job cache. This property is ignored if# i9 l+ H6 `$ @8 Q6 l* ]" J
the property mapreduce.jobhistory.loadedtasks.cache.size is set to a
7 }/ i" |% v4 H& [8 K positive value.8 x" J' U3 |- [$ _0 Y
</description>5 n: w+ _5 Z u* t0 Z
</property>
) d9 p6 i" M1 b2 N<property>( [. O, e v: [0 C4 E2 q
<name>mapreduce.jobhistory.loadedtasks.cache.size</name>+ t1 ^$ Q$ f# `
<value></value>
0 _0 e$ i4 H# l# h <description>Change the job history cache limit to be set in terms1 `4 p* x1 V+ Q M( i1 C2 Q
of total task count. If the total number of tasks loaded exceeds. t! ?) F3 ^: Z9 E& d
this value, then the job cache will be shrunk down until it is
: E) U) |/ [, M; B under this limit (minimum 1 job in cache). If this value is empty
- b8 ?& G* b* }2 e2 Q or nonpositive then the cache reverts to using the property4 u$ ]! a+ h/ K: [9 Z* r
mapreduce.jobhistory.loadedjobs.cache.size as a job cache size.
. [* W) A- @; c$ _ Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size
& z' E0 \ q+ f9 ?. A$ l. G property:8 _6 E5 H1 _8 j% ]) d. L
1) For every 100k of cache size, set the heap size of the Job History
' @9 S4 E/ Z& C' ]2 { Server to 1.2GB. For example,
1 \+ A2 I; q! s4 `0 V mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.
i8 t' V6 N1 o( }2 H) w/ y 2) Make sure that the cache size is larger than the number of tasks/ ?% K9 M) n4 ~; ], {+ W4 [
required for the largest job run on the cluster. It might be a good
3 C% S y* e* t% Y8 N+ e6 Z0 F idea to set the value slightly higher (say, 20%) in order to allow; x* C8 ?$ j0 [+ M+ A4 g
for job size growth.( y: F' D- x$ P, D7 N1 X+ ?3 h5 v/ O& e0 Y
</description>; N, T$ {2 x) n) t Y! V9 j3 S
</property>
9 X5 o$ S1 |3 E! f1 C7 E<property>
; L7 g7 V2 x4 s C; a! @ <name>mapreduce.jobhistory.move.interval-ms</name>6 ]+ v; b) d/ E
<value>180000</value>4 T. c ?+ H# T: A, Y( R/ N
<description>Scan for history files to more from intermediate done dir to done
6 \; X7 q, K6 m. D2 G& w dir at this frequency.
S* C( p% b7 A, C J- G; _. D X, e% K. T </description>
! L0 {- G) W5 k& y1 z5 J</property>
" Q% X; S4 B5 a {& {<property>
* O# ]0 c7 U$ R6 _ <name>mapreduce.jobhistory.move.thread-count</name>
4 P D$ F7 x( Q3 `$ o <value>3</value>
% q `8 G. Z) Z1 x2 m <description>The number of threads used to move files.</description>
0 s1 p& p' D* `9 t4 f: w8 w5 K</property>6 H' F% R+ y! p, E1 Q& i4 y" I
<property>/ H" m9 y. x( `) [* Q8 _
<name>mapreduce.jobhistory.store.class</name>
' S }4 d# \) X+ S9 G+ _# U <value></value>( b e* c1 g' R
<description>The HistoryStorage class to use to cache history data.</description>
1 N6 ]& P6 }0 p r0 f</property>5 }/ q# m4 y! C
<property>! |/ a; |1 s! {$ e2 i( M
<name>mapreduce.jobhistory.minicluster.fixed.ports</name>1 x0 M. Y7 U8 ?9 t
<value>false</value>
( S! V: p7 {8 A. c: d. p/ f <description>Whether to use fixed ports with the minicluster</description>
$ d8 M0 @( V5 ?$ o* h! b$ m</property>, {9 D( {, U3 A1 N8 m4 T! f
<property> ^; u3 ] k) x5 f8 ^
<name>mapreduce.jobhistory.admin.address</name>3 c8 ?( Z9 v' o/ [7 ?
<value>0.0.0.0:10033</value>
p s; O- ~5 t2 d% L <description>The address of the History server admin interface.</description>9 I2 `, l, J& O; Y& M
</property>
- V1 J$ M) _; N; U<property>
7 a9 h4 Y8 _* F1 l" C) n2 W <name>mapreduce.jobhistory.admin.acl</name>9 S$ Q$ A6 v& q+ e
<value>*</value>/ B- J5 x6 C# [/ X9 v
<description>ACL of who can be admin of the History server.</description>" x# U1 r5 ^4 J' S& T$ s, F
</property>
7 V5 k+ ?( P8 G1 J<property>! M8 M( P$ D5 |# ^0 f6 F6 B7 o
<name>mapreduce.jobhistory.recovery.enable</name>
6 U# o% F+ v% t( O( I! i <value>false</value>7 m+ k. }7 t8 q1 M$ u# g+ j
<description>Enable the history server to store server state and recover J* Q( H" j7 D) b
server state upon startup. If enabled then
* `+ w/ G# A$ g& M mapreduce.jobhistory.recovery.store.class must be specified.</description>' B: t( U* U- s& U( I
</property>3 c4 G5 x9 Z; J; j! c7 u( r( g
<property>
4 S y, N+ T) c) d( l; ?. z4 l <name>mapreduce.jobhistory.recovery.store.class</name>
$ ?' P4 K2 W9 n; F- q <value>org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService</value>
0 s- g) y' a1 w6 V <description>The HistoryServerStateStoreService class to store history server, Q" r8 p. q% T1 _. z1 `
state for recovery.</description>
1 ?) p3 o9 g; y1 F</property>
4 n& B2 j3 Q1 F( a- ?: e! Y$ G<property>" F" G9 E2 ]. I$ r! c
<name>mapreduce.jobhistory.recovery.store.fs.uri</name>2 J' `$ L3 x7 ]. A
<value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>
0 O6 T# Y& o A <!--value>hdfs://localhost:9000/mapred/history/recoverystore</value-->3 c) i1 U% \0 B3 E3 {
<description>The URI where history server state will be stored if0 L$ t8 t9 I4 u& F4 ]
HistoryServerFileSystemStateStoreService is configured as the recovery( W* H ^% E. D+ U7 u% h
storage class.</description>* y/ p! Z' B8 h1 J4 R& E
</property>
; x3 m% D7 u% s3 M: k0 L, N: C! J- H% ]<property>; G6 Z, H* q9 g. Y' c- z' R2 F
<name>mapreduce.jobhistory.recovery.store.leveldb.path</name>
$ ~: o) X4 r3 q5 C <value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>
) n9 k; ^5 ^$ D( P- \* E <description>The URI where history server state will be stored if
! y( p8 K5 a* f- S* H8 `) p HistoryServerLeveldbSystemStateStoreService is configured as the recovery
* V5 a, C2 @6 f4 H% @# Q: Z0 C storage class.</description>+ I2 ^9 ?3 F, A& ^
</property>
% Y+ F" D! E0 a s<property>% J2 R, g" A; b; j0 c, ~1 n3 l6 x
<name>mapreduce.jobhistory.http.policy</name>
+ s9 ~% E: o1 |0 a5 n, E <value>HTTP_ONLY</value>
4 A& @9 j. E' b& k. D% q% r" @) e4 o <description>* k4 ^ l: `( ~
This configures the HTTP endpoint for JobHistoryServer web UI.% ?- F8 y) P7 r% }- h
The following values are supported:0 K% L" s. \( n4 G( |# u
- HTTP_ONLY : Service is provided only on http& `. `6 T) r7 S% M2 G: | `6 n
- HTTPS_ONLY : Service is provided only on https7 H8 V$ Z; {& F) Y
</description>
' Z) M, v6 k; L+ t2 ^</property>9 h; L9 ~3 m' K. y2 u
<property>
- X* Y: k9 k q8 \ a <name>mapreduce.jobhistory.jobname.limit</name>7 y" n W' ?6 U) g, S! J& Y
<value>50</value>
0 y+ b! }" @1 G# f3 q/ u! A <description>! ~. c; e r: V* Q- @
Number of characters allowed for job name in Job History Server web page.
! V9 X9 ]: ]" v$ t+ M </description>; G, i5 S- a1 X6 y$ D4 P: i' Q
</property>
' O- r5 ` b' y<property>/ ?* i v/ S* Q! y
<description>
& B6 S% r2 x6 ]; _ File format the AM will use when generating the .jhist file. Valid
T% D2 r }# r% O. ^5 S values are "json" for text output and "binary" for faster parsing.3 {: H% Y" H( Q3 F
</description># h' e% q4 s0 b9 H2 |
<name>mapreduce.jobhistory.jhist.format</name>
& l! ]0 C- K* m0 F7 |% F <value>binary</value>& {3 U8 U1 e; Q- F, d; I& m
</property>. ^& Z7 K$ T4 J- [- V4 o. Y: y' p
<property>3 P' n* s) F- W% x' I
<name>mapreduce.job.heap.memory-mb.ratio</name>
+ g" T8 c4 |4 t" [5 K" E <value>0.8</value>
+ Z" `# p6 j9 W3 l3 z8 U: p/ c5 e! p <description>The ratio of heap-size to container-size. If no -Xmx is
$ U: u0 n: D# O% y' V" x+ g& w specified, it is calculated as
/ d+ c' i2 Z. s Q9 Z5 W7 [: [* q+ V4 G (mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio).) V6 Y/ e+ ^7 S, L0 l8 y5 R
If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is
: I5 O. M2 g2 l4 t$ b0 ~# p. r b5 q calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
4 b6 S K1 O- \$ a5 P </description>% A; t5 F/ \( _, ^ P, \$ K0 F2 o
</property>
# ~2 L6 y6 ^9 l$ l& W4 W) x0 c<property>. {4 B( n, b! Y# J; P4 f1 @
<name>yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size</name>" \; @! w' q& \ |/ Z" ?7 h
<value>10</value>
# g$ V. u+ v$ B! p3 E+ O <description>The initial size of thread pool to launch containers in the
$ K" `- E7 |& } app master.% Q) f% F5 x) b
</description>5 W J& m1 |* Q
</property>: n# I) I- G8 S# N0 f r$ x
<property>
4 s8 H$ U8 ~4 A <name>mapreduce.task.exit.timeout</name>& _ t! _- _3 W% H7 x6 I8 u
<value>60000</value>
+ m0 B9 S! F0 t8 T. ` <description>The number of milliseconds before a task will be! a$ {/ c: L! B( j
terminated if it stays in finishing state for too long.; O3 p0 D; u+ R
After a task attempt completes from TaskUmbilicalProtocol's point of view,. b2 G5 O+ `2 h$ H! d
it will be transitioned to finishing state. That will give a chance for the' l3 Z! Y) V' Q6 d3 {
task to exit by itself.
: u8 b2 f& n; @( }* z- V- l4 ^ </description>/ u+ i$ q. b' a) p$ x
</property>' K# v+ w6 H( _' x3 P' ~* S5 H
<property>0 l9 S% a/ \% M+ d- ]+ r
<name>mapreduce.task.exit.timeout.check-interval-ms</name>* U! c$ h. o9 `1 V
<value>20000</value> P3 F' s. q( m* t0 p
<description>The interval in milliseconds between which the MR framework6 V( r# c' s. g, E3 A
checks if task attempts stay in finishing state for too long.
; U% w- b2 ?7 t7 A+ Z0 h </description>( X$ z1 x2 W- ^8 j1 c( J( A2 q
</property>& C9 f d" D0 l/ c7 k8 @
<property>( c3 V* g& j9 U- U
<name>mapreduce.job.encrypted-intermediate-data</name>
K2 n) k' m5 j$ P, k <value>false</value>
5 P6 f) c z4 d6 z# ~ <description>Encrypt intermediate MapReduce spill files or not
! Z# F9 y2 q) N) I default is false</description>
5 R9 E7 Q" D. ~; ^/ \</property>, \8 W% i1 p g" G, p( x
<property>
+ H. u h/ j7 H0 n# [3 _) ]" y <name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
" o, \" a$ T4 @- H! p% U <value>128</value>7 y: H$ p$ `* I) ?# \8 L
<description>Mapreduce encrypt data key size default is 128</description>/ ~% ]# l8 T4 X6 ]/ t( D9 ^, G
</property>
- K0 X' D1 C2 N3 \; D<property>% x, X- c; u: @8 k( W
<name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
3 q3 T" u$ ]" P% U3 Y6 v <value>128</value>
( @, m% Q+ ^. @) @& N" @$ {1 d <description>Buffer size for intermediate encrypt data in kb" g+ J! t$ a- E! X# P; ~8 p
default is 128</description>
! P4 p4 l$ @) x9 Z. L& M. n</property># N9 Y7 s! U6 f5 z" b2 E
<property>
8 |9 x( t. E7 I# y6 v% n <name>mapreduce.task.local-fs.write-limit.bytes</name>* K( b' f' T8 v* h
<value>-1</value>7 Y/ D+ L. ?7 ` F% S5 u1 u2 n9 q
<description>Limit on the byte written to the local file system by each task.
9 T6 `4 L# I% p# V2 b l' \' g This limit only applies to writes that go through the Hadoop filesystem APIs
y C4 h0 V2 A within the task process (i.e.: writes that will update the local filesystem's
h: `" L; J4 B+ ] BYTES_WRITTEN counter). It does not cover other writes such as logging,
! O. t+ V; m# _4 B+ N7 W sideband writes from subprocesses (e.g.: streaming jobs), etc.
7 M7 Y$ C; k1 z/ [8 L: X& t$ `$ U Negative values disable the limit.. k8 x2 e: Y' U! C/ y# R
default is -1</description>: K" R/ C. i- }5 H) D
</property>8 ~$ K' ~1 m" j$ d% S& m! h
<property>
1 [9 A! |3 ?1 _* L m <description>
b, U' `! |7 f Enable the CSRF filter for the job history web app
" K. W. |% G) S# |$ e" H' b0 L </description>
2 R' p8 }2 k Z4 Z: W8 o/ H1 N) @ <name>mapreduce.jobhistory.webapp.rest-csrf.enabled</name>
' q8 u( e, {5 I8 z, U/ M8 A7 w <value>false</value>
; ~6 p! E4 d" ^# z" h! @</property>7 _, B8 K0 T! g# ?) Z" M9 k
<property>
$ g4 l' o( C1 r# s$ o% t <description>. O& e2 b4 ?* F: z
Optional parameter that indicates the custom header name to use for CSRF! q2 m+ s& f" }2 m
protection.7 o% \3 l! F2 v# Y8 C, ^; F
</description>
" R) {( V: P U# H; t) u X# U9 J <name>mapreduce.jobhistory.webapp.rest-csrf.custom-header</name>7 B/ T; l5 N8 p" [( e9 E
<value>X-XSRF-Header</value>" E& P' w2 z% `: m
</property>. X+ U5 m2 f+ g8 d( N
<property>
3 t8 {' H( V8 H x8 T$ ~; n5 P <description>
( [4 R& W- t4 Z$ I% o7 F Optional parameter that indicates the list of HTTP methods that do not: J% q' C4 n4 H: Q
require CSRF protection
9 M1 d3 d$ B4 I ] </description>/ A- \: v1 T8 C; J% G5 |0 }
<name>mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore</name>
+ |/ J' ^, l% ^/ @0 {( j <value>GET,OPTIONS,HEAD</value>; ?4 M1 ?7 p" ~8 _
</property>3 ^ ~, w& F6 _2 s4 x+ n
<property>
# B. W3 V4 i1 H. x7 l1 t7 q <name>mapreduce.job.cache.limit.max-resources</name>" W& v3 J6 i a' z7 M! I
<value>0</value>- B7 F9 T3 Y- ?3 w
<description>The maximum number of resources a map reduce job is allowed to- ~- p( D9 z1 D! k# [
submit for localization via files, libjars, archives, and jobjar command9 X9 d; @2 `0 r5 X
line arguments and through the distributed cache. If set to 0 the limit is
% P2 w* P8 m0 u# {. \/ P ignored.
1 Z# \/ p0 [3 g </description>3 k0 v6 J$ F+ R ^% O* @$ z
</property>
2 l* M+ {5 e A! ?5 p% X<property>
2 M! P' w1 n( \& P) V8 _- o <name>mapreduce.job.cache.limit.max-resources-mb</name>$ B0 a& ^3 ^) p& A' T
<value>0</value>3 u: @2 X _' g. Y% a* d
<description>The maximum size (in MB) a map reduce job is allowed to submit9 R& k( H7 B6 |
for localization via files, libjars, archives, and jobjar command line
' a( G1 r) D9 {) c6 T arguments and through the distributed cache. If set to 0 the limit is$ y! u: i$ i$ N- q/ v! S9 m
ignored.
6 A! p0 U8 X- q+ s4 O }% h5 w% `) l </description>
. O/ y% G2 U, V</property>1 B7 h* @) c% E. r8 m' u* o* q( m f
<property>3 d9 u2 Y7 w: C7 f# w. B2 b, t" N
<name>mapreduce.job.cache.limit.max-single-resource-mb</name>
$ x3 v0 I) ~" H/ {% `" [ <value>0</value>0 L: O& E& [7 u6 i# ~
<description>The maximum size (in MB) of a single resource a map reduce job% f" M5 n3 j5 l" N9 g+ Z3 u" J
is allow to submit for localization via files, libjars, archives, and* K5 _; d I C6 U
jobjar command line arguments and through the distributed cache. If set to
) G& I) o( Y- a3 l7 e 0 the limit is ignored.9 y6 Y/ B2 S# K N: w$ x. u
</description>
/ V9 N, b$ O4 m$ a# y' s& U6 |</property>
- s" ]/ K3 x3 g9 n7 Q' n1 i<property>
; E* P8 O9 V* ?/ g <description>+ x5 \0 [8 _. ~5 a, O- w
Value of the xframe-options; q, h. I- M& n* A0 o
</description>
! Z7 F1 W! j) L: ]7 Q& f+ b0 O9 P) c <name>mapreduce.jobhistory.webapp.xfs-filter.xframe-options</name>
9 X- y/ Y, I( |4 e+ d <value>SAMEORIGIN</value>
+ Y- M# b' h. F</property>
3 r% B9 J. h* s6 v+ J) d<property>: _; h/ l8 `4 t* [" }3 L
<description>
' d3 D4 T' n0 C; _; W The maximum number of tasks that a job can have so that the Job History' X2 h4 k* D1 U) b+ h
Server will fully parse its associated job history file and load it into
& }1 b# z1 L3 ^6 U @1 _7 u memory. A value of -1 (default) will allow all jobs to be loaded.
% Z z f1 v+ G! K0 @ </description>
+ Y- {8 c5 @8 L5 e* m2 r <name>mapreduce.jobhistory.loadedjob.tasks.max</name>" J+ X$ q, K7 l8 u9 e' f2 r
<value>-1</value>6 f" b2 o4 R9 w3 o
</property>
1 k G" N( x! }! [/ A<property>
/ s, }3 }' r2 n3 y5 h' O- g <description>& t/ J7 Q/ G- ]1 _( \1 S
The list of job configuration properties whose value will be redacted., v6 Z* Y6 U' u; E* N' m9 s
</description>
2 Q0 K$ G& Q0 R1 P. h* ^ <name>mapreduce.job.redacted-properties</name> g t/ B7 ?, G, S1 K
<value></value>
4 ~$ S: w- t+ h+ i5 ]" a9 P</property>- P. D' L) b7 a! W7 h
<property>. c; |: L; O* H& y
<description>: E; q; M& q+ O* ^2 | s
This configuration is a regex expression. The list of configurations that7 H+ a( h5 t& B7 ~2 _; h5 u; }
match the regex expression will be sent to RM. RM will use these& ] K/ p. _& O8 M# e. e1 p. b
configurations for renewing tokens.3 o/ M1 F$ W2 i- O
This configuration is added for below scenario: User needs to run distcp
6 I% U' R5 D' M) M+ ~' ]* ~4 @ jobs across two clusters, but the RM does not have necessary hdfs
* G0 v, Q5 W* S; B configurations to connect to the remote hdfs cluster. Hence, user relies on
3 e! S% E& c2 R+ Y; p this config to send the configurations to RM and RM uses these$ U7 v$ O4 c6 Q4 V. v
configurations to renew tokens.; X/ H5 z- T4 R
For example the following regex expression indicates the minimum required
* a2 x3 Z- ~" I+ `9 h+ ? configs for RM to connect to a remote hdfs cluster:
* j8 T' y- M! @7 c; k- Y dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$|dfs.namenode.kerberos.principal# N2 f5 ]2 W; f
</description> Q3 t* q; m6 a% `+ X+ q v
<name>mapreduce.job.send-token-conf</name>
, o+ k0 X* U2 `- A, ^/ U <value></value>
4 |7 H3 G5 l: X4 d6 x; H, M; d# F</property>6 X: E( `% W+ z
<property>% S, d% M* V j9 S
<description>
+ n' x! z" G( ~6 L% @7 }$ O5 f. S The name of an output committer factory for MRv2 FileOutputFormat to use3 L7 b1 _: w9 e
for committing work. If set, overrides any per-filesystem committer' Z0 P/ o6 w5 V4 y$ f4 e! |
defined for the destination filesystem.3 [% A3 \3 w8 R" B! l7 a
</description>
( e) q8 @" P4 @# [; N8 w <name>mapreduce.outputcommitter.factory.class</name>
. r5 z* L+ r) S0 T: K, [) n5 ` <value></value>1 j% n" v) O6 x2 v" M" v
</property>
* E( f1 i0 [+ n) j8 d<property>3 c. l) x* l9 T
<name>mapreduce.outputcommitter.factory.scheme.s3a</name>
4 U. [3 V) F; U' `! T <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value>8 y8 L' _' m9 B) ^) z/ f) h
<description>8 b! J! t" G$ m$ B8 O) P h& [
The committer factory to use when writing data to S3A filesystems.* R% M3 F# `; W2 \
If mapreduce.outputcommitter.factory.class is set, it will
& l; y; ^0 ] U3 D3 j% z n1 S' G override this property.
. ~ m! D# s `4 K3 B# M; M! } </description>$ r1 ~" [2 ~0 w# U1 r& }9 T4 m0 D
</property>
. r7 ]8 e ?. Z- v</configuration>* Q' X: m* U2 o' Z/ a5 c7 `% o
4.yarn-default.xml
! H0 H7 X# Q3 o% Q! I8 _
! s8 h: d0 K5 G/ X8 O) i<?xml version="1.0"?>8 A" J# h! T3 S6 p" I
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>" G5 ?/ z- h% r1 G6 q+ Y; f& F
<!--) K' I& {6 r$ C1 |$ F9 }
Licensed to the Apache Software Foundation (ASF) under one or more4 V7 C6 G2 @: v! s
contributor license agreements. See the NOTICE file distributed with
: Z4 z. W/ b1 x% e5 q9 m this work for additional information regarding copyright ownership.
" C+ J* H3 m }+ }$ H The ASF licenses this file to You under the Apache License, Version 2.0 m5 T. U1 o0 \* \
(the "License"); you may not use this file except in compliance with
5 D% G, N7 E" U$ b# ~: c& Z3 S the License. You may obtain a copy of the License at
9 e/ u9 d* |1 y& Z. H* N$ ^ http://www.apache.org/licenses/LICENSE-2.0
9 K: r3 w) S7 i: P# B Unless required by applicable law or agreed to in writing, software/ `& _# n5 [: C9 A! d$ t
distributed under the License is distributed on an "AS IS" BASIS,
' \6 b' D$ E# b& N5 W WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.( E" k4 x! J- m e- m/ z- e
See the License for the specific language governing permissions and) v& F0 s1 _3 F% V
limitations under the License.
) t4 @+ [: T# U% F* \4 ]5 @1 M0 Y-->
' R; I" W& F4 X3 P<!-- Do not modify this file directly. Instead, copy entries that you -->
/ Z7 q4 ~! P9 s+ T) z2 V<!-- wish to modify from this file into yarn-site.xml and change them -->" C( `& P7 f) l. h- G$ p
<!-- there. If yarn-site.xml does not already exist, create it. -->
- g# F) X0 t( f+ n& n, ^6 c( M. z<configuration>
: \) j+ i. }9 h' p( Q1 c <!-- IPC Configuration -->
# B, g; J6 l3 ^ <property>" _) N2 q# E/ Q' r0 U6 O8 s
<description>Factory to create client IPC classes.</description>
# N; r+ B5 N- |7 {+ M0 S <name>yarn.ipc.client.factory.class</name>
% f r6 i, J! s7 Q </property>
+ \6 S* q" X$ \; w5 f <property>
, W u- q: s) k2 Q. i <description>Factory to create server IPC classes.</description>
7 t. I* b q; m( {. Q4 b <name>yarn.ipc.server.factory.class</name>
/ T2 U: T3 {4 y+ U </property>) J, M6 |! n; c+ ?( @
<property>* g" V6 g/ T, J' a0 U
<description>Factory to create serializeable records.</description>9 L- [2 K! S2 r" n8 K# f) O! {% t
<name>yarn.ipc.record.factory.class</name>: Z t" b1 e0 R3 ]+ `" {
</property>; t7 @7 R. b, l* i
<property>
( U) {9 N0 q" _( q) P9 x6 Z( t <description>RPC class implementation</description>
. B% |; \- X0 V8 U <name>yarn.ipc.rpc.class</name>! ?4 Q. Q' s O
<value>org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC</value>
, |! ~0 C' S. D' a </property>9 O8 W* @$ P* b1 N$ r+ L
<!-- Resource Manager Configuration -->
0 ]2 O* g( l4 d, B" ]& G <property>
! Q' R% j. A: B3 I <description>The hostname of the RM.</description>
# E% S/ x% r) f0 _* X* _ <name>yarn.resourcemanager.hostname</name>
* W& M8 y% T8 ]& B& |$ h <value>0.0.0.0</value>
( |7 k8 D! ~8 a' M0 c6 |3 P0 t$ y </property>
) |' i6 I5 v0 ?! [ <property>
8 z. p( L' m( G <description>The address of the applications manager interface in the RM.</description>
- Q; ^' p7 M' R% r I& o9 T <name>yarn.resourcemanager.address</name>
( m0 }0 N" n" U$ |# ~9 S- R! D9 P7 \ <value>${yarn.resourcemanager.hostname}:8032</value> X5 U7 H5 i L6 L' u- q
</property>' f9 r8 V1 o- Z# P8 c, `8 U8 l# X: \
<property>/ R+ a( z7 B. o, g% M* A u. }) g J
<description>3 T0 U, ~ Z- P* O0 D
The actual address the server will bind to. If this optional address is8 x' H/ A7 m0 ?
set, the RPC and webapp servers will bind to this address and the port specified in
5 ]. k6 ~; C' L yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This
" V/ ]0 T+ v6 j: z* H0 c is most useful for making RM listen to all interfaces by setting to 0.0.0.0.) ^- k; g3 T7 s, z$ Q) `) {
</description>2 q3 E1 m' D& f' c7 m
<name>yarn.resourcemanager.bind-host</name>
' q/ S- a- D( P; J: ] <value></value>
F4 Q7 ], p/ n2 t& ~4 _ </property>; Y, [4 f) A! l
<property>
: u# Y9 T8 E! l# k1 V <description>- ^ \, J9 C3 @+ a
If set to true, then ALL container updates will be automatically sent to3 j2 Z( k* L0 C) z2 X C
the NM in the next heartbeat</description>
`2 M1 F$ m$ f) m <name>yarn.resourcemanager.auto-update.containers</name>
' Q2 |7 v0 o: F <value>false</value>
4 y- X; H. m7 J0 L$ |! r </property>
; l: t& i# h; [" s2 ^ <property>
( b u: ^8 \* n. n! t- G7 P# `# e <description>The number of threads used to handle applications manager requests.</description>% I$ M8 d+ B- G& k t7 i/ h
<name>yarn.resourcemanager.client.thread-count</name>4 t$ b1 _( r$ B& w; ^
<value>50</value>2 B5 e7 f" c8 Y' J) s
</property>3 C; g/ e( U5 u( L! C$ }& z
<property>
" k# r+ U# u3 W$ T: t6 p <description>Number of threads used to launch/cleanup AM.</description>
5 g% m' o* e6 `2 T3 A <name>yarn.resourcemanager.amlauncher.thread-count</name># p# Q- X C' c% n; M- J) ^
<value>50</value>
* N' B1 I, \' D% `- M7 P </property>
' }7 {0 X1 X+ ^# E <property>, g5 b0 i$ o: n8 n8 S9 t8 w
<description>Retry times to connect with NM.</description>
1 s/ `/ q0 g. Z7 L! p5 D Q' P2 ` <name>yarn.resourcemanager.nodemanager-connect-retries</name># s3 H4 ^, a% d
<value>10</value>0 O( h$ n( P0 C% V/ p# Y' _6 t
</property>: J( q- H) ^- f0 b S, K0 v$ i
<property>
1 x# W4 B3 s3 T( ~; C <description>Timeout in milliseconds when YARN dispatcher tries to drain the8 d) S V) u' M9 Z1 l
events. Typically, this happens when service is stopping. e.g. RM drains+ ~1 z, e m2 T+ V, X! U: |
the ATS events dispatcher when stopping.- L- `. @9 ]- {. R# |! x
</description>: E% G2 y/ L i
<name>yarn.dispatcher.drain-events.timeout</name>
: |, F& ]* e1 E6 y' t% i C1 v& p D& o <value>300000</value>
, r1 J8 [3 E; N0 p- k </property>
. w3 S% A: s" r; _1 s8 {* v0 Y1 H" Z+ X <property>+ w2 _ y. g# X( x" T
<description>The expiry interval for application master reporting.</description>! [, l& z2 U: l) w
<name>yarn.am.liveness-monitor.expiry-interval-ms</name>
! ^0 k3 a; ^! v) A! R! ^ <value>600000</value>% h" i3 B9 I8 U" I% }- {/ s | n
</property>1 R: s' f- O" q; O9 B6 F
<property>
& e* ^) c9 s% S, ] <description>The Kerberos principal for the resource manager.</description>7 ^" @. U: a+ r, p8 f3 I3 u+ y
<name>yarn.resourcemanager.principal</name>+ ]& \; B3 _9 a* l5 x% H
</property>0 n3 B; ?4 \# r. c6 D8 h/ U2 l) _6 T
<property># D+ N- z5 j% A" q5 L- e6 c
<description>The address of the scheduler interface.</description>9 p6 _ U J1 {, e- ?( V8 d
<name>yarn.resourcemanager.scheduler.address</name>
( H6 Y* V6 I/ p, U" ^' L5 y <value>${yarn.resourcemanager.hostname}:8030</value>) f3 u6 `3 D( C* k7 U$ R
</property>
& r$ \3 l; Z' n m4 O5 a! { <property>- v% N" I% u/ F* P {! I' j) D
<description>Number of threads to handle scheduler interface.</description>
0 M! v+ I! A) }5 u5 o7 G, w <name>yarn.resourcemanager.scheduler.client.thread-count</name> } M% B& i- z, c/ U+ c
<value>50</value>
6 ?+ \- q3 V0 Q3 \' k4 q! ~ </property>5 ~0 m1 l3 F5 u# d
<property>
" x2 Y# E9 `+ p6 a' n <description>* m% N4 H# t4 X* Q8 _
Specify which handler will be used to process PlacementConstraints.
~% ?/ ?9 c: p0 q/ C i Acceptable values are: `placement-processor`, `scheduler` and `disabled`. u- c3 u8 a5 q! b' k/ u, j2 X
For a detailed explanation of these values, please refer to documentation.
# b* n7 q. Q; R2 e3 n' D </description>" M* c9 W! u: @1 P! D3 N5 [
<name>yarn.resourcemanager.placement-constraints.handler</name>
" Z/ e& r" v# x' V3 ^ <value>disabled</value>, p6 O2 v( ?/ e& C; H# X! k
</property>& ~0 A$ i- n2 m6 e$ u! I" W
<property>
. F1 o) j3 K% o <description>Number of times to retry placing of rejected SchedulingRequests</description>0 m! X" l+ g# n Y2 l9 A
<name>yarn.resourcemanager.placement-constraints.retry-attempts</name>+ j; f2 j& a0 L0 d6 I+ Y
<value>3</value>4 O3 w$ j6 T9 Q w5 c4 \
</property>. b2 K7 s. q3 M# H
<property>
' s7 x4 M2 l. C T: k, j, j2 ^ <description>Constraint Placement Algorithm to be used.</description>+ s1 K6 E7 g H; _; z3 l
<name>yarn.resourcemanager.placement-constraints.algorithm.class</name>& U1 w/ d5 g: Z1 ^4 M
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm</value>3 f' i' ]. H" k& J8 J# g
</property>
# T0 `1 C) k* @' |3 ]/ S <property>" Z: x; t( }1 Z2 E2 U( ]8 G" y# e
<description>Placement Algorithm Requests Iterator to be used.</description>+ F5 k8 U3 I1 G& k0 u( d4 K
<name>yarn.resourcemanager.placement-constraints.algorithm.iterator</name> [- _1 V, A' K& S' \( r
<value>SERIAL</value>& E% J( V, S9 {% S6 Z* Y) X% m
</property>
" p/ j* }' O3 H5 I8 A1 t$ K <property>+ N8 `8 @5 l7 [: U) G6 p! e
<description>Threadpool size for the Algorithm used for placement constraint processing.</description>
+ Z) R3 p0 y+ \8 T6 I <name>yarn.resourcemanager.placement-constraints.algorithm.pool-size</name>
0 J3 u: p' k% b <value>1</value>
7 n$ R2 ]4 _( i# q9 m- O7 P+ @! q </property>
3 [, J+ L. @4 T! @/ a9 W1 \ <property>
( P- u8 O# }+ `: R, Y7 M <description>Threadpool size for the Scheduler invocation phase of placement constraint processing.</description>
* D, p4 N; R' k0 j+ ]9 Z% f <name>yarn.resourcemanager.placement-constraints.scheduler.pool-size</name>
* v/ S% Z% J; Y% `; g <value>1</value>
4 I7 X( B0 p- D& u' b- w </property>" h$ Y" e6 ^2 i, b+ g
<property>
! s# y' ]) R6 f2 i <description>
/ H! u* N p; B# z' u; y Comma separated class names of ApplicationMasterServiceProcessor3 E6 h" j5 I& m
implementations. The processors will be applied in the order
2 n% B( J6 ? e, z" i8 B they are specified.
1 x( P Y# o! D9 r </description>2 M3 W4 l, g& w6 s
<name>yarn.resourcemanager.application-master-service.processors</name>
# b1 X E5 V3 A: j$ V <value></value>
E8 i; q. s) ]6 V" ]" u1 C) C* @ </property> v) v; n+ R0 ]' l
<property>1 r: M3 I4 u. C2 j" c0 m8 f
<description>) [ s) g) I5 s/ p
This configures the HTTP endpoint for YARN Daemons.The following
2 G- S% M- e0 B$ r5 Q2 U values are supported:% y: p2 a: O, M/ M" S" j8 W X
- HTTP_ONLY : Service is provided only on http
) L( m% g; A W4 y3 ~3 k' s2 F - HTTPS_ONLY : Service is provided only on https' C/ y8 ~2 T, v* }2 y1 F; j
</description>7 h" ^* d& C0 H* D+ ~. u6 d2 P
<name>yarn.http.policy</name>9 D; f# c/ r" }9 _" Q
<value>HTTP_ONLY</value>
1 }8 M# M2 v& @3 j" L( [9 O </property>
2 I8 F9 f1 Q, T5 O9 H5 Q <property>9 z# h* L- l: o; P1 v. V/ W
<description>$ w4 G& A) g0 y& I, n( k" B
The http address of the RM web application.$ f c2 D& }4 r" z: h5 I
If only a host is provided as the value,& w$ ` L0 C$ H$ f, J
the webapp will be served on a random port.8 i W) G: @# n T3 v
</description>
# w, f% C0 V: ~) M2 S' Z <name>yarn.resourcemanager.webapp.address</name>0 L5 |. v9 C9 d3 c4 X
<value>${yarn.resourcemanager.hostname}:8088</value>
0 T4 x( u7 ]: H </property>
5 G* o( s3 Q* A4 v* @' M <property>
( y) L' J1 b k4 T S: y, l <description>
" q( ?# V3 x/ y1 V The https address of the RM web application.+ [) \0 i% r3 ` u' M4 w3 J7 Z+ p
If only a host is provided as the value,
* y7 R) ^! R& i5 G ^, X the webapp will be served on a random port.
?4 T5 }8 A. G; C' n </description>, l* F$ L( ?" b- E- V/ o0 y
<name>yarn.resourcemanager.webapp.https.address</name>
) ]8 K* g' U$ c7 t! ^ <value>${yarn.resourcemanager.hostname}:8090</value>
: Y" [6 q6 W! O3 M/ z: w/ x </property>
# h) e) g6 C* j1 V) S: S) m <property>
( ]6 v. E0 ?8 C; V( I4 A: I <description>; I1 J) _# z, y, \% Z9 K
The Kerberos keytab file to be used for spnego filter for the RM web* e0 |' `8 s& k
interface.
9 o* F$ S& q' G6 R2 w; \ </description>- }+ Y7 j9 `4 W) L4 D7 j
<name>yarn.resourcemanager.webapp.spnego-keytab-file</name>* b: J7 r9 D, h P( k
<value></value>
- V5 q8 U2 q# c1 ]0 W6 [ </property> p2 G E+ D4 G
<property>
6 U5 S: o5 I$ l- i6 U, q9 w <description>
* \8 ^2 u& ^) V9 z$ h, I The Kerberos principal to be used for spnego filter for the RM web4 X7 |4 X+ w$ K: ?, r: D T
interface.
- _6 b! J# M& _* [ </description>
+ j3 E" @& @ n" a) P <name>yarn.resourcemanager.webapp.spnego-principal</name>
* A: S+ P U$ H+ B! s8 ]6 ^) q* {7 ~ <value></value>
) K6 `) k L6 g+ h e </property>
% b% A" ?7 F& o" q$ Z- @ <property>
: L8 V P: y: y4 |1 P5 m, `. e <description>
6 j+ g- Q% d6 @4 f4 Y; {" Y Add button to kill application in the RM Application view.
3 m7 v L) z0 R' @; W </description>
) ?; D9 A m1 e; J0 q7 F <name>yarn.resourcemanager.webapp.ui-actions.enabled</name>
+ P" ?9 D7 V% @/ ?! E <value>true</value>
! u* R$ M2 b. R# Q8 L( M' u* L </property>
6 F$ r$ U. C4 h; X$ @ <property>
' J( B0 C1 [3 @ a2 B2 I7 v$ Q" ` <description>To enable RM web ui2 application.</description>
2 Q" j1 P/ M( r. z. }: W" l# T <name>yarn.webapp.ui2.enable</name>
r- O5 d6 F$ y3 S& l5 j& e <value>false</value>
2 n I( M! ]/ ?4 z$ a% C </property>+ {7 ^! B2 T+ D9 _
<property>- ?- ?, P/ v9 r Q7 H
<description>: A5 o1 o5 _& R* k( G4 O
Explicitly provide WAR file path for ui2 if needed.
) F @/ ?1 U6 p9 s- e- Y+ v </description>: @' l9 p. b# B+ V, D. S' [1 }
<name>yarn.webapp.ui2.war-file-path</name>5 T6 | ^, F3 ?* u; l
<value></value>
6 B s; K2 z9 {8 ~$ L9 S </property>- M' n. s+ a( n; i2 \6 f
<property>3 ^/ o; l5 K0 M7 I/ m3 Q
<description>" Q3 P! M2 M) q
Enable services rest api on ResourceManager., I/ w1 q/ S: h+ P9 s9 \+ A! c
</description>5 W1 p! R( m+ S: ~2 ~. Y
<name>yarn.webapp.api-service.enable</name>" A& Y/ Y& Z2 O9 c5 t& S) }
<value>false</value>2 j8 ?1 r' v2 c" N. u. V
</property>
: _3 P: ~) `5 ] <property># y; R. S0 d9 [4 x
<name>yarn.resourcemanager.resource-tracker.address</name>+ V8 I/ y* g4 ^6 h3 u9 |+ Q% }
<value>${yarn.resourcemanager.hostname}:8031</value>
, }" u. f1 B$ _0 f$ I </property>" e) w1 I/ ]* K
<property>: y* w2 e+ D# H& X$ m
<description>Are acls enabled.</description>
- M! T8 `8 V7 U# j0 e0 g& G <name>yarn.acl.enable</name>
( C; I' Y2 n* v" X B$ U3 M' J <value>false</value>
* H- Y& a( Q" c0 L </property>
: v+ P* ^9 \& ~7 K <property>
4 K! N' _& s) q- `/ Z <description>Are reservation acls enabled.</description>4 T! K' q& h" }9 e8 y6 C9 {3 `
<name>yarn.acl.reservation-enable</name>6 U, x$ v9 [7 g, ?* h! M
<value>false</value>: ]; V2 G) S' }8 ]9 k
</property>- F: f; |3 K; w( F" e. d
<property>* i* b: G. {0 P% S @! A
<description>ACL of who can be admin of the YARN cluster.</description>
0 C+ l) k8 V" Q0 _7 c; Q% h y <name>yarn.admin.acl</name>
2 V4 l2 W4 ]7 Y( ^ o( [ <value>*</value>' {. E6 C& |2 B$ q
</property> p) O! R! L; t. [6 @1 r
<property>
+ G7 p" t8 T/ P <description>The address of the RM admin interface.</description>
' w# \5 e: r/ Q0 ^6 d <name>yarn.resourcemanager.admin.address</name>
3 Y; a! _+ h# g* M, s* I3 S; U <value>${yarn.resourcemanager.hostname}:8033</value>
" y2 c, z6 @+ {" Q </property>; s, F. m. y* U7 z* b5 s
<property>
, Z4 ]/ | D; {2 x7 a% j7 I7 L+ ` <description>Number of threads used to handle RM admin interface.</description>: Q0 G0 I; P' _) b* ^4 j- g% W
<name>yarn.resourcemanager.admin.client.thread-count</name>
4 F ?+ i, f; [, E$ P( D! e <value>1</value>
! \ a, g% [& d) L </property>
9 O! l! Z$ C! j* n$ E$ m* q: H2 v <property>
. Z- S6 J7 L5 {! Q' p <description>Maximum time to wait to establish connection to
/ }* J* x* Z0 C ResourceManager.</description>
- R8 _+ W" n: Z$ J+ \6 T <name>yarn.resourcemanager.connect.max-wait.ms</name>! u0 n8 H1 \: A! d8 e- N
<value>900000</value>
- ?* E8 t( q* l( q s </property>; v7 j) z* y! [4 q/ ~. K4 s; h
<property>
6 ^$ M2 n( Y; P, P0 H7 y# J# w <description>How often to try connecting to the
" e6 p4 x6 m4 n4 q8 I0 K% c* J ResourceManager.</description>
: J, j0 |; \3 o <name>yarn.resourcemanager.connect.retry-interval.ms</name>
( B% Y$ A/ p. G <value>30000</value>4 l) A* i$ p/ Z2 p, ?4 G X
</property>
' l# T0 p8 n/ a9 p( A <property>
7 n8 m$ C2 \4 p, _6 J% } S# `3 n <description>The maximum number of application attempts. It's a global3 a/ Y. ?0 B, w; w
setting for all application masters. Each application master can specify
( w. U/ Y- o& s+ H0 x8 \, y its individual maximum number of application attempts via the API, but the+ m( D' X' Z' a+ q0 m
individual number cannot be more than the global upper bound. If it is,
. O4 ?3 ^. b( S0 L% q; D& W the resourcemanager will override it. The default number is set to 2, to
) g0 k# g: \6 Y$ e0 t allow at least one retry for AM.</description>
% s2 }( n8 [1 k8 `6 p <name>yarn.resourcemanager.am.max-attempts</name>: z6 v2 \+ d2 i" D" O) J, \
<value>2</value>
" q9 Z" p4 h* E* T" _* A </property>
M% N5 y. n; T# a <property>
. k) ^5 N8 ]9 V8 ^+ e0 T1 |8 \! w <description>How often to check that containers are still alive. </description>
( w& ?" Z8 U/ u! F/ k0 ~ <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>$ h. M7 H: K) V* O+ ? O
<value>600000</value>( X- G4 ^0 b" d; T2 m0 y' V; J
</property>
, U4 [& G5 [( B& G7 g <property>
$ e7 E+ i" f# i/ H/ i# m3 \ <description>The keytab for the resource manager.</description>
i, `( s- l' r/ V <name>yarn.resourcemanager.keytab</name>
, x$ J5 N7 k6 q" z# j <value>/etc/krb5.keytab</value>
* e, D8 W2 ~# i! Q3 ]; t. R# r/ z </property>. C$ D5 i% _1 U' a
<property>9 Y, k) t" f8 R5 P9 n
<description>Flag to enable override of the default kerberos authentication
7 o5 \3 i& C' W" ~* W filter with the RM authentication filter to allow authentication using3 h; l7 u7 L, x" f w- ]
delegation tokens(fallback to kerberos if the tokens are missing). Only- ?8 U9 o$ X/ e7 J9 x2 x
applicable when the http authentication type is kerberos.</description>9 t/ y9 v8 H' K/ p: [
<name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
7 I% s( U4 U1 z$ l <value>true</value>3 R0 e) e, `9 d, d' I5 D% q
</property>! P4 \: K0 _! {9 ]$ G& H9 d+ r
<property>
$ t: ]7 R$ j' E9 u <description>Flag to enable cross-origin (CORS) support in the RM. This flag# b- }9 t. v+ |& Z( \0 y5 A! _
requires the CORS filter initializer to be added to the filter initializers6 f2 u% W' D4 B4 `1 }& D( z
list in core-site.xml.</description>" \$ k& J, V3 C, B" z
<name>yarn.resourcemanager.webapp.cross-origin.enabled</name>
+ P, T2 O6 [! u$ w <value>false</value>
" D% h6 b4 ~% B4 R8 F7 r# @ </property> R7 [. d4 Z2 ? Y6 _/ ^2 h0 ?
<property>
) U* H$ M( @1 Y <description>How long to wait until a node manager is considered dead.</description>! {; B2 i, P( {' l; C8 O
<name>yarn.nm.liveness-monitor.expiry-interval-ms</name>( |: e! i0 d, U' {
<value>600000</value>* x+ O2 E) o$ ?0 q5 I
</property>
! L* `; k. c- T9 H9 \- n/ l <property>
- e( P( u* u/ ]1 _ H; B <description>Path to file with nodes to include.</description>
" m1 X8 f" j- R n1 } <name>yarn.resourcemanager.nodes.include-path</name>
; X# r$ ~, A! K5 b3 x <value></value>
# ~+ g0 Q5 `5 k1 E. J) Z ?* ^ </property>$ h2 N: _! c) v
<property>" G+ x2 w1 Z4 z$ N
<description>Path to file with nodes to exclude.</description>
3 v k! {/ l7 y* _- v/ F- z4 ]4 C <name>yarn.resourcemanager.nodes.exclude-path</name>
0 ?5 Y% {6 a& d4 u <value></value>
* s: W1 y+ Q; s; w! t/ X- F </property>. p% Z9 I e/ S; ~& O
<property>, U! l% y. a& w Q
<description>The expiry interval for node IP caching. -1 disables the caching</description>) Z# m, O" {3 l/ o5 _8 W! E) ~( K6 w
<name>yarn.resourcemanager.node-ip-cache.expiry-interval-secs</name>
' A, S. T6 O: R4 g <value>-1</value>
6 M, K9 I6 H, V" @/ G2 [! R </property>; f* D; I. P* ?0 H- A K
<property>4 D6 [/ a; `4 L* x" I) q
<description>Number of threads to handle resource tracker calls.</description>
( i8 ?8 S5 {8 O% g <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>" n# v+ t: t1 L& } [
<value>50</value>3 `* w R+ c; J) U$ t o3 ?0 A2 K
</property>
8 q. C. b U9 N: W4 \, w <property>
" Z/ V: Y; ]- h2 K <description>The class to use as the resource scheduler.</description>) u' x+ N6 p* o4 U5 C. n
<name>yarn.resourcemanager.scheduler.class</name>& s( B+ R' N: f1 @+ [9 e
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>: F8 X2 W, y A0 F' X+ Y$ x- p! ]$ E
</property>: h5 C( ^$ A" h9 v3 i% f
<property>/ U9 ?% |4 f! e7 l" Q: o6 l$ m
<description>The minimum allocation for every container request at the RM
$ [) g5 T3 ]4 b. q7 T, Q, u6 w0 [ in MBs. Memory requests lower than this will be set to the value of this
0 u. f2 C/ ?9 q: y/ p property. Additionally, a node manager that is configured to have less memory- l% J; {) T0 S7 B0 j4 S
than this value will be shut down by the resource manager.</description>
/ i9 u8 A) h+ z* R2 N <name>yarn.scheduler.minimum-allocation-mb</name>
' H; S7 y: a$ o7 Y <value>1024</value>
9 p; H0 h' E, W1 I </property>9 s/ q, J5 B& E: j8 f4 l. B
<property>* _3 y* `( T u$ I$ x: v1 D
<description>The maximum allocation for every container request at the RM
3 n9 M! z% D8 E' o4 j N in MBs. Memory requests higher than this will throw an
6 O i% i- a( b& j7 E& I6 R InvalidResourceRequestException.</description>
, P& z5 n1 ^# @. }& S$ p% l! V <name>yarn.scheduler.maximum-allocation-mb</name>
) R5 P- g6 @: _3 U+ [4 l <value>8192</value>
' c( ^; s2 D+ n8 H# P </property>( ]. C, f$ j* [+ P: Q0 t
<property>
3 k) k& ~& ~) y/ A6 V: S. ]( r6 @ A <description>The minimum allocation for every container request at the RM
/ K+ w [2 n! q4 f+ p M1 w5 K in terms of virtual CPU cores. Requests lower than this will be set to the
% s( X$ E' ]) J+ q9 O value of this property. Additionally, a node manager that is configured to
# s* o' B5 Z+ I: M have fewer virtual cores than this value will be shut down by the resource
% K$ @" Q. Y7 A. i5 B# c* P manager.</description>" c; b6 x9 ~% [* X* k5 s
<name>yarn.scheduler.minimum-allocation-vcores</name>- j& ?9 z9 Y! C/ c. Y) v
<value>1</value>
. s8 i& o! Z2 c+ U0 e. p/ C8 i+ c </property>
6 e" L: G! L, h% q! Y$ b. f* D/ v <property>1 ~1 Z, S! {7 \/ ?* \
<description>The maximum allocation for every container request at the RM
5 e" t/ y6 y* f) z7 E* R in terms of virtual CPU cores. Requests higher than this will throw an/ C+ X: n0 G. P; k
InvalidResourceRequestException.</description>
5 b/ Z! M% l& b$ N+ ?& ^ <name>yarn.scheduler.maximum-allocation-vcores</name>
; D0 f& i8 O+ P+ j* R \8 ~ <value>4</value>
8 q* R- u1 Q( a4 Z1 C </property>
$ W# ?" f' E0 Q0 z! O+ \2 P <property>. U7 P! \3 u# N- r& x( W5 C4 Q# ?4 z
<description>
' Y, u$ ]. X, W2 Q8 d, ? Used by node labels. If set to true, the port should be included in the+ [+ y# y( Y6 Q+ n" ]$ \9 X0 e! q$ Y1 @
node name. Only usable if your scheduler supports node labels.8 }- d+ \9 s( L& Y" K
</description>" T$ c0 z" I) L1 S. p# W
<name>yarn.scheduler.include-port-in-node-name</name>% B* J+ J; K" H; U E
<value>false</value>3 d# T, u3 Q5 k& y
</property>
& D' v" e: x' t& N <property>. B8 p" T1 m6 n! ~: E0 B; m8 c
<description>Enable RM to recover state after starting. If true, then$ w3 m9 `8 J2 R/ M/ B( p7 ~( P
yarn.resourcemanager.store.class must be specified. </description>9 ^4 L( F [7 c; N R; Z3 G! F
<name>yarn.resourcemanager.recovery.enabled</name>
9 f. G) I- t0 V$ L2 ] r7 h <value>false</value>; c* D) q% S5 ?3 F/ R
</property>
1 m) Q0 c- J8 d <property>
3 z8 d0 O P# i <description>Should RM fail fast if it encounters any errors. By defalt, it
4 p3 S* o- H6 m L0 J& L1 A" Q points to ${yarn.fail-fast}. Errors include:
' K$ v" I0 K6 Q) s" R 1) exceptions when state-store write/read operations fails.
J5 r6 o x" h$ G3 J </description>
# Y5 a" O( k% s) x2 {/ E/ p <name>yarn.resourcemanager.fail-fast</name># a6 H* ~3 \, |5 O) l* k4 g
<value>${yarn.fail-fast}</value>
1 _' \. W2 }7 E7 [ </property>* Q+ ?; r0 Y8 G/ C3 k0 m1 ]
<property>
5 ]) ~6 g; x: Q$ A. x- E) n, B <description>Should YARN fail fast if it encounters any errors.8 g1 ^& J# t# U: d# T
This is a global config for all other components including RM,NM etc.: P( W: ~$ ?8 l4 _( k4 x3 S
If no value is set for component-specific config (e.g yarn.resourcemanager.fail-fast),9 G% B" y& D; n" a" [6 e
this value will be the default.6 K7 b6 C7 h# B0 s2 ~" O, p% o
</description>
! Q# O" ?% N! V- g <name>yarn.fail-fast</name>
7 [+ h, B0 H0 c7 b2 G& a <value>false</value>
' b6 B* ~4 ]8 Y) A% H$ p! e </property>
( S- }. [* c1 N6 v- f9 F0 @ <property>5 J& t1 A4 ^- N1 Z. D4 v
<description>Enable RM work preserving recovery. This configuration is private
. ~" }- v. S$ A' U! X5 l to YARN for experimenting the feature.
4 L2 o" N) u F4 p8 ^# b </description>) s5 {% q7 K. q( s3 l7 D& p
<name>yarn.resourcemanager.work-preserving-recovery.enabled</name>
* b- }3 k, r3 B- D6 z0 ? C <value>true</value>4 n" b" E: k. [% |+ F
</property>! V# P% U; ~+ k X7 O
<property>' s2 \0 q$ L+ u3 |! z* j. d
<description>Set the amount of time RM waits before allocating new( @# P' F4 u& \$ g+ o$ t- _, T
containers on work-preserving-recovery. Such wait period gives RM a chance
. X' h: g% |. u- H) c' G( @ to settle down resyncing with NMs in the cluster on recovery, before assigning' |) t" t; l5 [9 Q
new containers to applications.' V6 {% u0 e( ?: F9 P
</description>
7 x2 R; }9 `# [* @: @5 P# H <name>yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms</name>
: q& A1 l3 W. N/ ~1 f4 M0 I$ v <value>10000</value>" T" j1 S/ L2 K7 z) J
</property>
; E/ x: I3 z: C8 c <property>
4 C) T# S6 N4 e <description>The class to use as the persistent store./ [+ m( s8 V' k) U9 B' h
If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
6 w+ t2 X- I2 C' c is used, the store is implicitly fenced; meaning a single ResourceManager
" R8 h+ j" D4 r0 { is able to use the store at any point in time. More details on this* h7 P6 R. S' ]
implicit fencing, along with setting up appropriate ACLs is discussed4 l* H, x0 P5 M1 T: I
under yarn.resourcemanager.zk-state-store.root-node.acl.
. }$ D: t4 H$ u3 _/ W </description>' C+ |8 U' t! `+ s
<name>yarn.resourcemanager.store.class</name>
/ i' E8 M# A& ], y <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
3 g2 E! ?" S" z; r E </property>8 E0 ~8 o% B8 F3 I0 b
<property>
& G: b3 m5 d& S+ v0 g <description>When automatic failover is enabled, number of zookeeper- ~; |& C& d) h& a U
operation retry times in ActiveStandbyElector</description>
, L. I% T, r8 | W1 e5 x$ Q <name>yarn.resourcemanager.ha.failover-controller.active-standby-elector.zk.retries</name>
' g7 K2 i- M _3 z- o8 D0 d <!--<value>3</value>-->3 p/ v5 v6 j7 Y9 S$ X9 E& Y$ u+ ?
</property>
9 K) S% ]$ m$ c! u l+ W* E/ \ <property>
. w7 w' M" ^0 N9 ^$ E4 `/ ? <description>The maximum number of completed applications RM state1 p/ x5 R+ o2 X5 F; i, h' ^
store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}.8 |! R0 B, y; Y: V8 @: r. O8 k
By default, it equals to ${yarn.resourcemanager.max-completed-applications}.: M7 d O8 s$ t. K ^8 h2 Q! ]2 C
This ensures that the applications kept in the state store are consistent with. b; _. u6 r3 b7 ]( Y
the applications remembered in RM memory.
8 b2 T9 b5 m, P! o- t9 D4 A, W0 N! [, v Any values larger than ${yarn.resourcemanager.max-completed-applications} will
1 K5 ^' t8 \. o, [+ Q O6 D, m be reset to ${yarn.resourcemanager.max-completed-applications}.5 X* g5 q) h& H- \& I2 D+ w) M
Note that this value impacts the RM recovery performance. Typically,
( }7 L' z8 } p- ^4 I7 l8 r a smaller value indicates better performance on RM recovery.' A0 z. w4 b$ a8 D' c I
</description>
/ h' w. T: [( B P' z2 H <name>yarn.resourcemanager.state-store.max-completed-applications</name>$ x- e# b. }9 v6 I. X$ q
<value>${yarn.resourcemanager.max-completed-applications}</value>
3 ~6 y% l7 p1 _ </property>
9 \# ~+ j' G4 Y6 k <property>
: [, w% N0 I$ h5 b6 M. z <description>Full path of the ZooKeeper znode where RM state will be
; O+ J; J% ^ L8 f stored. This must be supplied when using
/ s( R+ W+ X) ~$ x$ Q org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
# `2 J7 M7 I2 {0 r/ j. v) J as the value for yarn.resourcemanager.store.class</description># |& A" _& [7 @6 h
<name>yarn.resourcemanager.zk-state-store.parent-path</name>
2 ^& a' C- p& ]$ j: x* U <value>/rmstore</value>
6 r! o: l/ Y# W# f u </property>- r* f ~0 m9 n) V8 F. q
<property>/ l( O# C% a( x+ w7 I& M
<description>9 e( Z. Q' M; }( g
ACLs to be used for the root znode when using ZKRMStateStore in an HA
8 T. J$ ~& U. p3 [5 c6 Z scenario for fencing.- K/ D5 D& v6 S \+ ]' S
ZKRMStateStore supports implicit fencing to allow a single
3 D3 x j3 H, v9 U% w4 }" K ResourceManager write-access to the store. For fencing, the
- e' h/ Z$ K0 _: e ResourceManagers in the cluster share read-write-admin privileges on the0 ~. n( \+ `& B+ r; d
root node, but the Active ResourceManager claims exclusive create-delete
4 S. \9 Q) v. U5 c6 q& N" G9 h5 G permissions.; a6 Q* u2 v# d8 \+ w1 \: S# q
By default, when this property is not set, we use the ACLs from3 ^# J, E3 _6 ]5 c
yarn.resourcemanager.zk-acl for shared admin access and
7 L, w# S& ?- v% u$ E rm-address:random-number for username-based exclusive create-delete
, D/ n- X& W+ A+ i: e7 H* Q access.
; C$ x) C1 |) Y* x/ u This property allows users to set ACLs of their choice instead of using) b9 J% R! A; L; W" ]( g
the default mechanism. For fencing to work, the ACLs should be( ^8 l. M! S, t" m; t: l5 q
carefully set differently on each ResourceManger such that all the- z- I/ I8 ? h6 |
ResourceManagers have shared admin access and the Active ResourceManger7 k8 f: y" x8 P
takes over (exclusively) the create-delete access.2 ?$ P6 V$ a, p/ T) x
</description>
; r# X0 o* e% F' ? <name>yarn.resourcemanager.zk-state-store.root-node.acl</name>1 O+ ]4 W% v) j# t" s) q
</property>1 b6 ^" M/ L0 E/ @! S
<property>* N$ M. M' C$ P" p+ J8 w# a9 S' w
<description>URI pointing to the location of the FileSystem path where, C* W0 I' a( M( z; f
RM state will be stored. This must be supplied when using0 h; ^2 b. H) w4 O) t, I" e
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
5 M i: @5 O) Q, ~5 S0 ? as the value for yarn.resourcemanager.store.class</description>
, q7 @" w0 Z* T7 I# z <name>yarn.resourcemanager.fs.state-store.uri</name>
1 l7 y* b9 ?. o8 b( ? [% g <value>${hadoop.tmp.dir}/yarn/system/rmstore</value>( K6 p: s0 W C0 V% b8 L. e
<!--value>hdfs://localhost:9000/rmstore</value-->
6 R U; ?& n9 y- S </property>
: o! z- A8 \$ M. d- f, ?0 V. q# X <property>& N" t" V& J6 k4 V- [/ y# _; D
<description>the number of retries to recover from IOException in- i0 z: j$ Y5 R `! W4 R
FileSystemRMStateStore.2 A( w' D, L, I, w4 ~" L
</description>8 E8 v$ y3 ?7 t9 p1 }9 b" }
<name>yarn.resourcemanager.fs.state-store.num-retries</name>
0 w/ B5 ^ o; j3 ]- A* A+ f- U: ^9 B <value>0</value>
7 q! O' l7 V+ o& N) G* ?4 l- O </property>% O2 s, P2 C) _/ T0 ~# R
<property>; \0 x6 V4 b, h3 S V
<description>Retry interval in milliseconds in FileSystemRMStateStore.5 [' q6 Z( V3 B7 L# C
</description>
* v7 [9 [/ Q2 x! E5 D6 d <name>yarn.resourcemanager.fs.state-store.retry-interval-ms</name>$ n5 e( {8 E4 m) S0 M
<value>1000</value>
" p% ^! V" t: k3 t4 ` </property>$ Z, _& r9 A0 j) D/ {- Q
<property>5 I0 o0 |9 `, W
<description>Local path where the RM state will be stored when using
3 s5 u* f! L& X1 o# {1 [' e* c org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore9 @" t4 p" P! E7 F; E! L! |
as the value for yarn.resourcemanager.store.class</description>! y: W* ]( E, G# h- A$ q
<name>yarn.resourcemanager.leveldb-state-store.path</name>6 D9 _7 ]3 V C0 s
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>; E& q( J, Y K6 ^8 C
</property>* o/ R, b* H4 ^ E
<property>
3 I) ~$ I `. y, m& ? <description>The time in seconds between full compactions of the leveldb# z+ B: F4 \) A& n
database. Setting the interval to zero disables the full compaction9 u% ]( M$ ?- c
cycles.</description>, Y1 e# d8 F7 f
<name>yarn.resourcemanager.leveldb-state-store.compaction-interval-secs</name>, e( n% ^5 X7 x& g. r( s+ `
<value>3600</value>
- p( l- z# {" e5 r </property>. z# a9 s2 y! x# |5 [1 l
<property>
. q3 a' e8 F9 W6 Z <description>Enable RM high-availability. When enabled,
& Z5 j5 h% Q/ P" F$ ?8 }! { (1) The RM starts in the Standby mode by default, and transitions to" Q1 J2 @# K& ?
the Active mode when prompted to.- T4 }7 a9 V* b1 b6 p7 o
(2) The nodes in the RM ensemble are listed in, g( t' Q c7 F
yarn.resourcemanager.ha.rm-ids1 `: y, N! F6 a
(3) The id of each RM either comes from yarn.resourcemanager.ha.id8 Y$ H& t' Q6 _+ A+ R( t
if yarn.resourcemanager.ha.id is explicitly specified or can be' V9 e4 k# W$ ^
figured out by matching yarn.resourcemanager.address.{id} with local address9 j F$ k0 F' @, l
(4) The actual physical addresses come from the configs of the pattern; P4 N& B; F3 `2 f% ?
- {rpc-config}.{id}</description>
! z) A3 y$ u4 [( o( v* }" z <name>yarn.resourcemanager.ha.enabled</name>. w: o' [' B8 Q+ I# C
<value>false</value>! w9 L) i7 T# a2 z9 L
</property>
8 ]# @6 D( B: I. k- |' g <property>( [1 }) @' g: t) S! u, h
<description>Enable automatic failover.
. b$ G( v4 I0 |: `( V( S% Z By default, it is enabled only when HA is enabled</description>
9 h; F( }" o7 o4 d <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>& f" q/ ?7 Y" _# `' U# X
<value>true</value>
~4 _& T3 Q {4 n( U9 q$ J/ p </property>( M1 m0 }6 H; @* d3 I( {& M/ r
<property>6 \# F/ o5 J$ W' r
<description>Enable embedded automatic failover.! {4 r" m& l0 k6 {& u
By default, it is enabled only when HA is enabled.
4 ]% y' G3 U- ~# w5 R' r The embedded elector relies on the RM state store to handle fencing,$ [4 W6 x0 d' T( w' }
and is primarily intended to be used in conjunction with ZKRMStateStore.- {+ U( Y, o2 V5 r d
</description>
" e1 }4 u+ C7 E* O% m <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>! H8 m+ u- W; k8 o1 r
<value>true</value>
0 P+ R) [( J6 y </property>* x2 \8 y& q* }* k& m# C
<property>
9 F; h) c2 P3 l M2 } <description>The base znode path to use for storing leader information,( V$ L4 A0 u1 W# U. b5 W0 v* W
when using ZooKeeper based leader election.</description>
* O/ [& M/ m+ F6 `3 ?0 s; @, N <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>- h8 s9 o' e+ r: Z1 y; f
<value>/yarn-leader-election</value>3 a) z5 Z1 d9 b
</property>
' S5 n, y3 ^) ?9 R: i <property>
5 ~" g' p- o6 B, z; c2 q1 j <description>Index at which last section of application id (with each section- H# n$ L; u8 X
separated by _ in application id) will be split so that application znode2 W4 C6 F8 c% x
stored in zookeeper RM state store will be stored as two different znodes
+ s# `9 S% s$ a# X: C (parent-child). Split is done from the end.
% ]* t0 V4 h% q$ v' u For instance, with no split, appid znode will be of the form7 T, M# ?, R9 A8 O" z9 Z
application_1352994193343_0001. If the value of this config is 1, the
& G; W' T/ z8 k- m! e1 m! U appid znode will be broken into two parts application_1352994193343_000$ }; b$ ]/ c3 Z4 w
and 1 respectively with former being the parent node./ F2 K \4 T) ~
application_1352994193343_0002 will then be stored as 2 under the parent* d1 u/ z- c) R& R( b
node application_1352994193343_000. This config can take values from 0 to 4.5 F$ N* W0 Q9 s8 M2 H
0 means there will be no split. If configuration value is outside this
7 J2 f& ~# u" b/ L range, it will be treated as config value of 0(i.e. no split). A value2 g) i$ M; v4 Z( B& k( Z
larger than 0 (up to 4) should be configured if you are storing a large number
( c/ _1 Z& x4 N8 y2 b4 _3 q4 y' y' u of apps in ZK based RM state store and state store operations are failing due to9 ?+ H& `6 x$ V0 ?( u
LenError in Zookeeper.</description>
6 I l' A7 b/ }+ c <name>yarn.resourcemanager.zk-appid-node.split-index</name>
: u& z6 z j) j- T! K! D* y <value>0</value>
5 A% r; o2 @0 ^8 f6 Q' r6 B6 a </property>, h) e; L4 N6 b. J; I
<property>
2 ]8 t, b/ y0 {' x- [ <description>Index at which the RM Delegation Token ids will be split so
/ [. G) E* C) M& c e$ D- i that the delegation token znodes stored in the zookeeper RM state store' X8 @2 g- P8 O% r$ H, Y2 u7 a
will be stored as two different znodes (parent-child). The split is done
) h6 V& q; v: \! m. I" P from the end. For instance, with no split, a delegation token znode will
( D3 C/ \$ z2 Z ]5 R0 D be of the form RMDelegationToken_123456789. If the value of this config is" u4 S1 a; A% e6 v1 x/ g3 `, s
1, the delegation token znode will be broken into two parts: V2 f& w' t# q
RMDelegationToken_12345678 and 9 respectively with former being the parent
* D+ j6 j5 U8 k3 B node. This config can take values from 0 to 4. 0 means there will be no
! L$ C4 j0 h0 y% b% ?0 c) g split. If the value is outside this range, it will be treated as 0 (i.e.
( V8 R0 E) \7 m# S0 p0 [ no split). A value larger than 0 (up to 4) should be configured if you are
7 `4 u% d+ N, ~ Q8 ^2 y running a large number of applications, with long-lived delegation tokens! G0 Y1 O9 V6 P' P# w0 K& J) D
and state store operations (e.g. failover) are failing due to LenError in
. F- a& ^# Y7 l2 n } s9 `4 ~ Zookeeper.</description>$ }' K9 F& ?$ @2 m( p
<name>yarn.resourcemanager.zk-delegation-token-node.split-index</name>$ f s) o( M" l$ L, F
<value>0</value>2 {8 D$ M: G9 R- h+ {. L
</property> C; f, f- j: ]/ n! ?+ Y+ H$ M, ~
<property>% Q3 L7 c( r. C! w; x# _. f( a: t" q/ z
<description>Specifies the maximum size of the data that can be stored6 t7 J2 p0 k# l% s6 t) c4 w
in a znode. Value should be same or less than jute.maxbuffer configured
+ J% A4 ]) W1 r9 ` in zookeeper. Default value configured is 1MB.</description>; ?+ Y& Z( f, [7 d& P
<name>yarn.resourcemanager.zk-max-znode-size.bytes</name>
6 w% Y1 D4 G9 ?4 m' U <value>1048576</value>1 |0 _4 V D/ c. e3 m) m
</property>
! F. A* s# ?5 S: a& {& Q <property>! _+ w- r' U& l
<description>Name of the cluster. In a HA setting,
' i( I$ r2 t- E [ this is used to ensure the RM participates in leader
8 x! h3 C% z% |& B8 B8 W election for this cluster and ensures it does not affect3 p& w: T# k' O* b# {& h& u
other clusters</description># ?) ~+ U$ r! E$ h, m, |# @1 }
<name>yarn.resourcemanager.cluster-id</name>
6 C3 d( z: Z4 m) q' J" G! g <!--value>yarn-cluster</value-->
2 |( P& T: t" O% L </property>' D V/ r' a* g
<property>
4 H6 k' q0 V5 O) [% Y6 Z x' T# C <description>The list of RM nodes in the cluster when HA is
" h, R2 c* J! R% t3 k enabled. See description of yarn.resourcemanager.ha
7 O2 y4 F3 Q: T+ x7 v- j3 ] .enabled for full details on how this is used.</description>( s5 G" c' h- l, d' X7 T* _1 n
<name>yarn.resourcemanager.ha.rm-ids</name>
+ _8 I6 v, D0 s3 n; R4 d <!--value>rm1,rm2</value-->
! p& e. N; h# m! {0 s/ ~& {" x </property>
+ O( V3 x6 {" n4 E; f" B8 u' z <property>
1 P3 l! s& } B! j7 k F <description>The id (string) of the current RM. When HA is enabled, this
+ G0 P% `3 P6 H$ s" q" q is an optional config. The id of current RM can be set by explicitly6 w$ W4 u4 ?; s4 h# u
specifying yarn.resourcemanager.ha.id or figured out by matching0 N; i' ^3 `; ?$ z
yarn.resourcemanager.address.{id} with local address
" B& a- j# B. P$ t( X See description of yarn.resourcemanager.ha.enabled
) X v+ B( k1 |) a+ h- n for full details on how this is used.</description>
, S: d) f7 t1 o4 g$ }3 R) } <name>yarn.resourcemanager.ha.id</name>
* o w4 K- M2 G* t" T <!--value>rm1</value-->2 e7 G* E' ^5 P; O% G" W
</property>
1 p2 x' D2 m7 V: i! j <property>
1 u8 f" k6 i; } <description>When HA is enabled, the class to be used by Clients, AMs and
( s; {7 c" @# d: T3 ^5 r0 a NMs to failover to the Active RM. It should extend7 x0 V* v: z6 n# p8 I
org.apache.hadoop.yarn.client.RMFailoverProxyProvider</description>3 M. {6 F% n% _; S9 v) }5 E
<name>yarn.client.failover-proxy-provider</name>1 {$ d( h, C" l* n% K- `
<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
. `$ N. m0 N5 N" x) ]2 b </property>) T* A) j; w# z$ Z' H) v) H
<property>
1 F6 o* T/ x! z& m6 ~% S- { <description>When HA is enabled, the max number of times1 K, H+ h+ z$ y8 G) O
FailoverProxyProvider should attempt failover. When set,
9 P5 T( R. o y# r this overrides the yarn.resourcemanager.connect.max-wait.ms. When
+ _$ b( D; m: ?7 } not set, this is inferred from& z) ]4 [8 N7 p* S$ N1 }& ?
yarn.resourcemanager.connect.max-wait.ms.</description>
( V' s3 h8 U c; A <name>yarn.client.failover-max-attempts</name>
, r: H& o. m& i7 m <!--value>15</value-->; n! W5 z; a7 O0 F4 c- m5 s
</property>, y5 K- Z: u7 y
<property>& G' g6 A) N+ T- t3 B3 A8 x6 j
<description>When HA is enabled, the sleep base (in milliseconds) to be) t* ?8 ^% \! e9 f+ ^$ B# z/ L
used for calculating the exponential delay between failovers. When set,
8 Q2 b7 a7 Z/ P, G this overrides the yarn.resourcemanager.connect.* settings. When
h2 ~" Q6 g2 Y# ~ not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.- K; @6 d+ d8 ~. ^1 \
</description>
( T1 D, v. d# s% A& \, O/ L <name>yarn.client.failover-sleep-base-ms</name>
6 h$ @% n! ?. `4 |9 x# ? <!--value>500</value-->( u. M) R& I6 J2 P2 S+ m% ?
</property>" s3 k9 `" I, i
<property>
) _8 o1 A# B. Y5 l+ [ <description>When HA is enabled, the maximum sleep time (in milliseconds)7 m8 t [+ f7 W$ f4 f. K/ K. j
between failovers. When set, this overrides the
/ [' S4 f4 v# ]# _) C) O* X yarn.resourcemanager.connect.* settings. When not set,
# K; t A" j! d% x% A yarn.resourcemanager.connect.retry-interval.ms is used instead.</description> F% l0 M6 O3 u
<name>yarn.client.failover-sleep-max-ms</name>. r4 d3 e! A: p9 v: q& d
<!--value>15000</value-->
! Z f: f2 T6 Z& S4 v2 ~# k </property>
9 \% x- L" }$ i7 F. _6 U2 H1 L <property>
# K% l3 `: h7 M+ F, j/ j) [" R <description>When HA is enabled, the number of retries per
! u7 {% r5 x' X attempt to connect to a ResourceManager. In other words,' s w( W" T+ y8 L$ {- e' Y3 [
it is the ipc.client.connect.max.retries to be used during
. _. V5 s+ k- s. u) F4 D2 M failover attempts</description># N8 m* n, W1 D5 x& E1 A
<name>yarn.client.failover-retries</name>9 C. z1 o7 W! J* E) O" j
<value>0</value>, @) U* w* [+ a9 H/ o
</property> b8 S& _7 {& ?: E
<property>
' A q! Y$ I( j/ A( S' E0 j1 g <description>When HA is enabled, the number of retries per
+ {1 v" q! R$ c/ @ attempt to connect to a ResourceManager on socket timeouts. In other
$ h' Q! k# W) j l% r words, it is the ipc.client.connect.max.retries.on.timeouts to be used
2 u# G: G6 N. {8 p) ?% s* T( X3 Q' K during failover attempts</description>
1 m6 u' z- V7 y3 q8 }/ b <name>yarn.client.failover-retries-on-socket-timeouts</name>
% Y6 M# q$ a& U- {% Z. f <value>0</value>4 X+ A6 \ u o0 H7 S; T
</property>
! {" Y4 C- k# t" W <property>& `! e; W; E: Z. @- S6 q
<description>The maximum number of completed applications RM keeps. </description>
5 U2 n+ D2 w A$ S" ~: u <name>yarn.resourcemanager.max-completed-applications</name>
; M* G. T( U2 l; `1 Y6 @ <value>1000</value>9 |5 `) H4 G- N
</property>$ W8 o5 R6 M% P- |4 Y
<property>, y# z# u1 w( G
<description>Interval at which the delayed token removal thread runs</description>, t4 Y' X1 A9 U& e) A
<name>yarn.resourcemanager.delayed.delegation-token.removal-interval-ms</name>8 H; c* E! c+ x
<value>30000</value>7 M' I1 q% J7 W. Y; ^
</property>4 }0 i$ V' v8 |* ` u" v
<property>
0 v* n* [$ V) c1 Y7 s3 M <description>Maximum size in bytes for configurations that can be provided
* Z7 ^3 ~& F# K5 R Z( ^0 t by application to RM for delegation token renewal.9 F4 f+ I2 r0 C3 ?6 }0 s
By experiment, it's roughly 128 bytes per key-value pair." S$ X# O6 Z+ W4 R. J
The default value 12800 allows roughly 100 configs, may be less.
' l' q3 R' C: ]7 U </description>: H7 h: W7 o' ]2 G7 y3 E
<name>yarn.resourcemanager.delegation-token.max-conf-size-bytes</name>
8 o7 T$ s( B4 l @% A; z <value>12800</value>6 u! X1 a/ }, V
</property>( o. D' P5 J+ Z- Q7 b. w
<property>2 U2 v' o7 l, E3 F- Z, G$ S$ `
<description>If true, ResourceManager will have proxy-user privileges.
: Y1 T0 Y) Q( q# y" r* j3 n/ Z7 i Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to% V& Z4 x7 q& @+ Q
do localization and log-aggregation on behalf of the user. If this is set to true,: [- N# _, T2 _5 {7 m
ResourceManager is able to request new hdfs delegation tokens on behalf of
' s" T6 Y5 c% N# N6 G4 F, ^' J the user. This is needed by long-running-service, because the hdfs tokens% V: B8 X: t" p/ o
will eventually expire and YARN requires new valid tokens to do localization7 W2 ~; U# D6 {1 |1 ~
and log-aggregation. Note that to enable this use case, the corresponding
! R4 x, H4 P2 S: H HDFS NameNode has to configure ResourceManager as the proxy-user so that ] t% O2 Q. g
ResourceManager can itself ask for new tokens on behalf of the user when
$ u0 `& A+ X% a. i tokens are past their max-life-time.</description>! U9 G6 L# h/ O/ H- B' _% `
<name>yarn.resourcemanager.proxy-user-privileges.enabled</name>
7 V% ?7 ^. y; {- |+ n3 }, { <value>false</value>
* _/ Q+ C8 c( }' c- o </property>; d# t9 A) e E' Z1 t
<property>; o, e) w2 a; U: n j
<description>Interval for the roll over for the master key used to generate
' q, T; |) E4 X5 o( u" V. C5 q application tokens
' h# f6 @5 y+ x/ Y& O9 ~# V) A </description>6 {+ I3 i" D2 T- a9 ^3 N8 a7 Y
<name>yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs</name>
9 i! l# h9 a5 d2 S <value>86400</value>
) ^: m' a. T- }+ K </property>
! Y# E% N! o! q% [8 D, F <property>2 V+ j6 y' E' n: E2 `$ _
<description>Interval for the roll over for the master key used to generate
^4 n+ l% E, O container tokens. It is expected to be much greater than& u; k4 j/ u3 q1 L9 ?4 p+ A8 }
yarn.nm.liveness-monitor.expiry-interval-ms and
& f7 P. p1 J1 Y yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the+ j$ m6 Q& O* V' b. d* _( `5 S
behavior is undefined.
) X/ |1 F! ^. m v$ J: ?- @ </description>% I9 Q4 g1 m0 Y
<name>yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs</name>9 ]# E3 Q9 ]; M6 r& C8 b
<value>86400</value>) }7 ~. {4 {8 Y% Z- c5 p% C
</property>
1 `% g0 {0 s' a9 d4 V <property>$ H( O$ H3 t5 G3 o2 a# V7 m; U" I
<description>The heart-beat interval in milliseconds for every NodeManager in the cluster.</description>
5 n6 o" {" T# @2 K <name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>- [, [* q# f( w& y( H7 J! e2 ]
<value>1000</value>* f8 o; l" m! k
</property>1 `& X) P3 \" `1 {
<property>0 Z; y" r( _4 b' z X
<description>The minimum allowed version of a connecting nodemanager. The valid values are
5 E1 {/ t+ X, n. u0 ]* {" Q; `0 ` NONE (no version checking), EqualToRM (the nodemanager's version is equal to
5 M$ Y5 X7 M; h6 j6 e# ?+ X or greater than the RM version), or a Version String.</description>
' k) \. ]( w/ U <name>yarn.resourcemanager.nodemanager.minimum.version</name>
1 F+ i. y) s. A4 D <value>NONE</value>
& ?, {) ^' X0 w* d. ] </property>- D7 C$ V) A* ~5 F B7 m
<property>; C6 i* I7 N5 g/ n" q v0 n
<description>Enable a set of periodic monitors (specified in/ r3 I4 [" v& i; T" {
yarn.resourcemanager.scheduler.monitor.policies) that affect the
7 w: l% g5 w, O! [ scheduler.</description>- V. C3 w! d9 E" Z- r, T
<name>yarn.resourcemanager.scheduler.monitor.enable</name>% ~ r4 s' E d+ R: [/ M
<value>false</value>
( w H3 {* @5 D q, _: F6 { </property>% m! H$ G" |$ Y% b0 L3 C
<property>
9 `1 z+ L; k. M3 B3 {" f <description>The list of SchedulingEditPolicy classes that interact with5 y+ r( Z0 \2 K. g, ^
the scheduler. A particular module may be incompatible with the
8 i. n! j+ n9 a1 O6 b scheduler, other policies, or a configuration of either.</description>
( e7 @ `1 N$ A6 E <name>yarn.resourcemanager.scheduler.monitor.policies</name>
" _$ u) J: X5 L! N <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value>' Y+ @$ r6 l. M e: s! q( q8 g
</property># o' `7 y1 Q, h9 {0 H3 u
<property>
) E( n; O# S8 T. ]# y <description>The class to use as the configuration provider.+ Y/ I0 ?0 r+ m# c
If org.apache.hadoop.yarn.LocalConfigurationProvider is used,1 j" t3 e6 {7 ?9 v
the local configuration will be loaded.
' v! F2 D- L1 ?# k# y* {1 Q; b If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used,
' G7 M! g D3 [8 I$ [7 ] the configuration which will be loaded should be uploaded to remote File system first.1 C: Y3 @/ W2 ? n
</description>( C& m4 _. x h9 a4 G
<name>yarn.resourcemanager.configuration.provider-class</name>8 T3 k3 y( `6 e, G9 X) S0 J
<value>org.apache.hadoop.yarn.LocalConfigurationProvider</value>
) i& ? d0 g9 J! w. W0 V8 g <!-- <value>org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider</value> -->
& g, a; t7 Y B9 t! m/ F </property>
9 d h. u# V$ H L <property>
: {7 ]" D( m( L) M <description>7 o: c5 ^% }2 D' L' d
The value specifies the file system (e.g. HDFS) path where ResourceManager
: ?6 k2 p+ t2 a loads configuration if yarn.resourcemanager.configuration.provider-class
3 F, E. Y I4 | is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider.
7 n6 g' E- T( q* J! ? </description>
* R4 D+ {) o; H2 g) { <name>yarn.resourcemanager.configuration.file-system-based-store</name>2 }& [8 \7 h% r
<value>/yarn/conf</value>$ R! C/ c: d+ w9 }' y2 _8 ?7 H; v
</property>" s! f! E9 f4 Q: t7 F, v) E
<property>3 A' Z& g# t+ Y( k8 W. C
<description>The setting that controls whether yarn system metrics is; d7 r' l; H$ K7 |5 I
published to the Timeline server (version one) or not, by RM.
6 k4 M2 y+ @1 v# o, d5 } This configuration is now deprecated in favor of N1 T. B7 M+ v9 D5 X' B
yarn.system-metrics-publisher.enabled.</description>9 A' \1 }1 n' V
<name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
0 M# x$ g. W9 I' u% Q <value>false</value>6 s; ]7 u! E2 `: B6 g
</property>
" g% o: H# B; O* p) V5 E <property>* ^$ [" {- h5 |$ l
<description>The setting that controls whether yarn system metrics is4 |) H9 o4 }/ B( w1 L
published on the Timeline service or not by RM And NM.</description>- R, f& A' J* S4 S& x7 u: Q
<name>yarn.system-metrics-publisher.enabled</name>
7 m/ k) N Q0 y6 \3 R8 @+ G& O. o <value>false</value>4 M9 b( s# f+ z: n8 Y9 \, Q' D
</property>4 i+ ?, U1 k, F0 \( o. s
<property>
4 w, |9 ]2 N' C9 t <description>The setting that controls whether yarn container events are' t: e( g( R) Y6 J3 { g% J! S2 }
published to the timeline service or not by RM. This configuration setting
5 i5 d' r# k: @& P( Y+ c. v: h is for ATS V2.</description>
1 I& N+ ?: U. f$ }& _% S! \ <name>yarn.rm.system-metrics-publisher.emit-container-events</name> u6 f1 u+ U& n0 Z2 {
<value>false</value>
0 {8 g, x7 |: f0 N1 S </property>
4 W& Y! q. t- z- o( l9 h <property>9 U3 A5 E, ?. W0 K+ N- u
<description>Number of worker threads that send the yarn system metrics# e9 r2 h# n' `. D
data.</description>& j! R% @( K/ A/ ~- s7 M
<name>yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size</name>
. ~; Y1 L5 [( H' }, w2 d <value>10</value>
* {. Z' w4 Z I; U9 P# ?, _/ N </property>
, \! x! H V" o7 v) y/ L* S. i1 f <property>
% w b7 s* ~! j; _ <description>Number of diagnostics/failure messages can be saved in RM for
$ ]2 u! P' A* p4 v$ R0 ` log aggregation. It also defines the number of diagnostics/failure
: w; c' v' ?: @7 _; f messages can be shown in log aggregation web ui.</description>
- Y& r/ J. {4 R) s3 X+ G, @, d <name>yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory</name>- T, o8 `4 ]/ C2 \
<value>10</value>3 J. {0 L$ `- w7 x, j$ I
</property>. [3 A" w) T7 V, h' ^
<!-- Node Manager Configs -->* K- u2 w' S! x% a# v
<property>" V" D, z1 q* v/ U- \
<description>3 ]$ ]- t! d& M0 m9 ]; \
RM DelegationTokenRenewer thread count
+ e+ A7 W' h4 `) _0 B) |# ` </description>8 X" [# [, j, ?. E$ Q& e2 q
<name>yarn.resourcemanager.delegation-token-renewer.thread-count</name>1 {( `+ ^, D! @ W/ O
<value>50</value>6 a) g; \/ K+ v! M* F) q. V3 X7 F4 ]
</property>" g% U0 ?. W( d0 u" I
<property>% h: y/ o' x3 a$ E. {2 S+ `' k( j
<description>9 r# Y1 u( O- P/ f% J8 ]
RM secret key update interval in ms
: L( k& L. m9 y* W: t, s </description>5 z5 a, p/ g" p
<name>yarn.resourcemanager.delegation.key.update-interval</name>
' k$ X+ ?8 n% S& W <value>86400000</value>, R; e7 v/ r8 t% a1 Q& d
</property>
5 t, h4 e2 \5 d; b1 P1 e <property>
i0 f* u7 c |" C9 @* Z# \ <description>
; i! O# D! v/ H. c1 a; M RM delegation token maximum lifetime in ms, u9 q9 W# d5 N. A1 o
</description>: y$ p C1 K8 P5 c
<name>yarn.resourcemanager.delegation.token.max-lifetime</name>- ~! M. ^+ V! x2 {
<value>604800000</value>/ g! z4 h3 }; S$ F6 Y
</property>& `0 r# A2 U2 m2 }
<property>( [* R9 i. v1 j! X/ `' D' j3 E
<description>4 y3 v/ A. E3 `4 M7 c: ~
RM delegation token update interval in ms
# L/ i! M: k* {) @" [ </description>) u/ c( h) o& A! h, T
<name>yarn.resourcemanager.delegation.token.renew-interval</name>+ Y0 F5 n% j' ^9 G9 \
<value>86400000</value>
5 h! p% O' X' Q! l6 l3 H </property>
/ q- o( e0 o6 o, m! d <property>
6 \: o6 ]$ d, n3 [1 Z <description>
2 I) {* D9 h+ p Thread pool size for RMApplicationHistoryWriter.
/ I; q/ I, E/ s: h </description>$ b# {$ f; p* Z" \7 N
<name>yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size</name>1 R# z1 l* d; B8 f; B9 c e. l
<value>10</value>$ A: M ?) ~- I$ d7 d
</property>% P& y5 _7 k/ X2 e v% F
<property>
# j1 w, [8 o& R( {5 ~ <description>
+ Z& |' x- q/ Y, n. F) p" l: @ Comma-separated list of values (in minutes) for schedule queue related
4 r/ l7 ^3 t; v. @ metrics.5 ]' A( Z: u8 d, m5 {
</description>
" m7 g% u' U: k; X/ B( N# J" o <name>yarn.resourcemanager.metrics.runtime.buckets</name>- M x6 R, m& T; v0 {, Q0 X8 C
<value>60,300,1440</value>( A; ~, Z$ j; [
</property>
5 a* K0 c, k( B8 W/ L+ o <property>
! N7 @) ?& R" }, A. v5 h3 g <description>
" ?8 I9 F: A8 E Interval for the roll over for the master key used to generate; i7 E: @' l8 d- T j; b8 a
NodeManager tokens. It is expected to be set to a value much larger
& X' P/ {0 A$ n) w# I4 X& d than yarn.nm.liveness-monitor.expiry-interval-ms.
8 ^5 I! k" n) d' I </description>$ {" u7 U1 m4 m6 F1 G G
<name>yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs</name>" z! G! B3 \, r1 Y+ y
<value>86400</value> w/ M* L9 T2 ]/ N# F1 U" z- @8 b
</property>% N4 m' P& i. H: i/ c6 a) C
<property>, U' H. Q8 _; A3 p; u0 F @! d
<description>
4 M4 n2 t* V* s: Y6 U8 I Flag to enable the ResourceManager reservation system.
. u# d' e0 p2 Z, v </description>
) H3 H; m( `0 S+ g' t) V <name>yarn.resourcemanager.reservation-system.enable</name>
4 e$ e$ a* R: s* T0 P <value>false</value>
: P4 S) U! E4 k, e/ R8 K2 i! L </property>
' F) s3 z+ |8 v3 b4 Z <property>8 B% a% J. p7 x7 ?* l) n Z
<description>6 {! U$ ?- _9 ^% L
The Java class to use as the ResourceManager reservation system.2 p5 X; C5 L \. Y3 i1 \
By default, is set to) D( h( O. ?+ g% D, l& T
org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem
' }' L P X; C- l+ t4 j% v when using CapacityScheduler and is set to
! s6 Z1 F( k: I/ x org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem& ?0 i1 w; u$ a' i& d2 g, S
when using FairScheduler.: o8 n/ T& q2 n
</description>
! |7 {( s: g9 @3 I <name>yarn.resourcemanager.reservation-system.class</name>
2 M& I+ v: h. T <value></value>
0 z2 r0 I+ M- o( X9 f& E4 j& ^ </property>2 m) r# s2 o, s, R* K* p) E8 J
<property>- o7 _4 J% k: [" Y# ?/ ]
<description>" Q5 c5 T: P; x7 Z
The plan follower policy class name to use for the ResourceManager
! s( j! F% |, ~4 Z# I# L" J reservation system.& Y9 \- l9 u+ b9 g8 O: I9 z
By default, is set to
3 m- J- Y; i* c) ^; P' H2 q, D org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower
9 P o9 F3 k# l% S# k is used when using CapacityScheduler, and is set to% g0 m6 P+ D2 p& z0 g, @, [# Z
org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower; g' y2 _/ |" ~2 |& O1 O) [. s
when using FairScheduler.% R- n6 D' e; d8 g6 l4 a( M7 Y L
</description>8 U V$ ?0 x# `% ?
<name>yarn.resourcemanager.reservation-system.plan.follower</name>0 d9 M$ k3 z" e
<value></value>
: |1 Z7 ~% l: [8 q </property>
, {$ J z! F/ v# C2 O) s3 q6 ^ <property>
* S$ v- |5 L# h" n* I <description>
# d5 u j6 g9 G+ V' f. ?8 s$ o Step size of the reservation system in ms" W( m9 K6 W4 z0 T
</description> D6 n" T# A, J2 o6 T3 ^- V: n- P/ s! d
<name>yarn.resourcemanager.reservation-system.planfollower.time-step</name>1 `, h$ Q2 ~+ V4 p3 T
<value>1000</value>
( g5 N- l6 A! E: I2 s7 X </property>8 H5 [& D2 t4 K
<property>; s4 ?/ @! c4 x1 h8 C8 ^" H
<description>6 {6 [. J# Z" \: n6 ]: S
The expiry interval for a container
9 x# K* \5 x7 [; }8 j </description>
- j: x3 ^0 B* @( c4 o+ ?4 \ <name>yarn.resourcemanager.rm.container-allocation.expiry-interval-ms</name>
" y; J2 V: H6 \, p- U <value>600000</value>
/ y3 [7 `: j9 z+ M </property>
' D/ H# }# W5 a4 c; Q <property>
. l7 n4 g* W1 x <description>
. B4 c5 {1 Q+ k- C$ v, F Flag to enable/disable resource profiles% E! r4 E2 ^. z& U
</description>
" ]: C) D' X+ D2 t2 X% [+ E( ^& D <name>yarn.resourcemanager.resource-profiles.enabled</name>. S1 L* z, R0 N( y- a; |7 T
<value>false</value>
- p& e% i1 b* G7 @5 k5 N+ A </property>9 d5 {; q1 }' z8 M, M3 X
<property>
( M1 r& V* h% [6 K. i7 Y; @ <description>
! B0 J2 v4 N) \: b8 D0 l) X3 _ If resource profiles is enabled, source file for the profiles
7 b# a& O6 h4 `5 ? </description>
' {4 c/ U) O) |/ ?) V. h$ n <name>yarn.resourcemanager.resource-profiles.source-file</name>8 O9 P& v! N) }6 f* u' x
<value>resource-profiles.json</value>
% }, g6 N5 T" Y3 w; H* `( C) H </property>
* w$ Q7 v: _' [5 i; S <!-- Node Manager Configuration -->$ Y* ~) z* S; b7 y+ X. C
<property>% x p0 u. n) }( h, q
<description>The hostname of the NM.</description>* m: R' D7 a' s: g9 V# V$ z$ c
<name>yarn.nodemanager.hostname</name>
0 y, w: m$ w3 t% e- Y* `$ k9 r <value>0.0.0.0</value>
; A$ o. |( W& r) w& R' i7 F </property>
9 b1 _2 v ~6 V <property>
1 V0 E a4 u8 M% z" C! V8 K- C1 z <description>The address of the container manager in the NM.</description>4 t3 A7 j, n$ |* W! A5 o
<name>yarn.nodemanager.address</name>
0 Y3 b5 k B! j <value>${yarn.nodemanager.hostname}:0</value>) r' [$ m2 |) G& } _
</property>
( D/ @! B- D) h5 A% g <property>
0 t% E: }. _6 _0 S4 b <description>
. }& B- i5 E! r$ r The actual address the server will bind to. If this optional address is
! _8 g' J" Z8 ?! ^3 v' A set, the RPC and webapp servers will bind to this address and the port specified in
& ~( Z7 Y6 {0 g& h yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is* w0 h# F) ~9 t% u" h
most useful for making NM listen to all interfaces by setting to 0.0.0.0.. O7 U& ]* }$ S! Z! K
</description>
: Z9 n8 X7 x% n" V9 [0 @7 A6 p# n% G2 L <name>yarn.nodemanager.bind-host</name># r B% h% t/ e5 J) ^
<value></value>) d! R, {/ f0 F- y. Q- s4 @$ |
</property>* S5 E; ]3 |% m
<property>
* Z% @4 \! {7 W @ C <description>Environment variables that should be forwarded from the NodeManager's environment to the container's.</description>& M* ?6 A' l; x. o
<name>yarn.nodemanager.admin-env</name>
/ N% k* ?9 j% Q9 H3 K <value>MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX</value>3 f) t: E! k. K5 z
</property>) A! a. b% d/ [" v8 O
<property>
+ ]6 J1 w, _$ ?1 i( Y: ]% ]9 C2 J& N5 \ <description>Environment variables that containers may override rather than use NodeManager's default.</description>& v4 T& m' ]/ H% c
<name>yarn.nodemanager.env-whitelist</name>
* o$ Q* `& O8 e <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>* w: p" p7 P! z5 k
</property>, j$ K+ a0 ~7 E9 C/ S
<property>5 a& B; n( l& ]4 k8 Q% Q* [
<description>who will execute(launch) the containers.</description>
4 Y" w% G- o8 x <name>yarn.nodemanager.container-executor.class</name>+ y: N% a: u% ]. N, m/ J% ?% g
<value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value>
. v; V* M9 q* ^; L' ]% o </property>( w5 j1 n2 O" B! r1 p9 {! A' R0 Y
<property>
$ C$ b) }# H# ~) w8 A+ d4 L& w <description>Comma separated List of container state transition listeners.</description>( v9 q; X! Q) T9 {
<name>yarn.nodemanager.container-state-transition-listener.classes</name>- _" Q' `* e; A) L; g/ w0 V" s
<value></value>/ q% V* `6 N, j" ^ D5 W4 w
</property>
( h0 Y! }" l8 D* T- B' q! m& S n <property>
5 }* s9 |( v6 ~ X. z; V! A <description>Number of threads container manager uses.</description>8 b4 l& q, F0 o- Z# S
<name>yarn.nodemanager.container-manager.thread-count</name>
' H2 ?1 ~6 ]& s% l- A2 h: | <value>20</value>. V9 \/ u! D5 X3 V
</property>
4 h- f1 I7 t6 h8 B1 H5 y <property>' T! c6 b: W9 r; P: x) {
<description>Number of threads collector service uses.</description>
! {' x1 R: J0 Y <name>yarn.nodemanager.collector-service.thread-count</name>
! Q% @$ B x1 J! a <value>5</value>
. T% O& H0 N; W! a: I </property>
* K; E9 R6 P+ G. i <property>
4 J$ ]4 i( h% j <description>Number of threads used in cleanup.</description>
7 G6 ], G, J* h6 [, o. k1 Y <name>yarn.nodemanager.delete.thread-count</name>, \( W* E2 L* N& g5 [
<value>4</value>
|9 u( O2 J& [6 a X g </property>
" Z$ q+ {. o6 T; G <property>! }8 ]/ P& S- N! z. e2 G* u
<description>Max number of OPPORTUNISTIC containers to queue at the/ K( ~- @4 Y2 L4 n1 c3 D P
nodemanager.</description> V0 u# j' c) [- h9 m/ K" [
<name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>
9 x( T! E4 a: U6 N <value>0</value>* V4 g" h' M }0 n3 ]7 ?' v; Z# d
</property>) G( M0 R: H B
<property>
! d5 s; p1 h# S8 l/ f <description>9 L& ?& p! p- Y
Number of seconds after an application finishes before the nodemanager's # T( ] F/ d/ @9 a* y6 ?
DeletionService will delete the application's localized file directory
# a% F8 c- D5 j- d and log directory.
8 G% n/ x+ y' a* g+ Q To diagnose YARN application problems, set this property's value large
2 C8 d1 H; k( U' e* `% [1 y7 P enough (for example, to 600 = 10 minutes) to permit examination of these
( n5 E! ?4 x/ m) G directories. After changing the property's value, you must restart the
6 a: P, b0 F' g. i& O- B nodemanager in order for it to have an effect., |& R6 s1 Y, W9 ?* ~$ m
The roots of YARN applications' work directories is configurable with
]7 W; P' e Z( N+ p. Z the yarn.nodemanager.local-dirs property (see below), and the roots
, m' c8 U9 x' T+ o9 f, Q of the YARN applications' log directories is configurable with the
0 w F4 Z) Q( w3 W" x yarn.nodemanager.log-dirs property (see also below).% `4 T' E4 `: o5 d/ K
</description>
0 z4 O$ E+ c; N! x0 ?; L9 V6 N* i <name>yarn.nodemanager.delete.debug-delay-sec</name>5 V8 N4 ~" e+ c: v* k3 `/ T
<value>0</value>
2 o. j! z1 p' V, ^! v; Q$ P$ F </property>
! K; k I- o7 m8 U. n' d: U- H% b <property>6 ?7 I3 _ ]/ M6 j/ @1 h6 c3 p" }
<description>Keytab for NM.</description>
' |) @5 [7 l/ L- @# ]* r <name>yarn.nodemanager.keytab</name>% {$ M4 g/ Z4 U @0 w
<value>/etc/krb5.keytab</value>8 Q' B3 m j4 K3 ~5 d
</property>. ?' g7 B: X4 i
<property>
( ^9 x+ m+ d* \2 h- ~ <description>List of directories to store localized files in. An # V y8 K% [, b; C1 |1 F' p
application's localized file directory will be found in:
( }# L g* P S- U! [ ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
/ V" t) u$ b, @5 h5 f- o0 R2 w( g) K/ C Individual containers' work directories, called container_${contid}, will
: u% i" ?" Z( l, ~- e7 K be subdirectories of this.
3 r0 ?, f. R, E) m, K @+ [ </description>' o3 u5 `3 P3 V1 H4 Q: ^7 A- A
<name>yarn.nodemanager.local-dirs</name>' }9 Q# x6 ]8 g% V# z) X2 n
<value>${hadoop.tmp.dir}/nm-local-dir</value>6 B7 s) z5 q, A/ G v2 e2 |- \
</property>
/ b: @: a6 Q3 j2 C& n, o <property>5 |# U! c. w5 J$ Q0 ^+ W V4 s3 A
<description>It limits the maximum number of files which will be localized
! f+ m, [+ y' `/ B. \ in a single local directory. If the limit is reached then sub-directories9 k; X P6 j& v3 d9 I
will be created and new files will be localized in them. If it is set to
* `; Q* i8 B3 i5 F$ `3 N: ^( P! o a value less than or equal to 36 [which are sub-directories (0-9 and then0 l* C: U+ {! |, o% S
a-z)] then NodeManager will fail to start. For example; [for public
) h4 U. F1 Q9 J. j; P) T$ L cache] if this is configured with a value of 40 ( 4 files +
O1 } F& D& q% v, Z" h 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will
& ], C/ d' y7 ~( b6 u& @7 x allow 4 files to be created directly inside "/tmp/local-dir1/filecache".) k8 y+ F4 ~# b9 q
For files that are localized further it will create a sub-directory "0"
9 a2 j& @' ]7 V# z6 V3 s, Q inside "/tmp/local-dir1/filecache" and will localize files inside it
- H" w! p3 s# M' b+ R# K( H3 P2 E( r until it becomes full. If a file is removed from a sub-directory that
& v4 S0 H1 m. m, C* Y" ^ is marked full, then that sub-directory will be used back again to
7 ]' ]5 y) P& X localize files.
, y. f# K0 A9 K( y% i </description>
8 L) Q4 A% M, ` N <name>yarn.nodemanager.local-cache.max-files-per-directory</name>
5 P, Y- L2 t. Q4 }" l <value>8192</value>4 H" E# P, @$ b1 z. m
</property>
( ~) I0 o+ R) q9 W1 `/ ^2 c) \ <property>
m" C1 h0 f. G ]: h& [ <description>Address where the localizer IPC is.</description>8 w. ^2 N; \6 s% A
<name>yarn.nodemanager.localizer.address</name>
& r' `( F. Z0 l% O( l <value>${yarn.nodemanager.hostname}:8040</value>8 t: J8 c% C4 E( d
</property>
$ `" C3 M: k1 |5 F, a <property>' Z: b$ L- m3 I# ^0 s
<description>Address where the collector service IPC is.</description>2 q1 [/ ~4 R" P9 h. C
<name>yarn.nodemanager.collector-service.address</name>
3 H9 c" f5 s: a; G6 z" a5 V <value>${yarn.nodemanager.hostname}:8048</value>3 J, @" L, q4 z
</property>
9 j4 c+ y1 n" c' l% f$ }: E1 t <property>7 b5 B" p i6 F
<description>Interval in between cache cleanups.</description>; u$ s* K: z* y K& i% C
<name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
2 d" F$ o" ^$ e1 x+ N <value>600000</value>
0 v% ?- g9 `2 o8 k3 z- c4 r </property>6 [( ^# a2 y2 F5 h" U
<property>8 P6 Q4 C' l6 l7 E! t, e
<description>Target size of localizer cache in MB, per nodemanager. It is
% e, L$ M2 I) } a target retention size that only includes resources with PUBLIC and
9 Q l& |2 T! J9 T3 s PRIVATE visibility and excludes resources with APPLICATION visibility7 F, l; ^* k) X. \* W" `& {4 _ M
</description>; S% R' u2 r z2 Z0 O7 A7 R
<name>yarn.nodemanager.localizer.cache.target-size-mb</name>+ L* N/ X' L. ]! f G* C
<value>10240</value>' t2 H$ f3 s1 P a( j
</property>
* w4 @4 F" H5 _9 s# T @ <property>
7 S8 y; @5 F t( ^7 `1 R; \( R. T <description>Number of threads to handle localization requests.</description>
8 ]! j# K+ T6 H$ T <name>yarn.nodemanager.localizer.client.thread-count</name>
( V1 p! A4 P6 V* d3 K! H <value>5</value>
0 e! A: \) ~$ q. k% r </property>8 Q% [0 N: c/ Q; I1 z4 i( y
<property>
2 ~6 z1 `) k' m <description>Number of threads to use for localization fetching.</description>+ Y* p* ^: d* ?0 c+ j4 s
<name>yarn.nodemanager.localizer.fetch.thread-count</name>
f A3 o/ H5 \ <value>4</value>
$ U4 Z% Z! D- |4 w: b7 g/ V </property># L) H% H+ Y3 K
<property>* b; C2 S2 a& e& P. B% j
<description>: Y+ y4 l" l& e
</description>+ o n# d5 V1 {, d4 Q/ Y- |
<name>yarn.nodemanager.container-localizer.java.opts</name>
6 p9 s) v3 ]' t- a <value>-Xmx256m</value>
( K; F0 ^) h6 {5 W; T </property>
) f1 B, O0 h) ~0 l2 c <property>
1 z% `1 s- R" X4 D <description>
( f8 `. j# r+ t9 u The log level for container localizer while it is an independent process.
9 Y2 `+ j- {6 b8 \2 z: b </description>% v: x' o2 u- D
<name>yarn.nodemanager.container-localizer.log.level</name>
) d; C) I) j8 M6 t ] <value>INFO</value>
& c, R/ {0 Y. I/ n' l </property># l: p: c, m$ N( F( O) j8 t
<property>
* Z& e$ g6 w1 s <description>4 L7 H) @# u$ Y$ ^; {
Where to store container logs. An application's localized log directory9 h0 t2 L" v) [6 l1 w1 N
will be found in ${yarn.nodemanager.log-dirs}/application_${appid}.+ h6 X) y) B" X0 Z8 P
Individual containers' log directories will be below this, in directories ; n/ a$ @% @7 f8 P: ]9 W- _! @- D
named container_{$contid}. Each container directory will contain the files
% P e& C% ]" R6 R( z stderr, stdin, and syslog generated by that container.0 G/ s: R( j; w U1 M* N6 s
</description>! s! w' ~" O7 r% I
<name>yarn.nodemanager.log-dirs</name>
5 f* s8 U0 u% V f <value>${yarn.log.dir}/userlogs</value>4 G" A: S9 p6 y
</property>" v! _' V( k, M. z# `6 k
<property>
7 S6 G% {8 g2 x <description>6 R! x9 ]& j" W) N: l! S) b
The permissions settings used for the creation of container8 a3 d0 g( Y4 o6 c
directories when using DefaultContainerExecutor. This follows
1 `& Y2 t: g! T$ V5 v standard user/group/all permissions format.
0 c) v$ ?$ p' B1 k8 M4 [ </description>' Y6 L5 h$ |/ B6 T V, V3 a9 V
<name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>& A1 Z2 t1 ^) T( k1 }- {
<value>710</value>
" I% y& F* o: T" _. m </property>! @8 u7 n6 E8 y/ _! f* r0 L X4 ?
<property>' Z; D5 ]; ~, }, X
<description>Whether to enable log aggregation. Log aggregation collects
2 S) o. l+ a0 b5 E( L each container's logs and moves these logs onto a file-system, for e.g.* Z& r5 B7 N' x2 W; |& P! V
HDFS, after the application completes. Users can configure the
! ]% E; U: i2 a A$ Y "yarn.nodemanager.remote-app-log-dir" and
& k$ j: E+ i$ c7 }. [! u5 c "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine
5 p) F3 e: l2 F8 t$ V4 P' R where these logs are moved to. Users can access the logs via the2 F* X- l5 u! Y6 }; R" v
Application Timeline Server.
. s ?! c4 p3 M7 W3 I </description>" D+ P, ?& {. k8 G1 l) A& Y
<name>yarn.log-aggregation-enable</name>& A; f4 J2 l3 r# w4 L1 T7 F7 i
<value>false</value>
1 P. P: C( V6 A! z4 O3 T5 J </property>9 E( Q/ U) N) ^( K8 d6 k+ L" s# S
<property>, A' m9 p/ }- w) ]1 T, m# [0 k
<description>How long to keep aggregation logs before deleting them. -1 disables.
) S/ _- v. |9 e9 c Be careful set this too small and you will spam the name node.</description>
; Q0 W N A7 E& ^+ e <name>yarn.log-aggregation.retain-seconds</name>/ x* l) ^2 ]! S' S+ R8 [
<value>-1</value>+ s9 g; T3 g2 K, {) Q" D1 P' k
</property>
& {' w8 T5 L+ F0 [' |, j$ K% J <property>
5 P9 s/ ~- [1 { <description>How long to wait between aggregated log retention checks.
7 [7 A2 q( a9 m* W6 l) V- a2 d If set to 0 or a negative value then the value is computed as one-tenth" k5 N" u4 [1 A6 a' A
of the aggregated log retention time. Be careful set this too small and
# Y0 W! K0 l6 x- l2 x you will spam the name node.</description>
4 n) T, R0 F- F- ` <name>yarn.log-aggregation.retain-check-interval-seconds</name>. H$ s f6 O3 _4 ^. c7 p9 U
<value>-1</value># E5 R5 d+ `! e+ D2 O- K) {2 N
</property>8 D4 \# M" M/ X1 W
<property>$ n( f* L" z! Y/ I( L7 g
<description>Specify which log file controllers we will support. The first
6 f5 Q5 F! g! v9 ~2 v3 |; s6 g file controller we add will be used to write the aggregated logs.6 H; N7 j4 v/ ]/ @7 P! k D
This comma separated configuration will work with the configuration:& E+ l9 Z, X) F# D
yarn.log-aggregation.file-controller.%s.class which defines the supported
! G6 M D6 P- o* s% x. e/ J file controller's class. By default, the TFile controller would be used.
) [7 x' D2 a1 M' n+ H The user could override this configuration by adding more file controllers.
* \+ l0 [. d8 [; q( {3 t To support back-ward compatibility, make sure that we always
. W( a3 N* V( h$ M add TFile file controller.</description>6 S. V4 X8 k3 h9 v% e( b
<name>yarn.log-aggregation.file-formats</name>) E& p0 i7 \/ f- P
<value>TFile</value>
( ~. z/ ?; i: V8 C/ J+ S, _' }1 n </property>1 h$ ^, S2 j2 A# E1 W9 `
<property>7 w( [' P2 i! L7 W! ` P
<description>Class that supports TFile read and write operations.</description>$ n" u- ^% R$ K% g
<name>yarn.log-aggregation.file-controller.TFile.class</name>
4 t) u1 K% ~5 ?2 K3 W. o6 t* L <value>org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController</value>
5 j3 g9 `( V# ], `2 h1 l </property>
% V: E- Q/ k, v3 V& G: N: t* S& q! e <property>
u2 s- l7 l! { <description>
n! h( v; X I" n8 k3 W( j How long for ResourceManager to wait for NodeManager to report its
. I- Z9 P- l+ `; e, S log aggregation status. If waiting time of which the log aggregation4 K: W$ P; s5 S) a3 x
status is reported from NodeManager exceeds the configured value, RM
& `& M. @7 f1 q' {7 s { will report log aggregation status for this NodeManager as TIME_OUT.4 N R$ c t' [8 i0 j
This configuration will be used in NodeManager as well to decide: ~6 A* l) A% T: }% \
whether and when to delete the cached log aggregation status.! G, R! O# h0 i3 z+ W7 | Z
</description>
: u/ J- i6 [2 }2 E <name>yarn.log-aggregation-status.time-out.ms</name>; r" _2 F* e3 @% E: j
<value>600000</value>
2 F$ r1 h3 H$ Q# F( @. F5 c& B </property>
/ B1 H9 X' u, ?& V8 N N* J) |% u <property>2 l) O9 L( ^9 w( e5 F6 Q+ ]
<description>Time in seconds to retain user logs. Only applicable if
/ L# I, p, C! ? y3 z9 i log aggregation is disabled
' E S& r, T: O2 m </description> }; h2 w. q5 H% ~5 D4 v3 V
<name>yarn.nodemanager.log.retain-seconds</name>) C( O0 J+ v1 W
<value>10800</value># |0 K% P# _. k% r, @/ I4 w; u0 y
</property>
* L) u% `; B6 h9 ]& l, M7 A <property>8 a! b) f$ r* C% s& Q& R
<description>Where to aggregate logs to.</description>) n% p, b, c$ ~8 a0 y0 t
<name>yarn.nodemanager.remote-app-log-dir</name>; n, z! D* {- B" Y& z w2 R# g
<value>/tmp/logs</value> b# R4 M8 S; h- c) B: ~) H
</property>
2 m+ s( e: ?' P <property>
5 H' Q; R$ V2 N- f4 M3 R2 B2 Y <description>The remote log dir will be created at ! D$ k& r/ z& H( c
{yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
% R/ b3 \7 b/ E/ h, Q </description>, ~* c# f) i! Q" x# N+ U8 b+ Q# j7 R
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
- N# v: c- A$ _2 N5 k <value>logs</value>% l3 W G3 I2 U
</property>
# y& |: t* [7 M. T# e <property>9 x m( z( c9 m3 u
<description>Generate additional logs about container launches.
7 W3 l! n* d- L8 f' k& Q( `6 s7 [ Currently, this creates a copy of the launch script and lists the( E' n( e! W6 |
directory contents of the container work dir. When listing directory2 q/ Q) b% ?# x( U5 v3 X n2 c
contents, we follow symlinks to a max-depth of 5(including symlinks
: F5 G' J+ y4 o* i which point to outside the container work dir) which may lead to a b% ?2 B( V/ C8 x
slowness in launching containers.
9 C, _! F% j$ w* {/ F# V </description>% X! N: l3 T8 w' ?
<name>yarn.nodemanager.log-container-debug-info.enabled</name>5 H/ K- ~/ i! r9 O$ R
<value>true</value>2 _( z8 z% l2 i1 T
</property>7 S6 [6 v6 V" l, M5 Y# w
<property>( m7 Y* b' X J/ }
<description>Amount of physical memory, in MB, that can be allocated 4 R, k" x2 b | _5 d
for containers. If set to -1 and1 U( q/ s8 T( H- F4 V7 ]- k
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
5 t" G3 r) j) g0 O2 i automatically calculated(in case of Windows and Linux).* K. w1 W/ X/ [5 ^- r
In other cases, the default is 8192MB.
, Z! Q n6 `2 Q( S% l- ^ </description>
: q( V, D* {+ y) X <name>yarn.nodemanager.resource.memory-mb</name>9 o- _- ^. V# q2 E- G
<value>-1</value>/ P9 T. m/ G8 x8 f2 S+ y
</property>3 @ \2 f6 E2 i0 B
<property>) y( }, l# J+ n) m$ u7 O4 j0 ?# D' `& B
<description>Amount of physical memory, in MB, that is reserved% X) ~6 T8 r8 T4 T( H
for non-YARN processes. This configuration is only used if
5 K; D. |% X/ @/ P yarn.nodemanager.resource.detect-hardware-capabilities is set
+ K; N. a* U [& i to true and yarn.nodemanager.resource.memory-mb is -1. If set
3 ^+ W! |$ P% u5 k. {4 `/ E) [, ? to -1, this amount is calculated as( g2 N6 m6 P4 H- L7 H# r5 M5 ]$ r' y' g
20% of (system memory - 2*HADOOP_HEAPSIZE)# Q3 A. l1 o7 r! |8 x2 j
</description>
2 R& d/ o. K$ Q8 ^/ x' } <name>yarn.nodemanager.resource.system-reserved-memory-mb</name>1 p6 b$ K$ v0 y" {
<value>-1</value>! J( x* y3 I# m. T
</property>
* X* d# d; m. C, G <property>
9 W: V1 p2 x1 `6 u( p9 N! {" ^4 q/ L <description>Whether YARN CGroups memory tracking is enabled.</description>
8 P' M: W: R/ l& ^! {) \ <name>yarn.nodemanager.resource.memory.enabled</name>
/ @4 _2 _+ z0 t2 |' c <value>false</value>
; T6 ?8 y L7 v5 [ </property>/ c; i2 h( Y' i( ~
<property>
! n$ O3 W# v, c& Y8 e( v% W$ N <description>Whether YARN CGroups strict memory enforcement is enabled.
* c3 o. o, v- } i( r </description>& y" R- P/ y8 T* o1 R- U: f1 A
<name>yarn.nodemanager.resource.memory.enforced</name>9 ]: f( y, O1 z2 m: N( l- e
<value>true</value>
7 j" q ]' z$ [( ^5 a/ ^$ m </property>: J+ Y+ H3 c% S% o$ n
<property>5 `+ x2 w+ t0 B. C, S l0 i
<description>If memory limit is enforced, this the percentage of soft limit% @# l/ k6 `% L) R. Z* X# _
compared to the memory assigned to the container. If there is memory
$ x: v* u$ ]4 [ pressure container memory usage will be pushed back to its soft limit7 X k" E$ l1 ^2 s1 G
by swapping out memory.! D( u& ]) i# i N
</description>
0 M; W( N: S# I* i <name>yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage</name>7 W/ g. n0 R, e9 w o6 B
<value>90.0</value>
+ ]' s/ L! A. G7 y+ t </property>5 C$ Y1 |4 ?/ _8 [9 E* @& ~
<property>
n! B0 x2 j" x* o4 \* X: s5 }& c <description>Container swappiness is the likelihood a page will be swapped: \; U( b, S4 G( m. i
out compared to be kept in memory. Value is between 0-100.9 ^, s5 w: q! P1 f: \9 Z8 J
</description>
' \$ L2 C# N2 W2 Q% p$ o <name>yarn.nodemanager.resource.memory.cgroups.swappiness</name> o& { ?8 K1 }9 S$ ]
<value>0</value>1 y0 j1 J- G& e
</property>8 k, `$ u/ Y9 W& d" T$ L
<property>
" a9 ?5 F ]$ _2 x3 X <description>Whether physical memory limits will be enforced for$ G- A; s! d) A4 _& o2 ?& I
containers.</description>
6 t/ d5 q' G) o! \/ ? <name>yarn.nodemanager.pmem-check-enabled</name>$ b- Q- d& X; w+ Y9 O
<value>true</value>
; P* C2 M2 ?$ E$ O. I1 ? </property>6 C$ S/ r) h0 d. v0 K+ C5 O! z1 v
<property>1 N2 |0 P- X9 b5 X7 ~$ ^
<description>Whether virtual memory limits will be enforced for! X* j5 C; G; q5 k: P1 x, E
containers.</description>
$ T5 b: s" V) h2 D% Y2 n. J2 m9 ~ <name>yarn.nodemanager.vmem-check-enabled</name>" g( c- F i/ x; T7 K3 `
<value>true</value>+ }" U7 u$ e0 {
</property>4 u9 z: n5 p+ Y" V0 o
<property>
2 W+ P' n1 K* Y% y" v <description>Ratio between virtual memory to physical memory when+ ^: a. Q- M5 L2 G$ l
setting memory limits for containers. Container allocations are) g- d) A2 a$ M
expressed in terms of physical memory, and virtual memory usage
: q3 {% ~( V; a, I# J8 ^+ C is allowed to exceed this allocation by this ratio.
4 R# J) e, r8 ^% y, N) z </description>5 k, b9 m( D" M F
<name>yarn.nodemanager.vmem-pmem-ratio</name>/ j$ O% |) {% l; o
<value>2.1</value># d. g0 C4 ?4 c+ G
</property>7 `& y2 R" i' Z+ L( a
<property> w( _2 T+ C4 H3 ?9 h
<description>Number of vcores that can be allocated, B" D y5 b- m) \8 z
for containers. This is used by the RM scheduler when allocating
$ v" a) W6 S& e q resources for containers. This is not used to limit the number of
& \, w1 b3 u( q2 b+ B7 i3 f3 p! Q* i CPUs used by YARN containers. If it is set to -1 and
: }; H- U' Y% {- k# Q9 \) x yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
. E Y; X5 V% @7 {, h automatically determined from the hardware in case of Windows and Linux.; Q6 l/ b8 ^7 H- z# b2 e
In other cases, number of vcores is 8 by default.</description>( d* ^2 t/ m. g6 }" {& |
<name>yarn.nodemanager.resource.cpu-vcores</name>3 @2 w: P0 U' R1 a
<value>-1</value>
8 I" E; l/ \- [! T# _: b </property>
; @% J5 q% a& Q/ H5 ~' a <property>, g( i; E8 p. n# ?6 @( Y
<description>Flag to determine if logical processors(such as
5 S' K$ p1 n. a# k* { hyperthreads) should be counted as cores. Only applicable on Linux; t& _, F6 s* w5 f; O! D
when yarn.nodemanager.resource.cpu-vcores is set to -1 and' j8 \: F% u8 ~' }0 l
yarn.nodemanager.resource.detect-hardware-capabilities is true.) a4 D# x/ g$ A6 a# R% ^! R
</description>
: P1 P' K- I3 \# `+ ~# _ <name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>; ]0 W0 E) D1 U* b
<value>false</value>
5 h3 h' t% _4 A: o& j6 K </property>
5 X0 W4 b# \: I, |3 M' G <property>& \" {; C( M9 s5 j
<description>Multiplier to determine how to convert phyiscal cores to
3 m3 s1 i. ~9 N# E vcores. This value is used if yarn.nodemanager.resource.cpu-vcores1 H& H0 n7 l6 T) g+ ~
is set to -1(which implies auto-calculate vcores) and$ j) D3 @; l. g% M9 }' }; I) O
yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The- ~- j5 N4 F) r) s
number of vcores will be calculated as
3 S/ b5 o: F) A- k4 V# m number of CPUs * multiplier.( A+ Y3 k. D4 c' T5 D% s+ H
</description> K* T" h/ N' C7 ~
<name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>* }( ^) f1 c, o) g6 j& b* [
<value>1.0</value>. A: R0 V0 k4 y; T- L& r
</property>
7 s/ t. G, i4 M4 N. H <property>
7 K" b- \1 P+ d, J <description>
6 f) u! S b: r) v Thread pool size for LogAggregationService in Node Manager.9 Z* c: \: L( l! o# r
</description>
/ x% G& w' y; G4 D* C# a <name>yarn.nodemanager.logaggregation.threadpool-size-max</name>5 l3 r3 L4 u) l2 A9 n Q
<value>100</value>
4 @0 M( L( U5 H6 {( a </property>% t, z" b5 ?% [4 p. i
<property>
/ @; u7 @. t( O* [) Q* H; Y <description>Percentage of CPU that can be allocated
) F2 M3 q' H9 ?; h for containers. This setting allows users to limit the amount of' x% d; B) e* T4 p/ I
CPU that YARN containers use. Currently functional only
5 q* L: a, p) S6 X8 t7 O7 |" ~ on Linux using cgroups. The default is to use 100% of CPU. R! W1 u- M, t' k$ V
</description>8 R( e5 c: E5 j) g* x/ {
<name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>" G4 c0 b# }# u
<value>100</value>, S7 N j% C- U4 S
</property>
0 Y! n8 T6 ~4 r9 N4 x& B5 t <property>
) o) k, S! D6 o8 P6 Z <description>Enable auto-detection of node capabilities such as# Y7 Q0 H8 X' m9 _, P W' H, D
memory and CPU., H4 g) L1 C. K3 Y9 v: |* r. V
</description>, Q* b% W2 {) z: k) J
<name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
& { B- S1 g- f- b, P; q <value>false</value>
' I. [6 }. U) l" g </property>% X3 X, u+ C! H
<property>
8 q% y% B9 l2 _8 W7 t <description>NM Webapp address.</description>
; ]# {, B1 O6 c" \3 M <name>yarn.nodemanager.webapp.address</name>2 E& T' D" u9 @ s* {, U0 B: a
<value>${yarn.nodemanager.hostname}:8042</value>
- Z5 [8 M# U; I- }; } </property>% T1 D' W# R0 V% j
<property>
4 d7 e# ?3 t1 L7 y5 B/ T <description>- l$ w& R: [, J" @) p
The https adddress of the NM web application.5 Z) H0 F9 \3 D
</description>8 ~6 B& n, [4 J' T5 e( l5 |$ j; Y
<name>yarn.nodemanager.webapp.https.address</name>' Q8 }( c$ t4 p0 }+ y& j
<value>0.0.0.0:8044</value>5 s6 l# K3 k2 `9 C( H! S( D! p8 ~
</property>
. g& U. x& }. E; N& i <property>$ B* W1 i' `( J3 i- a- D
<description>
+ ~! g$ Z! G- y5 O The Kerberos keytab file to be used for spnego filter for the NM web) g) }/ D7 J7 u* [7 J" X! {% B
interface.
% D. F- F4 p4 i7 j </description>
% r" y' A! s! a M <name>yarn.nodemanager.webapp.spnego-keytab-file</name>! ]0 y. |1 _% C( B) I+ w
<value></value>
& w# ~& P: x; J- F$ L1 n8 {. z </property>
) w6 p; x! V2 L {/ w <property>
9 |! N6 ~% }& B6 I# L! R( s. @3 G* J/ O <description>! k" E- p$ q6 d9 j' o
The Kerberos principal to be used for spnego filter for the NM web& N" `9 }& S8 a8 I. r& [* V
interface., p+ I( X/ S6 r. u
</description>, w* G0 U& [: C- I5 C. S
<name>yarn.nodemanager.webapp.spnego-principal</name>3 P: }2 G; V( ~$ ]$ U1 Z
<value></value>7 O6 C: @0 P- ?% G5 V: V
</property>
5 f) y/ @' i1 ^5 J8 R% d8 F <property>
* @8 h% E4 V9 G) P- F <description>How often to monitor the node and the containers.; S2 s! y$ G* d. H* Z0 X1 w$ r
If 0 or negative, monitoring is disabled.</description>
8 `9 N% c5 a+ H. V% c0 _ <name>yarn.nodemanager.resource-monitor.interval-ms</name>8 q( i. W1 C6 L( F0 u- ^
<value>3000</value>
1 X1 d. R" n+ ~. ]4 A% n Z </property>* J" b) O8 p8 w! @* x$ } h( G
<property>
+ J; U/ F/ {6 y/ C <description>Class that calculates current resource utilization.</description>& b# ~# z7 _- V* e8 ~1 U/ h
<name>yarn.nodemanager.resource-calculator.class</name>
, Y- G1 b0 z' p4 i( J9 b8 |+ f: |8 ? </property>5 G0 x! ]: y" ]
<property>3 m4 S6 i+ S0 c* v! l+ n" T
<description>Enable container monitor</description>
9 G5 X' N9 H' w8 i <name>yarn.nodemanager.container-monitor.enabled</name>
; U$ j) J6 ~7 c( R$ K <value>true</value>
% S: k3 Q7 V9 v9 p( X9 ~$ T5 _ </property>
6 p) f" l4 K4 {. Z <property>1 n0 L3 K* E% g
<description>How often to monitor containers. If not set, the value for8 V% T z5 `& x' z8 L! w
yarn.nodemanager.resource-monitor.interval-ms will be used.- i- e- T; v! `3 j A/ O
If 0 or negative, container monitoring is disabled.</description>7 v0 c: e' i2 `. X8 H1 M0 _
<name>yarn.nodemanager.container-monitor.interval-ms</name>: _& x/ @+ E7 K$ l+ G
</property>( z4 b8 j) ^- B$ o, `
<property>" D T* E9 A5 G+ M5 q) Q
<description>Class that calculates containers current resource utilization.
# e6 k/ l: g `5 s4 H5 I0 o; N If not set, the value for yarn.nodemanager.resource-calculator.class will
5 N4 C8 i8 _' D: [' @ be used.</description>
9 H/ i) e; x: T/ \6 m7 D0 Z, ~ <name>yarn.nodemanager.container-monitor.resource-calculator.class</name>
$ ]: ~" X, _9 ?3 \! V+ n; t& A </property>3 \/ i O6 Z* X7 {
<property>, D0 i& L( \! g+ x4 [9 @ _
<description>Frequency of running node health script.</description>
- ?7 V; b# u% m$ R. @% L, h& o+ U8 i <name>yarn.nodemanager.health-checker.interval-ms</name>
$ V! X5 e+ I# N# M7 q7 x7 @ <value>600000</value>
9 v M* s4 U! X0 u2 d+ x0 X </property>
! _; ?: |+ _/ Q <property>7 Z. Q) r8 L$ j, D4 m7 y
<description>Script time out period.</description>
: R& t* g' Q Y# K& } <name>yarn.nodemanager.health-checker.script.timeout-ms</name>
' r6 G- `. R6 Z1 m/ F! P9 h <value>1200000</value>6 v- x: l) S+ _4 C, [. e
</property>6 f4 {5 q% C5 d0 w, |% t
<property>8 r1 i* {- K4 r9 C, x- y
<description>The health check script to run.</description>
; X, [* T2 D1 Q5 I$ F G, A <name>yarn.nodemanager.health-checker.script.path</name>
. [( G) K, T0 \# ~. S9 e <value></value>
9 k8 i, @* H9 |9 i </property>, ^ O L, z `* U
<property>
& t( h; K- p% J# c" ^2 h. n8 K <description>The arguments to pass to the health check script.</description>0 o+ G- X# t$ Q- V
<name>yarn.nodemanager.health-checker.script.opts</name>
' D- ]9 J( g. Q8 {3 v) G$ } <value></value>
5 d+ I8 l; [* u$ ^ </property>
" J7 W( N3 i: n/ L% Y <property>; W% M+ S4 e# S b) M
<description>Frequency of running disk health checker code.</description>6 U N k3 b* u3 f7 o( A
<name>yarn.nodemanager.disk-health-checker.interval-ms</name>
) N& p+ {) ^7 L4 ~ <value>120000</value>
. Q4 G1 U4 Y8 q L" }3 R; q7 Q$ ]1 E </property>
9 y9 J/ V' |1 d6 [' Y1 u1 g$ K4 g! T <property>
2 R. A8 ?% O6 n# `* K# ? <description>The minimum fraction of number of disks to be healthy for the
. j5 M4 v- J2 ` nodemanager to launch new containers. This correspond to both- [+ k8 ~1 s0 t& F
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there
' ]5 \6 J4 n* N are less number of healthy local-dirs (or log-dirs) available, then
, U- }6 L' D2 d new containers will not be launched on this node.</description>- a) W; T6 T: f/ a+ ~, H
<name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name># Z8 n k4 i" W6 S0 Z+ K
<value>0.25</value>
; E0 M: E5 P4 L$ ]) s4 i* I; ]3 a </property>
: [4 N1 V% X$ i: y) i4 ^ <property>( P, R) z1 }( K; x7 j, s8 E
<description>The maximum percentage of disk space utilization allowed after
0 t( [$ N7 p& i3 W. k which a disk is marked as bad. Values can range from 0.0 to 100.0. ! l9 w/ h% D9 k: @; o! L
If the value is greater than or equal to 100, the nodemanager will check
6 y5 `; ^' J" h& _# F. n for full disk. This applies to yarn.nodemanager.local-dirs and" N# b, E' @, Z! N" u$ W+ ^! C9 Q( K4 S
yarn.nodemanager.log-dirs.</description>
1 O2 o3 O6 V; e9 z' H <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>- V+ `( ?( y( ^; n$ V) b
<value>90.0</value>/ a c( T+ i1 ^
</property>$ N* ^. q3 j3 e6 d }3 a
<property>' y4 H; _/ T6 P8 l/ \" S3 S9 A
<description>The low threshold percentage of disk space used when a bad disk is
9 `# ~- R7 D) S0 v5 [- W2 j5 z9 U; K marked as good. Values can range from 0.0 to 100.0. This applies to& J ]% T. I5 C) v* E4 T- M
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.. p5 Y/ c% W; A9 [+ f
Note that if its value is more than yarn.nodemanager.disk-health-checker./ X# J/ U; K- p# I# |. V: Q6 u/ C
max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as3 L/ x- G2 _7 M
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage.</description> Y6 p/ t. Q9 u' j& u# |% N
<name>yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage</name>5 E, b7 Y6 e, _
<value></value>, U/ Z1 V% J; A
</property>
: | _+ W7 D4 d w* W# I% g5 h <property>
9 H' s1 J+ P% ]6 h* A) u <description>The minimum space that must be available on a disk for
5 Z* z* w Z: n* _ it to be used. This applies to yarn.nodemanager.local-dirs and* o8 f9 C. G' D$ L( V/ Y
yarn.nodemanager.log-dirs.</description>* X' x7 w: c0 O1 @
<name>yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb</name>
, @, S( Q0 O* Y' b0 W <value>0</value>% `/ m/ Z* {% N4 t6 ~, G
</property> N* h2 p* U- z8 F$ d. _5 o
<property>
2 x/ c2 @. v4 T <description>The path to the Linux container executor.</description>
/ p- ~- F! c8 j! w: r <name>yarn.nodemanager.linux-container-executor.path</name>
* p# y o' o: P/ m/ J8 S </property>$ t- D* x, [) Y1 J* I5 g w/ `5 v
<property>
$ d# j4 X! V% P t) `0 D <description>The class which should help the LCE handle resources.</description>
/ R' Z9 J3 c9 h7 O9 I <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>" Q' a# N3 x$ i1 R; c7 a# m6 A$ L
<value>org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler</value>8 ?0 C: s/ [6 E$ J
<!-- <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value> -->
' T# i0 f& B3 y </property>& h1 Y& s$ n" C7 A6 P0 u0 ~
<property>
& B/ @" n( L5 M <description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas).- b7 \4 L0 @7 y1 S6 j3 `
If yarn.nodemanager.linux-container-executor.cgroups.mount is false. ]8 m" n. A; o' a, u& A6 H
(that is, if cgroups have been pre-configured) and the YARN user has write3 n% R. M! b- p2 Q7 ~. C+ R
access to the parent directory, then the directory will be created.( ]0 x$ `/ T+ {/ Z# F
If the directory already exists, the administrator has to give YARN- y# M+ _& q }3 Z5 R; o: d$ M+ d
write permissions to it recursively.
" {* Z- N* |* q) V v. i0 w! Y2 q This property only applies when the LCE resources handler is set to
1 |' v0 O* c2 ~2 l5 b+ ]7 L( Z. s CgroupsLCEResourcesHandler.</description>
. Y% _) M8 T6 L: i1 B; Y <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
' m) x- R# S, Z, E7 z2 J7 J <value>/hadoop-yarn</value>
- w! X% X$ L( v </property>% D' I( G' e4 i) _
<property>9 _/ z' ~( k- {) _% v+ ]
<description>Whether the LCE should attempt to mount cgroups if not found.* J* k: i f" K% a* o+ U
This property only applies when the LCE resources handler is set to2 s$ Q# q% }2 m T/ |1 R+ m
CgroupsLCEResourcesHandler.
) D0 p. a. w1 N/ q </description>+ n% y! B" \$ M/ e v- P
<name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>/ p2 _: B7 ?" \. q' K) y
<value>false</value>
3 [3 E" u5 o9 |, W8 ]1 h/ l </property>' U$ E$ ]8 I2 j, G; n
<property>
! U2 i5 R0 g. e( y! L. j% j <description>This property sets the path from which YARN will read the) B! }# M X8 O8 S# E
CGroups configuration. YARN has built-in functionality to discover the) [9 G6 u0 \, y* M0 {+ @/ ^
system CGroup mount paths, so use this property only if YARN's automatic2 u5 Q+ n Y4 ~, P! O0 m
mount path discovery does not work.* k" I& M. k+ q; R# o$ H
The path specified by this property must exist before the NodeManager is9 h$ T6 a/ j* Z5 D/ j! x/ K
launched.1 M' v+ e5 M9 [% w
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,
7 T3 C! u4 q) M6 b! N1 i YARN will first try to mount the CGroups at the specified path before
( d- w, Q2 D; [% { G) _ reading them.
( Y' T! f% b" z g If yarn.nodemanager.linux-container-executor.cgroups.mount is set to
; M* V3 a! O. @" \$ F$ P7 _' R false, YARN will read the CGroups at the specified path.
) ?6 ]3 e4 e" E If this property is empty, YARN tries to detect the CGroups location.: {$ V* b5 v# |; m6 m
Please refer to NodeManagerCgroups.html in the documentation for further
9 S" u4 M) Q( U8 s& L$ m details.
. A8 y7 x: C1 Y3 j3 m9 T This property only applies when the LCE resources handler is set to
9 V$ a( r% ^9 e; w1 G9 O3 j7 T* L( c CgroupsLCEResourcesHandler./ K: R7 e1 {$ g: q
</description>
; z3 ^) u. m& K, o# |6 ? <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>; @4 x8 d d, F) v8 j: v
</property>
2 ^" e2 g% X3 v; L6 E. v <property>. h' e# ~ x. e
<description>Delay in ms between attempts to remove linux cgroup</description>! T9 H& I8 X4 K" \- M# R$ o) R
<name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name>5 U2 @( T9 ~0 ^! j% w" |) Y) f
<value>20</value>8 L2 a' j) k! F/ X1 e8 M
</property>* z' {/ Q; J0 l
<property>
- i% P' X: E' Z% B" ?4 F' }" x$ x <description>This determines which of the two modes that LCE should use on W; A9 g6 e/ u' ?% _0 A/ H& n3 N" a) O
a non-secure cluster. If this value is set to true, then all containers9 X$ q5 A$ P: ~: x: }" ]/ a) N1 G: P
will be launched as the user specified in
) E' H( f3 @/ |8 D1 Z yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If: `( a$ f2 ^% }) j# a
this value is set to false, then containers will run as the user who
" B( q* t1 U* N7 B submitted the application.</description>: x! ^7 q! i9 ~
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>2 e! r4 q m6 G
<value>true</value># V2 a. ~! Y1 I# ^
</property>
/ I& p- A8 J0 v% \" r* w2 O: Q <property>% V+ }& S) F2 x1 a8 n
<description>The UNIX user that containers will run as when
6 c# C0 x: p/ K* T Linux-container-executor is used in nonsecure mode (a use case for this
( N) d8 [; O$ y8 B is using cgroups) if the
7 a$ b% ~$ |( M# P: t% } yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is! Q+ @9 g9 x& T) O! ~
set to true.</description>2 p8 r9 i3 x* r
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name> i: O8 t: \4 B3 }1 }. ?/ v1 ^
<value>nobody</value>" ]2 X6 r8 ~+ {+ {, R' O
</property>
2 a H, R [8 J# A, B# A <property>7 u$ h& h! E4 y) W# O
<description>The allowed pattern for UNIX user names enforced by+ u7 b* y. Q& u5 ^2 G
Linux-container-executor when used in nonsecure mode (use case for this) T% P0 X, f5 N: ^, j r
is using cgroups). The default value is taken from /usr/sbin/adduser</description>
* _( M+ b; b; U" B+ { <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern</name>
& X& D3 j% F9 H6 g <value>^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$</value>8 c3 \5 P1 A! K: H# @0 u/ o) _- ?
</property>
8 W/ P* r( t5 ]& A6 H% N <property>
9 A- ^* }0 V1 b8 }7 e' W4 s <description>This flag determines whether apps should run with strict resource limits
7 W2 V8 e0 t+ @6 S) H9 X or be allowed to consume spare resources if they need them. For example, turning the1 W+ Q( G9 U9 h6 O) w
flag on will restrict apps to use only their share of CPU, even if the node has spare
; {/ ]4 e9 v$ j CPU cycles. The default value is false i.e. use available resources. Please note that& {+ v* T( Z( k# y
turning this flag on may reduce job throughput on the cluster. This setting does
) P: P' d' A( m: H5 x+ k& I. J not apply to other subsystems like memory.</description>
0 Q9 Q/ u; L0 R) `) C7 \ <name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>+ ~- M$ M, m7 G1 }4 M
<value>false</value>. V# W3 w4 T' }4 U
</property>+ b( @4 x) _* j$ V
<property>
& W( a2 }5 N% D% F8 k* m6 M6 w <description>Comma separated list of runtimes that are allowed when using3 o. V: o+ z# G
LinuxContainerExecutor. The allowed values are default, docker, and: I: U, m7 U, w# i0 b1 x2 A5 m
javasandbox.</description>0 P7 P' l# F1 {# R: N5 r, P! F! K
<name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
& a( S3 l% y% |! E <value>default</value># W" u% L6 \( S! r
</property>- J% c$ }/ S- t) b# M
<property>* i) K' F9 M* h0 J
<description>This configuration setting determines the capabilities
' W8 \3 \: N5 h( F# F: K assigned to docker containers when they are launched. While these may not
7 e4 N3 T4 R2 m3 B: Y be case-sensitive from a docker perspective, it is best to keep these! D0 }" |( x S/ R# v! K
uppercase. To run without any capabilites, set this value to
3 L& H! J* K5 O/ U* M* M "none" or "NONE"</description>
# T' D1 |; Q4 p <name>yarn.nodemanager.runtime.linux.docker.capabilities</name>9 ?+ X& b6 W' p$ p
<value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value>2 k1 |+ Q q- k0 V4 L
</property>
2 e& F$ a4 Y6 |3 Z7 ?2 S) K6 y; S9 ~3 Y1 f <property>3 t4 J- R# R% A$ e+ `, `7 \- F
<description>This configuration setting determines if+ _0 D1 [% g6 @0 C
privileged docker containers are allowed on this cluster., m9 n3 k# G" ]
Use with extreme care.</description>8 O: v; ~5 [5 f* Y
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name>: z. U3 E+ d7 K2 K
<value>false</value>
/ O$ F$ W7 s' {4 E3 y+ s& E( ^ </property>, U! s* a# I) P) @% J6 I1 M$ v
<property>
2 ]; T5 `3 E( u, q <description>This configuration setting determines who is allowed to run+ V$ |4 j" }+ E; V5 L; }
privileged docker containers on this cluster. Use with extreme care.
7 o. }/ m3 B' M$ x' D; N/ L </description>/ L* i8 y6 K7 N% \; G; x
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name>/ V/ O2 k. f8 ]; l
<value></value> y0 ^2 z ?# X3 t
</property>9 N: \8 a6 M. C1 r4 O9 Y; k' k
<property>* o7 D5 c; O# h8 B6 o m0 T) r" T
<description>The set of networks allowed when launching containers using the
w& \2 f6 s7 v0 o+ a DockerContainerRuntime.</description>3 Z2 _( [! \; k' u
<name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name>8 ^3 l( e' O/ ]. n; R& {% Y+ Y
<value>host,none,bridge</value>/ b2 d/ f2 y& I' f3 z& }1 ^4 Q- C
</property>/ A8 F1 s+ u/ d6 ]! U& f
<property>
: a( k( I G" Q& g8 ~( D6 W$ Z <description>The network used when launching containers using the+ i: i& j0 S( |
DockerContainerRuntime when no network is specified in the request% F, {" e# x% [) j2 _
. This network must be one of the (configurable) set of allowed container
. G; x- ?+ B4 M$ }# ^! ` networks.</description>" Z; B& C& Z- S
<name>yarn.nodemanager.runtime.linux.docker.default-container-network</name>
7 ]2 a2 b4 `% W* g <value>host</value>& e4 J( x1 K7 ]3 ~
</property>& e( q! R% W5 P! Z; L2 r
<property>3 L8 z7 A0 V- k# U2 G. j
<description>This configuration setting determines whether the host's PID! K# r+ }1 N+ h% B4 e! {6 l) e; C
namespace is allowed for docker containers on this cluster.# u) n4 _, H- g: W. d5 f
Use with care.</description>& E! K# R7 N7 b' ^% f: H
<name>yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed</name>
2 V- [. P* _# t0 Y) A <value>false</value>
; T) S* D) i, n) M5 J: M* h </property>
. C( z3 _9 R6 J' I& C" m& X+ C' s <property>
" k6 q1 j4 f$ s# C# k s <description>Property to enable docker user remapping</description>
3 g% @* c& Z" }. O: Y& s7 c <name>yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed</name>
9 w5 F( X2 ]2 Z7 V8 R <value>true</value>& M3 V( [& y# S+ q7 \: u- I# X3 t, s
</property>. {. j" Y0 q% {
<property>/ q& W% K9 e/ E: R) {1 f
<description>lower limit for acceptable uids of user remapped user</description>
; g( y- Y# L1 L1 K& D. C8 C, J# K7 _ <name>yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold</name>! }" v% O. n% n3 x; F4 \" g: K+ b" K
<value>1</value>
7 I6 n5 z% g- j2 W0 Y2 o </property>
1 D( P5 \* h& ]' F4 s- M <property>0 i& l: B. s- g$ C) S
<description>lower limit for acceptable gids of user remapped user</description>
8 I9 g& ?9 j1 N' K9 I# ]( u <name>yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold</name>
# U9 h- [4 w! x7 Y$ z# L" y/ v$ [ <value>1</value>2 Z4 o) s1 z. e2 B
</property>
9 F) I! |. q( ? s% A8 {" i, W <property>
0 N4 l3 z& d g: n% w <description>Whether or not users are allowed to request that Docker1 g: ~' }: h" I" R
containers honor the debug deletion delay. This is useful for
' Y, |/ ~6 J4 z+ F' [4 X# Y troubleshooting Docker container related launch failures.</description>
; |7 p: Q6 E9 u" G <name>yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed</name>
& p) l5 D8 @+ Q! u; ^ <value>false</value>" F% ]+ ^+ x* g
</property>7 z, ^% J( h: }5 `0 H( g0 r
<property>' v& B. C$ K6 K9 M; L d6 v3 c" I
<description>The default list of read-only mounts to be bind-mounted
) ]) Y0 n/ n4 a \ into all Docker containers that use DockerContainerRuntime.</description>
/ C, s( `, [! U) P+ x+ H <name>yarn.nodemanager.runtime.linux.docker.default-ro-mounts</name>
Y$ P! G! D Q <value></value>
6 B# j! }0 p+ x/ L& H# R) B </property>% i: [- M" k# I: v8 b" p! k
<property>( t' a, ?5 t" o; ^ l
<description>The default list of read-write mounts to be bind-mounted
) y( N! N% g) Z% I' V' J v% {1 x0 F9 ? into all Docker containers that use DockerContainerRuntime.</description>
: X' O( L+ _( t+ r4 d1 L6 z <name>yarn.nodemanager.runtime.linux.docker.default-rw-mounts</name>* _/ x3 R) h* o4 ]
<value></value>
) F; X, ~4 |4 U </property>9 j; n; B6 Q& G L& B5 l' o: p' n w
<property>& ~! [0 X6 p G( {- W! B& D
<description>The mode in which the Java Container Sandbox should run detailed by
2 H( ?2 ~4 E" t* g the JavaSandboxLinuxContainerRuntime.</description>4 |5 ^" P. V7 s9 J8 ~: t* g8 X1 H
<name>yarn.nodemanager.runtime.linux.sandbox-mode</name>
2 s6 G, _' S2 \) I6 D <value>disabled</value>
, S& i( ?6 m8 S* ^/ T" U) }( i# [ </property>" S# N4 Q/ C# H6 W9 ]
<property>
/ f) X$ M9 _/ F* | <description>Permissions for application local directories.</description>
$ |: E$ e7 u0 w. h <name>yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions</name>
7 \6 ]" _( d1 {) l <value>read</value>
: t4 `, D' o3 @2 J/ t/ h4 m. t7 x </property>8 b c, s2 ^% U; ?4 C
<property>! Y3 d# w. F/ r- F- j5 U8 p3 U0 Z
<description>Location for non-default java policy file.</description>
' J/ Z1 C9 I- U5 w5 Z* L! A2 P <name>yarn.nodemanager.runtime.linux.sandbox-mode.policy</name>
- @5 M9 p( a5 v: ~# Z2 {5 }) n <value></value>
2 r7 z8 u3 ^+ \* q- l$ ?8 N4 O; F: h </property>
; K, K/ c4 o3 E1 D" b <property>
+ e2 z6 X$ n0 t% W: m4 F* D# m' S, v <description>The group which will run by default without the java security
7 r0 G, Z, g/ w; j. q manager.</description>
/ R% Q' P0 u# r+ J% V+ S% c <name>yarn.nodemanager.runtime.linux.sandbox-mode.whitelist-group</name>
, W: }% ~6 R$ ]' n8 E' } <value></value>
0 k: F6 X2 R7 i/ y) y' x1 L9 L9 ~/ A </property>
: M. q! F( G" i8 O2 h. k& K6 ] <property>
; ~$ i( b5 V- ? <description>This flag determines whether memory limit will be set for the Windows Job0 ^8 e* H8 T$ K0 M% D$ a: [
Object of the containers launched by the default container executor.</description>9 w J( |) W0 T
<name>yarn.nodemanager.windows-container.memory-limit.enabled</name>0 D' |3 O* s8 g' |" T0 R
<value>false</value>6 y2 ~) j- i5 W. k6 {& G/ v
</property>
8 V, Z2 J/ t7 ^3 o' b) R <property>6 _! W4 K/ g, y( m, v
<description>This flag determines whether CPU limit will be set for the Windows Job
0 a% A( {" A% e2 f; T/ ~ Object of the containers launched by the default container executor.</description>9 u5 U' |8 ^9 @
<name>yarn.nodemanager.windows-container.cpu-limit.enabled</name>
0 F ?3 e- x: I3 n5 H <value>false</value>6 p X7 g7 P* ^4 P. a
</property>
7 a4 S6 ~# x' z% i <property>
! E+ `/ \! H* g <description>. Z% _. ~- P1 Q! {# s
Interval of time the linux container executor should try cleaning up
: Z* ]5 B& t) a1 P ]) o, m cgroups entry when cleaning up a container.
8 V) v# J( m$ {6 P7 o </description>
+ r2 P0 F6 Y' Z, d7 | <name>yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms</name>
( m- d5 ?4 V; h% K$ l: Y( h <value>1000</value>
0 F0 L' q/ g+ l3 T& }/ G4 ?; D# u </property>! h8 B4 t* |' W6 O. J
<property>/ b% z, x n* J- h" G3 p- a
<description>& R; X3 ]9 W3 b S
The UNIX group that the linux-container-executor should run as.* Z1 ?' l: r5 |# {
</description>4 `0 |) ?: I, T5 _) V
<name>yarn.nodemanager.linux-container-executor.group</name>
|* Y! o; `. g0 m/ _ <value></value>
: ?" k! I1 y$ n& i </property>
6 C0 h, ^: x A4 [ <property>
6 R# F6 p2 n# q <description>T-file compression types used to compress aggregated logs.</description>
: q3 j4 }1 C0 B$ U1 S8 C0 i; Y <name>yarn.nodemanager.log-aggregation.compression-type</name>
' B& n" R' B& P) M; h. J <value>none</value>
9 E! h, k/ h z h </property>* V4 t4 O2 @8 c" I, R
<property>
; w+ p6 @7 _4 X8 t6 z) c; b3 ? <description>The kerberos principal for the node manager.</description>
+ O" B$ w4 @! v" Z; X$ v1 s <name>yarn.nodemanager.principal</name>
) R2 w3 g' s( E. k f9 W" P+ @ <value></value>
" J. S# I- M" Z& f" y </property>6 G; Q6 v9 D! ]6 f9 T
<property>0 `2 {6 a* k- m
<description>A comma separated list of services where service name should only
& \( D/ U8 `. l N" @: m. ~) _ contain a-zA-Z0-9_ and can not start with numbers</description>! Q5 q6 J# H3 m( _. C. t, B7 i' y
<name>yarn.nodemanager.aux-services</name>+ }$ r% p" G6 W! D
<value></value>9 y2 }1 V, | g% q* Q
<!--<value>mapreduce_shuffle</value>-->, G$ a# Z3 d5 F+ b ?+ {! w. e0 c
</property>1 e. Y2 [& v; Y- N# x& k$ v$ F
<property>' s2 L& V7 U7 n" Y3 W/ r; |
<description>No. of ms to wait between sending a SIGTERM and SIGKILL to a container</description>. J# F/ o5 {; C# J
<name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
/ O4 y6 t3 s+ q/ [8 h2 i+ \ g <value>250</value>
$ y& |, \' ^# f$ w! h0 q" d4 Y </property>
3 Z4 a: A$ U6 l: J( N# ], x" X8 t <property>. R) E4 c& b8 V" M9 Z
<description>Max time to wait for a process to come up when trying to cleanup a container</description>! m1 U, P+ ~" ~8 G! ]
<name>yarn.nodemanager.process-kill-wait.ms</name>
+ c& a; r. h2 a C# V. ]" L <value>5000</value>
$ n3 x7 H0 ~$ P, i0 Y8 U </property>" E' \6 Y/ }) S
<property>
. ] {/ G; X8 O. x3 R9 e <description>The minimum allowed version of a resourcemanager that a nodemanager will connect to.
8 S( ]4 U9 m7 _) G/ _4 F The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is
1 r4 K$ ~# g+ N$ d equal to or greater than the NM version), or a Version String.</description>1 | a4 d) J* s( f
<name>yarn.nodemanager.resourcemanager.minimum.version</name>7 ~3 ?9 @2 K0 l0 @/ S3 g6 {
<value>NONE</value>
9 a/ Y4 o$ B0 o </property>
' d, B7 i% O/ r5 w <property>, r/ n! C( f7 ] J J8 ]
<description>Maximum size of contain's diagnostics to keep for relaunching
: L6 K1 z& m2 O* y container case.</description>
% r0 @9 [4 _) D& R1 D$ i; I# w <name>yarn.nodemanager.container-diagnostics-maximum-size</name>; H f' b' W) M6 W) r) ?
<value>10000</value>
3 O7 U X- h7 M+ c7 f8 f </property>% @( X7 T* B* i; _% V. z% J
<property>! }+ Z# z x2 b" D/ e6 P( k
<description>Minimum container restart interval in milliseconds.</description>
Q" G0 G& L3 F3 Y7 ] <name>yarn.nodemanager.container-retry-minimum-interval-ms</name>
, Y/ q/ N6 Q: m) A4 W: _" U <value>1000</value> p& D# ?3 u' R0 b
</property>
5 q# R$ @ k0 n- s) S <property>4 R. Z5 d0 R; _9 W) y- ]
<description>Max number of threads in NMClientAsync to process container% Q0 j$ G0 B3 m' J( o
management events</description>
" k, s6 ]8 c5 |0 x <name>yarn.client.nodemanager-client-async.thread-pool-max-size</name>! K+ E: v$ h& y2 R
<value>500</value>
6 c* e* g) V: ]$ M' l- A </property>
/ m4 K7 N6 @ \; Y9 a) h: g <property>
* X2 K, R0 W+ M6 s2 C! A- E3 D <description>Max time to wait to establish a connection to NM</description>
* ?$ @8 {( G+ p3 z# u <name>yarn.client.nodemanager-connect.max-wait-ms</name>
0 }9 R, I( {4 I& M0 y; V <value>180000</value>) h8 x8 @! h" @* _9 ~5 o) i
</property>. _ V2 L H& i- D5 B: e% A
<property>
6 }4 r: l& ~' R+ m <description>Time interval between each attempt to connect to NM</description>
" k. q& ]. A- o6 Z1 V <name>yarn.client.nodemanager-connect.retry-interval-ms</name>
9 U5 Q7 Y7 _% \0 w/ H6 j/ `+ b' _& m! _ <value>10000</value>
& m0 Y1 H2 `' f5 |+ ]& _+ f1 P </property>: D. h1 \- f; u& v, Z- y. M# u- k
<property>4 T1 m- c0 `( F7 E% p& B
<description>; q) h; L9 J/ G0 W" A4 G
Max time to wait for NM to connect to RM. g+ v5 W4 W/ {$ o) p1 C
When not set, proxy will fall back to use value of
_1 M) k. m" E& A yarn.resourcemanager.connect.max-wait.ms.
6 j! u; M, ~- G1 t </description>: j* f7 R3 \' ?1 B- G
<name>yarn.nodemanager.resourcemanager.connect.max-wait.ms</name>1 z: {& X1 A* B. b( H# F
<value></value>
7 s. A/ h& |9 O& J9 ~ </property>
! e+ i; q+ A, A9 u2 b% Y9 {& I3 p <property>
% [% l1 K8 z2 W5 ]% ? <description>
7 m. P' J8 A1 R Time interval between each NM attempt to connect to RM.
1 v* a# P3 u" K' R When not set, proxy will fall back to use value of
/ o: h" G* ?7 d" c1 I, t' g) ] yarn.resourcemanager.connect.retry-interval.ms." Q6 G" M6 z( o& S
</description>
- w" z. A4 J8 N: p9 ^ <name>yarn.nodemanager.resourcemanager.connect.retry-interval.ms</name>
: q& Z+ E/ K' y9 N6 F$ ` <value></value>. n0 `1 C: m. Z8 Z' |2 V7 {
</property>$ h/ c8 | P8 u" m2 C* l8 ?2 M
<property>( K& C! o3 g- C. W* P% C
<description>1 U0 B# ~% @0 F0 h
Maximum number of proxy connections to cache for node managers. If set
% p9 L3 F [! R8 {; D to a value greater than zero then the cache is enabled and the NMClient9 K$ ]! i; e. L8 e8 }/ d3 }
and MRAppMaster will cache the specified number of node manager proxies.; H9 I& u3 F! V
There will be at max one proxy per node manager. Ex. configuring it to a
8 U/ G+ N# i. W, i1 v value of 5 will make sure that client will at max have 5 proxies cached! L' W% p$ A# A/ u( q) G
with 5 different node managers. These connections for these proxies will
+ I0 Q0 `+ {& M7 N, ~ be timed out if idle for more than the system wide idle timeout period.! N9 C3 W; n" |" ]& F. f3 L' v% `) t
Note that this could cause issues on large clusters as many connections
2 D. H, ]! c6 ^' `7 B) L/ U could linger simultaneously and lead to a large number of connection
7 l( Y4 B9 ~! g4 _* m8 [" E threads. The token used for authentication will be used only at
6 O- v; {6 ~/ [ connection creation time. If a new token is received then the earlier
9 ?& r! h& s" }, a1 v connection should be closed in order to use the new token. This and
% ]' y) m* b. ?# E6 Y (yarn.client.nodemanager-client-async.thread-pool-max-size) are related$ ~* ?& F# J& u4 V1 S: F% d. ?
and should be in sync (no need for them to be equal).# K5 i N5 W f. G* ]7 G
If the value of this property is zero then the connection cache is
* J! D: g! V+ J disabled and connections will use a zero idle timeout to prevent too
( x8 b! k* T9 K& N8 X2 [1 L many connection threads on large clusters.! f' q3 u H( @' w7 l7 g
</description>. e- U) s2 U7 J$ l3 d Y
<name>yarn.client.max-cached-nodemanagers-proxies</name>5 P( j+ K5 S8 i' J6 L1 ~# {9 W
<value>0</value>& O: U/ F; m+ c0 e# S9 l
</property>
! Y+ }7 J& n9 |( o <property>
; }7 i# g4 @ r, A <description>Enable the node manager to recover after starting</description># t6 C# ~! j: {. p+ y, k
<name>yarn.nodemanager.recovery.enabled</name>
- j' J1 B1 `5 o' a- {( L6 i4 V <value>false</value>
7 y5 O6 _$ R- K8 U3 q" J </property> w" y9 v) O8 X# N: n0 X
<property>
/ \% q9 `2 q/ Z# O+ ?- n( W/ [ <description>The local filesystem directory in which the node manager will+ E) p0 o# u1 ~/ n( \
store state when recovery is enabled.</description>2 b. |3 S9 Z a& Q# {. k
<name>yarn.nodemanager.recovery.dir</name>% O1 e2 h/ {5 y+ @" T* m
<value>${hadoop.tmp.dir}/yarn-nm-recovery</value>/ C, I" e. M7 i4 @. C8 ~
</property> f1 j" g6 X- D( [( Q
<property>
9 p" u, A0 @& e; U) u& M8 u <description>The time in seconds between full compactions of the NM state! s8 X/ H# U3 l7 ?
database. Setting the interval to zero disables the full compaction+ T/ f G( y/ d6 [, x+ ~8 v, @8 b
cycles.</description>! f" S3 r" e7 F0 U+ _0 u
<name>yarn.nodemanager.recovery.compaction-interval-secs</name>5 n. ?8 N1 H% R7 [7 [- z7 r
<value>3600</value>- a; @! Z' v; U3 _4 F3 B/ z7 N
</property>) r- H7 S: Z) l9 \7 l, K8 {9 P
<property>
( a. w/ C, |7 J. ^ <description>Whether the nodemanager is running under supervision. A
/ f! Y' b; s! O nodemanager that supports recovery and is running under supervision
+ ^0 L3 b. K& N9 q w' z/ Z& Z will not try to cleanup containers as it exits with the assumption. K- F7 J. ?# G6 I
it will be immediately be restarted and recover containers.</description>
% t0 H: A# a u4 w L9 c <name>yarn.nodemanager.recovery.supervised</name>
& Z' D) ^5 S8 l& Z, k: \ <value>false</value>
5 ?4 c) Y( Q/ F- b& @& O </property>) H0 t3 _/ }- B' A
<!--Docker configuration-->
3 j( O# h) k1 o6 M; @/ Q9 H <property>7 b/ ?6 u( z* \! W# o" y' f
<description> ]: t! m2 ]! D0 [; @. [6 g
Adjustment to the container OS scheduling priority. In Linux, passed
. A3 Z" t1 j* K& q! E7 h$ m/ h3 T directly to the nice command. If unspecified then containers are launched K0 N' W4 X# f9 k4 T3 m
without any explicit OS priority.
J. J2 e* t9 k+ x! e7 e/ e6 j# h </description>
: X, s3 f9 P9 P. m% J <name>yarn.nodemanager.container-executor.os.sched.priority.adjustment</name>
7 R: k9 o& y9 P; I0 W </property>1 n1 [# T/ [5 M- q3 O
<property>
! x T5 x" [1 q, S: e <description>: [; Y: l1 W0 u8 j* z5 H7 T
Flag to enable container metrics& Q7 ?8 ^+ h: s
</description>
1 G/ o8 U, ?# H/ t <name>yarn.nodemanager.container-metrics.enable</name>
0 t4 R6 H7 R' Z4 z, ` <value>true</value>: W; K" \0 Q6 Y6 a
</property>5 m1 q( G+ p0 `2 a! [
<property>" O0 y1 C0 l7 V: E
<description>; S8 o$ H9 ^! _" D: A9 B/ _
Container metrics flush period in ms. Set to -1 for flush on completion.8 v* G6 W$ ]: N1 P: A- M. r( n; z
</description>, J ?% |7 n4 a% S- L F
<name>yarn.nodemanager.container-metrics.period-ms</name>
& T0 j1 {5 L6 {. d% v( _0 G! E9 Q <value>-1</value>' T) {* [% O' |6 @; d
</property>
" ]& D' |& ], E8 ^) ] <property>. C1 z! m9 ~2 Q1 l7 u3 A
<description> G% M: \4 G% x/ @
The delay time ms to unregister container metrics after completion.+ F" ~* x* i1 E4 R; `, q# S2 c
</description>) R0 h2 j: x5 c& W: u% }. C" z# i
<name>yarn.nodemanager.container-metrics.unregister-delay-ms</name>
! U$ V" V' Z- J8 |2 g0 F( R <value>10000</value>& I' N. y; `: j$ ^# X6 Q; y: @
</property>
5 q; r5 G7 K& v3 @ <property>& Z, }; t9 ?; L
<description>0 D& I1 l) r' d! h6 m! i
Class used to calculate current container resource utilization.' `! A$ B( b$ S( S& w/ V4 N" L
</description>
! q$ z: W3 O, G* K6 I( P <name>yarn.nodemanager.container-monitor.process-tree.class</name>
9 a U, P; E8 O) ]: ] <value></value>
' m' q q3 D7 x# d2 m- p </property>
# V% E- C% K* j; ]: a <property>3 p: u2 s& u% w/ W
<description>
# } }. ~: l+ w- ]) V- H4 G2 J# N+ N Flag to enable NodeManager disk health checker; k/ j1 b% G( L$ P+ ?
</description>
# v- r* ?" d4 `) n& k+ _ <name>yarn.nodemanager.disk-health-checker.enable</name>3 G8 W4 c; z' l: D) `
<value>true</value>- @7 q. ?) A5 o* z9 m; Z2 n
</property>
5 v- N6 g7 Y8 y1 ^8 q0 c1 M- i <property>
; P/ a' x& A: e: N6 y, L2 z <description>
; r% l* o$ n) D# |6 y Number of threads to use in NM log cleanup. Used when log aggregation
# v" @/ [+ S8 U) \# h2 [, ] is disabled.# C- w8 ]$ q7 x" k
</description>1 N ^7 | y% {/ X
<name>yarn.nodemanager.log.deletion-threads-count</name>6 R0 X# ~% T! Z7 z; R" F+ X4 L
<value>4</value>
; R ~7 X' G& M! d6 r8 M </property>
5 r2 k" `$ U: g5 B$ W <property>
$ P: n. @2 G" m4 N) [# @ <description>. g& M+ X! q. ~2 E( G3 n; b
The Windows group that the windows-container-executor should run as.
9 Y$ B) W. s3 B ^9 H% P0 E </description>
, H; m' K, } O <name>yarn.nodemanager.windows-secure-container-executor.group</name>
" G, B' x0 z% X/ U7 p% H7 \5 \ <value></value>
; q/ Y9 Y) Q, ~9 P </property>( W) J+ e' r; U. `9 A) K, T
<!-- Map Reduce Configuration -->
3 z' F0 c# B7 X& o, E4 i6 C/ q" T/ T <property>
3 J! u8 B* C( i: R3 \ <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
9 {% X) ]( D/ V }. K% o y <value>org.apache.hadoop.mapred.ShuffleHandler</value>7 B/ J: ?& i6 X( }: [( X
</property>) E5 X& b; M9 O9 F
<!-- WebAppProxy Configuration -->
1 L$ t0 Q$ R a( O4 S+ I& S <property>
3 R1 l7 v2 O1 G7 z <description>The kerberos principal for the proxy, if the proxy is not& K( R$ \9 |. x1 q
running as part of the RM.</description>6 O. C1 r" k# v6 m
<name>yarn.web-proxy.principal</name>: p# ~/ K/ w. f
<value/>
8 T$ W0 X1 f/ H N; J2 V# | </property>" x/ a& ^- z6 H) N$ c
<property>
( z, V' v$ ?, \* F; W7 T* r5 h* X <description>Keytab for WebAppProxy, if the proxy is not running as part of 0 k4 h. o# m% x! w' @* f
the RM.</description>4 d! Z8 e: Y: k
<name>yarn.web-proxy.keytab</name>
4 t2 Z8 o7 O0 B. U </property>9 y/ I- e/ Q4 g2 N
<property># U! l7 D9 n. F
<description>The address for the web proxy as HOST:PORT, if this is not! z6 G( ~0 v4 d5 A5 T( o5 o
given then the proxy will run as part of the RM</description>3 \( G0 `8 N9 b; r# G6 @
<name>yarn.web-proxy.address</name># V1 Q. _" Z2 b
<value/>' |. q. U8 ?' @$ D$ N/ H' e% S' m; H( C
</property>' s- j0 q4 ]3 r [. h3 G3 v7 Z, u2 w2 M
<!-- Applications' Configuration -->1 T! H/ }% t/ G: [0 D' [
<property>% |* L9 ]" e) j( j& i4 P
<description>
L6 f$ b: a4 _( l: h" E7 y CLASSPATH for YARN applications. A comma-separated list/ u+ P: l2 P7 ]( ]; Z3 i
of CLASSPATH entries. When this value is empty, the following default
9 }/ f9 F) S. h6 _ CLASSPATH for YARN applications would be used. ) w# ?; _3 N3 u3 G5 [% Z
For Linux:
$ A; \$ S, o4 {/ n' I' M $HADOOP_CONF_DIR,6 c& N9 P/ e' A
$HADOOP_COMMON_HOME/share/hadoop/common/*,
* @5 X5 s6 h! M+ q$ S8 U+ _ $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
, b2 s- e9 A2 c" T9 W $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,) g+ p% Q9 B& O$ C' k; C9 E6 U+ G
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,6 N6 [& ?" g; w
$HADOOP_YARN_HOME/share/hadoop/yarn/*,, e9 A$ @' `0 E. m" y
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
3 d k# U* z% s+ ~9 K For Windows:$ a" R1 z! d" r9 z: ]$ i4 w8 \
%HADOOP_CONF_DIR%,
) @- Y/ p4 F0 [5 v %HADOOP_COMMON_HOME%/share/hadoop/common/*,9 b) l5 E; X8 A! c
%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,) f8 s9 t& }0 g& I+ E! t5 P
%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,$ v* m! b. j9 S$ w2 d1 P W1 p2 I
%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,
) j" y! } B. I8 B" |0 L %HADOOP_YARN_HOME%/share/hadoop/yarn/*,5 K+ f- e4 Q$ {6 N$ C8 ^, b0 j6 R
%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*" j$ ^: k5 O: A* s5 C
</description>0 c& W8 v M: J- _8 o
<name>yarn.application.classpath</name>
! |% {2 i5 Q4 q/ W+ M9 M, g <value></value>
+ k+ y Z. B u7 P* X7 b! o </property>
% \7 i7 |( S! ~* q( ] <!-- Timeline Service Configuration -->2 c( L, l1 T) H: s
<property>) q8 V+ p* W/ I6 [8 Y2 }7 F, I8 A
<description>Indicate what is the current version of the running' l) h7 v. J/ X- R5 S- L
timeline service. For example, if "yarn.timeline-service.version" is 1.5,
5 \9 P9 k( t9 d$ F and "yarn.timeline-service.enabled" is true, it means the cluster will and
% P" W* f ~, R4 o4 H) p% P: H should bring up the timeline service v.1.5 (and nothing else).
% t6 y2 T B) D5 j8 J9 o; U* B On the client side, if the client uses the same version of timeline service,
. N9 j0 K" H/ p0 A4 v' R it should succeed. If the client chooses to use a smaller version in spite of this,
6 K8 |5 y& z. b- Q, J then depending on how robust the compatibility story is between versions,
( R$ p3 S" j# e- D& D7 B- U the results may vary.
+ J) k* v, s6 X4 K' A4 @ </description>
3 X0 E1 a# G$ f8 g% Z <name>yarn.timeline-service.version</name>0 n. x7 `* U( x) Y7 f. b7 I
<value>1.0f</value>
, u0 q, n2 z7 q0 V5 M+ y' g </property>, m+ ?+ q: M( J8 c5 K
<property>
. @+ N+ E0 u ]3 } <description>/ B4 H* g- _9 b- ~* L+ i. X. p6 ?
In the server side it indicates whether timeline service is enabled or not.
2 B& r+ D5 a+ m8 F And in the client side, users can enable it to indicate whether client wants& d( d9 C( ~: l% h4 }/ Y8 J
to use timeline service. If it's enabled in the client side along with" t! \6 N" l8 Z w- @
security, then yarn client tries to fetch the delegation tokens for the9 C; Q& l% _% |6 B$ ]
timeline server.9 x0 W) p( h7 _0 M
</description>2 E+ ~) p9 u9 W x$ [
<name>yarn.timeline-service.enabled</name>
$ B& [$ G/ ?! u$ X8 l, g2 y2 q7 _6 J <value>false</value>2 R# n* ?. G) F) j8 S$ x' j
</property>
. m/ V. G' t1 x% p9 V# X <property>( Z* O( g% u3 y% Z1 ]) y
<description>The hostname of the timeline service web application.</description>) H9 G/ Z! D- p. T$ r3 b/ l* e
<name>yarn.timeline-service.hostname</name>
, X1 Q2 U3 a% N' u {! f# R' E <value>0.0.0.0</value>
C3 v' x; L+ ]( b& u( J! D m </property> Q9 c9 Z) E0 o
<property>, J) H, F4 K6 _/ E+ Y+ [- @
<description>This is default address for the timeline server to start the
, s: D) a* J. X+ P RPC server.</description>0 M5 [3 _5 k5 m( J% m7 u
<name>yarn.timeline-service.address</name>8 F4 x4 f4 G# u
<value>${yarn.timeline-service.hostname}:10200</value>$ r2 r8 m$ @: L- k# Z1 Q
</property>
6 _7 B6 a0 J3 Z8 A* |2 |' e <property>) L/ I( `" ]8 t9 c
<description>The http address of the timeline service web application.</description>4 w# e# N, r2 a. G& i+ T
<name>yarn.timeline-service.webapp.address</name>
5 Y5 q& ^) i, l4 k7 [, X8 j <value>${yarn.timeline-service.hostname}:8188</value>* ]7 Y4 m' [$ Q& }
</property>
' v! A: _7 Z- j, f7 g <property>" B' s6 r S5 S1 U7 F
<description>The https address of the timeline service web application.</description>7 A3 H8 {" ^8 u* b
<name>yarn.timeline-service.webapp.https.address</name>
& D$ H! [' _6 g) s5 Y, G <value>${yarn.timeline-service.hostname}:8190</value>
. T: Z' d6 |" A* Y </property>
8 b# g: {0 K: A+ a/ {# a: e <property>
$ b$ J; U8 E6 B! J4 X9 | <description>
, A' y+ A6 _7 E) t o0 N The actual address the server will bind to. If this optional address is
6 u3 [! U9 I7 I" ~ set, the RPC and webapp servers will bind to this address and the port specified in
/ \9 Z8 `1 ?3 m% h/ M. d yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively.5 ?/ \3 c0 j! v" w" @3 j
This is most useful for making the service listen to all interfaces by setting to
& @2 x' O& R7 m% [ J 0.0.0.0.% y, B1 ^- B. V" Q: @
</description>% @& {9 ?2 A: ?* \( e% L
<name>yarn.timeline-service.bind-host</name>' |. O4 e5 _6 }
<value></value>
8 J* L* N# D$ e# A) ~ </property>
5 v: p. u* ?# m# \2 x <property>; l) o3 W; Z- e4 _! L
<description>
* G q( c% l0 U Defines the max number of applications could be fetched using REST API or
+ [' t# l9 r5 E1 m application history protocol and shown in timeline server web ui.
/ i8 @6 f4 N7 c2 n& d+ z </description>( B; J2 c' H, b# E: R9 k. o7 l' A
<name>yarn.timeline-service.generic-application-history.max-applications</name>: s# H8 N5 Z8 [
<value>10000</value>! b, q; Q+ J1 ?* V9 `
</property>
5 ^* U* z, v' S+ M <property>. {9 b P. p: c( H2 @
<description>Store class name for timeline store.</description># `$ s1 {5 H0 E8 K K V1 ^8 D
<name>yarn.timeline-service.store-class</name>' |2 B5 k' R1 p+ z6 o- e! u' v" o( L
<value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>9 A' i% {7 J6 C J' g8 v" k
</property>
* H3 U' V3 ?: l" O <property>
z; y/ K5 N2 }- g <description>Enable age off of timeline store data.</description>+ z# O. l, ^7 [7 I
<name>yarn.timeline-service.ttl-enable</name>, u( v! R$ ~' j$ L* w
<value>true</value>" l2 I0 y( W9 k6 T- U! w
</property>9 O7 L& _3 Z6 c9 D
<property>5 d, R+ D7 g* P: y; [$ @, q
<description>Time to live for timeline store data in milliseconds.</description>7 `( p6 e. S8 E! }
<name>yarn.timeline-service.ttl-ms</name>8 C5 D# ^# R! C/ k. X( t0 Y; s, t
<value>604800000</value>
, x& w7 O' d( T9 ?6 t- v$ Y U </property>; M8 F, u2 O# e
<property> `" R2 H4 m! Z* L1 J8 c3 o
<description>Store file name for leveldb timeline store.</description>/ |( R/ u) v+ K/ n- `* H5 R3 d
<name>yarn.timeline-service.leveldb-timeline-store.path</name>
1 c! x- G' Q4 J. V% H/ F G <value>${hadoop.tmp.dir}/yarn/timeline</value>
; t4 [; ~9 ~( s </property>9 J h' A' \; x! r3 }
<property>% H+ f, U) d9 ^% n( K% ^! r9 J
<description>Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.</description>$ O6 \# p: b+ b9 T/ |7 l& F* m$ s
<name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name>8 ~9 R* ]: @$ C" z
<value>300000</value>* x, f8 v: V: j" S) _' S; a% Z
</property>
- p9 t$ K5 P0 x' m <property>& [2 R- D8 ~* V* `
<description>Size of read cache for uncompressed blocks for leveldb timeline store in bytes.</description># q7 q& c" D6 {# J3 @9 T
<name>yarn.timeline-service.leveldb-timeline-store.read-cache-size</name>
/ T' @( `" w: o) M$ d <value>104857600</value>, B$ ~* ?% Z4 y1 S+ K
</property>2 y; H- `! z* f" Y* a
<property>
4 |& l/ y6 |, w! d. l3 i/ ] <description>Size of cache for recently read entity start times for leveldb timeline store in number of entities.</description>0 z2 ^( C4 N2 h. `9 @: L( B1 d4 |
<name>yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size</name>
2 H0 o+ l4 F2 j <value>10000</value>5 e5 K7 Y) [6 o* [# M
</property>
) F4 S, x: @& a3 n4 U5 R0 ^9 ] <property>: p% V! j% u5 [; b& ^
<description>Size of cache for recently written entity start times for leveldb timeline store in number of entities.</description>, ? @# x( w7 D( i. O; D/ y$ I
<name>yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size</name>
2 g( m& w" h2 A1 T; ?0 _ <value>10000</value>
/ D! n/ e9 k2 [) P$ o; D% p2 m </property>' p( R, @: o* ]* B
<property>; X, \" G! t" J! o/ {
<description>Handler thread count to serve the client RPC requests.</description>* n, s. c2 R+ s# Y( D$ Q$ f
<name>yarn.timeline-service.handler-thread-count</name>
c- u. c3 z1 i <value>10</value>
6 {8 c7 W/ |" ]; K# Z2 c </property>
* _/ r* c* H) M8 M G' J <property>) X- z6 J P; ?" ^4 ~! [
<name>yarn.timeline-service.http-authentication.type</name>
, c6 u7 \# x0 \ <value>simple</value>/ a0 A, d3 X2 t* ?
<description>
% G H$ y; e% e3 |; O4 r Defines authentication used for the timeline server HTTP endpoint.
" _) J9 S/ S6 }7 z Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
: ]9 ~# I" O) W6 Z </description>1 D+ T" R' ?/ x+ p' ?; s1 |
</property>: E) v( ~% W. g, R9 U/ l
<property>1 Q+ u0 r8 s: x0 {. i
<name>yarn.timeline-service.http-authentication.simple.anonymous.allowed</name>, o! e2 W1 A: a
<value>true</value>/ i) c1 o; D1 Q$ D6 O
<description>/ V: B: \) l1 T( H, F9 _4 [2 n
Indicates if anonymous requests are allowed by the timeline server when using! _4 [( l- [* h. l% L& M
'simple' authentication.
. b# b% D! q/ { </description>
6 I y- ~; U# @6 g8 D/ W( p& r </property>! W# Z& m! k& F7 \
<property>' W/ u' g$ g* L, g$ h
<description>The Kerberos principal for the timeline server.</description>
5 u6 C8 m" y) R- q# {4 I" e <name>yarn.timeline-service.principal</name>
8 e/ _ N) P8 |6 N l <value></value>" m" t% F9 s" \$ R$ |
</property>9 ~- A6 B/ O) m3 Y
<property>
7 S- q/ `& j0 b W2 s <description>The Kerberos keytab for the timeline server.</description>
& a* f7 A2 n/ v% S; s <name>yarn.timeline-service.keytab</name>& I, O. y. k! @3 t- ~3 n
<value>/etc/krb5.keytab</value>) a6 Y4 g/ @% b5 a R) O- M
</property>
$ S' B- B f: \ <property>7 @) Z+ G; Y5 Q" _
<description>Comma separated list of UIs that will be hosted</description>5 J ]7 j7 B9 W h' Y, m
<name>yarn.timeline-service.ui-names</name>
2 y4 _6 _' }' b. \ <value></value>& s O, r! D& k& q& ?5 ^. d% {4 L$ T8 E
</property>! D# r7 ?) r0 h1 X- I" Y5 W
<property>
0 H1 P4 R: j4 P <description>7 G6 F! z; w3 J" \1 l0 ~
Default maximum number of retries for timeline service client7 S: ?$ y. ^3 N* M6 r
and value -1 means no limit.6 w6 G+ M/ A/ D s
</description>
) j$ j2 F! u8 Y6 q7 i( Z5 M, y <name>yarn.timeline-service.client.max-retries</name>
, Y* v | x g7 { <value>30</value>
' i/ K) F+ O/ {1 Y </property>
) e3 [* f$ p l8 Q6 _- F) m <property>6 j" d, }( z$ c+ ~
<description>Client policy for whether timeline operations are non-fatal." w" z: ^5 Y! d: h
Should the failure to obtain a delegation token be considered an application
8 G" G) m- F8 c) G# P" J failure (option = false), or should the client attempt to continue to
! Z z; O/ V k; v- q+ V publish information without it (option=true)</description>
& ]% n5 n5 t+ g% B& }. \8 P <name>yarn.timeline-service.client.best-effort</name>
. L* m* E" ~; \: U4 C <value>false</value>4 C: v. `- E4 E1 G8 |- h# v: b
</property>2 u1 T% j$ N% {* @" }# ]- G) I
<property>% r2 N6 R1 K* [' j7 y
<description>9 |+ J( e" A9 l% N. P
Default retry time interval for timeline servive client.
) t. }4 o$ A' m! M3 M </description>
. r" C. }5 O! d- _- {" j <name>yarn.timeline-service.client.retry-interval-ms</name>
* s$ q4 ?/ }' B' B7 y <value>1000</value>& D% D' Z9 s, @
</property>8 @' T4 `/ w" f: O9 }1 K, R% d
<property>/ i: z3 c v. V6 }
<description>
5 r" Q6 f6 k! Z/ I The time period for which timeline v2 client will wait for draining- k2 z5 G$ A2 \3 P( g) K! n
leftover entities after stop.; B# x: z" {$ Z% a1 Q% M
</description>
+ X4 r& @6 f6 }; }5 ^7 H# ] <name>yarn.timeline-service.client.drain-entities.timeout.ms</name>
7 }( j8 |* B3 y8 F <value>2000</value>
3 J- Z) c! w" J </property>5 G6 r7 k' ]8 s. h/ T3 ~* k
<property>
5 m$ `. T. W w# s9 T9 M. `. ^ <description>Enable timeline server to recover state after starting. If
- I( | w$ A7 n* T: Y9 T! ~ true, then yarn.timeline-service.state-store-class must be specified.. Q7 ^) W/ O4 k
</description>% o0 _, s$ r) c1 A j
<name>yarn.timeline-service.recovery.enabled</name>
9 y3 D; M0 `) G- Y1 A <value>false</value>
) i- j; J6 [" G5 D1 {% {' p </property>. q; x0 L- Y3 E. v" h
<property>
, P7 k$ t1 A0 x- G <description>Store class name for timeline state store.</description>! Q' }0 V" u: j8 M
<name>yarn.timeline-service.state-store-class</name>
* Q8 A; ^" }3 M; Z+ ^& K5 g r7 f5 i d <value>org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore</value>9 b s- {/ N. L) I' R2 \7 [$ j& J
</property>
/ _, I" z* h4 @- o& K5 f; X3 J; d <property>0 Y: u K/ k5 c! F6 j
<description>Store file name for leveldb state store.</description>
6 T' _2 l& k+ j4 b$ \5 e6 n <name>yarn.timeline-service.leveldb-state-store.path</name>/ E$ [# Q+ r* r
<value>${hadoop.tmp.dir}/yarn/timeline</value>
7 ^5 D- P% E) S! g! A6 x8 @ </property>
$ e* Q5 W& a6 s0 W% d3 d, h. z2 L( M <!-- Timeline Service v1.5 Configuration -->* C& R# B. Y6 N( i( a: i* G
<property>
& J6 g2 u/ u, M7 r' q$ S: j <name>yarn.timeline-service.entity-group-fs-store.cache-store-class</name>
" @9 M" R+ Y3 Z$ g( w% Z" J, { <value>org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore</value>) Y; a& V9 `% P! O a0 O* e! O/ `5 |
<description>Caching storage timeline server v1.5 is using. </description>
4 E- w/ j& ^, m F, g2 @ </property>* [$ M" O$ Z& U6 v* b
<property>, @- l- C1 W, @
<name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
2 }) Q$ E- F- |+ K! }% _9 m <value>/tmp/entity-file-history/active</value>
& S# l# E g5 T- l' G% D: I, v# z7 [" c <description>HDFS path to store active application’s timeline data</description>
) w. X0 ]4 ~/ p. A: { </property> B6 {+ }" F3 m
<property>
! D# @3 p/ d2 T& P2 Q <name>yarn.timeline-service.entity-group-fs-store.done-dir</name>6 K7 g1 I; F7 }+ b1 U
<value>/tmp/entity-file-history/done/</value>
0 e) X+ ?$ s9 m! v <description>HDFS path to store done application’s timeline data</description>
" U8 [% b+ @" D# D7 g# h ^6 v$ d </property>0 o. W' {/ `- E' F! c
<property>
( y3 O& q/ g0 d9 { d/ | <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
+ X. B }0 l5 U# u; u9 s: a2 c <value></value>- x$ y+ R7 k* \# k* g9 i# v# X
<description>
7 Y+ \. p3 ^9 Y* A Plugins that can translate a timeline entity read request into
( g3 m4 k( j4 x+ G a list of timeline entity group ids, separated by commas.$ N1 K2 [% \8 t
</description>- l+ k6 a$ x2 ]2 ?" C+ i, w* g
</property>. b' P+ c" M$ L. F
<property>
. ]& f9 w0 D- ^( _% }3 {2 I <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath</name>
& ?! C* e- G0 b; | <value></value>
% J# l& q7 Z7 @4 w <description>' l7 Q+ _2 \- t0 |
Classpath for all plugins defined in
4 `% ?7 |" S6 A$ ?/ U yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes.
" W8 X/ O6 Q7 ^0 g: F" G9 O </description>
9 n! M% A1 _/ k/ `* x! H6 X </property>9 C& X5 ?) @& @" q) E
<property>
9 c* p7 X- v. w( Q, J <name>yarn.timeline-service.entity-group-fs-store.summary-store</name>
, ^' }. z7 X# y1 G1 b4 p <description>Summary storage for ATS v1.5</description>( E4 G B* M% }5 }. _! K
<value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>. I2 i' O1 R" N
</property>
3 O) q! ~. I1 s: h) b <property>
0 G a% y! z- x0 e9 u <name>yarn.timeline-service.entity-group-fs-store.scan-interval-seconds</name>/ A0 q% ?$ x5 c
<description>6 \$ k! ~" S8 o# j! N/ I
Scan interval for ATS v1.5 entity group file system storage reader.This
/ u; x$ L$ E# F+ `! R value controls how frequent the reader will scan the HDFS active directory: M! J3 n/ D& r Q9 z9 Y. v, S
for application status.1 g% h1 [# L) c @4 D
</description>' J+ w- o* E4 D1 ]: E
<value>60</value>/ L& z, u( M8 Q: ]: u
</property>
4 z! A, c; I6 _8 n$ e <property>
# j* y/ ?+ O& n0 r( P/ _7 d <name>yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds</name>' N' d- v9 O9 S; [
<description>+ q& D' l% Q8 s1 n0 F& G
Scan interval for ATS v1.5 entity group file system storage cleaner.This
# }% e* M1 f: o5 y, d value controls how frequent the reader will scan the HDFS done directory
5 y2 l8 b& g% @8 t for stale application data.
' ^- ]# R4 b7 P/ E2 D' r </description>
$ W4 h$ s0 F3 I <value>3600</value>8 [6 _# q. j- T3 S& C
</property>
$ J+ @; F+ i% a. ?( l- { <property>
0 U" s$ z) d# m' c# a! O1 s) Y <name>yarn.timeline-service.entity-group-fs-store.retain-seconds</name>4 Z3 T% O. _+ n. {% Q/ K
<description>" n# h9 [$ X0 d) N4 y
How long the ATS v1.5 entity group file system storage will keep an
7 l- V4 ]; g+ g1 f: B6 o application's data in the done directory.% _/ O! V! x6 e6 ?) z
</description>$ ~" N. H2 r" ^) ]
<value>604800</value>& B8 a8 q/ R8 P; V
</property>* K: o# Z- ~3 k% R2 {( t. ~: }. v
<property>/ {0 T% r; O" w: M
<name>yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size</name>
, u% Z$ s6 s$ H8 M <description>
4 }8 b# t1 y( D: i* R$ L) i( R" _ Read cache size for the leveldb cache storage in ATS v1.5 plugin storage.9 h+ w: A, R& i) {. e" Y
</description>9 V0 F, n7 n; ]6 n" d. i1 t2 n
<value>10485760</value>( f/ {/ n6 N- ]4 h4 N
</property>' C: r/ p# w8 x# [
<property>
6 z1 K7 ]+ e0 p: i) M) @ <name>yarn.timeline-service.entity-group-fs-store.app-cache-size</name> q# ?5 D# L( O
<description>9 N( L# W/ A$ b* \/ p, V( {/ p
Size of the reader cache for ATS v1.5 reader. This value controls how many) J& H: D8 m( r# r. N: i5 O: h1 P
entity groups the ATS v1.5 server should cache. If the number of active
; m; q0 ~9 d8 D0 p/ s" j read entity groups is greater than the number of caches items, some reads" t4 a1 K6 d5 u; \
may return empty data. This value must be greater than 0.1 H( x7 T& R' C4 N% J
</description>
4 A4 {' _, }& `# E <value>10</value>& D# {1 J6 O9 ~0 W
</property>
3 s: s! ?+ E# v' o <property>
( v& q9 z% X3 i a. g <name>yarn.timeline-service.client.fd-flush-interval-secs</name>: H- K5 Z8 P0 X# d% B
<description>
7 M! p: ]' J1 m5 c5 V: t- H- g* w Flush interval for ATS v1.5 writer. This value controls how frequent
1 T# F7 }+ F& j) w7 ^1 x the writer will flush the HDFS FSStream for the entity/domain.
+ M0 y9 |4 l+ M5 B% s </description>- G4 i% S7 p: ^, i
<value>10</value>" W5 L" Q) s& w9 ]( l5 B$ _5 W0 |
</property>
) j1 S9 U! P# {; G! Z <property>( v1 b& C6 D2 I. D5 g; Y4 a- i0 F) `1 Q
<name>yarn.timeline-service.client.fd-clean-interval-secs</name>
) l7 i0 A! w" P7 _$ P- F <description>
' u+ T1 b9 t* F9 B, l2 L5 C Scan interval for ATS v1.5 writer. This value controls how frequent
8 _ q0 T$ [' t. o- j( Q" v the writer will scan the HDFS FSStream for the entity/domain.! i9 Z5 S0 ~4 g% M( C
If the FSStream is stale for a long time, this FSStream will be close.
( @, ^: f! Z+ V3 O. B; v </description>) {+ O. _( L; a1 U! S9 ]& u
<value>60</value>* ~2 y9 j `& l: X6 H2 z
</property>
- I% v) v1 R" s! O3 e+ J <property>
) ~3 I2 u/ L* a- M6 y& E1 ^: Q <name>yarn.timeline-service.client.fd-retain-secs</name>
: `3 H2 V% h! z& b <description>& T( ~, U2 t% G* ]. A$ P
How long the ATS v1.5 writer will keep an FSStream open.7 b7 e+ z2 f% o- i' a
If this fsstream does not write anything for this configured time,- G7 S6 H1 J# b
it will be close.9 L5 E% \2 o T t \; k3 X
</description>* n. U6 x" z( m
<value>300</value>
) X+ k" o9 m; j. h$ x+ [ </property>7 |0 v- j, l) R* G5 F6 |& |
<!-- Timeline Service v2 Configuration -->) h" r5 H& ^4 s+ v; Y8 u& d
<property>; u9 p- J3 t9 U; Y
<name>yarn.timeline-service.writer.class</name>
" F' E9 J7 Q6 J" s/ {0 D <description>
2 G- N0 l) ~; f8 S Storage implementation ATS v2 will use for the TimelineWriter service.1 W3 X8 w+ C H$ t
</description>, Y+ p* y6 A+ G2 c! s( g
<value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl</value>1 [9 b% X+ X8 G0 ~/ \' R" P
</property>
' X1 u" u0 W. F! {7 v$ M0 g8 t <property>2 e) F3 _- k) Q B1 f& C" e& S
<name>yarn.timeline-service.reader.class</name>
8 b, q5 r* J# c0 O0 v* V$ O8 ] <description>
* L; a' l/ h2 A' m0 m Storage implementation ATS v2 will use for the TimelineReader service.7 U" O/ e& Q% e1 q. z! R* X, A
</description>
' p3 l+ C5 {7 d- d+ n4 O <value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl</value># r1 a3 r8 L" U/ V+ Q
</property>
; t G4 n: e2 X; O& j! i <property>% r7 U: c9 r% S- x6 g9 h r
<name>yarn.timeline-service.client.internal-timers-ttl-secs</name>
- v. k) i/ _& @8 l0 O! D4 i <description>* P' B/ P" t& i) z- n
How long the internal Timer Tasks can be alive in writer. If there is no* k4 u' }: o1 z: V/ \$ R
write operation for this configured time, the internal timer tasks will" d" s4 N' c0 ?( Q) [
be close.5 o; C, \/ }1 R4 X7 V: t+ q
</description>7 i& B' x3 z, |9 Y; W S0 @
<value>420</value>1 B3 X5 f) _) \: ^- X6 j4 k2 v
</property> W: N+ O1 k$ a; k' I( B3 q/ u
<property>5 o+ v0 Z$ j( h4 o) k' |
<description>The setting that controls how often the timeline collector( w( s) d# Y7 l* E
flushes the timeline writer.</description>
6 `* L; Q. p3 {( ]3 M+ Z* D <name>yarn.timeline-service.writer.flush-interval-seconds</name>8 w' U/ f. h3 w9 A+ F0 |+ p" i4 Q3 F% C
<value>60</value>" M$ w2 h# y2 O: {3 k( H
</property>/ W: m) L4 U1 p5 Z4 o
<property>
1 [0 M* E' {5 b+ E <description>Time period till which the application collector will be alive8 J+ o6 e1 [- Y- @
in NM, after the application master container finishes.</description>6 L& u o- D/ w4 u* V
<name>yarn.timeline-service.app-collector.linger-period.ms</name>
6 {& O' Q" N3 z4 s! ^0 z; v0 }+ _ <value>60000</value>
6 D( y$ { Q5 D$ {. _6 I4 H3 k+ U </property>
2 i8 M$ b& i1 C# L) Q3 D <property>
, A: f" Y. Z; e0 c8 N: k* W# G <description>Time line V2 client tries to merge these many number of6 r3 }2 F/ C) }8 C1 l c6 G8 A' k3 t
async entities (if available) and then call the REST ATS V2 API to submit.$ Q9 x7 E% r5 O2 c9 K1 a
</description>; q8 d* }: d' B8 P b+ `
<name>yarn.timeline-service.timeline-client.number-of-async-entities-to-merge</name>8 v* X2 U7 b/ H% |
<value>10</value>' U, j9 _, q+ \# U$ b
</property>% L& T% T$ I, z2 [9 R1 Y+ u' ]! \
<property>
' H: J; q* ?% J# |# n <description>
$ r& J' K0 s8 Z2 W8 O Z/ U! ` The setting that controls how long the final value
1 G; l' R) X# {, p6 q0 [+ E of a metric of a completed app is retained before merging into1 W- w% c! V. @3 F3 J9 V/ H
the flow sum. Up to this time after an application is completed
0 P! m2 g) M7 g; K2 c. `( ]% P out-of-order values that arrive can be recognized and discarded at the R( Q; j# ~' A' b, V9 \- y/ p' {
cost of increased storage.$ N/ M3 \' s# j% R- E# s3 l
</description>
8 w9 M- N+ X! ^2 i <name>yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds
) ^' J/ g0 M0 ~% `! x4 q( v </name>
- r O7 X+ Z2 r% [5 Q <value>259200000</value>
* Y4 q& t3 ~5 @2 d. d! E </property>
, S3 ^. B0 @& Q$ J, ^8 X) @ <property>
3 A6 C9 Z0 R* C. X <description>$ j$ s+ E4 g1 u7 m! ?# o
The default hdfs location for flowrun coprocessor jar.3 P: ^( m! |+ v2 {
</description>+ Y4 n( n9 a7 s
<name>yarn.timeline-service.hbase.coprocessor.jar.hdfs.location" Z* D L: l: X0 `4 b V. l
</name>
6 k" h0 I0 |0 s c% _; @ <value>/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar</value>2 D" }. e, l& ^' a* R" [
</property>
! O+ F1 _4 j% g: Z <property>, ^8 O" K- y5 V3 b% ~
<description>
6 v& \' R6 o$ x. H r! l2 o! F The value of this parameter sets the prefix for all tables that are part of+ g J( y( r0 W: v: X! `( [
timeline service in the hbase storage schema. It can be set to "dev."
3 c1 X. {! z/ g' y7 Z or "staging." if it is to be used for development or staging instances.
; S9 h/ c1 h' J This way the data in production tables stays in a separate set of tables
; R; m$ W) H2 p& Z) m- e prefixed by "prod.".
4 z8 c' F7 a7 m- }$ E </description>7 h3 b. S0 T; g' a
<name>yarn.timeline-service.hbase-schema.prefix</name>
0 c8 W8 \3 y0 w# O$ r% V6 K <value>prod.</value>, w5 q; l4 }* \" ~) O
</property>
: \: A9 p: j3 V% e6 D <property>) T3 i8 Y/ Q8 D4 X$ j+ ?' B
<description> Optional URL to an hbase-site.xml configuration file to be
+ I; a# _' C4 i) y& O, x' k# q, J3 _ used to connect to the timeline-service hbase cluster. If empty or not8 S: v1 g+ i- v# S3 F
specified, then the HBase configuration will be loaded from the classpath.! F* z% N# G+ N- [6 \9 c" e
When specified the values in the specified configuration file will override
; Z5 j4 d+ N$ `: T* G# K/ s+ C( X+ H those from the ones that are present on the classpath.
6 J( Q, M" v4 c1 r" x- w4 u </description>4 M3 P; t0 t$ \) w) J; U# U: Z+ Z7 T
<name>yarn.timeline-service.hbase.configuration.file6 y6 A" n' Q2 W4 T# Y
</name>
" S4 C' B% \6 f1 R <value></value>+ I7 O- e5 y) b
</property>- G0 V: h, I" U& n3 u- f( }' A9 H
<!-- Shared Cache Configuration -->1 L/ c; d5 R* J+ t# C! [* l6 [
<property>
( P$ ~8 L4 h f6 h2 ? <description>Whether the shared cache is enabled</description>
. a5 s% M$ y) e2 _% h) V, e <name>yarn.sharedcache.enabled</name>4 ~. T b4 T- F: a& F
<value>false</value>
/ E" n" @$ j. H5 X Q# `) F6 ^ </property>
; Q8 N8 _: u0 \" U <property>
$ x$ ]" f3 X# f; T; H0 } <description>The root directory for the shared cache</description>4 C: a2 y( J; w, [
<name>yarn.sharedcache.root-dir</name>% d2 J. T- g4 S4 e, f2 r a
<value>/sharedcache</value>
" J4 Q3 o) A& \ R+ ]# h </property>
# [9 l2 L$ N9 B# V <property>$ ?( d J, ?6 j1 s& }: O
<description>The level of nested directories before getting to the checksum* |: v" Z# B" m F/ g1 n# f
directories. It must be non-negative.</description>3 \: l! O0 R) w* n1 e$ T
<name>yarn.sharedcache.nested-level</name>" h, ?8 d$ ]4 r/ r7 J4 \" z+ a) o
<value>3</value>8 [" g7 d8 B* I9 W- k1 z' P
</property>2 K5 u. g0 l8 C1 {+ Y& S
<property>
, h. b, ]. \& d( f0 `" I: G. N2 C <description>The implementation to be used for the SCM store</description>
1 Z0 Z; D, J! K" K. U3 b8 V <name>yarn.sharedcache.store.class</name>
% y/ E, b c- r* {7 J, H <value>org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore</value># y, k* T5 V( j# f, h; i* b9 A
</property>8 V3 c9 c) w6 N+ C! p% u
<property>
. T" r; f, k$ r u7 g( E) g <description>The implementation to be used for the SCM app-checker</description>, }/ i- Q! o) `: ~
<name>yarn.sharedcache.app-checker.class</name>
! T& z4 `* w3 c& w <value>org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker</value>' A, z6 Y) w( F
</property>$ s) f4 M4 a' [5 o, J6 s2 F
<property>( V* e; l' s1 y9 ^# ]
<description>A resource in the in-memory store is considered stale
5 y) v$ j5 l5 q' L" M if the time since the last reference exceeds the staleness period.0 V- }* a& m+ `
This value is specified in minutes.</description>" t& H5 [9 r5 m' y+ S w
<name>yarn.sharedcache.store.in-memory.staleness-period-mins</name>
* t8 \; q; K: E <value>10080</value>: s! B7 R! B" g9 ]6 ]& ~6 n
</property>8 h3 _2 Y6 X/ V' o
<property>: A6 C) z* ^$ j; E
<description>Initial delay before the in-memory store runs its first check( u' x) J/ q( i4 g1 T
to remove dead initial applications. Specified in minutes.</description>
+ D3 t) S( ?0 ]; D- o% L <name>yarn.sharedcache.store.in-memory.initial-delay-mins</name>3 r' }, Y, T; G! B5 g6 o4 p0 s' N
<value>10</value>% X7 a/ L3 H3 ^( `* v& ]) W# \
</property># d4 z+ U8 `+ g$ M: `* s/ z
<property>5 K/ a2 z) F& m
<description>The frequency at which the in-memory store checks to remove
* |2 q& Z; b2 D* F dead initial applications. Specified in minutes.</description>" q- v- a8 D" ?' P/ t
<name>yarn.sharedcache.store.in-memory.check-period-mins</name>
/ j6 q: M1 b0 E <value>720</value>
8 }9 H, x* L8 D1 z. u* G+ h3 t </property>1 e0 Q: p; B! e
<property>
( y- R) a5 x9 a' t( O* V) C( j3 D <description>The address of the admin interface in the SCM (shared cache manager)</description>
9 T2 Y. R' w* d/ [- {* Y <name>yarn.sharedcache.admin.address</name>
5 q7 o# N/ U, }! q& ^& @" w V <value>0.0.0.0:8047</value>5 F5 \1 p* [2 M% d
</property> s* C! a2 X0 E$ l
<property>7 x% K3 q2 g) z' x0 Y2 r ^& l
<description>The number of threads used to handle SCM admin interface (1 by default)</description>
" ~0 ]8 R# C' v+ {3 B9 C! P Q0 D o <name>yarn.sharedcache.admin.thread-count</name>
1 r2 ?- A* u! T0 `) b <value>1</value>
; [# k! o* M' k. ~ Z </property>9 t9 M1 C; _* i# @# A
<property>
4 e) b, Y, h2 u, y# M0 O <description>The address of the web application in the SCM (shared cache manager)</description>
3 K( s- w5 K1 h9 _4 v* { <name>yarn.sharedcache.webapp.address</name>
o6 T$ i8 W2 M5 Z: _9 {. `5 d <value>0.0.0.0:8788</value>
2 A6 w; N/ |/ W) `& y </property>
) I4 w) H2 H ^4 z; K <property>
* g9 l7 Z, B) L& p( s+ o <description>The frequency at which a cleaner task runs.
' g2 K! ]/ r, g! ^+ V j- _8 @+ ~ Specified in minutes.</description>" H' F2 M+ \; Y c9 A
<name>yarn.sharedcache.cleaner.period-mins</name>' x; e: X6 ` j P3 }& V- h% Q$ l
<value>1440</value>
' _1 v7 @$ d+ l0 H </property>0 G/ j6 n+ T# [
<property>
2 L! w+ q l8 Z6 Y2 W( C; b <description>Initial delay before the first cleaner task is scheduled.9 y/ {, m) P% |& q
Specified in minutes.</description># q, A5 _3 d: n5 ?/ e
<name>yarn.sharedcache.cleaner.initial-delay-mins</name>7 v7 |' O; k& _( k
<value>10</value>: f, }. B# M; A. E& ?
</property>% b4 M2 B2 t& H* {7 s! i( v6 W* q
<property>5 ]8 v4 s8 M [, i
<description>The time to sleep between processing each shared cache h/ `9 Y, e7 A. |
resource. Specified in milliseconds.</description>, P& m I2 }7 ^3 a; H$ B
<name>yarn.sharedcache.cleaner.resource-sleep-ms</name>
' N0 R( K/ l$ D <value>0</value>
) g `1 _% e# _+ E4 h6 s </property>
0 b* ]3 t& R2 q' F& q6 H5 t <property>: k: Q4 Y, x4 V8 M3 g$ d
<description>The address of the node manager interface in the SCM$ X3 W0 V0 b9 b8 X, F L
(shared cache manager)</description>( c/ N8 x. x# p1 a% n$ H" m$ j
<name>yarn.sharedcache.uploader.server.address</name>
# j% [1 W+ B$ d" I9 _2 U! n <value>0.0.0.0:8046</value>& m* ?( N! n- `' s5 C
</property>4 J |3 w' p* d* w2 P' J
<property>- T; T n. z$ w; D: Y
<description>The number of threads used to handle shared cache manager0 K, T, M2 E! z
requests from the node manager (50 by default)</description>- U5 l4 ?# E* ]* P% I
<name>yarn.sharedcache.uploader.server.thread-count</name>1 i2 @6 i( r/ Q+ [+ ?
<value>50</value>
) S6 ]# r0 A1 K0 Y </property>
{. P h9 D) L: [# C% e <property>
% b* |, t. b1 g <description>The address of the client interface in the SCM
. [' [( R. h' B* Q (shared cache manager)</description>% Z C2 |3 m4 ]4 L# p8 S/ \
<name>yarn.sharedcache.client-server.address</name>
5 T' p0 E$ n+ `" T% q8 W0 e# a <value>0.0.0.0:8045</value>
( J+ `, i! k) e7 K5 X8 \ </property>. X* g% p6 \6 D/ s
<property>
0 I& ^3 `+ b0 d# v) v* Q; n+ B9 C <description>The number of threads used to handle shared cache manager- S: y, Q+ U1 c( y$ \8 L }! W
requests from clients (50 by default)</description>
3 z( B( A4 o0 u* o/ L8 s! J <name>yarn.sharedcache.client-server.thread-count</name>- u" q6 E' P% t a5 D0 K" m' B. g
<value>50</value>
) U/ M9 s: D, |2 h) X </property>3 W4 h$ O3 d1 g) T; e' C7 M* b; H
<property>
6 c- Z8 d, W2 q( M <description>The algorithm used to compute checksums of files (SHA-256 by5 d# [! j6 W. G4 `1 Z* I8 _
default)</description>
. v8 x0 b' l0 \ _ <name>yarn.sharedcache.checksum.algo.impl</name>/ ]3 l* O: H6 \1 O
<value>org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl</value>
! V9 W" B+ G9 G6 o </property>
1 H& m7 x: O! y2 x4 r. g* {: F <property>% n5 E: P! l, o( X y5 ~
<description>The replication factor for the node manager uploader for the
$ H4 K0 \( I; F6 {2 y shared cache (10 by default)</description> z, M: U* G; [; N
<name>yarn.sharedcache.nm.uploader.replication.factor</name>
) g) ?- T v" i <value>10</value>/ K5 _ N3 Z' U3 c4 M
</property>
/ d4 b" M7 G' J3 T4 p0 |: n <property>0 ?, D2 G4 A4 J B
<description>The number of threads used to upload files from a node manager
1 E* O3 I: \5 E; L, Z: e% } instance (20 by default)</description>) }8 E9 u: X# e1 [( [" L
<name>yarn.sharedcache.nm.uploader.thread-count</name>
8 l. Y9 j$ x% P <value>20</value>
( f0 v& T+ s/ G, k0 m </property>7 E9 E( \/ t3 w, |$ K4 D
<property>
1 C7 W* c7 V0 q9 V8 u <description>
4 l/ ]! V4 ~8 {' I+ N/ m# u ACL protocol for use in the Timeline server.: p: J0 O& x" i5 q4 _, }1 o
</description>& p7 {+ A L8 b1 Q" x
<name>security.applicationhistory.protocol.acl</name>/ R. j; x: K( T/ c0 k
<value></value>
3 m1 h8 o% C0 W ~ </property>
* G! ?( {5 O) A* d& f+ } <!-- Minicluster Configuration (for testing only!) -->! ]' \1 E0 k1 k
<property>
$ f# P3 M1 o, g( f <description>; B& Y3 H7 ?5 ~2 S5 {
Set to true for MiniYARNCluster unit tests) `, t; h% ~0 f6 r( `, }! v$ t8 \
</description>. v- u1 G. \- v& a
<name>yarn.is.minicluster</name>
" m0 {6 E0 `% P: I$ } <value>false</value>
9 ? ~' V& U) D, R" h </property>* I; q; c% @8 s* J- ?
<property>
% y0 S6 w) k: B0 i/ S& d6 G% b <description>
V( q0 a" _! K# M/ n; O1 g Set for MiniYARNCluster unit tests to control resource monitoring
2 r3 k& h/ H: } </description>, O5 \: }$ ~* Y2 C3 E
<name>yarn.minicluster.control-resource-monitoring</name>
8 ]7 ^* I. w: [- l. U7 e3 A <value>false</value>! A& [0 \5 R2 U/ i c$ Z2 w% Q( H
</property>" t- u5 t# v1 Y* V8 U- B9 g- ^
<property>
$ U- ?) K& Y0 q <description>
6 f9 q! \4 U& [* ]7 ^5 H9 M Set to false in order to allow MiniYARNCluster to run tests without
0 J( E& s4 a8 O3 u port conflicts.
' X+ ]0 C- O0 s </description>
' s$ l9 ~" i7 Y4 [! O <name>yarn.minicluster.fixed.ports</name>3 K, i! Y+ { j2 o
<value>false</value>
, S$ i, X9 `+ F </property>
' S) e$ u4 M$ x <property>
' k1 q# f# \$ h+ k <description>
1 [3 @ b) U4 d9 U9 a: m Set to false in order to allow the NodeManager in MiniYARNCluster to
6 c% b4 y U% O6 z4 J use RPC to talk to the RM.9 H( M+ x6 D1 W5 f2 e
</description>
% V: Y5 ` i) U. w <name>yarn.minicluster.use-rpc</name>: U; q8 v' E: V& Z& z, Y- H5 k5 g) t
<value>false</value>
: g2 e* b0 N- d2 X9 {, b </property>
( _9 I) y& t1 G; |( L9 j7 ? r <property>
u" l; A- T4 ^8 k, x7 |' j <description>
4 Z5 O- i/ N+ L( G$ N# y/ E As yarn.nodemanager.resource.memory-mb property but for the NodeManager5 ?3 O- `9 U& D' S) t2 t+ V' O0 q u
in a MiniYARNCluster.- S* v" J# Z& g) x Q/ i# T
</description>
5 n$ F- y: |2 a/ y. L u; t <name>yarn.minicluster.yarn.nodemanager.resource.memory-mb</name>
* G/ C5 ~) T5 @& V& q4 k <value>4096</value>" T, c& C. b& c2 o* Y6 V
</property> | U1 _/ g( l8 V
<!-- Node Labels Configuration -->: r9 C4 m7 B! X/ @7 |! D
<property>
( A( V0 V3 z0 w/ V9 I" B <description>/ S3 m, g! X. k& |; F
Enable node labels feature! u5 A0 R2 H" c" v2 W( G- L* m
</description>
( D7 K8 p+ Z: `( ^" ^ <name>yarn.node-labels.enabled</name>' ]- [4 l( ^! d
<value>false</value>; v: J9 g' a- z/ Z
</property>& G8 V2 `: u; m& c2 J
<property>( d. W; v# p" k. y$ @9 O8 Z
<description>
5 }7 S/ ]& ~* N& }; _ URI for NodeLabelManager. The default value is
* T6 s) e; w7 s/ R /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem.
- C& o8 ^. s N1 U4 i- W9 f7 D6 v </description>$ F3 r! D2 N J/ ~
<name>yarn.node-labels.fs-store.root-dir</name>
' F% C/ V( h% I <value></value>
$ R4 N7 g- m F, R# v2 ]. i </property>6 i5 P; V& t# j% H
<property>7 X1 P% b5 Q+ V6 {5 n+ ^' a
<description>1 D( R% Y$ j( P& j' P
Set configuration type for node labels. Administrators can specify4 u* {) t4 ^; ~# x) s& x' c, N
"centralized", "delegated-centralized" or "distributed".
3 Y0 G& ~+ l y, y6 c </description>9 m0 F% Y, p- o) n+ R! @! P8 d* u
<name>yarn.node-labels.configuration-type</name>
8 ^- S( J4 e- T1 J7 s <value>centralized</value>
7 q t3 c4 M$ s* r; x8 ~ </property>/ \4 l( g W) o7 g) l+ ?( z
<!-- Distributed Node Labels Configuration -->
; o) B4 }; R B$ i% u <property>
8 |- q- q6 }- ^ <description>! D! [3 N5 U% Z2 b* V7 x- S
When "yarn.node-labels.configuration-type" is configured with "distributed"
5 ?+ J+ Z* V: M5 s1 k9 K in RM, Administrators can configure in NM the provider for the3 X9 Y) G( N1 V5 K+ W% u
node labels by configuring this parameter. Administrators can
( Z7 j2 w! ^. j5 @' n configure "config", "script" or the class name of the provider. Configured, V9 b5 p9 j/ u/ v S) J* E
class needs to extend- L& i: P, |7 ^% a4 g" x' ?& ~5 C
org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider.. _! P R% F+ N. ^% a' f4 Y
If "config" is configured, then "ConfigurationNodeLabelsProvider" and if$ K% R8 D1 B1 r: ]( ?
"script" is configured, then "ScriptNodeLabelsProvider" will be used.
1 @, U# d. M! W% l, q </description> W5 z4 k. x6 O1 w- S
<name>yarn.nodemanager.node-labels.provider</name>
& n. H( y4 J" L; y9 G' ^) W7 L </property>( H. _* |( c( J- _ T& {+ ^# F' S
<property>7 b/ e; Q' x, w D7 ? e
<description>/ Z& j1 M- c! L C9 a
When "yarn.nodemanager.node-labels.provider" is configured with "config",8 z) L( E$ G% J: n9 @1 I5 t) _
"Script" or the configured class extends AbstractNodeLabelsProvider, then
8 r7 C' H# v8 `* m2 K l' | periodically node labels are retrieved from the node labels provider. This' P6 y+ ?. G. e' N
configuration is to define the interval period.! q, W; ?( Z, |& ]6 `
If -1 is configured then node labels are retrieved from provider only
# N4 `9 @+ R) j3 [ during initialization. Defaults to 10 mins.* y2 B# `: t6 Z
</description>- A- V" K- I/ R- p6 t
<name>yarn.nodemanager.node-labels.provider.fetch-interval-ms</name>
% a, I3 ?+ f [0 c <value>600000</value>$ Z% Y2 j5 n: @/ N; }
</property>
; t/ g# \& n' b6 ~: v <property>8 y o8 d% a" |9 ~* { Z
<description>
2 Q G/ q/ P' n0 l& ?0 \3 c Interval at which NM syncs its node labels with RM. NM will send its loaded
) b+ j6 A6 w0 x' ]# ]" R, _ labels every x intervals configured, along with heartbeat to RM.
5 J- Z$ p, _& `" @ </description># ?* X+ g" F3 h; f; x/ M( f$ r
<name>yarn.nodemanager.node-labels.resync-interval-ms</name>
* l* f- n0 k; \0 t7 Y4 j' W- G, n# l& w' P <value>120000</value>8 V/ B) M$ p2 ]" R. Q3 ^
</property>1 z$ Z r7 {1 t$ b$ O6 _" v/ \8 N
<property>7 i6 G: J' k R/ q
<description>2 g4 V) n$ d7 \. w
When "yarn.nodemanager.node-labels.provider" is configured with "config"
. x! C$ j! e- p- O3 K& n0 C then ConfigurationNodeLabelsProvider fetches the partition label from this3 I( H( D) V* Q! x0 a/ B
parameter.
' q A( U/ { a/ ?0 w# v( e- |7 a b3 p0 W </description>; p6 |: q) P7 P: }. I, ?' u
<name>yarn.nodemanager.node-labels.provider.configured-node-partition</name>
; n6 G7 ]! x6 l9 Y9 j </property>- c2 n0 ?7 V2 T; V
<property>2 M* E2 w# i: M) U! `
<description>7 M2 \8 T7 k& O+ i. J
When "yarn.nodemanager.node-labels.provider" is configured with "Script"
, f# z2 A. H" o, ~+ a: U" F then this configuration provides the timeout period after which it will/ ~& l/ z! [5 C+ S( U$ v: {! w. F- T
interrupt the script which queries the Node labels. Defaults to 20 mins.3 x( ^5 f" M T! H# E
</description>; x" R* _. Y6 ?) q
<name>yarn.nodemanager.node-labels.provider.fetch-timeout-ms</name>
# }1 Y N! d8 w: E/ F; ?# l <value>1200000</value>
# m& Y- B4 J/ Y </property>( T5 ~5 b( a6 q/ {( I7 F6 l
<!-- Delegated-centralized Node Labels Configuration -->: X' o2 M9 X( ^, B/ o
<property>/ ]4 N' l0 }; z" |
<description>$ y4 z: n7 a1 {4 T, X* w
When node labels "yarn.node-labels.configuration-type" is2 U- ~! J- Y |
of type "delegated-centralized", administrators should configure
( v' r Y0 W' J5 s4 j) m/ b the class for fetching node labels by ResourceManager. Configured/ N2 l6 j9 A+ m( G. A
class needs to extend s1 W# F/ L4 `/ j7 d
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.
! P) k4 S4 B4 Z$ ]+ p RMNodeLabelsMappingProvider.
7 W& c. n% L6 }' H </description>% I$ }# e$ t) S& Z
<name>yarn.resourcemanager.node-labels.provider</name>, w }6 _+ ~, W5 {3 }- t
<value></value>% Q5 _, b- ^% `; m5 q7 A! M
</property>
6 E) [# y5 a+ k( f8 z" m <property>3 O5 Z0 l1 u J2 w
<description>
' U; |2 i+ s+ O) z When "yarn.node-labels.configuration-type" is configured with0 X! [( s7 ~# F+ K! [0 a
"delegated-centralized", then periodically node labels are retrieved
" z& g# C0 J; S8 t4 {8 ?( { from the node labels provider. This configuration is to define the
$ K6 p- y1 G# \! C; w/ W interval. If -1 is configured then node labels are retrieved from L, X& Z7 }6 d! y
provider only once for each node after it registers. Defaults to 30 mins.: ^& a; u" z' N. c; I E$ s$ G
</description>
; r7 T! _. B4 { <name>yarn.resourcemanager.node-labels.provider.fetch-interval-ms</name>
) Z) T. @! b- _ <value>1800000</value>
3 j* g& d( e1 w- Y' P2 k E </property>) I: F% a; c* V% U1 f& w! K
<property>4 W9 q2 n7 T2 V$ s
<description>
* i9 b" `, z1 U7 ~5 P0 s( f4 [ Timeout in seconds for YARN node graceful decommission.
7 ]6 P$ Z, R8 b8 j4 w This is the maximal time to wait for running containers and applications to complete a7 f6 j1 p9 X
before transition a DECOMMISSIONING node into DECOMMISSIONED.
2 k" z6 U# ?1 M" ?7 l </description>
. |- f0 ~( R! D. z' C# _& t <name>yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs</name>0 g* ~! X) {0 W
<value>3600</value>
6 S) N% y& ?/ {8 _; [- K8 s </property>
2 i# E5 c8 M I. q) u: M$ |+ ] <property>
" P5 L% ~% x2 b% Z <description>5 C7 \7 |9 ^1 r
Timeout in seconds of DecommissioningNodesWatcher internal polling./ F: t$ g1 A' J; d
</description>' e" N; m" n3 o& r3 \
<name>yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs</name>! u9 T" M" w1 w! y g: s* o
<value>20</value>
' o# I3 B8 v: J" u6 I7 O2 m </property>
7 l) a0 u; Q1 r( P <property>2 t3 Y" r2 u6 t/ C3 V$ A* n( _8 |
<description>The Node Label script to run. Script output Line starting with2 l7 u2 k) ?! q
"NODE_PARTITION:" will be considered as Node Label Partition. In case of
; |6 `7 r( O; ]* P& x multiple lines have this pattern, then last one will be considered
! l0 G6 E) a% H1 I6 y </description>9 L6 u5 W0 U. ]5 I, L
<name>yarn.nodemanager.node-labels.provider.script.path</name>
9 U) r1 G! H/ v H3 @2 o </property>' M* Q' b H: ~: h3 z" d U
<property># Q7 {& b& n3 {* k; H5 {0 o* o+ K* k# X
<description>The arguments to pass to the Node label script.</description>% y1 {- p3 ^8 V2 G C2 C4 S1 x
<name>yarn.nodemanager.node-labels.provider.script.opts</name>: p9 I0 Q1 X' L% e# }/ y
</property>$ R3 E, V) [1 o" C5 q% o; y
<!-- Federation Configuration -->, o5 I+ j& K" M+ K. r, F3 g, ^' P3 P
<property>
6 D/ J1 {# F4 r2 N# r5 G <description>
" c$ H9 L" R7 ~ m Flag to indicate whether the RM is participating in Federation or not.
, p" \; S' S1 t* C3 P3 H8 L </description>
. s l0 x* a1 O1 U# W6 ?' ^1 V; T& h& | <name>yarn.federation.enabled</name>2 ] I, M( \, l
<value>false</value>9 P- R& W3 u$ f: s! X7 c, C
</property>
1 V6 @4 E0 d3 D+ ` <property>4 a) h2 A) _# C* c$ [) ]8 X
<description>
1 v$ h* ^8 Y8 |- }! T+ U Machine list file to be loaded by the FederationSubCluster Resolver4 o, N9 Q+ o$ G. {4 M
</description>
" m$ b7 T& q+ [" { <name>yarn.federation.machine-list</name>
! W) O- X" G Y* W& G </property>
0 y8 k% o& O+ i4 o' a5 p3 m <property>
+ s6 f w8 X4 X+ U3 v( l `6 Y <description>1 Q0 A4 f8 z, T) |: r% E
Class name for SubClusterResolver
`( P; [0 G( U1 S d& g2 I </description>5 @5 s6 o) k4 {) `2 K0 F
<name>yarn.federation.subcluster-resolver.class</name>
4 _9 v( w# O: N- K <value>org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl</value>' K1 L) s/ X, ~! H$ Y; t* I
</property>( w& ]# P- B# }1 {; g& u
<property>
4 V( r$ v+ t; o# E <description>
4 b% s* b8 `+ c% Y ` Store class name for federation state store2 b1 E) w% l9 k; O* ]5 s; F1 ~, T6 v
</description>9 u5 c2 H( A' k# z8 l. {* }
<name>yarn.federation.state-store.class</name>
& H! T8 j: A! `* E3 l" \ <value>org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore</value>
' z& g# }5 H0 v, M5 ]0 J6 N! r: ` </property>) o, ]1 l3 r ^, X- w2 T4 T1 y: _
<property>' z( U1 {( S2 [9 Q3 ~- w
<description>7 @. h3 C* V( V, y
The time in seconds after which the federation state store local cache7 H, Z' W8 b4 m( X* E' v4 l
will be refreshed periodically5 h9 G) C, \) h% F8 W
</description>
0 E- o$ N3 W5 k1 D! N5 z4 M <name>yarn.federation.cache-ttl.secs</name>& Y, w) ^. x4 A) p
<value>300</value>
- j8 c9 T1 ~# A5 i: Q </property>4 X! ?- T- S" s; C+ l
<property>9 W% {! J+ m3 ^
<description>The registry base directory for federation.</description>3 f" F. _! e3 ~8 v8 r2 r
<name>yarn.federation.registry.base-dir</name>8 @, r: s- n" L' l
<value>yarnfederation/</value>
: h8 |, Z3 Q& Z3 B4 T </property>+ Q4 l& x/ L) y- O, D) ^2 Y
<!-- Other Configuration -->$ f0 n: ?. K% V5 H0 h' j
<property>: B. G3 b+ `) A8 K2 _ l# S+ l
<description>The registry implementation to use.</description>
; x8 `$ h8 M. L9 B% D <name>yarn.registry.class</name># m& n- r7 ]+ T6 |( V' l$ `$ F& ?9 G _
<value>org.apache.hadoop.registry.client.impl.FSRegistryOperationsService</value>
$ W8 Y, x6 T" Q0 k! v K9 B7 t9 K </property>
6 j' z" Z/ H( \ <property>
( b# T% M) k7 f; G: K1 a" P* f5 I <description>The interval that the yarn client library uses to poll the
! S) I4 f7 x. ^' ]7 X% j, _ completion status of the asynchronous API of application client protocol.
) \8 {# u2 v+ |: c2 S </description>
( G3 Y- y7 |" M <name>yarn.client.application-client-protocol.poll-interval-ms</name>
9 v* r2 s' q5 G <value>200</value>8 V4 J: A2 f" s+ z' [8 z
</property>+ c) C" X0 {7 X' b. ]* ^6 Y
<property>
# o" }- s% h. K t- d1 T <description>
4 d9 ^9 F5 S3 z5 S: g- j The duration (in ms) the YARN client waits for an expected state change$ q% J" ]* z7 a8 ^9 H! |' I
to occur. -1 means unlimited wait time.
1 ~4 o4 U5 k8 Z( T2 r9 T+ L7 x( X </description>5 N& |& `& |9 Z3 U) _, r& G `
<name>yarn.client.application-client-protocol.poll-timeout-ms</name>
1 L5 n% F9 c/ [! ~2 N <value>-1</value>
6 `3 V$ O& c" Y$ q( y6 ` </property>
' B s- X. k* l( n <property>
, I3 l. @8 z, L- i$ E <description>RSS usage of a process computed via
6 d2 Q1 w. m P /proc/pid/stat is not very accurate as it includes shared pages of a- Q3 {' z9 e* H: K
process. /proc/pid/smaps provides useful information like
$ v2 A3 M$ }8 K2 o) B+ w Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used; |6 g& K5 \' m* ^ _
for computing more accurate RSS. When this flag is enabled, RSS is computed
" S! E+ K. O, V* C( w as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes
! h; _# h+ x( G read-only shared mappings in RSS computation.
2 ]6 H0 l$ K/ V4 c3 y" ?" K$ ~0 { </description>
5 p4 ]3 K# f3 r$ m% B+ g3 _ <name>yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled</name>7 w+ e, `/ E/ l* j" S0 o$ r" m
<value>false</value>4 s/ k) X' g: z6 B5 Q1 ]( B3 T
</property>1 b2 s1 [' p: S1 t% F+ l
<property>
3 L( ^# U0 q& w; w6 c( _ <description>
1 v! N- c' p( X# L2 Q; ? URL for log aggregation server
- t% s: A- i" X </description>
9 }; k4 i. A' Z, [ <name>yarn.log.server.url</name>
! T" Y$ q* {& b$ a* M& f <value></value>5 s: f" @5 k" m/ n
</property>
3 }1 s+ g* X/ M+ t <property>
* {+ U6 U0 j X: m <description>
( U) K; y! F0 m% L" j3 L URL for log aggregation server web service
4 S u- T% K5 W: V </description>4 E, N! G/ ]* X
<name>yarn.log.server.web-service.url</name>
6 X4 Z* b4 y' x0 [0 M <value></value>% j9 ^: U1 Z- ~3 M
</property>* p8 P1 T, v& @" ?
<property>
) g( ^1 Y* L2 G0 V# Q <description>6 v9 m, J. }& V! B, h
RM Application Tracking URL
. F- q: [# v8 F" r$ t </description>/ _0 X$ A" r7 G, L( R
<name>yarn.tracking.url.generator</name>
' X1 f1 V5 N1 w5 k <value></value>2 E* t+ a. {+ f) _& q Y
</property>
7 [9 l5 i) ^. ?: h% Y <property>. ?2 f9 Y9 z% Z( \% \9 W# b
<description>
# m& u+ ~: g; g( @6 |; s Class to be used for YarnAuthorizationProvider, W) W1 L. k/ `3 y, [
</description>* T8 S n: N7 y' C
<name>yarn.authorization-provider</name>
7 ]# f- q0 n* {4 M' W! g( ` <value></value>$ l1 Y- B, l/ J# ^
</property>
0 {; ~& Y! M& A B5 U+ ~& t* H <property>+ ^. d; j* R8 i6 M) i9 N
<description>Defines how often NMs wake up to upload log files.5 R& r. b* o! r+ o" m+ }
The default value is -1. By default, the logs will be uploaded when. w% N' E' }* L" R2 B7 h
the application is finished. By setting this configure, logs can be uploaded
3 c O, f. T- } H4 P( _" \! L* K9 p periodically when the application is running. The minimum rolling-interval-seconds
6 r z" q5 [: a) \9 E0 v* l$ g can be set is 3600., t7 o: ~' j: k6 B; K: e! l
</description>
& D" x5 I6 t. h* @) v <name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name># A; _3 B4 |, s6 Z- l# p
<value>-1</value>
8 z6 j3 j9 ?7 s: i0 {( c </property>' v6 p" m; G W3 D) v3 ~# w: T. O
<property>
; }0 U2 E- d, T1 r1 B* w <description>Define how many aggregated log files per application per NM
8 |* l+ C9 Q8 U we can have in remote file system. By default, the total number of
* t0 x- A9 G/ p" b aggregated log files per application per NM is 30.3 C2 {2 B s: n( m" U6 a
</description>* l- t/ J2 B2 X2 a
<name>yarn.nodemanager.log-aggregation.num-log-files-per-app</name>+ q' {/ H8 M( H* r- h; E
<value>30</value>2 [2 x, ~6 J! l
</property>8 B" T5 d. E% k& _+ y. ]! q
<property>& M1 R% n& L- ?5 r) H" T! @, V
<description>
& p1 i9 {$ k9 w" [8 _& S, [: z7 d7 W2 p Enable/disable intermediate-data encryption at YARN level. For now,
3 s7 b3 j- O J0 N/ @) y this only is used by the FileSystemRMStateStore to setup right% u P+ w3 m" D; C5 M
file-system security attributes.
+ i7 ]5 I: u' g: V# W/ X </description>
" ?, y! g& \& H( Y <name>yarn.intermediate-data-encryption.enable</name>% t2 v( P+ o3 x# S/ c( O: [+ U$ p- b
<value>false</value>
" m+ i( H! G1 d1 T$ k! P2 {" [ </property>
' e8 t7 {8 K. Y: P2 m; C <property>
/ h, ?& R, U6 T( ? <description>Flag to enable cross-origin (CORS) support in the NM. This flag
. j9 P( a$ S8 Q; g requires the CORS filter initializer to be added to the filter initializers* r( x+ s* Z2 ?5 L# m$ |, y
list in core-site.xml.</description>
) r( [1 p8 a5 g- w4 y9 Q: a <name>yarn.nodemanager.webapp.cross-origin.enabled</name>: p0 i! s9 L, d: a0 {
<value>false</value>
* q1 P6 k/ z+ j) g/ r </property>$ X. I; N" o4 Q+ H
<property>& _! x+ g- ~' {; v" G7 Q8 C6 z7 F
<description>
! _+ c) e* n. k& @. J0 h Defines maximum application priority in a cluster.4 x. D6 ~- H, O
If an application is submitted with a priority higher than this value, it will be
3 ?% }6 E. i+ w3 a( T) N reset to this maximum value.5 z; h0 W. z. ?) l J5 F) f
</description>
! k; b- x! G% {& {' Y <name>yarn.cluster.max-application-priority</name>
2 T4 n: c2 |) W, y$ Q <value>0</value>& D4 [' m; m( L4 h2 M8 t
</property>
& l5 c- o2 z/ {# a3 M0 Y <property>& X8 ?' ^; d/ E2 t7 U5 ^9 b: w- p
<description>& g4 [0 | C7 L; ?, J& C+ k, c/ Z
The default log aggregation policy class. Applications can
8 ^1 C0 T5 Q- m7 A6 _, e' G, e override it via LogAggregationContext. This configuration can provide
) u) H9 X& ]1 F; f$ U) A some cluster-side default behavior so that if the application doesn't/ N) \- A, Z+ e6 C3 D
specify any policy via LogAggregationContext administrators of the cluster& G8 j& j! a, p* r1 g
can adjust the policy globally.
% T5 D6 c" g' f5 [# R/ Y5 j3 Q% ~ </description>
7 Z5 Z" m; _0 V+ ?. L" s& v <name>yarn.nodemanager.log-aggregation.policy.class</name>
2 {) ?, T5 K7 P" @$ `6 v3 w& N <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy</value>
" h& I2 \7 U- X% _6 \ </property>9 m: p9 t3 c5 m! Q+ p
<property>7 S, Z) O" |' O" V7 c
<description>" ^$ F i1 k4 [* l7 @1 X) f
The default parameters for the log aggregation policy. Applications can
0 ^! s3 \ D! s* J5 z0 x" v0 l override it via LogAggregationContext. This configuration can provide
! B) E$ C9 C$ l: ]) w. V8 G, U some cluster-side default behavior so that if the application doesn't1 q+ T6 o+ i! N, ?0 a
specify any policy via LogAggregationContext administrators of the cluster
, X3 ~0 a1 b1 ` can adjust the policy globally.
; M% N/ N9 C( P$ L9 \2 W6 e" \ </description>
, d' f( p* F( r* d0 F5 l <name>yarn.nodemanager.log-aggregation.policy.parameters</name>
9 h+ w( ]; E0 \9 G6 Q& Z <value></value>, h+ r# Z( J9 m. E6 f+ o
</property>1 ]8 u- H7 P$ r- o
<property>
* h8 X- {" g0 O( r <description>- \5 O, Q( M9 L' T
Enable/Disable AMRMProxyService in the node manager. This service is used to( r: F k. K) d" k+ D8 f
intercept calls from the application masters to the resource manager.
[5 `. u% k, S- I6 ] </description>
+ l' H* f A' C6 Z$ ` <name>yarn.nodemanager.amrmproxy.enabled</name>
) ?0 P, M+ [) G7 |: g, b <value>false</value>9 |3 c C* N& X: g( u7 ^
</property>0 Q3 ]: V' B9 Z
<property>, z& l' e! d5 s- B* e- Z/ j
<description>, }2 V8 @- Y) A1 _$ i4 X, @
The address of the AMRMProxyService listener.
+ x1 g- b6 _/ a' l </description>
7 V- o- A# [2 w* z7 v- c! u5 I <name>yarn.nodemanager.amrmproxy.address</name>& R; Z6 ?: u9 S$ Y
<value>0.0.0.0:8049</value>$ M6 ^/ `8 ~4 }3 K
</property>9 B1 C( V% K s+ n6 P: H: g
<property>
5 u) o; Y! u/ ]$ ~& B5 c <description>
* Z& j# ?; C% M The number of threads used to handle requests by the AMRMProxyService.
9 k @9 n& y) a- T </description>6 J' A% G0 \* A" M) M( \
<name>yarn.nodemanager.amrmproxy.client.thread-count</name>
5 w2 W1 [1 ]9 M- x) }: K' l <value>25</value>" m$ N; A3 N) C; t# ~+ B8 q
</property>& k0 s! D9 l) q. t3 ]8 {
<property>
4 F. i' b' ]% Z& P- F" ]: D <description>
6 J2 z# V" N6 a The comma separated list of class names that implement the! w6 D0 @0 V+ R6 g1 f& g
RequestInterceptor interface. This is used by the AMRMProxyService to create
% t$ P$ ]. V6 K0 n/ V- i the request processing pipeline for applications.( i, m+ l5 h P% n! c( n0 g
</description>
q8 M: r0 J# a3 S/ o5 b <name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>& Q- i! g- `5 f3 S
<value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor</value>' h! j% {" L9 K" s- o, ^4 x& ~
</property>* t. C6 ]+ Y1 u, S
<property>
, n4 X- n/ D' r# }5 w$ W* B: |" d% d <description>9 {9 B* T$ f" c
Whether AMRMProxy HA is enabled.
6 m. ~- O' B b </description>
& N0 O y$ Y! q2 Z$ w. _! ~% L <name>yarn.nodemanager.amrmproxy.ha.enable</name>
. \( y. k: m' c2 Y& D2 @/ {5 X <value>false</value>! W4 T F0 Q: S- C+ N
</property>; N! L7 k5 X, @" X+ i# \
<property>
, v( ]1 W6 J! U b3 V S. r) R5 } <description>
( p1 ?4 ]9 v1 o' J& b Setting that controls whether distributed scheduling is enabled.
4 o0 b0 q2 E `. B0 ]3 v, _9 Q </description>
9 s3 E, R& @ Z5 Y <name>yarn.nodemanager.distributed-scheduling.enabled</name>) }2 ?; x4 p" \- x7 P0 p. }
<value>false</value>
% H: J8 J8 E; M; C </property>& z6 O8 y- V" l. h/ r& D& c6 h
<property>
. G1 C' m$ w: _. V3 ?, V0 |& S9 ^! @ <description>
& D3 M8 Z) X& o# { Setting that controls whether opportunistic container allocation- e- F) }, ~7 E( ~
is enabled.
# z/ l5 m( q, n' a </description>8 I8 K1 }# d. @3 l l1 d& K& d
<name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>
5 v+ K5 d3 r. Q3 l( z <value>false</value>
5 x; V4 R4 n; ]! W9 y </property>7 W; V+ }8 d% U" @, ], q
<property>
4 P4 C8 @& v" |, j K$ ?. m$ N <description>
- F* R( l a4 |( ] Number of nodes to be used by the Opportunistic Container Allocator for F6 m6 L5 _8 l/ Q, Q! b) a
dispatching containers during container allocation.
# X( j2 P! Q$ T. } </description>
, R1 l5 _7 \, f0 O4 N <name>yarn.resourcemanager.opportunistic-container-allocation.nodes-used</name>
9 j5 V! k O# X <value>10</value>
$ R/ ~7 d$ P: q a* C) a/ {! O </property>
% Z: V# B; H- t9 G( w( w& n8 I+ |1 L' H <property>
& a! N3 j( T( }$ i* s <description>
% v8 n9 g. [% C4 L& Y8 P+ s; x Frequency for computing least loaded NMs.- u/ M5 w% t" r- m7 f/ x9 f
</description>
5 @/ j0 ^" ]$ H1 q2 J" q: x <name>yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms</name>% E" h e' a, P2 p' M' q2 f
<value>1000</value>
2 a1 \0 ?' g6 | I </property>6 ]. X9 P& m/ R1 F8 R
<property>8 F* j" j; Y8 x% O- [
<description>, S7 h3 m4 I. o) n2 G, {2 \4 I4 e
Comparator for determining node load for Distributed Scheduling.
) _. g, k Q! e2 V/ _ v2 R </description>
3 ~5 [0 X( I* a; K2 l; V <name>yarn.resourcemanager.nm-container-queuing.load-comparator</name>% D6 ?: T% p0 x: U z o% J! _
<value>QUEUE_LENGTH</value>
- w3 r2 y H; h# w9 G7 ]% K </property>5 J+ z+ z" I3 V/ Z* G0 b. ~7 \) n
<property>
3 W% m- y- X/ ]2 s <description>
6 h5 M6 y. [1 @2 F, \ Value of standard deviation used for calculation of queue limit thresholds.
' V7 O4 Q; }6 N& \: d8 v </description>
/ u: R* q, P3 w x <name>yarn.resourcemanager.nm-container-queuing.queue-limit-stdev</name>
* r* K; e- m7 ?8 H <value>1.0f</value>/ J9 O( h3 Q9 A4 Y B3 i
</property>
; j2 j8 ~0 i9 z% t, h3 w) m+ m7 X <property>. Z: l5 C1 R, Q2 N7 P
<description>9 t; W: j; Q0 n
Min length of container queue at NodeManager.
5 i' v6 p' u/ R' J2 p1 ~1 e </description>8 w+ p" l7 W5 D+ b+ }
<name>yarn.resourcemanager.nm-container-queuing.min-queue-length</name>
6 N! ~+ t5 q, x, F7 p/ Z% }' j <value>5</value>
4 ?/ _6 J2 S$ P7 o7 J+ j3 t </property>
* f3 O/ D( g, K0 v, X( s <property>+ W6 E% _8 g* w% ]0 c1 z
<description>
( c- ~% u5 ^3 H Max length of container queue at NodeManager.
( H. q. L9 M/ h0 T; ^) I3 y </description>/ S/ w: \+ T8 K5 T4 V9 f' v2 s
<name>yarn.resourcemanager.nm-container-queuing.max-queue-length</name>
0 C( {/ ?- ]) C$ Z+ c. z <value>15</value>
w. j& `+ T, `# {7 g; v </property>/ k( ]6 `7 r! H) _4 N
<property>
. c3 j% E2 A9 b, Y1 y/ p <description>
4 `6 R! o" q& w9 b H Min queue wait time for a container at a NodeManager.
" R' Q1 l6 Q5 m' H, ~/ P </description>
% q4 v' g1 j/ f l( u* V9 { <name>yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms</name>
8 o: [2 Z. a" ?1 e8 Z <value>10</value>0 |6 _; R3 O) m1 }6 I/ k
</property>. C& @) D \% d+ B/ X! {* b
<property># H& d/ @% m; {- _
<description>
, J: y2 E4 P! g7 ?; D: j, H O Max queue wait time for a container queue at a NodeManager.: f- k" o6 ?+ |* |
</description>
* A, G0 A: Y8 Y, ^5 A1 | <name>yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms</name>
! _. o4 H; Z" X! f$ q- k1 o( V <value>100</value>
! l4 J& i) G! j; ?" t4 \ </property>) j& j) e2 Q; B
<property>
' w8 v; N5 C4 `# I <description>/ s6 r! m( K ~" Y. F; K( k4 g
Use container pause as the preemption policy over kill in the container6 f" C# U# h! ?1 f! J6 \) b9 [
queue at a NodeManager.
* L& J3 W1 m: u# [ </description>! @& e% _2 u) o& e
<name>yarn.nodemanager.opportunistic-containers-use-pause-for-preemption</name>7 u9 W( `7 a$ R6 y' ~
<value>false</value>7 w" l' F! c. Y( @; G1 I
</property>/ z" f, a+ C: ^, k O# O1 w, d
<property>
/ `2 x$ \- g s3 z) X K3 U. j1 G <description>
$ [/ Z8 y9 S& ~ Error filename pattern, to identify the file in the container's6 l2 k, T$ O1 J [
Log directory which contain the container's error log. As error file
5 H* C5 g5 V: e% @9 L$ } redirection is done by client/AM and yarn will not be aware of the error
- \# ~& R2 X t+ P. _ ^* {% w- h file name. YARN uses this pattern to identify the error file and tail' ~6 C; _, H( M* p6 G: G/ a
the error log as diagnostics when the container execution returns non zero+ {' I% M) s2 z7 f3 K0 k
value. Filename patterns are case sensitive and should match the
1 J) Y* W9 n' {8 d* V specifications of FileSystem.globStatus(Path) api. If multiple filenames4 g9 `2 g) u7 W% b& `1 T+ b0 v
matches the pattern, first file matching the pattern will be picked.
3 v1 m( e& p, U! k* f; G& [ </description>
9 O7 v( r3 T; \2 J <name>yarn.nodemanager.container.stderr.pattern</name>
5 C6 ]4 O' X$ R; |; y" G# A2 Z/ | <value>{*stderr*,*STDERR*}</value>
6 M1 x5 k+ I6 d1 k9 W0 P9 o8 U* E </property>
# b* s+ o, b4 n& o <property>
' Y1 Y! s3 }/ {1 Z <description>& w& D( V2 A5 D# q! D
Size of the container error file which needs to be tailed, in bytes.
5 L+ j! K; V7 b! _* \# E </description>( @5 d2 l5 {# ?$ _/ S
<name>yarn.nodemanager.container.stderr.tail.bytes </name>
! X9 }2 g9 d+ Q4 |1 D) k7 | <value>4096</value># M! p$ ?% y0 b1 D9 i1 r$ \+ o5 `
</property>
. k$ N; c- g' T% |6 o <property>4 E4 y2 f$ G/ O
<description>
/ Q- M2 B. H$ X, L Choose different implementation of node label's storage6 V6 p8 ^* a' F
</description>
# B7 f# Z, M$ A* N <name>yarn.node-labels.fs-store.impl.class</name>5 r" d5 O' s+ H* ]* v y/ W
<value>org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore</value>. n( @$ I* \9 w6 Y
</property>& _3 ^' T! k# R! G
<property>1 g- I( D; }9 Z- J
<description>" Q* b" i2 _; g9 M7 {2 Q! D& D
Enable the CSRF filter for the RM web app. d& b% n6 r, h/ h$ H, n2 G
</description>& C; W, t5 w9 U* i4 X
<name>yarn.resourcemanager.webapp.rest-csrf.enabled</name>& |# Z2 O# G% J; _7 p
<value>false</value>$ q" a( B! z2 @% N+ W/ ~: {
</property>
' g5 W. i J! a" t8 X( g. W <property>
n' |& P, e2 i% Q; k+ d. A, q7 a <description>
0 A$ J" G1 x9 O/ w( k& I* z$ P, N Optional parameter that indicates the custom header name to use for CSRF
' a U4 I I' K: d4 g protection.3 I3 p" t' z7 F% g3 j5 k
</description>& l/ ~6 |: l& _5 I' C$ y- u
<name>yarn.resourcemanager.webapp.rest-csrf.custom-header</name>" A+ F2 N( A8 G* b5 J
<value>X-XSRF-Header</value>6 ~6 V8 @/ Y7 ~' \8 I& E
</property>
8 j5 `8 f8 O" p+ W) t" k2 y <property>
$ g1 Q# ^$ \* z% m0 c, s6 Y6 R <description>9 K& J4 W) X+ \( @6 X
Optional parameter that indicates the list of HTTP methods that do not
7 F! l7 Y) {: w, G require CSRF protection
& |( T) M( ^( c$ `1 d/ h9 W5 w, @ </description>
$ ]4 x3 u- W5 e- D <name>yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore</name>9 Q _4 G: l9 j; q2 c' V& N" q& r3 |
<value>GET,OPTIONS,HEAD</value>
2 H/ x/ s* R/ O F' Q9 c </property>1 s9 _* T; @) l- `6 N, }
<property>8 q$ z X4 A" Q/ |
<description>5 a% T* H6 D' O7 E& h, y
Enable the CSRF filter for the NM web app8 {2 @0 S4 _1 }" _0 l
</description>
9 g) V! L* q* g6 E0 q <name>yarn.nodemanager.webapp.rest-csrf.enabled</name>
) X# ?5 Y0 B/ g, O+ i <value>false</value>
' r; H+ W9 l3 E4 F) s* W& H </property>6 n: x7 @& a; Y# [/ |9 S. |2 K
<property>
9 X( D2 M5 R2 d5 K( h <description>6 t* p: @: y4 R6 D- {; m
Optional parameter that indicates the custom header name to use for CSRF
6 t$ x) d z* t9 o protection.& M0 g; K$ y _- k+ B! N
</description>
0 \, P+ N% X1 b) ]1 Y2 _ <name>yarn.nodemanager.webapp.rest-csrf.custom-header</name>
: I f! q2 O$ u1 A/ `* L7 ` <value>X-XSRF-Header</value>' g7 ]6 O2 |9 Y: A/ h4 G: c/ o
</property>3 a. [% ? `' A9 o3 {6 {2 z
<property>
2 v0 \6 Y- k! \* e, T: N <description>( I8 }. J C& C, S/ t! J
Optional parameter that indicates the list of HTTP methods that do not6 x$ M" F, r+ h. W
require CSRF protection
0 [ e5 ]8 v9 I! A# x </description>5 j: O# P: m$ T7 G6 M8 `
<name>yarn.nodemanager.webapp.rest-csrf.methods-to-ignore</name>, a7 t% r. t4 B2 c
<value>GET,OPTIONS,HEAD</value>2 H$ G, ^. A1 m8 G
</property>
, q4 T; I1 s9 q! | <property>
+ C$ C. l) u' c7 L$ N <description>
* Q. _ a: }! q" h The name of disk validator.' E; [6 J9 D9 g/ T# G
</description>
1 t: a5 X! ~1 }% ?3 B <name>yarn.nodemanager.disk-validator</name>
: z! a7 t" Z/ q/ c; i <value>basic</value>
& F& B, `( u8 Z& A* m </property>
, B$ I) T" G7 ]2 x# E <property>
5 s7 G% [5 y1 [- K" a# g1 D <description>
! z/ j) H" ~/ I1 @ Enable the CSRF filter for the timeline service web app+ H0 h6 N+ w3 ^4 H7 @* @$ H
</description>
1 f- d$ T5 }+ R- Q$ R* v4 V: h <name>yarn.timeline-service.webapp.rest-csrf.enabled</name>! h. e' O" X* \! `) {! [
<value>false</value>2 c$ ]/ Y/ b/ C
</property>
" Z+ A. ?% ^( r6 _ <property>3 H& x( H, K: o" V( x: y5 I
<description>" A1 h" d% N+ Q! _! V8 \# x+ B
Optional parameter that indicates the custom header name to use for CSRF
. ?+ G4 U7 T% o; t# u! F9 k) g protection.5 ]. I2 D$ x0 j+ O0 ]4 g
</description>
9 O |2 |* g% ^" _3 K <name>yarn.timeline-service.webapp.rest-csrf.custom-header</name>
& m) r5 [6 V, ~, A6 [ <value>X-XSRF-Header</value>! k) [9 z7 v+ ~4 X7 f" x9 a
</property>$ p' e- m: M& J
<property>
) o' |; D) f& e# n <description>) `% u$ z1 i$ n
Optional parameter that indicates the list of HTTP methods that do not3 ~9 n0 Z$ |7 H/ G1 w% Z! |2 c& Z
require CSRF protection' J2 e) }8 [7 [# `/ H% I, U' B
</description>
+ ^0 \+ v j( F" y/ U1 U% X <name>yarn.timeline-service.webapp.rest-csrf.methods-to-ignore</name>
6 u6 x" E' _+ \* C$ q <value>GET,OPTIONS,HEAD</value>- p1 f+ j' S% ]* N: S
</property> H* Q" Y2 q- R
<property>1 M$ p& B& J; g! C# e
<description>- s2 {) r& n# I- P
Enable the XFS filter for YARN
5 i, ]. b8 O- L/ M% ~: K </description>
9 R+ Y$ D& B8 v# }- n <name>yarn.webapp.xfs-filter.enabled</name>0 I! y$ e& F# a$ K: Z/ p
<value>true</value>, A; ], f0 e6 f) H8 W$ m( u8 L" `
</property>
! h' Z7 M5 [% l0 | <property>
# d$ m$ R. Z8 G3 u% ~& L <description>
0 {4 q+ G- W+ c6 k e R Property specifying the xframe options value.
8 i2 z. Z7 x9 `% m4 Q' J+ v4 J </description>- I; x& a& b! I6 K
<name>yarn.resourcemanager.webapp.xfs-filter.xframe-options</name> w/ Y9 U; U \
<value>SAMEORIGIN</value>$ o4 S/ }1 d9 _* A/ O0 a
</property>
- `# g+ z& K) q <property>
# _7 q5 p& J r' n <description>
! F: z6 ? Y) ^& P Property specifying the xframe options value.
; i6 O+ m. g' M! B </description>
* l3 D2 _6 H* ~1 k1 z$ q4 ]3 D <name>yarn.nodemanager.webapp.xfs-filter.xframe-options</name>' ^- `+ Y8 M$ Q/ ]
<value>SAMEORIGIN</value> Y, ?8 e8 Y* } |* u+ N% o/ r
</property>* o& B4 q, Y! g2 a0 q& h
<property>
7 x1 T M$ f3 B0 b9 d# q! y' F <description>
3 x6 m# t. \) I3 y% V Property specifying the xframe options value.
5 W0 ?5 i; n3 r </description>
; p- Y6 e$ y, I1 u <name>yarn.timeline-service.webapp.xfs-filter.xframe-options</name>6 v6 u4 v, l. h9 I& Q, }% n/ }
<value>SAMEORIGIN</value>5 ~9 J V# n8 ]
</property>8 Z$ Q7 T7 r* @
<property>
: E# T# ]2 V& S; s, Z <description>
1 v4 E/ H7 j% B' q& X The least amount of time(msec.) an inactive (decommissioned or shutdown) node can# Y8 f4 C0 e" u
stay in the nodes list of the resourcemanager after being declared untracked.6 D7 O, x& {& n! O# N
A node is marked untracked if and only if it is absent from both include and/ }% ]1 |( f, m2 u5 T
exclude nodemanager lists on the RM. All inactive nodes are checked twice per
6 b' V, _2 I' M$ W4 ?# ] timeout interval or every 10 minutes, whichever is lesser, and marked appropriately.
2 o0 L8 {1 V( @: F: E The same is done when refreshNodes command (graceful or otherwise) is invoked.+ [& M+ H. H3 {+ `
</description># b) [' G8 a6 n8 K0 x
<name>yarn.resourcemanager.node-removal-untracked.timeout-ms</name>
' x$ Y. |" m( P) w <value>60000</value>! a& Q: W% D/ _$ X! m# S S$ K
</property>/ S( q& i! G& q- [& A
<property>
9 s& `: |5 E0 v% _7 r/ ] <description>6 v" {7 \- b+ j- p
The RMAppLifetimeMonitor Service uses this value as monitor interval* d. M# \( ?3 W) G! V
</description>* c6 y# R' j$ r! z
<name>yarn.resourcemanager.application-timeouts.monitor.interval-ms</name>
3 S: D, w1 v& l2 ~0 l/ I. P3 I <value>3000</value>
( N3 G" z* z% ~7 ?/ }% } </property>
$ z- } k/ r, x4 H <property>1 X4 b0 E2 i4 E
<description>
0 p" c% ^3 _+ J4 u$ I Defines the limit of the diagnostics message of an application1 P% f2 [$ h! U6 [+ z( I+ [
attempt, in kilo characters (character count * 1024).
1 Q. E- {- H; F. j/ O When using ZooKeeper to store application state behavior, it's- A! n% \( U% s( J5 ^* E
important to limit the size of the diagnostic messages to
: D% ?# O1 E/ f7 C4 ], b1 Q prevent YARN from overwhelming ZooKeeper. In cases where
/ z: R9 @5 y' N# E. o7 F yarn.resourcemanager.state-store.max-completed-applications is set to
# x* S2 k0 o0 }2 W a large number, it may be desirable to reduce the value of this property
4 W0 P0 \1 ?) K/ Y" W& Y* Z$ j6 { to limit the total data stored.5 e" a- }: d" q& s
</description>
0 X+ C' o Y8 l; f7 S <name>yarn.app.attempt.diagnostics.limit.kc</name>
1 z" S) r& \/ Q$ I! V; H <value>64</value>
& X+ t' Z9 x0 L* _5 d </property>; F& P( k' z' _5 y0 _& @3 ^
<property>/ H% Q0 o) ^: Z7 b( S6 R. q
<description>
' s2 l; F* d! Q3 p( x6 r3 l( \4 _ Flag to enable cross-origin (CORS) support for timeline service v1.x or- o/ Q: \. W- G# R' p6 r
Timeline Reader in timeline service v2. For timeline service v2, also add
U: S5 h5 @0 ?3 o% R org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the
( L' X# B8 n4 G7 h8 ]/ L configuration hadoop.http.filter.initializers in core-site.xml.4 p3 h( I; C, A9 l* C& @+ J8 k q
</description>; R. X& \6 X% H0 k, W- a
<name>yarn.timeline-service.http-cross-origin.enabled</name>
5 l0 j. W! C8 w+ Z8 E <value>false</value>( \2 Y' g" k' y) p6 w& I
</property>. ]. |$ r# N7 g* R3 R7 u$ n
<property>
& m; z; p) a) n1 ^5 q <description>: J4 \* D6 Z* C$ c! g, K4 R; Z: N6 C
Flag to enable cross-origin (CORS) support for timeline service v1.x or
6 \9 i, N9 }% ^9 h: R Timeline Reader in timeline service v2. For timeline service v2, also add
+ r+ S I/ A5 }( b$ c, V% A" W org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the/ c' g: H" l3 g4 p% l
configuration hadoop.http.filter.initializers in core-site.xml.
' n* |# ?4 G1 M </description>
2 e% F8 A: D- A) x! G0 [* t% O <name>yarn.timeline-service.http-cross-origin.enabled</name>
7 [8 X. |1 A+ | <value>false</value>6 h S! U' _0 x: l4 x) w/ @
</property>
. \! [* T2 ~3 V& B. k <property>
$ S$ T- E, @$ \& G' f/ D# r <description>
' Q$ E& |8 F' g8 y5 Z# x The comma separated list of class names that implement the: a3 L7 ]! h2 |/ H/ m0 A5 t
RequestInterceptor interface. This is used by the RouterClientRMService* w0 I' \/ T0 F6 F) l: H" a
to create the request processing pipeline for users.( O% T9 C, ^$ j- N- h
</description> F# M- A/ {! I9 H
<name>yarn.router.clientrm.interceptor-class.pipeline</name>+ F1 S; G C1 k: ]. R
<value>org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor</value>1 {1 c2 q5 [' O4 T' M
</property>; f$ e0 z# W1 J% j% C" i- c" q+ Y2 @
<property>
( G5 `' z' q( d. Q <description>+ y& G! S. s N- o6 p
Size of LRU cache for Router ClientRM Service and RMAdmin Service.: w% E) ~% p" t+ a1 E" R
</description>
# S. ], k% J- m <name>yarn.router.pipeline.cache-max-size</name>% @. t0 s- W' _1 J5 {
<value>25</value>& p4 n& K: F+ E( v4 L7 ]* y
</property>
$ \1 A! n t( z <property>6 y; z# _' f2 R! w
<description>8 w/ Q/ t' f. G! t1 ~# t [
The comma separated list of class names that implement the
4 b- D: P1 W2 a! s2 _% F% l RequestInterceptor interface. This is used by the RouterRMAdminService+ P) g# _! F3 V* ^9 i5 U0 E8 D
to create the request processing pipeline for users.& ?0 s( y" [1 J) Q: F
</description>
, g- i" l: n; H, I <name>yarn.router.rmadmin.interceptor-class.pipeline</name>- e2 J z ^% y2 k7 \1 ?( Q
<value>org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor</value>9 `- }" h) T# k) v
</property>; s3 h% n u1 z, O/ ]
<property>; B8 Y6 h0 S4 N( T" U
<description>
! H' y3 z+ `9 o/ ?6 d# e The actual address the server will bind to. If this optional address is$ v' i' o4 x4 u' ?
set, the RPC and webapp servers will bind to this address and the port specified in. ]* _. B9 h& x6 y% N
yarn.router.address and yarn.router.webapp.address, respectively. This is
# I3 W/ G. {" O. g2 y most useful for making Router listen to all interfaces by setting to 0.0.0.0.
/ m0 Y/ s- u3 c$ w </description>; {- q0 d- g$ x1 R: \
<name>yarn.router.bind-host</name> b/ \# j' B) R, M! n
<value></value>* \" a0 y# e5 Q B
</property>. p0 B& a# K [1 z2 l# ~" Z
<property>1 t4 ?$ K% j2 q! K9 A" x$ ?4 P
<description>
4 ^0 L+ A7 A5 } Comma-separated list of PlacementRules to determine how applications+ P& c. h5 r W: A
submitted by certain users get mapped to certain queues. Default is3 D6 t8 M# A3 p8 [9 {) A
user-group, which corresponds to UserGroupMappingPlacementRule.
% _( V* R u4 H0 w8 X. _# M </description>
. y$ L* P5 d2 V <name>yarn.scheduler.queue-placement-rules</name>
0 I4 D9 Q% o: T4 ~ <value>user-group</value>3 K" M1 |! u8 ~5 n0 F
</property>
2 J& o1 z# K, m <property>
5 V% T: g2 \, a. D <description>
! [. _7 {2 @3 [ The comma separated list of class names that implement the( N/ I: P2 I# L) n$ M" h
RequestInterceptor interface. This is used by the RouterWebServices
4 Q( I7 {3 ]! p to create the request processing pipeline for users.
( _- D1 L6 y- j9 c </description>
4 F! _5 n! Z J2 T <name>yarn.router.webapp.interceptor-class.pipeline</name>
3 `( w: g1 G6 T5 J" C& z4 W4 z( a! Y, z <value>org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST</value>/ [/ |; b9 |' S9 a
</property>
3 D) a! Y* I" I) I. Z4 p4 Z Y <property>" ]9 M! ]) g' O- }
<description>
% _3 B U+ g0 H6 F The http address of the Router web application.
& { {5 l, z0 p* i& y! O- e If only a host is provided as the value,
. j* G4 ~$ L( O$ G% ^6 y& ?! g" ? the webapp will be served on a random port.7 J9 v1 k5 \6 s! \5 {# ~
</description>
5 q) L- d$ A! H2 s3 d4 S- |2 `; `- Z <name>yarn.router.webapp.address</name>
/ j7 B! S. O3 p# ?5 P <value>0.0.0.0:8089</value>
' Q2 K$ o+ S/ E: k$ B/ | </property>+ p4 Z' ~4 v) ~
<property>
6 y0 O3 K! S& T! R/ p' s <description>
+ l" K8 r& x& h2 | The https address of the Router web application.
4 s$ [2 v* J- ~! ~ If only a host is provided as the value,
" S+ i( _0 w& ^1 Q) N the webapp will be served on a random port.
/ A% L3 ~# S+ ~0 ]4 M4 g( k </description>
0 ~( D5 y! C4 R% }* F <name> yarn.router.webapp.https.address</name>1 }9 u9 S# D$ l* w6 l" N8 B; u8 g
<value>0.0.0.0:8091</value>
+ R. I- @( l1 ` </property>
1 S o) q) [" r# l- q! X <property>
6 |& c" y' a! G* _# o$ D$ Q* T <description>1 f& k$ d! k# c, X4 ~, J$ G
It is TimelineClient 1.5 configuration whether to store active
4 V4 d' D$ w9 ~+ c" F9 U0 A+ ~ application’s timeline data with in user directory i.e
- o) a3 d; ~8 b( o ${yarn.timeline-service.entity-group-fs-store.active-dir}/${user.name}
^# a, O8 Y6 G# y$ Q7 |. e" ~ </description>
! P6 U/ F' }9 ] <name>yarn.timeline-service.entity-group-fs-store.with-user-dir</name>
$ Z h- R9 K) q <value>false</value>: \8 S) B0 e/ ]9 N" K: A
</property>! c# h2 O% m, D; k) U' Q* s+ g
<!-- resource types configuration -->
' o+ V( G0 D, V9 v( q( w+ D <property>7 z5 I0 A3 p" t8 Q f6 m1 u
<name>yarn.resource-types</name>
8 E0 ?% f4 p1 _- t% W5 V/ N6 G R <value></value>
) H/ G) y( @' A" o, _ <description>- s, F+ f" G2 q: j/ P$ o; b8 H8 ^
The resource types to be used for scheduling. Use resource-types.xml
+ Z' x+ X2 ~9 H: U! k0 F1 e9 d* ~. E to specify details about the individual resource types.8 Q/ O' S$ m% D8 H$ u% k
</description>
0 a) W4 W0 X- v! t </property>1 j5 \) w# t# O
<property>2 h8 L0 @/ j. q) ]( x' M% ^
<name>yarn.webapp.filter-entity-list-by-user</name>, T+ a/ o$ q) a+ f% Z v
<value>false</value>+ C% ?% f/ O" E, _7 I/ e0 w3 Z
<description>
% v6 i* B" {: y+ l7 N" ?' v. l Flag to enable display of applications per user as an admin: N$ j- h/ w7 b3 d: Z: ?
configuration.
2 j0 H# q% ^ u- F </description>$ o& D& m+ ~! w0 _$ A
</property>
. v1 J( w8 H2 Q$ v/ z' `2 n <property>1 m* a6 S# q& R
<description>
6 n: |5 b' W" o$ Q+ P The type of configuration store to use for scheduler configurations.
1 ` Z2 C+ o/ p. y Default is "file", which uses file based capacity-scheduler.xml to
. m9 T6 p6 j# y( X K retrieve and change scheduler configuration. To enable API based" Q. h' f0 N9 S- q8 j
scheduler configuration, use either "memory" (in memory storage, no
`8 B }/ f7 N2 [. X: z persistence across restarts), "leveldb" (leveldb based storage), or
7 p6 p/ u5 @# E, W* e "zk" (zookeeper based storage). API based configuration is only useful
! \% |. q* a0 q' m. K1 e when using a scheduler which supports mutable configuration. Currently
2 G# }9 G2 f. ?7 D0 o9 }$ W only capacity scheduler supports this.0 `5 C5 A' C8 g# \9 `' a1 H
</description>. F) j) S1 v: k, W3 m' C6 O
<name>yarn.scheduler.configuration.store.class</name>
$ u( A9 O/ V; Y1 V3 x1 j <value>file</value>. v* k: u" R, a9 |. o2 q) y
</property>- g/ a1 [* ]0 X5 \
<property>
. Q/ V: p# D6 k# R( h9 n) x <description>+ L7 G- b1 r+ v; b; n/ l% q3 {
The class to use for configuration mutation ACL policy if using a mutable/ h0 m* G" q. u# q6 p
configuration provider. Controls whether a mutation request is allowed.
# G4 f; u4 `# Z0 C The DefaultConfigurationMutationACLPolicy checks if the requestor is a+ Z/ ], _1 } j3 D$ H/ H. _
YARN admin./ K$ v( G4 X% s! S
</description>; W/ P- ~, x- G- o
<name>yarn.scheduler.configuration.mutation.acl-policy.class</name>) C0 X5 _6 C- I- b- y5 ^" D# B
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy</value>
3 g; _# t1 [* Y/ g# t </property>' G% d* G& M W" u9 e# H/ ?2 X: Z' G
<property>
. m( m) m7 o$ `' n* @" e1 K7 c <description>/ p( ^2 v9 U$ b- d; d, X5 `
The storage path for LevelDB implementation of configuration store,4 e, [. Y' p, y. G8 F3 M& @
when yarn.scheduler.configuration.store.class is configured to be3 K; P5 C6 E/ b' \4 c# J
"leveldb".4 b. t4 a7 w! J8 U# c- y
</description>
2 A/ b" r# I+ _. D- a) V <name>yarn.scheduler.configuration.leveldb-store.path</name>5 r8 _ z9 N9 |# d1 o" u
<value>${hadoop.tmp.dir}/yarn/system/confstore</value>) y& [: i, X' F
</property> j. ^ @3 w6 N$ O: V
<property>4 F: q q1 Z( b
<description>
- R! |9 ]4 f6 A/ i: O The compaction interval for LevelDB configuration store in secs,
% m E1 ?' k' |3 y7 u& \ when yarn.scheduler.configuration.store.class is configured to be' I& H/ r% h* Q6 N5 a
"leveldb". Default is one day.# i! }: l6 ~! F8 V- ?7 U
</description>6 H$ ?! z% T0 j8 j2 L
<name>yarn.scheduler.configuration.leveldb-store.compaction-interval-secs</name>
# {0 g/ u* G: K/ D5 ~! E/ } <value>86400</value>
( n9 D3 F4 a2 ^- A# q </property>
3 K% {2 |4 M3 |0 G$ J <property>6 Z$ m+ Y! M; \; d
<description>& f2 L+ |# m! N
The max number of configuration change log entries kept in config
. C0 S% w) l- A, J9 a store, when yarn.scheduler.configuration.store.class is configured to be
1 x. M3 x# D" P6 S "leveldb" or "zk". Default is 1000 for either.
! m$ t" U' g* E3 l </description>$ q" T8 {, E7 p' _
<name>yarn.scheduler.configuration.store.max-logs</name>3 o) A1 u6 t P y
<value>1000</value># J& S2 ?0 L: w0 B7 ]% D: G. y
</property>
4 l) ~5 }: p$ L# z' _( A <property>
/ v3 x) J7 n" _3 { <description>
7 U: X; u! F0 E- v ZK root node path for configuration store when using zookeeper-based
# C( `( }( U+ y7 O' { configuration store.
. e# U) R- r; I: X </description>2 j5 ^" h% q5 w
<name>yarn.scheduler.configuration.zk-store.parent-path</name>7 l3 [; j) \ _. t
<value>/confstore</value>
0 q1 b, b2 a" W </property>
; J5 |% b* o+ {( j E4 j) o <property>: W W. c$ s! C8 y* L. E
<description>4 f5 r/ B0 E4 m& I1 s+ s3 ^, C
Provides an option for client to load supported resource types from RM
! A0 h- M7 [4 M ~0 \2 | instead of depending on local resource-types.xml file.
# f2 |3 A: n+ A0 |$ T </description>
& R2 a2 S* h! x \9 G" m <name>yarn.client.load.resource-types.from-server</name># h. m( k H3 t0 K; g
<value>false</value>: `! [% J: h! r0 ^/ m
</property>! ^/ h; n: V3 J6 b
<property>+ J. y, E# p+ y% e* Z* D
<description>
8 R' h& i. Z J When yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto specified,
+ T! E' o- k% W YARN NodeManager needs to run GPU discovery binary (now only support& \* n7 R/ K. R/ H! T! a
nvidia-smi) to get GPU-related information.
1 x0 f4 c) v5 x4 ^& R When value is empty (default), YARN NodeManager will try to locate% [8 p9 G$ J1 _5 J
discovery executable itself.
( F$ @0 A4 g# s6 o7 A( M An example of the config value is: /usr/local/bin/nvidia-smi
; q: R" p0 v9 |3 Y3 i </description>
. {/ C& V- x8 ]/ P. D <name>yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables</name>
: ?1 _/ D0 G4 e4 T, [4 x; z6 G9 A <value></value>
3 t( z6 K' b/ m& s$ I* n$ I& I2 k </property>) {( Q' n* b+ @# b" {6 A. V
<property>2 m" x( J- r0 l' A' X v
<description>2 p+ U0 ~- t; {* |& q9 s
Enable additional discovery/isolation of resources on the NodeManager,
8 b" f/ n7 w5 f8 d/ i split by comma. By default, this is empty.' ]) l) J; L' q# E3 l" f
Acceptable values: { "yarn-io/gpu", "yarn-io/fpga"}.7 O# h! [) B2 A
</description>
, U8 r* C4 J! f2 y4 C1 ?: B1 p+ ^2 B <name>yarn.nodemanager.resource-plugins</name>
9 S& Z2 e# `$ m0 Y- x <value></value>
, G; k" G6 G4 R* x* K7 b' e </property>
+ P% h0 n: L- U0 S+ L" O+ `, h <property>, e3 x- l+ |6 N
<description>
% m0 i9 i/ _4 Y ?7 V4 H Specify GPU devices which can be managed by YARN NodeManager, split by comma0 y; z' a5 L+ P( c1 ]) t. p
Number of GPU devices will be reported to RM to make scheduling decisions.* _! m- g H( O: a
Set to auto (default) let YARN automatically discover GPU resource from* M- _ w9 U8 G8 N: E$ k
system.
( N! i% P, }6 d% J( h Manually specify GPU devices if auto detect GPU device failed or admin; k; I1 a- ~- V# w. S
only want subset of GPU devices managed by YARN. GPU device is identified# R3 z! v4 W" _$ F1 Q/ }4 K1 i
by their minor device number and index. A common approach to get minor _: d% h0 _( N6 J# H9 p
device number of GPUs is using "nvidia-smi -q" and search "Minor Number"1 x' K. x ]( x! A" E, V
output.
/ S9 W- G2 e1 N0 m6 a; l When manual specify minor numbers, admin needs to include indice of GPUs, e5 d/ q) D- O3 I' a/ ^) e
as well, format is index:minor_number[,index:minor_number...]. An example: G8 |2 f% }; l, D. C# g0 L( m
of manual specification is "0:0,1:1,2:2,3:4" to allow YARN NodeManager to
: P% E+ g+ I; o manage GPU devices with indice 0/1/2/3 and minor number 0/1/2/4.
& x9 H% Z8 P, h e numbers .3 A" R! Y# x) X$ V1 \# B, U
</description>' I+ g4 F0 }, ^% ~
<name>yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices</name>
7 o$ I) Z9 O$ a: g8 [ <value>auto</value>2 a' U. w( t: ^8 q# U' o( }
</property>, W4 [, N+ ?5 h3 `- \
<property>' |. T9 `* W, t6 s$ [' A) g
<description>
: Q! P* f i# G9 m Specify docker command plugin for GPU. By default uses Nvidia docker V1.
; q$ q: Q& L2 c: ~4 g </description>. k( h- }- X2 [3 N( R4 J
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin</name>
! _6 C$ e2 ~' f: ^0 A' H <value>nvidia-docker-v1</value>8 d0 S0 D4 L, m8 F$ l' {
</property>2 U v: k' D" w& N
<property>+ O- E/ `4 g! e
<description>
& x. \! }) h' s Specify end point of nvidia-docker-plugin.. g& t$ Y0 k- H
Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki
( `; L- y1 A- u+ s* t For more details.& B. e/ F+ }& F. c; A5 d
</description>% A7 w% M7 I B+ G4 H0 s- e
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint</name>1 l7 S" Q! O, e, f& m! [2 k" u* k5 k
<value>http://localhost:3476/v1.0/docker/cli</value>
# q) ~6 X$ E/ N9 @5 | </property>1 ?3 f' Y8 X% Q* M# Y8 s
<property>
2 \9 e6 f( L: g: M <description>
: g: c# Y* Z1 Y2 L, p2 P1 U% k Specify one vendor plugin to handle FPGA devices discovery/IP download/configure.
) ~: T2 |7 k1 r; ^# W4 L; e Only IntelFpgaOpenclPlugin is supported by default.. W5 f0 a; X& n9 u$ x
We only allow one NM configured with one vendor FPGA plugin now since the end user can put the same
, j2 K; D3 R& A9 h5 ?8 C# O9 d! N vendor's cards in one host. And this also simplify our design.
; H' f) H4 }6 y! D1 t </description>
. y9 b) r7 O) ~ I <name>yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class</name>. V# ` t( r! @4 L
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin</value>, e1 c- ~* r6 R! E
</property>
9 j6 j4 Y `1 k8 k# \( `3 c <property>
0 m0 }4 i! I" a- W8 n' K <description>2 d4 l0 f2 ?- c% K+ `
When yarn.nodemanager.resource.fpga.allowed-fpga-devices=auto specified,9 `: v( m8 `3 S/ X& `
YARN NodeManager needs to run FPGA discovery binary (now only support
* R/ S. J: R8 u! E IntelFpgaOpenclPlugin) to get FPGA information.4 n& W- F' `7 p! o2 ~: d% s& K
When value is empty (default), YARN NodeManager will try to locate7 g# J! w- h& L9 S) D' f
discovery executable from vendor plugin's preference
& F- j# q- ]3 P </description>
* N+ X3 {- _( F: v+ x) @% _! s <name>yarn.nodemanager.resource-plugins.fpga.path-to-discovery-executables</name>
" m# a7 J0 f# t' P; _ <value></value>7 V& x# ^+ H+ x( g9 m
</property>) ?2 U; O% ?! ^& m2 A, s0 v
<property>& C4 T6 r- y: v3 \+ Z4 |0 F q( y
<description>
, g8 @" x: \2 A+ A" n Specify FPGA devices which can be managed by YARN NodeManager, split by comma1 B( t7 j+ H6 Q; H% j
Number of FPGA devices will be reported to RM to make scheduling decisions.2 b0 i t( ~; y: i
Set to auto (default) let YARN automatically discover FPGA resource from
6 j1 r" x9 U8 c! m system.9 r ]4 m/ s7 _9 m# e2 `: }" l
Manually specify FPGA devices if admin only want subset of FPGA devices managed by YARN.
! j$ c$ [0 f5 i4 g2 ^$ y4 t At present, since we can only configure one major number in c-e.cfg, FPGA device is
& n. c+ i) i) ^( E& [% c identified by their minor device number. A common approach to get minor
9 f" Q( g- b0 S+ m0 g$ G; ]4 m( E device number of FPGA is using "aocl diagnose" and check uevent with device name.; K0 r' y) W9 j1 O4 L
</description>
6 s- C5 J5 Y1 U <name>yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices</name>9 @6 q3 h8 o5 s
<value>0,1</value>
" s2 \' W2 t# {. t- ^ </property>
7 S; T: ~9 \% q! [ <property>
% p, W& j6 S* ~ <description>The http address of the timeline reader web application.</description>
4 q& u. A, n- R8 S <name>yarn.timeline-service.reader.webapp.address</name>
& }( Z$ T) I* ], P5 n# B) s% ? <value>${yarn.timeline-service.webapp.address}</value>
$ _9 }: ~$ Q. F+ f3 H </property>
$ b2 c$ G, v% o6 m1 F {- B4 J6 i <property>0 c& G2 r, |- `& f6 Z% s
<description>The https address of the timeline reader web application.</description>
: f1 o4 J8 `4 o0 k <name>yarn.timeline-service.reader.webapp.https.address</name>2 }) N& j! H, A8 u+ D! ]
<value>${yarn.timeline-service.webapp.https.address}</value>
( p, F. ^6 w* F2 f </property>3 d7 _* W; l% t
<property>
6 @1 A; p2 I M' }( X <description>4 l' G/ X! t: O% q3 Y1 ?4 @
The actual address timeline reader will bind to. If this optional address is* |! s3 a, v4 ]1 q% i! v
set, the reader server will bind to this address and the port specified in
, C- N7 c( M7 w7 ]/ ]" S yarn.timeline-service.reader.webapp.address.
' |9 T4 q, V2 ^: n; F2 C o; \) R' F3 I This is most useful for making the service listen to all interfaces by setting to
0 z1 c/ W% D6 E5 Q# A 0.0.0.0.
+ y# c( X7 Q& C </description>
/ [; u& N) ~) K0 W! G <name>yarn.timeline-service.reader.bind-host</name>
5 W& l/ @; a t9 j; V; |- Y <value></value>
8 G5 U0 z: ]) n </property>$ }8 i; `0 I* V0 u- Q
<property>% x. ]1 v' ^2 |6 r6 Z* T# ?# W
<description>
& o: h: {; f. Z- @' l Whether to enable the NUMA awareness for containers in Node Manager.# y. B9 X' ^6 g. K5 w: A1 ~) Y
</description>+ l; t) j9 G) P L
<name>yarn.nodemanager.numa-awareness.enabled</name>5 w( s* W- q6 c8 g1 R
<value>false</value>
# y! k9 c# i5 F) v# s </property>
8 m" p& t& p7 f <property>8 X# e4 B$ d) m3 z; V, t* ^: E
<description>
0 J+ G5 l/ q; G Whether to read the NUMA topology from the system or from the
7 W2 U) Z' b' P configurations. If the value is true then NM reads the NUMA topology from
; V Y& l8 z8 [' N+ \' M+ {4 A9 | system using the command 'numactl --hardware'. If the value is false then NM0 w$ }9 W" G! ]
reads the topology from the configurations
; E" j3 A9 m. {+ [ 'yarn.nodemanager.numa-awareness.node-ids'(for node id's),2 P1 t. X4 W# D; _
'yarn.nodemanager.numa-awareness.<NODE_ID>.memory'(for each node memory),
& I* M; W* @% [ e7 b; } 'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus'(for each node cpus).' X3 \8 [- ^' @
</description>4 U& T$ z, N. J
<name>yarn.nodemanager.numa-awareness.read-topology</name>
. b- w8 a" _: U0 s) T <value>false</value>- d) w9 ?( Y E( n3 [4 y
</property>
- K6 S& J. L- { <property># l% b& Y. W& c3 J8 N+ ^/ i
<description>
0 s) F: q- H# V! e1 [: w. P. P5 g NUMA node id's in the form of comma separated list. Memory and No of CPUs& `7 V. o* t. a- B r
will be read using the properties
% T6 \$ f- i0 _* O+ f 'yarn.nodemanager.numa-awareness.<NODE_ID>.memory' and1 ?' F) E `2 D2 U9 F- L
'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus' for each id specified
8 V$ q' Q. ? z$ G. r4 s+ U in this value. This property value will be read only when
; ]3 G. p$ c% x% e+ f$ d 'yarn.nodemanager.numa-awareness.read-topology=false'.' g5 X$ @9 X9 H- ]3 N# R, n' ]& c8 ^
For example, if yarn.nodemanager.numa-awareness.node-ids=0,1' l4 C3 e7 x: W( M+ u# U
then need to specify memory and cpus for node id's '0' and '1' like below,# z. Z" S2 d* U. X' S/ Z
yarn.nodemanager.numa-awareness.0.memory=73717
. j X9 c( w1 Q- z0 ` yarn.nodemanager.numa-awareness.0.cpus=4
" v( N- N4 ?) V( h yarn.nodemanager.numa-awareness.1.memory=73727) j6 S5 Z& K: `' ?% K4 N
yarn.nodemanager.numa-awareness.1.cpus=43 ?* j. y% l; \( S
</description>; t+ G. h, |# q1 ?& x
<name>yarn.nodemanager.numa-awareness.node-ids</name>4 k2 c" B; u, s/ \5 `; T& @7 ?
<value></value>: h1 I) M' C6 R
</property>
3 V G( \, c0 X% ? <property>
5 r: ?( E7 H3 |# i) r& p <description>
2 }4 a& Q) `$ G The numactl command path which controls NUMA policy for processes or
; O8 `- l. |5 a( i/ j. U& Y shared memory.
. b0 A% I. Q/ b0 Q6 ? </description>
. v2 p' Y M" L4 H6 W S m/ r <name>yarn.nodemanager.numa-awareness.numactl.cmd</name>
. W9 O! Q& C/ g+ l <value>/usr/bin/numactl</value>
( S" E5 `& k2 @3 R# S </property>$ K! u5 [5 i" P' H
</configuration>8 l+ J) h# A0 n$ G V
" Z9 B6 w% Q4 E1 |% n* B/ a N6 a |
|
发表于 2022-11-7 11:18:00